Anton Malevich created MESOS-8703:
-------------------------------------
Summary: Mesos master can`t reconnect to zookeeper
Key: MESOS-8703
URL: https://issues.apache.org/jira/browse/MESOS-8703
Project: Mesos
Issue Type: Bug
Components: master
Affects Versions: 1.4.1
Reporter: Anton Malevich
Mesos master can`t reconnect to zookeeper after zookeeper hangs.
{noformat}
2018-03-20
10:16:45,608:1(0x2ae675db6700):ZOO_ERROR@handle_socket_error_msg@1666: Socket
[<zknode1>:2181] zk retcode=-7, errno=110(Connection timed out): connection to
<zknode1>:2181 timed out (exceeded timeout by 3ms)
2018-03-20 10:16:45,609:1(0x2ae675db6700):ZOO_INFO@check_events@1728: initiated
connection to server [<zknode2>:2181]
2018-03-20
10:16:45,619:1(0x2ae675db6700):ZOO_ERROR@handle_socket_error_msg@1764: Socket
[<zknode2>:2181] zk retcode=-112, errno=116(Stale file handle):
sessionId=0x5623d0e483dd435 has expired.
I0320 10:16:45.620604 18 group.cpp:511] ZooKeeper session expired
I0320 10:16:45.620802 16 detector.cpp:152] Detected a new leader: None
I0320 10:16:45.620957 16 master.cpp:2176] The newly elected leader is None
mesos-master: ../../3rdparty/stout/include/stout/option.hpp:112: T&
Option<T>::get() & [with T = mesos::MasterInfo]: Assertion `isSome()' failed.
*** Aborted at 1521541005 (unix time) try "date -d @1521541005" if you are
using GNU date ***
PC: @ 0x2ae63d2b9428 (unknown)
*** SIGABRT (@0x1) received by PID 1 (TID 0x2ae648ffa700) from PID 1; stack
trace: ***
@ 0x2ae63d078390 (unknown)
@ 0x2ae63d2b9428 (unknown)
@ 0x2ae63d2bb02a (unknown)
@ 0x2ae63d2b1bd7 (unknown)
@ 0x2ae63d2b1c82 (unknown)
2018-03-20 10:16:45,622:1(0x2ae649ffc700):ZOO_INFO@zookeeper_close@2543:
Freeing zookeeper resources for sessionId=0x5623d0e483dd435
2018-03-20 10:16:45,623:1(0x2ae6477f7700):ZOO_INFO@log_env@726: Client
environment:zookeeper.version=zookeeper C client 3.4.8
2018-03-20 10:16:45,623:1(0x2ae6477f7700):ZOO_INFO@log_env@730: Client
environment:host.name=<mesos_hostname>
2018-03-20 10:16:45,623:1(0x2ae6477f7700):ZOO_INFO@log_env@737: Client
environment:os.name=Linux
2018-03-20 10:16:45,623:1(0x2ae6477f7700):ZOO_INFO@log_env@738: Client
environment:os.arch=4.8.15-1.el7.wg.x86_64
2018-03-20 10:16:45,623:1(0x2ae6477f7700):ZOO_INFO@log_env@739: Client
environment:os.version=#1 SMP Mon Dec 26 14:34:45 UTC 2016
2018-03-20 10:16:45,624:1(0x2ae6477f7700):ZOO_INFO@log_env@747: Client
environment:user.name=(null)
2018-03-20 10:16:45,624:1(0x2ae6477f7700):ZOO_INFO@log_env@755: Client
environment:user.home=/root
2018-03-20 10:16:45,624:1(0x2ae6477f7700):ZOO_INFO@log_env@767: Client
environment:user.dir=/
2018-03-20 10:16:45,624:1(0x2ae6477f7700):ZOO_INFO@zookeeper_init@800:
Initiating client connection, host=<zk_pool> sessionTimeout=10000
watcher=0x2ae63b3711e0 sessionId=0 sessionPasswd=<null> context=0x2ae6900036f8
flags=0
@ 0x2ae63ad6b55b mesos::internal::master::Master::detected()
@ 0x2ae63b9e4cfc process::ProcessBase::visit()
2018-03-20 10:16:45,634:1(0x2ae6765b7700):ZOO_INFO@check_events@1728: initiated
connection to server [<zknode1>:2181]
@ 0x2ae63b9fac84 process::ProcessManager::resume()
@ 0x2ae63b9fd5e6
_ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
@ 0x2ae63c87ec80 (unknown)
@ 0x2ae63d06e6ba start_thread
@ 0x2ae63d38b3dd (unknown)
2018-03-20 10:16:45,651:1(0x2ae6765b7700):ZOO_INFO@check_events@1775: session
establishment complete on server [<zknode1>:2181], sessionId=0x1623f43348692c7,
negotiated timeout=10000
I0320 10:16:45.651684 15 group.cpp:341] Group process
(zookeeper-group(2)@<mesos4>:5050) connected to ZooKeeper
I0320 10:16:45.651733 15 group.cpp:831] Syncing group operations: queue size
(joins, cancels, datas) = (0, 0, 0)
I0320 10:16:45.651743 15 group.cpp:419] Trying to create path '/mesos' in
ZooKeeper
I0320 10:16:45.676736 15 detector.cpp:152] Detected a new leader: (id='704')
I0320 10:16:45.676844 15 group.cpp:700] Trying to get
'/mesos/json.info_0000000704' in ZooKeeper
I0320 10:16:45.683346 15 zookeeper.cpp:262] A new leading master
(UPID=master@<mesos4>:5050) is detected
{noformat}
After this, mesos master do not answer for http requests, and leader election
do not happens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
