[
https://issues.apache.org/jira/browse/MESOS-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978599#comment-13978599
]
Jie Yu commented on MESOS-1231:
-------------------------------
Tests:
https://reviews.apache.org/r/20618/
> CHECK failed in log coordinator
> -------------------------------
>
> Key: MESOS-1231
> URL: https://issues.apache.org/jira/browse/MESOS-1231
> Project: Mesos
> Issue Type: Bug
> Components: replicated log
> Affects Versions: 0.19.0
> Reporter: Jie Yu
> Priority: Critical
> Fix For: 0.19.0
>
>
> We have seen the following check failure in tests:
> {noformat}
> F0423 02:22:33.860647 58518 coordinator.cpp:399] Check failed: !missing Not
> expecting local replica to be missing position 4 after the writing is done
> {noformat}
> This is due to a bug in replicated log that the coordinator assume the local
> replica is always in the network (may not be true). Here is what happened:
> 1) Coordinator writes position 4
> 2) Write request is accepted and persisted
> 3) Coordinator tries to send learned message to the local replica
> 4) ZK down, network becomes empty, the local replica never receives the
> learned message
> 5) Coordinator thinks that the local replica should have learned position 4
> The full log is in the following:
> {noformat}
> [ RUN ] MasterZooKeeperTest.LostZooKeeperCluster
> I0423 02:22:33.207391 58498 leveldb.cpp:174] Opened db in 128.980841ms
> I0423 02:22:33.226372 58498 leveldb.cpp:181] Compacted db in 18.88168ms
> I0423 02:22:33.226482 58498 leveldb.cpp:196] Created db iterator in 26045ns
> I0423 02:22:33.226506 58498 leveldb.cpp:202] Seeked to beginning of db in
> 1578ns
> I0423 02:22:33.226570 58498 leveldb.cpp:271] Iterated through 0 keys in the
> db in 327ns
> I0423 02:22:33.226619 58498 replica.cpp:729] Replica recovered with log
> positions 0 -> 0 with 1 holes and 0 unlearned
> I0423 02:22:33.265805 58515 leveldb.cpp:304] Persisting metadata (8 bytes) to
> leveldb took 38.364737ms
> I0423 02:22:33.265853 58515 replica.cpp:320] Persisted replica status to
> VOTING
> I0423 02:22:33.400118 58498 leveldb.cpp:174] Opened db in 133.685751ms
> I0423 02:22:33.476763 58498 leveldb.cpp:181] Compacted db in 76.545296ms
> I0423 02:22:33.476889 58498 leveldb.cpp:196] Created db iterator in 18192ns
> I0423 02:22:33.476927 58498 leveldb.cpp:202] Seeked to beginning of db in
> 13800ns
> I0423 02:22:33.477010 58498 leveldb.cpp:271] Iterated through 1 keys in the
> db in 24427ns
> I0423 02:22:33.477082 58498 replica.cpp:729] Replica recovered with log
> positions 0 -> 0 with 1 holes and 0 unlearned
> 2014-04-23 02:22:33,477:58498(0x7f0c0bc98940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> 2014-04-23 02:22:33,477:58498(0x7f0c0bc98940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> 2014-04-23 02:22:33,477:58498(0x7f0c0bc98940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2014-04-23 02:22:33,477:58498(0x7f0c0bc98940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> 2014-04-23 02:22:33,477:58498(0x7f0c0bc98940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> 2014-04-23 02:22:33,477:58498(0x7f0c0bc98940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> 2014-04-23 02:22:33,477:58498(0x7f0c07c90940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> 2014-04-23 02:22:33,477:58498(0x7f0c0bc98940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> I0423 02:22:33.477603 58529 log.cpp:226] Attempting to join replica to
> ZooKeeper group
> 2014-04-23 02:22:33,477:58498(0x7f0c07c90940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> 2014-04-23 02:22:33,477:58498(0x7f0c07c90940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2014-04-23 02:22:33,477:58498(0x7f0c07c90940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> 2014-04-23 02:22:33,477:58498(0x7f0c07c90940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> 2014-04-23 02:22:33,477:58498(0x7f0c0bc98940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,477:58498(0x7f0c0bc98940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null>
> context=0x7f0bbc033970 flags=0
> 2014-04-23 02:22:33,477:58498(0x7f0c07c90940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> 2014-04-23 02:22:33,477:58498(0x7f0c07c90940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2014-04-23 02:22:33,477:58498(0x7f0c07c90940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,477:58498(0x7f0c07c90940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null> context=0x1d01ad0
> flags=0
> I0423 02:22:33.477948 58524 recover.cpp:293] Starting replica recovery
> 2014-04-23 02:22:33,478:58498(0x7f0bc4a75940):ZOO_INFO@check_events@1703:
> initiated connection to server [127.0.0.1:52633]
> 2014-04-23 02:22:33,478:58498(0x7f0bc5478940):ZOO_INFO@check_events@1703:
> initiated connection to server [127.0.0.1:52633]
> 2014-04-23 02:22:33,478:58498(0x7f0c0bc98940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> 2014-04-23 02:22:33,478:58498(0x7f0c0bc98940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> 2014-04-23 02:22:33,478:58498(0x7f0c0bc98940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2014-04-23 02:22:33,478:58498(0x7f0c0bc98940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> 2014-04-23 02:22:33,478:58498(0x7f0c0bc98940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> 2014-04-23 02:22:33,478:58498(0x7f0c0bc98940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> 2014-04-23 02:22:33,478:58498(0x7f0c0bc98940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2014-04-23 02:22:33,478:58498(0x7f0c0bc98940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,478:58498(0x7f0c0b497940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> 2014-04-23 02:22:33,478:58498(0x7f0c0bc98940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null>
> context=0x7f0bf80be8b0 flags=0
> 2014-04-23 02:22:33,478:58498(0x7f0c0b497940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> 2014-04-23 02:22:33,478:58498(0x7f0c0b497940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2014-04-23 02:22:33,478:58498(0x7f0c0b497940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> 2014-04-23 02:22:33,478:58498(0x7f0c0b497940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> 2014-04-23 02:22:33,478:58498(0x7f0c0b497940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> I0423 02:22:33.479634 58533 recover.cpp:319] Replica is in VOTING status
> 2014-04-23 02:22:33,479:58498(0x7f0c0b497940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2014-04-23 02:22:33,479:58498(0x7f0c0b497940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,479:58498(0x7f0c0b497940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null> context=0x1d2b120
> flags=0
> I0423 02:22:33.480101 58533 recover.cpp:308] Recover process terminated
> 2014-04-23 02:22:33,480:58498(0x7f0bc3a73940):ZOO_INFO@check_events@1703:
> initiated connection to server [127.0.0.1:52633]
> 2014-04-23 02:22:33,480:58498(0x7f0bc2a71940):ZOO_INFO@check_events@1703:
> initiated connection to server [127.0.0.1:52633]
> I0423 02:22:33.481199 58519 master.cpp:264] Master
> 20140423-022233-1828659978-34577-58498 (smfd-atr-11-sr1.devel.twitter.com)
> started on 10.35.255.108:34577
> I0423 02:22:33.481271 58519 master.cpp:298] Master only allowing
> authenticated frameworks to register!
> W0423 02:22:33.481387 58519 master.cpp:328] Permissions on credentials file
> '/tmp/MasterZooKeeperTest_LostZooKeeperCluster_KXTDKq/credentials' are too
> open. It is recommended that your credentials file is NOT accessible by
> others.
> I0423 02:22:33.487786 58532 contender.cpp:127] Joining the ZK group
> 2014-04-23 02:22:33,518:58498(0x7f0bc4a75940):ZOO_INFO@check_events@1750:
> session establishment complete on server [127.0.0.1:52633],
> sessionId=0x1458c6446740000, negotiated timeout=10000
> I0423 02:22:33.518537 58529 group.cpp:310] Group process
> ((100)@10.35.255.108:34577) connected to ZooKeeper
> I0423 02:22:33.518580 58529 group.cpp:781] Syncing group operations: queue
> size (joins, cancels, datas) = (0, 0, 0)
> I0423 02:22:33.518615 58529 group.cpp:382] Trying to create path
> '/znode/replicas' in ZooKeeper
> 2014-04-23 02:22:33,540:58498(0x7f0bc5478940):ZOO_INFO@check_events@1750:
> session establishment complete on server [127.0.0.1:52633],
> sessionId=0x1458c6446740001, negotiated timeout=10000
> I0423 02:22:33.540578 58526 group.cpp:310] Group process
> ((101)@10.35.255.108:34577) connected to ZooKeeper
> I0423 02:22:33.540634 58526 group.cpp:781] Syncing group operations: queue
> size (joins, cancels, datas) = (1, 0, 0)
> I0423 02:22:33.540653 58526 group.cpp:382] Trying to create path
> '/znode/replicas' in ZooKeeper
> 2014-04-23 02:22:33,540:58498(0x7f0bc3a73940):ZOO_INFO@check_events@1750:
> session establishment complete on server [127.0.0.1:52633],
> sessionId=0x1458c6446740002, negotiated timeout=10000
> I0423 02:22:33.541086 58527 group.cpp:310] Group process
> ((104)@10.35.255.108:34577) connected to ZooKeeper
> I0423 02:22:33.541121 58527 group.cpp:781] Syncing group operations: queue
> size (joins, cancels, datas) = (1, 0, 0)
> I0423 02:22:33.541143 58527 group.cpp:382] Trying to create path '/znode' in
> ZooKeeper
> 2014-04-23 02:22:33,541:58498(0x7f0bc2a71940):ZOO_INFO@check_events@1750:
> session establishment complete on server [127.0.0.1:52633],
> sessionId=0x1458c6446740003, negotiated timeout=10000
> I0423 02:22:33.541852 58519 group.cpp:310] Group process
> ((106)@10.35.255.108:34577) connected to ZooKeeper
> I0423 02:22:33.541923 58519 group.cpp:781] Syncing group operations: queue
> size (joins, cancels, datas) = (0, 0, 0)
> I0423 02:22:33.541944 58519 group.cpp:382] Trying to create path '/znode' in
> ZooKeeper
> I0423 02:22:33.565755 58514 contender.cpp:243] New candidate (id='0') has
> entered the contest for leadership
> I0423 02:22:33.574548 58515 detector.cpp:134] Detected a new leader: (id='0')
> I0423 02:22:33.574959 58527 group.cpp:655] Trying to get
> '/znode/info_0000000000' in ZooKeeper
> I0423 02:22:33.583420 58529 network.hpp:413] ZooKeeper group memberships
> changed
> I0423 02:22:33.583534 58528 detector.cpp:376] A new leading master
> ([email protected]:34577) is detected
> I0423 02:22:33.583587 58513 group.cpp:655] Trying to get
> '/znode/replicas/0000000000' in ZooKeeper
> I0423 02:22:33.584410 58520 master.cpp:940] The newly elected leader is
> [email protected]:34577 with id 20140423-022233-1828659978-34577-58498
> I0423 02:22:33.584434 58520 master.cpp:950] Elected as the leading master!
> I0423 02:22:33.584473 58520 master.cpp:773] Recovering from registrar
> I0423 02:22:33.584812 58529 registrar.cpp:217] Recovering registrar
> I0423 02:22:33.585399 58514 network.hpp:451] ZooKeeper group PIDs: {
> log-replica(6)@10.35.255.108:34577 }
> I0423 02:22:33.586002 58524 log.cpp:639] Attempting to start the writer
> I0423 02:22:33.587383 58529 replica.cpp:474] Replica received implicit
> promise request with proposal 1
> I0423 02:22:33.617635 58529 leveldb.cpp:304] Persisting metadata (8 bytes) to
> leveldb took 30.198423ms
> I0423 02:22:33.617684 58529 replica.cpp:342] Persisted promised to 1
> I0423 02:22:33.618604 58520 coordinator.cpp:229] Coordinator attemping to
> fill missing position
> I0423 02:22:33.621388 58518 replica.cpp:375] Replica received explicit
> promise request for position 0 with proposal 2
> I0423 02:22:33.642787 58518 leveldb.cpp:341] Persisting action (8 bytes) to
> leveldb took 21.299332ms
> I0423 02:22:33.642882 58518 replica.cpp:664] Persisted action at 0
> I0423 02:22:33.644448 58518 replica.cpp:508] Replica received write request
> for position 0
> I0423 02:22:33.644603 58518 leveldb.cpp:436] Reading position from leveldb
> took 23568ns
> I0423 02:22:33.651110 58518 leveldb.cpp:341] Persisting action (14 bytes) to
> leveldb took 6.450127ms
> I0423 02:22:33.651197 58518 replica.cpp:664] Persisted action at 0
> I0423 02:22:33.651711 58533 replica.cpp:643] Replica received learned notice
> for position 0
> I0423 02:22:33.659396 58533 leveldb.cpp:341] Persisting action (16 bytes) to
> leveldb took 7.652094ms
> I0423 02:22:33.659435 58533 replica.cpp:664] Persisted action at 0
> I0423 02:22:33.659462 58533 replica.cpp:649] Replica learned NOP action at
> position 0
> I0423 02:22:33.660080 58523 log.cpp:655] Writer started with ending position 0
> I0423 02:22:33.662197 58515 leveldb.cpp:436] Reading position from leveldb
> took 42564ns
> I0423 02:22:33.666170 58533 registrar.cpp:243] Successfully recovered
> registrar
> I0423 02:22:33.666329 58533 registrar.cpp:314] Attempting to update the
> 'registry'
> I0423 02:22:33.670188 58523 log.cpp:663] Attempting to append 155 bytes to
> the log
> I0423 02:22:33.670377 58520 coordinator.cpp:339] Coordinator attempting to
> write APPEND action at position 1
> I0423 02:22:33.672163 58522 replica.cpp:508] Replica received write request
> for position 1
> I0423 02:22:33.676118 58522 leveldb.cpp:341] Persisting action (174 bytes) to
> leveldb took 3.910783ms
> I0423 02:22:33.676208 58522 replica.cpp:664] Persisted action at 1
> I0423 02:22:33.677208 58533 replica.cpp:643] Replica received learned notice
> for position 1
> I0423 02:22:33.684420 58533 leveldb.cpp:341] Persisting action (176 bytes) to
> leveldb took 7.123224ms
> I0423 02:22:33.684505 58533 replica.cpp:664] Persisted action at 1
> I0423 02:22:33.684530 58533 replica.cpp:649] Replica learned APPEND action at
> position 1
> I0423 02:22:33.685788 58533 registrar.cpp:356] Successfully updated 'registry'
> I0423 02:22:33.686305 58520 log.cpp:682] Attempting to truncate the log to 1
> I0423 02:22:33.686362 58528 master.cpp:800] Recovered 0 slaves from the
> Registry (117B) ; allowing 10mins for slaves to re-register
> I0423 02:22:33.686494 58519 coordinator.cpp:339] Coordinator attempting to
> write TRUNCATE action at position 2
> I0423 02:22:33.687501 58529 replica.cpp:508] Replica received write request
> for position 2
> 2014-04-23 02:22:33,688:58498(0x7f0c104a1940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> 2014-04-23 02:22:33,688:58498(0x7f0c104a1940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> 2014-04-23 02:22:33,688:58498(0x7f0c104a1940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2014-04-23 02:22:33,688:58498(0x7f0c104a1940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> 2014-04-23 02:22:33,688:58498(0x7f0c104a1940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> 2014-04-23 02:22:33,688:58498(0x7f0c104a1940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> 2014-04-23 02:22:33,688:58498(0x7f0c104a1940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2014-04-23 02:22:33,689:58498(0x7f0c104a1940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,689:58498(0x7f0c104a1940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null>
> context=0x7f0c002804f0 flags=0
> I0423 02:22:33.689520 58527 slave.cpp:129] Slave started on
> 1)@10.35.255.108:34577
> 2014-04-23 02:22:33,689:58498(0x7f0bc1a6f940):ZOO_INFO@check_events@1703:
> initiated connection to server [127.0.0.1:52633]
> I0423 02:22:33.689576 58527 slave.cpp:138] Moving slave process into its own
> cgroup
> 2014-04-23 02:22:33,690:58498(0x7f0c08491940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> I0423 02:22:33.690764 58498 sched.cpp:121] Version: 0.19.0
> 2014-04-23 02:22:33,690:58498(0x7f0c08491940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> 2014-04-23 02:22:33,690:58498(0x7f0c08491940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2014-04-23 02:22:33,690:58498(0x7f0c08491940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> 2014-04-23 02:22:33,690:58498(0x7f0c08491940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> 2014-04-23 02:22:33,691:58498(0x7f0c08491940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> 2014-04-23 02:22:33,691:58498(0x7f0c08491940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2014-04-23 02:22:33,691:58498(0x7f0c08491940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,691:58498(0x7f0c08491940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null> context=0x1c9b7d0
> flags=0
> 2014-04-23 02:22:33,691:58498(0x7f0bc0a6d940):ZOO_INFO@check_events@1703:
> initiated connection to server [127.0.0.1:52633]
> I0423 02:22:33.692736 58529 leveldb.cpp:341] Persisting action (16 bytes) to
> leveldb took 5.127297ms
> I0423 02:22:33.692821 58529 replica.cpp:664] Persisted action at 2
> I0423 02:22:33.693465 58524 replica.cpp:643] Replica received learned notice
> for position 2
> 2014-04-23 02:22:33,698:58498(0x7f0bc1a6f940):ZOO_INFO@check_events@1750:
> session establishment complete on server [127.0.0.1:52633],
> sessionId=0x1458c6446740004, negotiated timeout=10000
> I0423 02:22:33.699019 58516 group.cpp:310] Group process
> ((116)@10.35.255.108:34577) connected to ZooKeeper
> I0423 02:22:33.699064 58516 group.cpp:781] Syncing group operations: queue
> size (joins, cancels, datas) = (0, 0, 0)
> I0423 02:22:33.699084 58516 group.cpp:382] Trying to create path '/znode' in
> ZooKeeper
> I0423 02:22:33.701099 58524 leveldb.cpp:341] Persisting action (18 bytes) to
> leveldb took 7.583491ms
> I0423 02:22:33.701223 58524 leveldb.cpp:399] Deleting ~1 keys from leveldb
> took 33262ns
> I0423 02:22:33.701306 58524 replica.cpp:664] Persisted action at 2
> I0423 02:22:33.701359 58524 replica.cpp:649] Replica learned TRUNCATE action
> at position 2
> 2014-04-23 02:22:33,706:58498(0x7f0bc0a6d940):ZOO_INFO@check_events@1750:
> session establishment complete on server [127.0.0.1:52633],
> sessionId=0x1458c6446740005, negotiated timeout=10000
> I0423 02:22:33.707068 58522 group.cpp:310] Group process
> ((122)@10.35.255.108:34577) connected to ZooKeeper
> I0423 02:22:33.707278 58522 group.cpp:781] Syncing group operations: queue
> size (joins, cancels, datas) = (0, 0, 0)
> I0423 02:22:33.707310 58522 group.cpp:382] Trying to create path '/znode' in
> ZooKeeper
> I0423 02:22:33.709638 58516 detector.cpp:134] Detected a new leader: (id='0')
> I0423 02:22:33.709689 58529 detector.cpp:134] Detected a new leader: (id='0')
> I0423 02:22:33.709972 58534 group.cpp:655] Trying to get
> '/znode/info_0000000000' in ZooKeeper
> I0423 02:22:33.710321 58524 group.cpp:655] Trying to get
> '/znode/info_0000000000' in ZooKeeper
> I0423 02:22:33.712035 58531 detector.cpp:376] A new leading master
> ([email protected]:34577) is detected
> I0423 02:22:33.712049 58523 detector.cpp:376] A new leading master
> ([email protected]:34577) is detected
> I0423 02:22:33.712280 58517 sched.cpp:217] New master detected at
> [email protected]:34577
> I0423 02:22:33.712394 58517 sched.cpp:268] Authenticating with master
> [email protected]:34577
> I0423 02:22:33.712681 58532 authenticatee.hpp:104] Initializing client SASL
> I0423 02:22:33.714192 58532 authenticatee.hpp:128] Creating new client SASL
> connection
> I0423 02:22:33.714577 58533 master.cpp:2715] Authenticating framework at
> scheduler(1)@10.35.255.108:34577
> I0423 02:22:33.715448 58535 authenticator.hpp:91] Initializing server SASL
> I0423 02:22:33.716045 58535 authenticator.hpp:148] Creating new server SASL
> connection
> I0423 02:22:33.716263 58535 authenticatee.hpp:219] Received SASL
> authentication mechanisms: CRAM-MD5
> I0423 02:22:33.716431 58535 authenticatee.hpp:245] Attempting to authenticate
> with mechanism 'CRAM-MD5'
> I0423 02:22:33.716552 58526 authenticator.hpp:254] Received SASL
> authentication start
> I0423 02:22:33.716640 58526 authenticator.hpp:342] Authentication requires
> more steps
> I0423 02:22:33.716739 58516 authenticatee.hpp:265] Received SASL
> authentication step
> I0423 02:22:33.716943 58533 authenticator.hpp:282] Received SASL
> authentication step
> I0423 02:22:33.717118 58533 authenticator.hpp:334] Authentication success
> I0423 02:22:33.717270 58526 authenticatee.hpp:305] Authentication success
> I0423 02:22:33.717320 58533 master.cpp:2755] Successfully authenticated
> framework at scheduler(1)@10.35.255.108:34577
> I0423 02:22:33.717632 58526 sched.cpp:342] Successfully authenticated with
> master [email protected]:34577
> I0423 02:22:33.717903 58533 master.cpp:997] Received registration request
> from scheduler(1)@10.35.255.108:34577
> I0423 02:22:33.718073 58533 master.cpp:1015] Registering framework
> 20140423-022233-1828659978-34577-58498-0000 at
> scheduler(1)@10.35.255.108:34577
> I0423 02:22:33.718497 58530 sched.cpp:392] Framework registered with
> 20140423-022233-1828659978-34577-58498-0000
> I0423 02:22:33.718652 58513 hierarchical_allocator_process.hpp:332] Added
> framework 20140423-022233-1828659978-34577-58498-0000
> I0423 02:22:33.766744 58527 slave.cpp:138] Moving slave process into its own
> cgroup
> I0423 02:22:33.817608 58527 slave.cpp:217] Slave resources: cpus(*):2;
> mem(*):1024; disk(*):1024; ports(*):[31000-32000]
> I0423 02:22:33.818116 58527 slave.cpp:245] Slave hostname:
> smfd-atr-11-sr1.devel.twitter.com
> I0423 02:22:33.818147 58527 slave.cpp:246] Slave checkpoint: false
> I0423 02:22:33.819965 58526 state.cpp:33] Recovering state from
> '/tmp/MasterZooKeeperTest_LostZooKeeperCluster_AwmIH4/meta'
> I0423 02:22:33.820495 58533 status_update_manager.cpp:193] Recovering status
> update manager
> I0423 02:22:33.821004 58533 slave.cpp:2791] Finished recovery
> I0423 02:22:33.821710 58521 slave.cpp:496] New master detected at
> [email protected]:34577
> I0423 02:22:33.821869 58533 status_update_manager.cpp:167] New master
> detected at [email protected]:34577
> I0423 02:22:33.822451 58521 slave.cpp:521] Detecting new master
> I0423 02:22:33.823314 58514 registrar.cpp:314] Attempting to update the
> 'registry'
> I0423 02:22:33.826674 58524 log.cpp:663] Attempting to append 382 bytes to
> the log
> I0423 02:22:33.826882 58524 coordinator.cpp:339] Coordinator attempting to
> write APPEND action at position 3
> I0423 02:22:33.828371 58514 replica.cpp:508] Replica received write request
> for position 3
> I0423 02:22:33.842730 58514 leveldb.cpp:341] Persisting action (401 bytes) to
> leveldb took 14.324743ms
> I0423 02:22:33.842820 58514 replica.cpp:664] Persisted action at 3
> I0423 02:22:33.843473 58531 replica.cpp:643] Replica received learned notice
> for position 3
> I0423 02:22:33.851027 58531 leveldb.cpp:341] Persisting action (403 bytes) to
> leveldb took 7.469229ms
> I0423 02:22:33.851112 58531 replica.cpp:664] Persisted action at 3
> I0423 02:22:33.851140 58531 replica.cpp:649] Replica learned APPEND action at
> position 3
> I0423 02:22:33.852147 58517 registrar.cpp:356] Successfully updated 'registry'
> I0423 02:22:33.852347 58530 log.cpp:682] Attempting to truncate the log to 3
> I0423 02:22:33.852566 58522 master.cpp:2150] Admitted slave on
> smfd-atr-11-sr1.devel.twitter.com at slave(1)@10.35.255.108:34577
> I0423 02:22:33.852670 58522 master.cpp:3204] Adding slave
> 20140423-022233-1828659978-34577-58498-0 at smfd-atr-11-sr1.devel.twitter.com
> with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
> I0423 02:22:33.852726 58533 coordinator.cpp:339] Coordinator attempting to
> write TRUNCATE action at position 4
> I0423 02:22:33.853090 58517 slave.cpp:539] Registered with master
> [email protected]:34577; given slave ID
> 20140423-022233-1828659978-34577-58498-0
> I0423 02:22:33.854205 58515 hierarchical_allocator_process.hpp:445] Added
> slave 20140423-022233-1828659978-34577-58498-0
> (smfd-atr-11-sr1.devel.twitter.com) with cpus(*):2; mem(*):1024;
> disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024;
> disk(*):1024; ports(*):[31000-32000] available)
> I0423 02:22:33.854622 58518 replica.cpp:508] Replica received write request
> for position 4
> 2014-04-23
> 02:22:33,855:58498(0x7f0bc3a73940):ZOO_ERROR@handle_socket_error_msg@1721:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=112(Host is down): failed while
> receiving a server response
> I0423 02:22:33.855417 58515 master.cpp:2676] Sending 1 offers to framework
> 20140423-022233-1828659978-34577-58498-0000
> 2014-04-23
> 02:22:33,855:58498(0x7f0bc5478940):ZOO_ERROR@handle_socket_error_msg@1721:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=112(Host is down): failed while
> receiving a server response
> I0423 02:22:33.855680 58517 group.cpp:415] Lost connection to ZooKeeper,
> attempting to reconnect ...
> 2014-04-23
> 02:22:33,855:58498(0x7f0bc4a75940):ZOO_ERROR@handle_socket_error_msg@1721:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=112(Host is down): failed while
> receiving a server response
> 2014-04-23
> 02:22:33,856:58498(0x7f0bc1a6f940):ZOO_ERROR@handle_socket_error_msg@1721:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=112(Host is down): failed while
> receiving a server response
> 2014-04-23
> 02:22:33,856:58498(0x7f0bc2a71940):ZOO_ERROR@handle_socket_error_msg@1721:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=112(Host is down): failed while
> receiving a server response
> I0423 02:22:33.856515 58526 group.cpp:415] Lost connection to ZooKeeper,
> attempting to reconnect ...
> I0423 02:22:33.856559 58525 group.cpp:415] Lost connection to ZooKeeper,
> attempting to reconnect ...
> I0423 02:22:33.856650 58530 group.cpp:415] Lost connection to ZooKeeper,
> attempting to reconnect ...
> 2014-04-23
> 02:22:33,856:58498(0x7f0bc0a6d940):ZOO_ERROR@handle_socket_error_msg@1721:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=112(Host is down): failed while
> receiving a server response
> I0423 02:22:33.856796 58535 group.cpp:415] Lost connection to ZooKeeper,
> attempting to reconnect ...
> I0423 02:22:33.856910 58529 group.cpp:415] Lost connection to ZooKeeper,
> attempting to reconnect ...
> I0423 02:22:33.857614 58498 zookeeper_test_server.cpp:122] Shutdown
> ZooKeeperTestServer on port 52633
> W0423 02:22:33.858124 58532 group.cpp:453] Timed out waiting to reconnect to
> ZooKeeper. Forcing ZooKeeper session (sessionId=1458c6446740001) expiration
> W0423 02:22:33.858194 58521 group.cpp:453] Timed out waiting to reconnect to
> ZooKeeper. Forcing ZooKeeper session (sessionId=1458c6446740004) expiration
> I0423 02:22:33.858209 58532 group.cpp:469] ZooKeeper session expired
> I0423 02:22:33.858232 58521 group.cpp:469] ZooKeeper session expired
> W0423 02:22:33.858266 58535 group.cpp:453] Timed out waiting to reconnect to
> ZooKeeper. Forcing ZooKeeper session (sessionId=1458c6446740002) expiration
> W0423 02:22:33.858356 58526 group.cpp:453] Timed out waiting to reconnect to
> ZooKeeper. Forcing ZooKeeper session (sessionId=1458c6446740000) expiration
> I0423 02:22:33.858410 58535 group.cpp:469] ZooKeeper session expired
> W0423 02:22:33.858453 58525 group.cpp:453] Timed out waiting to reconnect to
> ZooKeeper. Forcing ZooKeeper session (sessionId=1458c6446740005) expiration
> W0423 02:22:33.858391 58531 group.cpp:453] Timed out waiting to reconnect to
> ZooKeeper. Forcing ZooKeeper session (sessionId=1458c6446740003) expiration
> I0423 02:22:33.858507 58526 group.cpp:469] ZooKeeper session expired
> I0423 02:22:33.858486 58525 group.cpp:469] ZooKeeper session expired
> I0423 02:22:33.858474 58533 log.cpp:360] Renewing replica group membership
> 2014-04-23 02:22:33,858:58498(0x7f0c0f49f940):ZOO_INFO@zookeeper_close@2522:
> Freeing zookeeper resources for sessionId=0x1458c6446740004
> I0423 02:22:33.858518 58531 group.cpp:469] ZooKeeper session expired
> 2014-04-23 02:22:33,858:58498(0x7f0c0f49f940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> I0423 02:22:33.858690 58513 detector.cpp:134] Detected a new leader: None
> 2014-04-23 02:22:33,858:58498(0x7f0c0f49f940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> I0423 02:22:33.858649 58520 network.hpp:413] ZooKeeper group memberships
> changed
> 2014-04-23 02:22:33,858:58498(0x7f0c0f49f940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2014-04-23 02:22:33,858:58498(0x7f0c0f49f940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> I0423 02:22:33.858753 58519 contender.cpp:192] Membership cancelled: 0
> 2014-04-23 02:22:33,858:58498(0x7f0c0f49f940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> I0423 02:22:33.858796 58517 detector.cpp:134] Detected a new leader: None
> 2014-04-23 02:22:33,858:58498(0x7f0c0f49f940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> I0423 02:22:33.858533 58516 detector.cpp:134] Detected a new leader: None
> 2014-04-23 02:22:33,859:58498(0x7f0c09c94940):ZOO_INFO@zookeeper_close@2522:
> Freeing zookeeper resources for sessionId=0x1458c6446740001
> I0423 02:22:33.859089 58529 sched.cpp:233] No master detected
> 2014-04-23 02:22:33,859:58498(0x7f0c08491940):ZOO_INFO@zookeeper_close@2522:
> Freeing zookeeper resources for sessionId=0x1458c6446740002
> 2014-04-23 02:22:33,859:58498(0x7f0c09c94940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> 2014-04-23 02:22:33,859:58498(0x7f0c0f49f940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2014-04-23 02:22:33,859:58498(0x7f0c09c94940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> 2014-04-23 02:22:33,859:58498(0x7f0c09c94940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> I0423 02:22:33.859365 58527 slave.cpp:517] Lost leading master
> 2014-04-23 02:22:33,859:58498(0x7f0c08491940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> 2014-04-23 02:22:33,859:58498(0x7f0c0cc9a940):ZOO_INFO@zookeeper_close@2522:
> Freeing zookeeper resources for sessionId=0x1458c6446740000
> 2014-04-23 02:22:33,859:58498(0x7f0c09c94940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> I0423 02:22:33.859379 58518 leveldb.cpp:341] Persisting action (16 bytes) to
> leveldb took 4.705146ms
> 2014-04-23 02:22:33,859:58498(0x7f0c09c94940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> 2014-04-23 02:22:33,859:58498(0x7f0c0f49f940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,859:58498(0x7f0c0d49b940):ZOO_INFO@zookeeper_close@2522:
> Freeing zookeeper resources for sessionId=0x1458c6446740005
> I0423 02:22:33.859505 58527 slave.cpp:521] Detecting new master
> 2014-04-23 02:22:33,859:58498(0x7f0c0a495940):ZOO_INFO@zookeeper_close@2522:
> Freeing zookeeper resources for sessionId=0x1458c6446740003
> 2014-04-23 02:22:33,859:58498(0x7f0c0cc9a940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> I0423 02:22:33.859549 58520 network.hpp:451] ZooKeeper group PIDs: { }
> I0423 02:22:33.859591 58518 replica.cpp:664] Persisted action at 4
> 2014-04-23 02:22:33,859:58498(0x7f0c09c94940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> 2014-04-23 02:22:33,859:58498(0x7f0c08491940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> 2014-04-23 02:22:33,859:58498(0x7f0c0d49b940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> 2014-04-23 02:22:33,859:58498(0x7f0c08491940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2014-04-23 02:22:33,859:58498(0x7f0c0a495940):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> 2014-04-23 02:22:33,859:58498(0x7f0c08491940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> 2014-04-23 02:22:33,859:58498(0x7f0c0f49f940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null> context=0x1cc2d40
> flags=0
> 2014-04-23 02:22:33,859:58498(0x7f0c08491940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> 2014-04-23 02:22:33,859:58498(0x7f0c09c94940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2014-04-23 02:22:33,859:58498(0x7f0c0cc9a940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> 2014-04-23 02:22:33,859:58498(0x7f0c0d49b940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> 2014-04-23 02:22:33,859:58498(0x7f0c0cc9a940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2014-04-23 02:22:33,859:58498(0x7f0c0d49b940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2014-04-23 02:22:33,859:58498(0x7f0c0d49b940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> 2014-04-23 02:22:33,859:58498(0x7f0c0a495940):ZOO_INFO@log_env@716: Client
> environment:host.name=smfd-atr-11-sr1.devel.twitter.com
> 2014-04-23 02:22:33,859:58498(0x7f0c0d49b940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> 2014-04-23 02:22:33,859:58498(0x7f0c0a495940):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2014-04-23 02:22:33,859:58498(0x7f0c09c94940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,859:58498(0x7f0c08491940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> 2014-04-23 02:22:33,859:58498(0x7f0c0cc9a940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> 2014-04-23 02:22:33,859:58498(0x7f0c0a495940):ZOO_INFO@log_env@724: Client
> environment:os.arch=2.6.44-t14.el5
> 2014-04-23
> 02:22:33,860:58498(0x7f0bc2270940):ZOO_ERROR@handle_socket_error_msg@1697:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=111(Connection refused): server
> refused to accept the client
> 2014-04-23 02:22:33,860:58498(0x7f0c0a495940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> 2014-04-23 02:22:33,860:58498(0x7f0c0cc9a940):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Thu Dec 19 12:29:49 PST 2013
> 2014-04-23 02:22:33,859:58498(0x7f0c0d49b940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> 2014-04-23 02:22:33,860:58498(0x7f0c0cc9a940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> 2014-04-23 02:22:33,860:58498(0x7f0c09c94940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null>
> context=0x7f0bbc05b7d0 flags=0
> 2014-04-23 02:22:33,860:58498(0x7f0c0a495940):ZOO_INFO@log_env@733: Client
> environment:user.name=vinod
> 2014-04-23 02:22:33,860:58498(0x7f0c08491940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2014-04-23 02:22:33,860:58498(0x7f0c0d49b940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2014-04-23 02:22:33,860:58498(0x7f0c08491940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,860:58498(0x7f0c0a495940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2014-04-23 02:22:33,860:58498(0x7f0c0d49b940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,860:58498(0x7f0c0cc9a940):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2014-04-23 02:22:33,860:58498(0x7f0c0a495940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,860:58498(0x7f0c0d49b940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null>
> context=0x7f0bf80080a0 flags=0
> 2014-04-23 02:22:33,860:58498(0x7f0c0cc9a940):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/vinod/mesos/build
> 2014-04-23 02:22:33,860:58498(0x7f0c0a495940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null> context=0x1d2b120
> flags=0
> 2014-04-23 02:22:33,860:58498(0x7f0c08491940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null>
> context=0x7f0c0031cda0 flags=0
> 2014-04-23
> 02:22:33,860:58498(0x7f0bc4274940):ZOO_ERROR@handle_socket_error_msg@1697:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=111(Connection refused): server
> refused to accept the client
> 2014-04-23 02:22:33,860:58498(0x7f0c0cc9a940):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=127.0.0.1:52633 sessionTimeout=10000
> watcher=0x7f0c18b10d5f sessionId=0 sessionPasswd=<null>
> context=0x7f0c0027bdf0 flags=0
> F0423 02:22:33.860647 58518 coordinator.cpp:399] Check failed: !missing Not
> expecting local replica to be missing position 4 after the writing is done
> 2014-04-23
> 02:22:33,860:58498(0x7f0bc647a940):ZOO_ERROR@handle_socket_error_msg@1697:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=111(Connection refused): server
> refused to accept the client
> 2014-04-23
> 02:22:33,860:58498(0x7f0bc4c77940):ZOO_ERROR@handle_socket_error_msg@1697:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=111(Connection refused): server
> refused to accept the client
> 2014-04-23
> 02:22:33,860:58498(0x7f0bc1a6f940):ZOO_ERROR@handle_socket_error_msg@1697:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=111(Connection refused): server
> refused to accept the client
> 2014-04-23
> 02:22:33,860:58498(0x7f0bc3272940):ZOO_ERROR@handle_socket_error_msg@1697:
> Socket [127.0.0.1:52633] zk retcode=-4, errno=111(Connection refused): server
> refused to accept the client
> *** Check failure stack trace: ***
> @ 0x7f0c18f91754 google::LogMessage::Fail()
> @ 0x7f0c18f916a3 google::LogMessage::SendToLog()
> @ 0x7f0c18f910b4 google::LogMessage::Flush()
> @ 0x7f0c18f93de8 google::LogMessageFatal::~LogMessageFatal()
> @ 0x7f0c18cbc0c5
> mesos::internal::log::CoordinatorProcess::updateIndexAfterWritten()
> @ 0x7f0c18cc4bf7
> _ZZN7process8dispatchI6OptionImEN5mesos8internal3log18CoordinatorProcessEbbEENS_6FutureIT_EERKNS_3PIDIT0_EEMSB_FS9_T1_ET2_ENKUlPNS_11ProcessBaseEE_clESK_
> @ 0x7f0c18cd2227
> _ZNSt17_Function_handlerIFvPN7process11ProcessBaseEEZNS0_8dispatchI6OptionImEN5mesos8internal3log18CoordinatorProcessEbbEENS0_6FutureIT_EERKNS0_3PIDIT0_EEMSF_FSD_T1_ET2_EUlS2_E_E9_M_invokeERKSt9_Any_dataS2_
> @ 0x7f0c18e58b4d std::function<>::operator()()
> @ 0x7f0c18e38295 process::ProcessBase::visit()
> @ 0x7f0c18e42898 process::DispatchEvent::visit()
> @ 0x7ba322 process::ProcessBase::serve()
> @ 0x7f0c18e345a6 process::ProcessManager::resume()
> @ 0x7f0c18e2b36b process::schedule()
> @ 0x7f0c1699883d start_thread
> @ 0x7f0c156ff26d clone
> /var/tmp/sclUrjwpu: line 8: 58498 Aborted
> './bin/mesos-tests.sh' '--gtest_filter=*ZooKeeper*' '--verbose'
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
