Hi,
I've setup a test mesos cluster on a few VM's running locally. I have
three masters and two slaves
masters : 192.168.48.14[5 - 7]
slaves : 192.168.48.15[0 - 1]
The masters startup correctly and are able to elect a leader. The slaves
can find the master and register, but for some reason they immediately
disconnect.
*On the master (mesos-master.INFO)*
master.cpp:3122] Registered slave 20141215-160321-2435885248-5050-20424-S68
at slave(1)@127.0.1.1:5051 (192.168.48.150) with cpus(*):1; mem(*):489;
disk(*):13901; ports(*):[31000-32000]
I1215 16:15:51.970082 20448 hierarchical_allocator_process.hpp:442] Added
slave 20141215-160321-2435885248-5050-20424-S68 (192.168.48.150) with
cpus(*):1; mem(*):489; disk(*):13901; ports(*):[31000-32000] (and
cpus(*):1; mem(*):489; disk(*):13901; ports(*):[31000-32000] available)
I1215 16:15:51.970474 20454 master.cpp:839] Slave
20141215-160321-2435885248-5050-20424-S68 at slave(1)@127.0.1.1:5051
(192.168.48.150) disconnected
I1215 16:15:51.970546 20454 master.cpp:1789] Disconnecting slave
20141215-160321-2435885248-5050-20424-S68 at slave(1)@127.0.1.1:5051
(192.168.48.150)
I1215 16:15:51.970612 20454 master.cpp:1808] Deactivating slave
20141215-160321-2435885248-5050-20424-S68 at slave(1)@127.0.1.1:5051
(192.168.48.150)
I1215 16:15:51.970772 20454 hierarchical_allocator_process.hpp:481] Slave
20141215-160321-2435885248-5050-20424-S68 deactivated
I1215 16:15:51.975980 20453 replica.cpp:655] Replica received learned
notice for position 276
I1215 16:15:51.977501 20453 leveldb.cpp:343] Persisting action (20 bytes)
to leveldb took 1.475474ms
I1215 16:15:51.977625 20453 leveldb.cpp:401] Deleting ~2 keys from leveldb
took 50280ns
*On the slave (mesos-slave.INFO)*
Dec 15 16:06:09 ubuntu mesos-slave[18118]: 2014-12-15
16:06:09,209:18118(0x7fa67d700700):ZOO_INFO@check_events@1750: session
establishment complete on server [192.168.48.147:2181],
sessionId=0x34a5067fd9e0001, negotiated timeout=10000
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.210183 18140
group.cpp:313] Group process (group(1)@127.0.1.1:5051) connected to
ZooKeeper
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.210248 18140
group.cpp:790] Syncing group operations: queue size (joins, cancels, datas)
= (0, 0, 0)
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.210270 18140
group.cpp:385] Trying to create path '/mesos' in ZooKeeper
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.213835 18140
detector.cpp:138] Detected a new leader: (id='55')
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.214570 18140
group.cpp:659] Trying to get '/mesos/info_0000000055' in ZooKeeper
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.215833 18141
detector.cpp:433] A new leading master ([email protected]:5050) is
detected
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.220592 18141
state.cpp:33] Recovering state from '/home/agh/mesos-work/meta'
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.220757 18141
state.cpp:62] Failed to find the latest slave from
'/home/agh/mesos-work/meta'
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.226416 18136
status_update_manager.cpp:197] Recovering status update manager
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.226963 18134
containerizer.cpp:281] Recovering containerizer
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.228973 18135
slave.cpp:3466] Finished recovery
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.230242 18137
status_update_manager.cpp:171] Pausing sending status updates
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.230450 18135
slave.cpp:602] New master detected at [email protected]:5050
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.230873 18135
slave.cpp:627] No credentials provided. Attempting to register without
authentication
Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.231045 18135
slave.cpp:638] Detecting new master
Dec 15 16:07:09 ubuntu mesos-slave[18118]: I1215 16:07:09.225389 18141
slave.cpp:3321] Current usage 12.01%. Max allowed age: 5.459239732780289days
Dec 15 16:08:09 ubuntu mesos-slave[18118]: I1215 16:08:09.228869 18141
slave.cpp:3321] Current usage 12.01%. Max allowed age: 5.459239732780289days
Dec 15 16:09:09 ubuntu mesos-slave[18118]: I1215 16:09:09.252048 18141
slave.cpp:3321] Current usage 12.01%. Max allowed age: 5.459239732780289days
Dec 15 16:09:27 ubuntu mesos-slave[18118]: I1215 16:09:27.288277 18141
http.cpp:330] HTTP request for '/slave(1)/state.json'
Dec 15 16:10:09 ubuntu mesos-slave[18118]: I1215 16:10:09.271672 18138
slave.cpp:3321] Current usage 12.01%. Max allowed age: 5.459239732780289days
It does not look like the slave is disconnecting, so why does the master
think the slave keeps disconnecting and deactivate the slave ?
Thanks,
Arunabha
