Is there anything in the ERROR/WARNING logs? Tim
On Mon, Dec 15, 2014 at 4:22 PM, Arunabha Ghosh <[email protected]> wrote: > > Hi, > I've setup a test mesos cluster on a few VM's running locally. I have > three masters and two slaves > > masters : 192.168.48.14[5 - 7] > slaves : 192.168.48.15[0 - 1] > > The masters startup correctly and are able to elect a leader. The slaves > can find the master and register, but for some reason they immediately > disconnect. > > > *On the master (mesos-master.INFO)* > > master.cpp:3122] Registered slave > 20141215-160321-2435885248-5050-20424-S68 at slave(1)@127.0.1.1:5051 > (192.168.48.150) with cpus(*):1; mem(*):489; disk(*):13901; > ports(*):[31000-32000] > I1215 16:15:51.970082 20448 hierarchical_allocator_process.hpp:442] Added > slave 20141215-160321-2435885248-5050-20424-S68 (192.168.48.150) with > cpus(*):1; mem(*):489; disk(*):13901; ports(*):[31000-32000] (and > cpus(*):1; mem(*):489; disk(*):13901; ports(*):[31000-32000] available) > I1215 16:15:51.970474 20454 master.cpp:839] Slave > 20141215-160321-2435885248-5050-20424-S68 at slave(1)@127.0.1.1:5051 > (192.168.48.150) disconnected > I1215 16:15:51.970546 20454 master.cpp:1789] Disconnecting slave > 20141215-160321-2435885248-5050-20424-S68 at slave(1)@127.0.1.1:5051 > (192.168.48.150) > I1215 16:15:51.970612 20454 master.cpp:1808] Deactivating slave > 20141215-160321-2435885248-5050-20424-S68 at slave(1)@127.0.1.1:5051 > (192.168.48.150) > I1215 16:15:51.970772 20454 hierarchical_allocator_process.hpp:481] Slave > 20141215-160321-2435885248-5050-20424-S68 deactivated > I1215 16:15:51.975980 20453 replica.cpp:655] Replica received learned > notice for position 276 > I1215 16:15:51.977501 20453 leveldb.cpp:343] Persisting action (20 bytes) > to leveldb took 1.475474ms > I1215 16:15:51.977625 20453 leveldb.cpp:401] Deleting ~2 keys from leveldb > took 50280ns > > *On the slave (mesos-slave.INFO)* > > Dec 15 16:06:09 ubuntu mesos-slave[18118]: 2014-12-15 > 16:06:09,209:18118(0x7fa67d700700):ZOO_INFO@check_events@1750: session > establishment complete on server [192.168.48.147:2181], > sessionId=0x34a5067fd9e0001, negotiated timeout=10000 > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.210183 18140 > group.cpp:313] Group process (group(1)@127.0.1.1:5051) connected to > ZooKeeper > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.210248 18140 > group.cpp:790] Syncing group operations: queue size (joins, cancels, datas) > = (0, 0, 0) > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.210270 18140 > group.cpp:385] Trying to create path '/mesos' in ZooKeeper > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.213835 18140 > detector.cpp:138] Detected a new leader: (id='55') > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.214570 18140 > group.cpp:659] Trying to get '/mesos/info_0000000055' in ZooKeeper > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.215833 18141 > detector.cpp:433] A new leading master ([email protected]:5050) > is detected > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.220592 18141 > state.cpp:33] Recovering state from '/home/agh/mesos-work/meta' > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.220757 18141 > state.cpp:62] Failed to find the latest slave from > '/home/agh/mesos-work/meta' > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.226416 18136 > status_update_manager.cpp:197] Recovering status update manager > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.226963 18134 > containerizer.cpp:281] Recovering containerizer > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.228973 18135 > slave.cpp:3466] Finished recovery > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.230242 18137 > status_update_manager.cpp:171] Pausing sending status updates > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.230450 18135 > slave.cpp:602] New master detected at [email protected]:5050 > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.230873 18135 > slave.cpp:627] No credentials provided. Attempting to register without > authentication > Dec 15 16:06:09 ubuntu mesos-slave[18118]: I1215 16:06:09.231045 18135 > slave.cpp:638] Detecting new master > Dec 15 16:07:09 ubuntu mesos-slave[18118]: I1215 16:07:09.225389 18141 > slave.cpp:3321] Current usage 12.01%. Max allowed age: 5.459239732780289days > Dec 15 16:08:09 ubuntu mesos-slave[18118]: I1215 16:08:09.228869 18141 > slave.cpp:3321] Current usage 12.01%. Max allowed age: 5.459239732780289days > Dec 15 16:09:09 ubuntu mesos-slave[18118]: I1215 16:09:09.252048 18141 > slave.cpp:3321] Current usage 12.01%. Max allowed age: 5.459239732780289days > Dec 15 16:09:27 ubuntu mesos-slave[18118]: I1215 16:09:27.288277 18141 > http.cpp:330] HTTP request for '/slave(1)/state.json' > Dec 15 16:10:09 ubuntu mesos-slave[18118]: I1215 16:10:09.271672 18138 > slave.cpp:3321] Current usage 12.01%. Max allowed age: 5.459239732780289days > > It does not look like the slave is disconnecting, so why does the master > think the slave keeps disconnecting and deactivate the slave ? > > Thanks, > Arunabha > > > >
