Running mesos #1205738 (should I upgrade? if so, will it still work with the git head of Spark?)
I find often mesos-slave's are getting registered multiple times: I0420 23:52:37.429045 20020 master.cpp:814] Attempting to register slave 201204202347-0-8 at [email protected]:50961 I0420 23:52:37.429446 20020 master.cpp:1057] Master now considering a slave at ip-10-252-94-24.us-west-2.compute.internal:50961 as active I0420 23:52:37.429523 20020 master.cpp:1588] Adding slave 201204202347-0-8 at ip-10-252-94-24.us-west-2.compute.internal with cpus=2; mem=1184 I0420 23:52:37.429679 20020 simple_allocator.cpp:71] Added slave 201204202347-0-8 with cpus=2; mem=1184 I0420 23:52:38.429553 20020 master.cpp:814] Attempting to register slave 201204202347-0-9 at [email protected]:50961 I0420 23:52:38.429792 20020 master.cpp:1588] Adding slave 201204202347-0-9 at ip-10-252-94-24.us-west-2.compute.internal with cpus=2; mem=1184 I0420 23:52:38.429898 20020 simple_allocator.cpp:71] Added slave 201204202347-0-9 with cpus=2; mem=1184 I0420 23:52:39.430135 20020 master.cpp:814] Attempting to register slave 201204202347-0-10 at [email protected]:50961 I0420 23:52:39.430416 20020 master.cpp:1588] Adding slave 201204202347-0-10 at ip-10-252-94-24.us-west-2.compute.internal with cpus=2; mem=1184 I0420 23:52:39.430531 20020 simple_allocator.cpp:71] Added slave 201204202347-0-10 with cpus=2; mem=1184 I0420 23:52:40.431375 20020 master.cpp:814] Attempting to register slave 201204202347-0-11 at [email protected]:50961 I0420 23:52:40.431562 20020 master.cpp:1588] Adding slave 201204202347-0-11 at ip-10-252-94-24.us-west-2.compute.internal with cpus=2; mem=1184 I0420 23:52:40.431658 20020 simple_allocator.cpp:71] Added slave 201204202347-0-11 with cpus=2; mem=1184 I know for a fact mesos-slave is only being run once; I added a 'date >> file' line to the script that launches it and checked the file. It doesn't matter if I add a sleep, nohup, and/or stdin/out/err redirections. I have attached the tcpdump from that session. -- Scott
23:52:37.428402 IP ip-10-252-94-24.us-west-2.compute.internal.34456 > ip-10-252-3-41.us-west-2.compute.internal.mmcc: Flags [S], seq 1506576508, win 14600, options [mss 1460,sackOK,TS val 4294957978 ecr 0,nop,wscale 5], length 0 E..<S.@.;.t. .^. ..)....Y..|......9.?.......... ............ 23:52:37.428437 IP ip-10-252-3-41.us-west-2.compute.internal.mmcc > ip-10-252-94-24.us-west-2.compute.internal.34456: Flags [S.], seq 3358128289, ack 1506576509, win 14480, options [mss 1460,sackOK,TS val 27068077 ecr 4294957978,nop,wscale 5], length 0 E..<..@.@... ..) .^......(..Y..}..8.wg......... ............ 23:52:37.428733 IP ip-10-252-94-24.us-west-2.compute.internal.34456 > ip-10-252-3-41.us-west-2.compute.internal.mmcc: Flags [.], ack 1, win 457, options [nop,nop,TS val 4294957978 ecr 27068077], length 0 E..4S.@.;.t. .^. ..)....Y..}.(.......n..... ........ 23:52:37.428772 IP ip-10-252-94-24.us-west-2.compute.internal.34456 > ip-10-252-3-41.us-west-2.compute.internal.mmcc: Flags [P.], seq 1:309, ack 1, win 457, options [nop,nop,TS val 4294957978 ecr 27068077], length 308 E..hS.@.;.so .^. ..)....Y..}.(......p...... ........POST /master/mesos.internal.RegisterSlaveMessage HTTP/1.0 User-Agent: libprocess/[email protected]:50961 Connection: Keep-Alive Transfer-Encoding: chunked 87 .. *ip-10-252-94-24.us-west-2.compute.internal.*ip-10-252-94-24.us-west-2.compute.internal.. .cpus... .......@.. .mem... .......@ .? 0 23:52:37.428784 IP ip-10-252-3-41.us-west-2.compute.internal.mmcc > ip-10-252-94-24.us-west-2.compute.internal.34456: Flags [.], ack 309, win 486, options [nop,nop,TS val 27068077 ecr 4294957978], length 0 E..4..@.@..{ ..) .^......(..Y.......w_..... ........ 23:52:37.429625 IP ip-10-252-3-41.us-west-2.compute.internal.60900 > ip-10-252-94-24.us-west-2.compute.internal.50961: Flags [S], seq 835726631, win 14600, options [mss 1460,sackOK,TS val 27068078 ecr 0,nop,wscale 5], length 0 E..<+.@.@... ..) .^.....1.-'......9.wg......... ............ 23:52:38.429349 IP ip-10-252-94-24.us-west-2.compute.internal.34456 > ip-10-252-3-41.us-west-2.compute.internal.mmcc: Flags [P.], seq 309:617, ack 1, win 457, options [nop,nop,TS val 4294958228 ecr 27068077], length 308 E..hS.@.;.sn .^. ..)....Y....(......n...... ........POST /master/mesos.internal.RegisterSlaveMessage HTTP/1.0 User-Agent: libprocess/[email protected]:50961 Connection: Keep-Alive Transfer-Encoding: chunked 87 .. *ip-10-252-94-24.us-west-2.compute.internal.*ip-10-252-94-24.us-west-2.compute.internal.. .cpus... .......@.. .mem... .......@ .? 0 23:52:38.429369 IP ip-10-252-3-41.us-west-2.compute.internal.mmcc > ip-10-252-94-24.us-west-2.compute.internal.34456: Flags [.], ack 617, win 520, options [nop,nop,TS val 27068328 ecr 4294958228], length 0 E..4..@[email protected] ..) .^......(..Y.......w_..... ........ 23:52:39.429825 IP ip-10-252-94-24.us-west-2.compute.internal.34456 > ip-10-252-3-41.us-west-2.compute.internal.mmcc: Flags [P.], seq 617:925, ack 1, win 457, options [nop,nop,TS val 4294958478 ecr 27068328], length 308 E..hS.@.;.sm .^. ..)....Y....(......k...... ........POST /master/mesos.internal.RegisterSlaveMessage HTTP/1.0 User-Agent: libprocess/[email protected]:50961 Connection: Keep-Alive Transfer-Encoding: chunked 87 .. *ip-10-252-94-24.us-west-2.compute.internal.*ip-10-252-94-24.us-west-2.compute.internal.. .cpus... .......@.. .mem... .......@ .? 0 23:52:39.429857 IP ip-10-252-3-41.us-west-2.compute.internal.mmcc > ip-10-252-94-24.us-west-2.compute.internal.34456: Flags [.], ack 925, win 553, options [nop,nop,TS val 27068578 ecr 4294958478], length 0 E..4..@[email protected] ..) .^......(..Y......)w_..... ........ 23:52:40.431148 IP ip-10-252-94-24.us-west-2.compute.internal.34456 > ip-10-252-3-41.us-west-2.compute.internal.mmcc: Flags [P.], seq 925:1233, ack 1, win 457, options [nop,nop,TS val 4294958728 ecr 27068578], length 308 E..hS.@.;.sl .^. ..)....Y....(......hf..... ........POST /master/mesos.internal.RegisterSlaveMessage HTTP/1.0 User-Agent: libprocess/[email protected]:50961 Connection: Keep-Alive Transfer-Encoding: chunked 87 .. *ip-10-252-94-24.us-west-2.compute.internal.*ip-10-252-94-24.us-west-2.compute.internal.. .cpus... .......@.. .mem... .......@ .? 0 23:52:40.431170 IP ip-10-252-3-41.us-west-2.compute.internal.mmcc > ip-10-252-94-24.us-west-2.compute.internal.34456: Flags [.], ack 1233, win 587, options [nop,nop,TS val 27068828 ecr 4294958728], length 0 E..4..@[email protected] ..) .^......(..Y..M...Kw_..... .. ..... 23:52:40.436926 IP ip-10-252-3-41.us-west-2.compute.internal.60900 > ip-10-252-94-24.us-west-2.compute.internal.50961: Flags [S], seq 835726631, win 14600, options [mss 1460,sackOK,TS val 27068830 ecr 0,nop,wscale 5], length 0 E..<+.@.@... ..) .^.....1.-'......9.wg......... .. ......... 23:52:40.437263 IP ip-10-252-94-24.us-west-2.compute.internal.50961 > ip-10-252-3-41.us-west-2.compute.internal.60900: Flags [S.], seq 3088019921, ack 835726632, win 14480, options [mss 1460,sackOK,TS val 4294958730 ecr 27068830,nop,wscale 5], length 0 E..<..@.;... .^. ..)......q.1.-(..8.p.......... ...... ..... 23:52:40.437291 IP ip-10-252-3-41.us-west-2.compute.internal.60900 > ip-10-252-94-24.us-west-2.compute.internal.50961: Flags [.], ack 1, win 457, options [nop,nop,TS val 27068830 ecr 4294958730], length 0 E..4+.@.@... ..) .^.....1.-(..q.....w_..... .. ..... 23:52:40.437336 IP ip-10-252-3-41.us-west-2.compute.internal.60900 > ip-10-252-94-24.us-west-2.compute.internal.50961: Flags [P.], seq 1:194, ack 1, win 457, options [nop,nop,TS val 27068830 ecr 4294958730], length 193 E...+.@.@..> ..) .^.....1.-(..q.....x ..... .. .....POST /slave/mesos.internal.SlaveRegisteredMessage HTTP/1.0 User-Agent: libprocess/[email protected]:5050 Connection: Keep-Alive Transfer-Encoding: chunked 14 . .201204202347-0-8 0 23:52:40.437601 IP ip-10-252-94-24.us-west-2.compute.internal.50961 > ip-10-252-3-41.us-west-2.compute.internal.60900: Flags [.], ack 194, win 486, options [nop,nop,TS val 4294958730 ecr 27068830], length 0 E..4.X@.;..3 .^. ..)......q.1.-......F..... ...... . 23:52:40.437617 IP ip-10-252-3-41.us-west-2.compute.internal.60900 > ip-10-252-94-24.us-west-2.compute.internal.50961: Flags [P.], seq 194:1167, ack 1, win 457, options [nop,nop,TS val 27068830 ecr 4294958730], length 973 E...+.@[email protected] ..) .^.....1.-...q.....{,..... .. .....POST /slave/PING HTTP/1.0 User-Agent: libprocess/[email protected]:5050 Connection: Keep-Alive POST /slave/mesos.internal.SlaveRegisteredMessage HTTP/1.0 User-Agent: libprocess/[email protected]:5050 Connection: Keep-Alive Transfer-Encoding: chunked 14 . .201204202347-0-9 0 POST /slave/PING HTTP/1.0 User-Agent: libprocess/[email protected]:5050 Connection: Keep-Alive POST /slave/mesos.internal.SlaveRegisteredMessage HTTP/1.0 User-Agent: libprocess/[email protected]:5050 Connection: Keep-Alive Transfer-Encoding: chunked 15 . .201204202347-0-10 0 POST /slave/PING HTTP/1.0 User-Agent: libprocess/[email protected]:5050 Connection: Keep-Alive POST /slave/mesos.internal.SlaveRegisteredMessage HTTP/1.0 User-Agent: libprocess/[email protected]:5050 Connection: Keep-Alive Transfer-Encoding: chunked 15 . .201204202347-0-11 0 POST /slave/PING HTTP/1.0 User-Agent: libprocess/[email protected]:5050 Connection: Keep-Alive 23:52:40.437899 IP ip-10-252-94-24.us-west-2.compute.internal.50961 > ip-10-252-3-41.us-west-2.compute.internal.60900: Flags [.], ack 1167, win 547, options [nop,nop,TS val 4294958730 ecr 27068830], length 0 E..4.Y@.;..2 .^. ..)......q.1.1....#.<..... ...... . 23:52:40.438305 IP ip-10-252-94-24.us-west-2.compute.internal.34456 > ip-10-252-3-41.us-west-2.compute.internal.mmcc: Flags [P.], seq 1233:1333, ack 1, win 457, options [nop,nop,TS val 4294958730 ecr 27068828], length 100 E...S.@.;.t; .^. ..)....Y..M.(......o...... ...... .POST /338/PONG HTTP/1.0 User-Agent: libprocess/[email protected]:50961 Connection: Keep-Alive 23:52:40.438314 IP ip-10-252-3-41.us-west-2.compute.internal.mmcc > ip-10-252-94-24.us-west-2.compute.internal.34456: Flags [.], ack 1333, win 587, options [nop,nop,TS val 27068830 ecr 4294958730], length 0 E..4..@[email protected] ..) .^......(..Y......Kw_..... .. ..... 23:52:40.438582 IP ip-10-252-94-24.us-west-2.compute.internal.34456 > ip-10-252-3-41.us-west-2.compute.internal.mmcc: Flags [P.], seq 1333:1633, ack 1, win 457, options [nop,nop,TS val 4294958730 ecr 27068830], length 300 E..`S.@.;.sr .^. ..)....Y....(.......Z..... ...... .POST /342/PONG HTTP/1.0 User-Agent: libprocess/[email protected]:50961 Connection: Keep-Alive POST /346/PONG HTTP/1.0 User-Agent: libprocess/[email protected]:50961 Connection: Keep-Alive POST /350/PONG HTTP/1.0 User-Agent: libprocess/[email protected]:50961 Connection: Keep-Alive 23:52:40.438595 IP ip-10-252-3-41.us-west-2.compute.internal.mmcc > ip-10-252-94-24.us-west-2.compute.internal.34456: Flags [.], ack 1633, win 620, options [nop,nop,TS val 27068830 ecr 4294958730], length 0 E..4..@[email protected] ..) .^......(..Y......lw_..... .. .....
