stderr on the slave: WARNING: Logging before InitGoogleLogging() is written to STDERR I0818 19:06:55.033699 22 fetcher.cpp:73] Fetching URI 'http://downloads.mesosphere.io/storm/storm-mesos-0.9.tgz' I0818 19:06:55.033994 22 fetcher.cpp:123] Downloading 'http://downloads.mesosphere.io/storm/storm-mesos-0.9.tgz' to '/tmp/mesos/slaves/20140818-190538-2466255276-5050-11-0/frameworks/20140818-190538-2466255276-5050-11-0002/executors/wordcount-1-1408388814/runs/69496890-fc18-43f3-be87-198bceba7226/storm-mesos-0.9.tgz' I0818 19:07:11.567514 22 fetcher.cpp:61] Extracted resource '/tmp/mesos/slaves/20140818-190538-2466255276-5050-11-0/frameworks/20140818-190538-2466255276-5050-11-0002/executors/wordcount-1-1408388814/runs/69496890-fc18-43f3-be87-198bceba7226/storm-mesos-0.9.tgz' into '/tmp/mesos/slaves/20140818-190538-2466255276-5050-11-0/frameworks/20140818-190538-2466255276-5050-11-0002/executors/wordcount-1-1408388814/runs/69496890-fc18-43f3-be87-198bceba7226' --2014-08-18 19:07:12-- http://master:35468/conf/storm.yaml Resolving master (master)... 172.17.0.147 Connecting to master (master)|172.17.0.147|:35468... connected. HTTP request sent, awaiting response... 404 Not Found 2014-08-18 19:07:12 ERROR 404: Not Found.
root@master:/# cat /var/log/supervisor/mesos-master-stderr.log ... I0818 19:11:10.456274 19 master.cpp:2704] Executor wordcount-1-1408388814 of framework 20140818-190538-2466255276-5050-11-0002 on slave 20140818-190538-2466255276-5050-11-0 at slave(1)@172.17.0.149:5051 (slave) has exited with status 8 I0818 19:11:10.457824 19 master.cpp:2628] Status update TASK_LOST (UUID: ddd2a5c6-39d6-4450-824b-2ddc5b39869b) for task slave-31000 of framework 20140818-190538-2466255276-5050-11-0002 from slave 20140818-190538-2466255276-5050-11-0 at slave(1)@172.17.0.149:5051 (slave) I0818 19:11:10.457898 19 master.hpp:673] Removing task slave-31000 with resources cpus(*):1; mem(*):1000; ports(*):[31000-31000] on slave 20140818-190538-2466255276-5050 root@master:/# cat /var/log/supervisor/nimbus-stderr.log I0818 19:06:23.683955 190 sched.cpp:126] Version: 0.19.1 2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@716: Client environment:host.name=master 2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@724: Client environment:os.arch=3.15.3-tinycore64 2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@725: Client environment:os.version=#1 SMP Fri Aug 15 09:11:44 UTC 2014 2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@741: Client environment:user.home=/root 2014-08-18 19:06:23,685:26(0x7f3575014700):ZOO_INFO@log_env@753: Client environment:user.dir=/ 2014-08-18 19:06:23,685:26(0x7f3575014700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=zookeeper:2181 sessionTimeout=10000 watcher=0x7f3576f9cf80 sessionId=0 sessionPasswd=<null> context=0x7f3554000e00 flags=0 2014-08-18 19:06:23,712:26(0x7f3573010700):ZOO_INFO@check_events@1703: initiated connection to server [172.17.0.145:2181] 2014-08-18 19:06:23,724:26(0x7f3573010700):ZOO_INFO@check_events@1750: session establishment complete on server [172.17.0.145:2181], sessionId=0x147ea82a658000c, negotiated timeout=10000 I0818 19:06:23.729141 242 group.cpp:310] Group process ((3)@172.17.0.147:49673) connected to ZooKeeper I0818 19:06:23.729308 242 group.cpp:784] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0818 19:06:23.729367 242 group.cpp:382] Trying to create path '/mesos' in ZooKeeper I0818 19:06:23.745023 242 detector.cpp:135] Detected a new leader: (id='1') I0818 19:06:23.745312 242 group.cpp:655] Trying to get '/mesos/info_0000000001' in ZooKeeper I0818 19:06:23.752063 242 detector.cpp:377] A new leading master ([email protected]:5050) is detected I0818 19:06:23.752250 242 sched.cpp:222] New master detected at [email protected]:5050 I0818 19:06:23.752893 242 sched.cpp:230] No credentials provided. Attempting to register without authentication I0818 19:06:23.755734 242 sched.cpp:397] Framework registered with 20140818-190538-2466255276-5050-11-0002 W0818 19:06:54.991662 245 sched.cpp:901] Attempting to launch task slave-31001 with an unknown offer 20140818-190538-2466255276-5050-11-18 2014-08-18 19:09:10,656:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 28ms W0818 19:10:58.976002 248 sched.cpp:901] Attempting to launch task slave-31001 with an unknown offer 20140818-190538-2466255276-5050-11-57 2014-08-18 19:11:40,927:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 107ms 2014-08-18 19:12:07,700:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 72ms 2014-08-18 19:15:54,659:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 20ms W0818 19:16:41.581099 241 sched.cpp:901] Attempting to launch task slave-31001 with an unknown offer 20140818-190538-2466255276-5050-11-259 W0818 19:19:52.968051 242 sched.cpp:901] Attempting to launch task slave-31001 with an unknown offer 20140818-190538-2466255276-5050-11-367 2014-08-18 19:20:14,970:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 24ms 2014-08-18 19:20:31,676:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 13ms 2014-08-18 19:20:48,375:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 12ms W0818 19:22:33.935534 244 sched.cpp:901] Attempting to launch task slave-31001 with an unknown offer 20140818-190538-2466255276-5050-11-395 (Y) On Aug 20, 2014, at 8:43 AM, Ankit Toshniwal <[email protected]> wrote: > On the storm UI are you seeing your topology getting the required number of > workers and is the topology emitting any tuples? > > Also Storm UI should tell you if any of the supervisors were started and if > they were on which machine and you can ssh to that machine to look at > supervisor logs. > > But the general behavior is that once workers start up they will report their > heartbeats into ZK, and Nimbus will pull this data from ZK to confirm which > worker is up and which is down. > > Hope this helps! > > Ankit > > > On Fri, Aug 15, 2014 at 10:15 PM, Yaron Rosenbaum <[email protected]> > wrote: > That's the thing - I don't know. > Mesos is supposed to start the workers, so either they don't start, or they > start - but they can't communicate with Nimbus. > Or do they need to communicate with Zookeeper directly? > > This is a slightly more complicated setup than usual. > > (Y) > > On Aug 15, 2014, at 8:07 PM, Ankit Toshniwal <[email protected]> wrote: > >> Are the workers starting fine? I have seen this in the past when the workers >> were not reporting their heartbeats leading to this message in nimbus.log >> >> Ankit >> >> >> On Fri, Aug 15, 2014 at 9:18 AM, Yaron Rosenbaum <[email protected]> >> wrote: >> Hi >> >> I've set up a Docker based Mesos setup, including a mesos-storm framework >> installed (plus chronos, marathon, ..). >> You can find it here: https://github.com/yaronr/docker-mesos >> >> It works fine, storm UI is accessible. >> I am able to deploy topologies, and storm-ui reflects these deployments >> correctly. >> However, from the storm logs, it appears that the executors are not able to >> communicate with nimbus. Or they don't exist (mesos is responsible for >> spawning 'supervisors' or executors). >> 2014-08-15 16:03:11 b.s.d.nimbus [INFO] Executor wordcount-1-1408118461:[3 >> 3] not alive >> >> I'm suspecting this has something to do with some ports / links that I have >> to define, but I have nothing to go on but this wild guess. >> >> Any help would be appreciated! >> >> (Y) >> >> > >
