Here is the slave log. I0917 00:16:45.759867 10209 slave.cpp:3057] Current usage 43.77%. Max allowed age: 3.235877672809236days I0917 00:17:31.145267 10212 slave.cpp:1011] Got assigned task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 for framework 20140915-230424-326871232-5050-13574-0000 I0917 00:17:31.145606 10212 gc.cpp:84] Unscheduling '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000' from gc I0917 00:17:31.145678 10212 slave.cpp:1121] Launching task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 for framework 20140915-230424-326871232-5050-13574-0000 I0917 00:17:31.146811 10212 slave.cpp:1231] Queuing task 'dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000' for executor production-topology-1-1410913050 of framework '20140915-230424-326871232-5050-13574-0000 I0917 00:17:31.146898 10212 containerizer.cpp:394] Starting container 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' for executor 'production-topology-1-1410913050' of framework '20140915-230424-326871232-5050-13574-0000' I0917 00:17:31.147886 10212 launcher.cpp:137] Forked child with pid '20604' for container 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' I0917 00:17:31.148232 10212 containerizer.cpp:510] Fetching URIs for container 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' using command '/home/ubuntu/mesos/build/src/mesos-fetcher' I0917 00:17:45.761070 10210 slave.cpp:3057] Current usage 43.77%. Max allowed age: 3.235862073509317days I0917 00:17:48.932791 10214 slave.cpp:2542] Monitoring executor 'production-topology-1-1410913050' of framework '20140915-230424-326871232-5050-13574-0000' in container 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' I0917 00:17:58.360540 10215 slave.cpp:1741] Got registration for executor 'production-topology-1-1410913050' of framework 20140915-230424-326871232-5050-13574-0000 I0917 00:17:58.360841 10215 slave.cpp:1859] Flushing queued task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 for executor 'production-topology-1-1410913050' of framework 20140915-230424-326871232-5050-13574-0000 I0917 00:17:58.461596 10215 slave.cpp:2093] Handling status update TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework 20140915-230424-326871232-5050-13574-0000 from executor(1)@ 192.168.123.29:41892 I0917 00:17:58.461767 10215 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework 20140915-230424-326871232-5050-13574-0000 I0917 00:17:58.461879 10215 status_update_manager.cpp:373] Forwarding status update TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework 20140915-230424-326871232-5050-13574-0000 to [email protected]:5050 I0917 00:17:58.462002 10215 slave.cpp:2256] Sending acknowledgement for status update TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework 20140915-230424-326871232-5050-13574-0000 to executor(1)@ 192.168.123.29:41892 I0917 00:17:58.470211 10213 status_update_manager.cpp:398] Received status update acknowledgement (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework 20140915-230424-326871232-5050-13574-0000 I0917 00:18:11.776726 10209 http.cpp:330] HTTP request for '/slave(1)/state.json' I0917 00:18:45.616175 10211 http.cpp:330] HTTP request for '/slave(1)/state.json' I0917 00:18:45.762120 10213 slave.cpp:3057] Current usage 43.92%. Max allowed age: 3.225739484327836days I0917 00:19:43.761559 10210 http.cpp:330] HTTP request for '/slave(1)/state.json' I0917 00:19:45.762634 10208 slave.cpp:3057] Current usage 43.92%. Max allowed age: 3.225736771406111days I0917 00:19:59.021102 10208 containerizer.cpp:997] Executor for container 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' has exited I0917 00:19:59.021178 10208 containerizer.cpp:882] Destroying container 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' I0917 00:19:59.026793 10208 slave.cpp:2600] Executor 'production-topology-1-1410913050' of framework 20140915-230424-326871232-5050-13574-0000 exited with status 0 I0917 00:19:59.027725 10208 slave.cpp:2093] Handling status update TASK_LOST (UUID: 8e3543de-161d-4a5a-bcd7-b7be6f053b24) for task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework 20140915-230424-326871232-5050-13574-0000 from @0.0.0.0:0 W0917 00:19:59.027933 10213 containerizer.cpp:788] Ignoring update for unknown container: dfbf47af-c8b7-481f-b8fc-6a33a8f115d9 I0917 00:19:59.028105 10213 status_update_manager.cpp:320] Received status update TASK_LOST (UUID: 8e3543de-161d-4a5a-bcd7-b7be6f053b24) for task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework 20140915-230424-326871232-5050-13574-0000 I0917 00:19:59.028143 10213 status_update_manager.cpp:373] Forwarding status update TASK_LOST (UUID: 8e3543de-161d-4a5a-bcd7-b7be6f053b24) for task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework 20140915-230424-326871232-5050-13574-0000 to [email protected]:5050 I0917 00:19:59.033900 10213 slave.cpp:2736] Cleaning up executor 'production-topology-1-1410913050' of framework 20140915-230424-326871232-5050-13574-0000 I0917 00:19:59.034037 10213 slave.cpp:2811] Cleaning up framework 20140915-230424-326871232-5050-13574-0000 I0917 00:19:59.034162 10211 status_update_manager.cpp:282] Closing status update streams for framework 20140915-230424-326871232-5050-13574-0000 I0917 00:19:59.034098 10213 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' for gc 6.99999960653926days in the future I0917 00:19:59.034209 10213 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050' for gc 6.99999960616889days in the future I0917 00:19:59.034255 10213 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000' for gc 6.99999960553185days in the future
-Luyi. On Wed, Sep 17, 2014 at 12:56 PM, Benjamin Mahler <[email protected] > wrote: > Can you show us the the slave log and more of the master log? > > There should be a TASK_LOST somewhere within them. > > On Wed, Sep 17, 2014 at 10:43 AM, Luyi Wang <[email protected]> > wrote: > >> Have anyone experience TASK_LOST status for storm tasks on mesos. >> >> >> I checked the stderr. Everything seems normal. >> WARNING: Logging before InitGoogleLogging() is written to STDERR >> I0917 00:21:36.164840 4831 fetcher.cpp:76] Fetching URI 'hdfs:// >> 192.168.123.27/storm-mesos-0.9.2-incubating.tgz' >> I0917 00:21:36.165225 4831 fetcher.cpp:105] Downloading resource from >> 'hdfs://192.168.123.27/storm-mesos-0.9.2-incubating.tgz' to >> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm-mesos-0.9.2-incubating.tgz' >> I0917 00:21:52.202791 4831 fetcher.cpp:64] Extracted resource >> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm-mesos-0.9.2-incubating.tgz' >> into >> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9' >> I0917 00:21:52.206581 4831 fetcher.cpp:76] Fetching URI ' >> http://mesos:45579/conf/storm.yaml' >> I0917 00:21:52.206626 4831 fetcher.cpp:126] Downloading ' >> http://mesos:45579/conf/storm.yaml' to >> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm.yaml' >> I0917 00:22:01.829298 4984 exec.cpp:132] Version: 0.21.0 >> I0917 00:22:01.832185 5006 exec.cpp:206] Executor registered on slave >> 20140915-185627-326871232-5050-8074-3 >> >> >> >> >> I also checked the mesos info log. Here is what it logged. >> >> I0917 00:19:59.020755 5145 master.cpp:3261] Executor >> production-topology-1-1410913050 of framework >> 20140915-230424-326871232-5050-13574-0000 on slave >> 20140915-185627-326871232-5050-8074-2 at slave(1)@192.168.123.29:5051 ( >> dev10-cdh5-03.int.dev10.smcl.pure-breeze.com) exited with status 0 >> >> >> Any idea on this? >> >> >> >> >> -Luyi. >> >> >> >> >
