[
https://issues.apache.org/jira/browse/MESOS-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345440#comment-14345440
]
Niklas Quarfot Nielsen commented on MESOS-2419:
-----------------------------------------------
What is the different between the two logs?
In mesos-chronos.log, the slave doesn't attempt to recover the chronos tasks
(were checkpointing enabled? You enabled it by default recently:
https://github.com/mesos/chronos/pull/380/files)
There are roughly 139 cassandra tasks being recovered but the chronos tasks
seems to not be attempted to be recovered.
Can you provide the master log of chronos registering with the master?
In the mesos.log, I stumbled upon this:
{code}
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.289890 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.1023ea18-be14-11e4-b8d6-566d21d75321/runs/1792f862-0feb-4b4e-bc96-b729093333d1/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.295059 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.733b4ec6-be13-11e4-b8d6-566d21d75321/runs/384ff848-819e-4001-91e4-aaecd91a99ba/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.297408 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/hdfs-dry-run.7b7c4f70-be21-11e4-85f4-566d21d75321/runs/0e1dcf53-a0b0-4a31-bf39-0774796d6a16/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.300632 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.b2a1a590-be1b-11e4-85f4-566d21d75321/runs/cca5e12e-130a-49e9-8ff8-6e00b1f65646/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.304968 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.fd540830-be19-11e4-a854-566d21d75321/runs/a897a998-0337-4e89-ad29-05b2e7365ad7/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.307368 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.42a9fca8-be13-11e4-817b-566d21d75321/runs/c8904748-9fb9-41cb-91ae-93e6c20f8a42/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.310233 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.c9a5e328-be15-11e4-b8d6-566d21d75321/runs/df18e954-cc16-4951-9149-a6070b6e7951/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.311167 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.7b8fe333-be17-11e4-905b-566d21d75321/runs/aa100f4b-afc6-48d7-8567-f52efff44b16/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.317438 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.b739c87a-be17-11e4-905b-566d21d75321/runs/bb284c6c-5545-4f77-bbaf-daad19e27999/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.320307 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.addbb90d-be1b-11e4-85f4-566d21d75321/runs/9b176f32-f86d-4553-b648-ac73f4e560f7/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.333884 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.c8937263-be1e-11e4-85f4-566d21d75321/runs/355c839b-c698-42c6-af43-f495543a0558/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.334828 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.87836a81-be13-11e4-b8d6-566d21d75321/runs/d1169ac7-6831-43d8-99c8-d30e2bfa8f38/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.336736 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.3e6a504d-be16-11e4-811b-566d21d75321/runs/c7702bcc-4d2e-4b9a-b1c6-9a877bcecd8c/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.339596 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.b05d1ec1-be19-11e4-a854-566d21d75321/runs/d33ed7f6-7131-45aa-965a-8bcc32a6609a/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.340558 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.39080816-be0c-11e4-8fda-56847afe9799/runs/746a5851-9c09-4ef6-bcc0-9c8c41eaa927/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.348759 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.1a508fe8-be17-11e4-905b-566d21d75321/runs/1ad7fedc-eb9b-4a35-911d-6394585d55a8/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.350206 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.62cbb672-be0c-11e4-8fda-56847afe9799/runs/c814d23d-0bdf-4a91-9440-257411300059/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.360815 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.5fbd0805-be12-11e4-a6bd-566d21d75321/runs/c397d7e9-9b1e-4b62-b49e-5082a58e79cf/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.361299 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.124420d1-be16-11e4-811b-566d21d75321/runs/d6712650-ce69-47c4-aba7-2d58eaa55c75/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.372257 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.aa1e0f10-be13-11e4-b8d6-566d21d75321/runs/df954fce-0202-4eba-b83b-9a22d1ce3cd7/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.374622 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.0c23f7ac-be12-11e4-9fdd-566d21d75321/runs/7ba4e9fe-ec3d-4655-87d5-c3371ac24ca8/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.375586 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.db33d8d3-be1b-11e4-85f4-566d21d75321/runs/2fc6f755-9c43-4ebd-b44a-1dc093c157c2/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.376596 10451 state.cpp:581] Failed to find status updates file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/hdfs-dry-run.51130eb5-be28-11e4-b59b-566d21d75321/runs/0b2a249f-81dd-48b2-b7c4-f2b454ee6a8e/tasks/hdfs-dry-run.51130eb5-be28-11e4-b59b-566d21d75321/task.updates'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.381433 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.11822c91-be12-11e4-9fdd-566d21d75321/runs/4ead7946-ff04-4197-9c6e-ebcfcd113017/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.385275 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.3d353708-be0c-11e4-8fda-56847afe9799/runs/0020b106-4c23-4f52-afd9-11e717c5eeb3/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.387636 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/hdfs-dry-run.83d6ad53-be21-11e4-85f4-566d21d75321/runs/8f79efbf-a773-4b18-b45e-f861908a7ebd/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.389041 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.07f6efc8-be12-11e4-9fdd-566d21d75321/runs/4bafa879-2dad-469b-b7bf-c8fc07680166/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.398073 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.9ae4c870-be16-11e4-811b-566d21d75321/runs/400e9e7c-5007-4145-9c79-f56663e8ea1e/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.398560 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0000/executors/cassandra-mesos.ae9510f2-be16-11e4-811b-566d21d75321/runs/dd8dcc4a-4466-4299-adcd-0b95901d8fb1/pids/libprocess.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.443220 10451 state.cpp:455] Failed to find executor forked pid file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0001/executors/ct:1424995134352:0:spark-perf:/runs/5457f0f0-9148-465a-b70c-072fd52c402a/pids/forked.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.443609 10451 state.cpp:455] Failed to find executor forked pid file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150226-230228-2931198986-5050-717-0001/executors/ct:1425003977577:0:spark-perf:/runs/6a4dcfa5-d900-41f3-ab56-d6df131b806f/pids/forked.pid'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.457813 10451 state.cpp:581] Failed to find status updates file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150227-014043-2931198986-5050-7728-0002/executors/cassandra.test-suite.node.113.executor/runs/b6ab2310-ad99-4f0f-a8e5-4de904b835d1/tasks/cassandra.test-suite.node.113.executor/task.updates'
Feb 27 02:30:23 ip-10-178-203-215.ec2.internal mesos-slave[10450]: W0227
02:30:23.457921 10451 state.cpp:496] Failed to find executor libprocess pid
file
'/tmp/mesos/meta/slaves/20150226-230228-2931198986-5050-717-S0/frameworks/20150227-014043-2931198986-5050-7728-0002/executors/cassandra.test-suite.node.113.executor/runs/b6ab2310-ad99-4f0f-a8e5-4de904b835d1/pids/libprocess.pid'
{code}
Not sure what caused the pid files to 1) Not be written at all 2) Be corrupted
or inaccessible during recovery.
> Slave recovery not recovering tasks
> -----------------------------------
>
> Key: MESOS-2419
> URL: https://issues.apache.org/jira/browse/MESOS-2419
> Project: Mesos
> Issue Type: Bug
> Components: slave
> Affects Versions: 0.22.0, 0.23.0
> Reporter: Brenden Matthews
> Assignee: Niklas Quarfot Nielsen
> Attachments: mesos-chronos.log.gz, mesos.log.gz
>
>
> In a recent build from master (updated yesterday), slave recovery appears to
> have broken.
> I'll attach the slave log (with GLOG_v=1) showing a task called
> `long-running-job` which is a Chronos job that just does `sleep 1h`. After
> restarting the slave, the task shows as `TASK_FAILED`.
> Here's another case, which is for a docker task:
> {noformat}
> Feb 27 00:09:49 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:49.247159 10022 docker.cpp:421] Recovering Docker containers
> Feb 27 00:09:49 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:49.247207 10022 docker.cpp:468] Recovering container
> 'f2001064-e076-4978-b764-ed12a5244e78' for executor
> 'chronos.55ffc971-be13-11e4-b8d6-566d21d75321' of framework
> 20150226-230228-2931198986-5050-717-0000
> Feb 27 00:09:49 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:49.254791 10022 docker.cpp:1333] Executor for container
> 'f2001064-e076-4978-b764-ed12a5244e78' has exited
> Feb 27 00:09:49 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:49.254812 10022 docker.cpp:1159] Destroying container
> 'f2001064-e076-4978-b764-ed12a5244e78'
> Feb 27 00:09:49 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:49.254844 10022 docker.cpp:1248] Running docker stop on container
> 'f2001064-e076-4978-b764-ed12a5244e78'
> Feb 27 00:09:49 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:49.262481 10027 containerizer.cpp:310] Recovering containerizer
> Feb 27 00:09:49 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:49.262565 10027 containerizer.cpp:353] Recovering container
> 'f2001064-e076-4978-b764-ed12a5244e78' for executor
> 'chronos.55ffc971-be13-11e4-b8d6-566d21d75321' of framework
> 20150226-230228-2931198986-5050-717-0000
> Feb 27 00:09:49 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:49.263675 10027 linux_launcher.cpp:162] Couldn't find freezer cgroup
> for container f2001064-e076-4978-b764-ed12a5244e78, assuming already destroyed
> Feb 27 00:09:49 ip-10-81-189-232.ec2.internal mesos-slave[10018]: W0227
> 00:09:49.265467 10020 cpushare.cpp:199] Couldn't find cgroup for container
> f2001064-e076-4978-b764-ed12a5244e78
> Feb 27 00:09:49 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:49.266448 10022 containerizer.cpp:1147] Executor for container
> 'f2001064-e076-4978-b764-ed12a5244e78' has exited
> Feb 27 00:09:49 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:49.266466 10022 containerizer.cpp:938] Destroying container
> 'f2001064-e076-4978-b764-ed12a5244e78'
> Feb 27 00:09:50 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:50.593585 10021 slave.cpp:3735] Sending reconnect request to executor
> chronos.55ffc971-be13-11e4-b8d6-566d21d75321 of framework
> 20150226-230228-2931198986-5050-717-0000 at executor(1)@10.81.189.232:43130
> Feb 27 00:09:50 ip-10-81-189-232.ec2.internal mesos-slave[10018]: E0227
> 00:09:50.597843 10024 slave.cpp:3175] Termination of executor
> 'chronos.55ffc971-be13-11e4-b8d6-566d21d75321' of framework
> '20150226-230228-2931198986-5050-717-0000' failed: Container
> 'f2001064-e076-4978-b764-ed12a5244e78' not found
> Feb 27 00:09:50 ip-10-81-189-232.ec2.internal mesos-slave[10018]: E0227
> 00:09:50.597949 10025 slave.cpp:3429] Failed to unmonitor container for
> executor chronos.55ffc971-be13-11e4-b8d6-566d21d75321 of framework
> 20150226-230228-2931198986-5050-717-0000: Not monitored
> Feb 27 00:09:50 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:50.598785 10024 slave.cpp:2508] Handling status update TASK_FAILED
> (UUID: d8afb771-a47a-4adc-b38b-c8cc016ab289) for task
> chronos.55ffc971-be13-11e4-b8d6-566d21d75321 of framework
> 20150226-230228-2931198986-5050-717-0000 from @0.0.0.0:0
> Feb 27 00:09:50 ip-10-81-189-232.ec2.internal mesos-slave[10018]: E0227
> 00:09:50.599093 10023 slave.cpp:2637] Failed to update resources for
> container f2001064-e076-4978-b764-ed12a5244e78 of executor
> chronos.55ffc971-be13-11e4-b8d6-566d21d75321 running task
> chronos.55ffc971-be13-11e4-b8d6-566d21d75321 on status update for terminal
> task, destroying container: Container 'f2001064-e076-4978-b764-ed12a5244e78'
> not found
> Feb 27 00:09:50 ip-10-81-189-232.ec2.internal mesos-slave[10018]: W0227
> 00:09:50.599148 10024 composing.cpp:513] Container
> 'f2001064-e076-4978-b764-ed12a5244e78' not found
> Feb 27 00:09:50 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:50.599220 10024 status_update_manager.cpp:317] Received status update
> TASK_FAILED (UUID: d8afb771-a47a-4adc-b38b-c8cc016ab289) for task
> chronos.55ffc971-be13-11e4-b8d6-566d21d75321 of framework
> 20150226-230228-2931198986-5050-717-0000
> Feb 27 00:09:50 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:50.599256 10024 status_update_manager.hpp:346] Checkpointing UPDATE for
> status update TASK_FAILED (UUID: d8afb771-a47a-4adc-b38b-c8cc016ab289) for
> task chronos.55ffc971-be13-11e4-b8d6-566d21d75321 of framework
> 20150226-230228-2931198986-5050-717-0000
> Feb 27 00:09:50 ip-10-81-189-232.ec2.internal mesos-slave[10018]: W0227
> 00:09:50.607086 10022 slave.cpp:2706] Dropping status update TASK_FAILED
> (UUID: d8afb771-a47a-4adc-b38b-c8cc016ab289) for task
> chronos.55ffc971-be13-11e4-b8d6-566d21d75321 of framework
> 20150226-230228-2931198986-5050-717-0000 sent by status update manager
> because the slave is in RECOVERING state
> Feb 27 00:09:52 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:52.594267 10021 slave.cpp:2457] Cleaning up un-reregistered executors
> Feb 27 00:09:52 ip-10-81-189-232.ec2.internal mesos-slave[10018]: I0227
> 00:09:52.594379 10021 slave.cpp:3794] Finished recovery
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
