[
https://issues.apache.org/jira/browse/MESOS-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306819#comment-14306819
]
Matt Christiansen edited comment on MESOS-1837 at 2/5/15 8:26 AM:
------------------------------------------------------------------
I am also getting this error:
E0205 00:07:42.387457 4456 slave.cpp:2344] Failed to update resources for
container 8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207 of executor
8004091e-e93b-4415-6c7b-3e52d62abef9 running task
8004091e-e93b-4415-6c7b-3e52d62abef9 on status update for terminal task,
destroying container: Failed to determine cgroup for the 'cpu' subsystem:
Failed to read /proc/4519/cgroup: Failed to open file '/proc/4519/cgroup': No
such file or directory
E0205 00:13:42.638646 4454 slave.cpp:2344] Failed to update resources for
container 29890198-88e8-4809-91a1-590546bee1fd of executor
8005091e-e93b-4415-6c7b-3e52d62abef9 running task
8005091e-e93b-4415-6c7b-3e52d62abef9 on status update for terminal task,
destroying container: Failed to determine cgroup for the 'cpu' subsystem:
Failed to read /proc/4891/cgroup: Failed to open file '/proc/4891/cgroup': No
such file or directory
it doesn't end in a TASK_KILL or fail, instead TASK_FINISHED but the job never
moves from Active Tasks to Completed Tasks but the resources are freed
This is on Centos 6.6, Mesos 0.21.1, Docker 1.4.1 with a custom in house
framework, and I have tried jobs with both custom executors or use the built in
one made no difference.
This seems to happen every time I run a job and regardless if the job succeeds
or fails.
here is the full log for task :8004091e-e93b-4415-6c7b-3e52d62abef9
{noformat}
I0205 00:07:13.098376 4454 slave.cpp:1083] Got assigned task
8004091e-e93b-4415-6c7b-3e52d62abef9 for framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:13.114512 4454 slave.cpp:1193] Launching task
8004091e-e93b-4415-6c7b-3e52d62abef9 for framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:13.115538 4454 slave.cpp:3997] Launching executor
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 in work directory
'/tmp/mesos/slaves/20150204-233402-654050058-5050-15891-S2/frameworks/20150204-233402-654050058-5050-15891-0000/executors/8004091e-e93b-4415-6c7b-3e52d62abef9/runs/8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207'
I0205 00:07:13.115799 4454 slave.cpp:1316] Queuing task
'8004091e-e93b-4415-6c7b-3e52d62abef9' for executor
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
'20150204-233402-654050058-5050-15891-0000
I0205 00:07:13.118450 4452 docker.cpp:927] Starting container
'8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207' for task
'8004091e-e93b-4415-6c7b-3e52d62abef9' (and executor
'8004091e-e93b-4415-6c7b-3e52d62abef9') of framework
'20150204-233402-654050058-5050-15891-0000'
I0205 00:07:14.945565 4454 docker.cpp:633] Checkpointing pid 4527 to
'/tmp/mesos/meta/slaves/20150204-233402-654050058-5050-15891-S2/frameworks/20150204-233402-654050058-5050-15891-0000/executors/8004091e-e93b-4415-6c7b-3e52d62abef9/runs/8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207/pids/forked.pid'
I0205 00:07:14.946475 4450 slave.cpp:2840] Monitoring executor
'8004091e-e93b-4415-6c7b-3e52d62abef9' of framework
'20150204-233402-654050058-5050-15891-0000' in container
'8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207'
I0205 00:07:15.000608 4454 slave.cpp:1860] Got registration for executor
'8004091e-e93b-4415-6c7b-3e52d62abef9' of framework
20150204-233402-654050058-5050-15891-0000 from executor(1)@10.3.0.112:53291
I0205 00:07:15.001273 4454 slave.cpp:1979] Flushing queued task
8004091e-e93b-4415-6c7b-3e52d62abef9 for executor
'8004091e-e93b-4415-6c7b-3e52d62abef9' of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:15.010057 4457 slave.cpp:2215] Handling status update TASK_RUNNING
(UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 from executor(1)@10.3.0.112:53291
I0205 00:07:15.010246 4457 status_update_manager.cpp:317] Received status
update TASK_RUNNING (UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:15.010418 4457 status_update_manager.hpp:346] Checkpointing UPDATE
for status update TASK_RUNNING (UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for
task 8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:15.047556 4455 docker.cpp:1298] Updated 'cpu.shares' to 1126 at
/cgroup/cpu/docker/a8d803bfb154165cfb0491fd7d17f66e2697e09f3908a89e2cf1e9e3647ab3e0
for container 8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207
I0205 00:07:15.047868 4455 docker.cpp:1333] Updated
'memory.soft_limit_in_bytes' to 4544MB for container
8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207
I0205 00:07:15.048305 4455 docker.cpp:1359] Updated 'memory.limit_in_bytes' to
4544MB at
/cgroup/memory/docker/a8d803bfb154165cfb0491fd7d17f66e2697e09f3908a89e2cf1e9e3647ab3e0
for container 8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207
I0205 00:07:15.159543 4450 slave.cpp:2458] Forwarding the update TASK_RUNNING
(UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 to [email protected]:5050
I0205 00:07:15.159682 4450 slave.cpp:2391] Sending acknowledgement for status
update TASK_RUNNING (UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 to executor(1)@10.3.0.112:53291
I0205 00:07:15.162772 4452 status_update_manager.cpp:389] Received status
update acknowledgement (UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:15.162827 4452 status_update_manager.hpp:346] Checkpointing ACK
for status update TASK_RUNNING (UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for
task 8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:42.387159 4450 slave.cpp:2215] Handling status update
TASK_FINISHED (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 from executor(1)@10.3.0.112:53291
E0205 00:07:42.387457 4456 slave.cpp:2344] Failed to update resources for
container 8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207 of executor
8004091e-e93b-4415-6c7b-3e52d62abef9 running task
8004091e-e93b-4415-6c7b-3e52d62abef9 on status update for terminal task,
destroying container: Failed to determine cgroup for the 'cpu' subsystem:
Failed to read /proc/4519/cgroup: Failed to open file '/proc/4519/cgroup': No
such file or directory
I0205 00:07:42.398448 4456 status_update_manager.cpp:317] Received status
update TASK_FINISHED (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:42.398532 4456 status_update_manager.hpp:346] Checkpointing UPDATE
for status update TASK_FINISHED (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9)
for task 8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:42.398494 4457 docker.cpp:1501] Destroying container
'8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207'
I0205 00:07:42.398794 4457 docker.cpp:1593] Running docker stop on container
'8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207'
I0205 00:07:42.508761 4450 slave.cpp:2458] Forwarding the update TASK_FINISHED
(UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 to [email protected]:5050
I0205 00:07:42.508932 4450 slave.cpp:2391] Sending acknowledgement for status
update TASK_FINISHED (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 to executor(1)@10.3.0.112:53291
I0205 00:07:42.516126 4454 status_update_manager.cpp:389] Received status
update acknowledgement (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:42.516222 4454 status_update_manager.hpp:346] Checkpointing ACK
for status update TASK_FINISHED (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9)
for task 8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
{noformat}
was (Author: nikore):
I am also getting this error:
E0205 00:07:42.387457 4456 slave.cpp:2344] Failed to update resources for
container 8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207 of executor
8004091e-e93b-4415-6c7b-3e52d62abef9 running task
8004091e-e93b-4415-6c7b-3e52d62abef9 on status update for terminal task,
destroying container: Failed to determine cgroup for the 'cpu' subsystem:
Failed to read /proc/4519/cgroup: Failed to open file '/proc/4519/cgroup': No
such file or directory
E0205 00:13:42.638646 4454 slave.cpp:2344] Failed to update resources for
container 29890198-88e8-4809-91a1-590546bee1fd of executor
8005091e-e93b-4415-6c7b-3e52d62abef9 running task
8005091e-e93b-4415-6c7b-3e52d62abef9 on status update for terminal task,
destroying container: Failed to determine cgroup for the 'cpu' subsystem:
Failed to read /proc/4891/cgroup: Failed to open file '/proc/4891/cgroup': No
such file or directory
it doesn't end in a TASK_KILL or fail, instead TASK_FINISHED but the job never
moves from Active Tasks to Completed Tasks but the resources are freed
This is on Centos 6.6, Mesos 0.21.1, Docker 1.4.1 with a custom in house
framework, and I have tried jobs with both custom executors or use the built in
one made no difference.
This seems to happen every time I run a job and regardless if the job succeeds
or fails.
here is the full log for task :8004091e-e93b-4415-6c7b-3e52d62abef9
I0205 00:07:13.098376 4454 slave.cpp:1083] Got assigned task
8004091e-e93b-4415-6c7b-3e52d62abef9 for framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:13.114512 4454 slave.cpp:1193] Launching task
8004091e-e93b-4415-6c7b-3e52d62abef9 for framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:13.115538 4454 slave.cpp:3997] Launching executor
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 in work directory
'/tmp/mesos/slaves/20150204-233402-654050058-5050-15891-S2/frameworks/20150204-233402-654050058-5050-15891-0000/executors/8004091e-e93b-4415-6c7b-3e52d62abef9/runs/8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207'
I0205 00:07:13.115799 4454 slave.cpp:1316] Queuing task
'8004091e-e93b-4415-6c7b-3e52d62abef9' for executor
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
'20150204-233402-654050058-5050-15891-0000
I0205 00:07:13.118450 4452 docker.cpp:927] Starting container
'8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207' for task
'8004091e-e93b-4415-6c7b-3e52d62abef9' (and executor
'8004091e-e93b-4415-6c7b-3e52d62abef9') of framework
'20150204-233402-654050058-5050-15891-0000'
I0205 00:07:14.945565 4454 docker.cpp:633] Checkpointing pid 4527 to
'/tmp/mesos/meta/slaves/20150204-233402-654050058-5050-15891-S2/frameworks/20150204-233402-654050058-5050-15891-0000/executors/8004091e-e93b-4415-6c7b-3e52d62abef9/runs/8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207/pids/forked.pid'
I0205 00:07:14.946475 4450 slave.cpp:2840] Monitoring executor
'8004091e-e93b-4415-6c7b-3e52d62abef9' of framework
'20150204-233402-654050058-5050-15891-0000' in container
'8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207'
I0205 00:07:15.000608 4454 slave.cpp:1860] Got registration for executor
'8004091e-e93b-4415-6c7b-3e52d62abef9' of framework
20150204-233402-654050058-5050-15891-0000 from executor(1)@10.3.0.112:53291
I0205 00:07:15.001273 4454 slave.cpp:1979] Flushing queued task
8004091e-e93b-4415-6c7b-3e52d62abef9 for executor
'8004091e-e93b-4415-6c7b-3e52d62abef9' of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:15.010057 4457 slave.cpp:2215] Handling status update TASK_RUNNING
(UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 from executor(1)@10.3.0.112:53291
I0205 00:07:15.010246 4457 status_update_manager.cpp:317] Received status
update TASK_RUNNING (UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:15.010418 4457 status_update_manager.hpp:346] Checkpointing UPDATE
for status update TASK_RUNNING (UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for
task 8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:15.047556 4455 docker.cpp:1298] Updated 'cpu.shares' to 1126 at
/cgroup/cpu/docker/a8d803bfb154165cfb0491fd7d17f66e2697e09f3908a89e2cf1e9e3647ab3e0
for container 8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207
I0205 00:07:15.047868 4455 docker.cpp:1333] Updated
'memory.soft_limit_in_bytes' to 4544MB for container
8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207
I0205 00:07:15.048305 4455 docker.cpp:1359] Updated 'memory.limit_in_bytes' to
4544MB at
/cgroup/memory/docker/a8d803bfb154165cfb0491fd7d17f66e2697e09f3908a89e2cf1e9e3647ab3e0
for container 8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207
I0205 00:07:15.159543 4450 slave.cpp:2458] Forwarding the update TASK_RUNNING
(UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 to [email protected]:5050
I0205 00:07:15.159682 4450 slave.cpp:2391] Sending acknowledgement for status
update TASK_RUNNING (UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 to executor(1)@10.3.0.112:53291
I0205 00:07:15.162772 4452 status_update_manager.cpp:389] Received status
update acknowledgement (UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:15.162827 4452 status_update_manager.hpp:346] Checkpointing ACK
for status update TASK_RUNNING (UUID: d4a33cd4-d09f-423a-8dce-31f485854e5d) for
task 8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:42.387159 4450 slave.cpp:2215] Handling status update
TASK_FINISHED (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 from executor(1)@10.3.0.112:53291
E0205 00:07:42.387457 4456 slave.cpp:2344] Failed to update resources for
container 8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207 of executor
8004091e-e93b-4415-6c7b-3e52d62abef9 running task
8004091e-e93b-4415-6c7b-3e52d62abef9 on status update for terminal task,
destroying container: Failed to determine cgroup for the 'cpu' subsystem:
Failed to read /proc/4519/cgroup: Failed to open file '/proc/4519/cgroup': No
such file or directory
I0205 00:07:42.398448 4456 status_update_manager.cpp:317] Received status
update TASK_FINISHED (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:42.398532 4456 status_update_manager.hpp:346] Checkpointing UPDATE
for status update TASK_FINISHED (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9)
for task 8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:42.398494 4457 docker.cpp:1501] Destroying container
'8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207'
I0205 00:07:42.398794 4457 docker.cpp:1593] Running docker stop on container
'8ac5e3d1-edc9-4d74-8658-9b0b4fd6f207'
I0205 00:07:42.508761 4450 slave.cpp:2458] Forwarding the update TASK_FINISHED
(UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 to [email protected]:5050
I0205 00:07:42.508932 4450 slave.cpp:2391] Sending acknowledgement for status
update TASK_FINISHED (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000 to executor(1)@10.3.0.112:53291
I0205 00:07:42.516126 4454 status_update_manager.cpp:389] Received status
update acknowledgement (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9) for task
8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
I0205 00:07:42.516222 4454 status_update_manager.hpp:346] Checkpointing ACK
for status update TASK_FINISHED (UUID: d8a4a00c-8c91-4311-af16-f3dc5f7cd8f9)
for task 8004091e-e93b-4415-6c7b-3e52d62abef9 of framework
20150204-233402-654050058-5050-15891-0000
> failed to determine cgroup for the 'cpu' subsystem
> --------------------------------------------------
>
> Key: MESOS-1837
> URL: https://issues.apache.org/jira/browse/MESOS-1837
> Project: Mesos
> Issue Type: Bug
> Components: docker
> Affects Versions: 0.20.1
> Environment: Ubuntu 14.04
> Reporter: Chris Fortier
> Assignee: Timothy Chen
>
> Attempting to launch Docker container with Marathon. Container is launched
> then fails.
> A search of /var/log/syslog reveals:
> Sep 27 03:01:43 vagrant-ubuntu-trusty-64 mesos-slave[1409]: E0927
> 03:01:43.546957 1463 slave.cpp:2205] Failed to update resources for
> container 8c2429d9-f090-4443-8108-0206ca37f3fd of executor
> hello-world.970dbe74-45f2-11e4-8b1d-56847afe9799 running task
> hello-world.970dbe74-45f2-11e4-8b1d-56847afe9799 on status update for
> terminal task, destroying container: Failed to determine cgroup for the 'cpu'
> subsystem: Failed to read /proc/9792/cgroup: Failed to open file
> '/proc/9792/cgroup': No such file or directory
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
