[
https://issues.apache.org/jira/browse/MESOS-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258446#comment-14258446
]
Albert Strasheim commented on MESOS-1837:
-----------------------------------------
[~tnachen] Yep, seems to repro most or all of the time.
Looking closer now, it seems as if things actually run successfully most of the
time, despite this error.
Here's a snapshot of the logs:
{code}
Dec 24 18:48:03 mesos-slave[695]: I1224 708 slave.cpp:2898] Executor
'inky.633ff636-8b9d-11e4-9c79-56847afe9799' of framework
20141224-001522-352986122-5050-5505-0000 exited with status 0
Dec 24 18:48:03 mesos-slave[695]: I1224 708 slave.cpp:3007] Cleaning up
executor 'inky.633ff636-8b9d-11e4-9c79-56847afe9799' of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:03 mesos-slave[695]: I1224 703 gc.cpp:56] Scheduling
'/state/var/lib/mesos/slaves/20141224-001522-352986122-5050-5505-S2/frameworks/20141224-001522-352986122-5050-5505-0000/executors/inky.633ff636-8b9d-11e4-9c79-56847afe9799/runs/08dbe7c4-d1ea-4428-906d-cc9e3da26891'
for gc 6.99999124332148days in the future
Dec 24 18:48:03 mesos-slave[695]: I1224 703 gc.cpp:56] Scheduling
'/state/var/lib/mesos/slaves/20141224-001522-352986122-5050-5505-S2/frameworks/20141224-001522-352986122-5050-5505-0000/executors/inky.633ff636-8b9d-11e4-9c79-56847afe9799'
for gc 6.99999124047407days in the future
Dec 24 18:48:03 mesos-slave[695]: I1224 712 slave.cpp:1083] Got assigned task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 for framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:03 mesos-slave[695]: I1224 712 slave.cpp:1193] Launching task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 for framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:03 mesos-slave[695]: I1224 712 slave.cpp:3997] Launching executor
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 in work directory
'/state/var/lib/mesos/slaves/20141224-001522-352986122-5050-5505-S2/frameworks/20141224-001522-352986122-5050-5505-0000/executors/inky.6471bf78-8b9d-11e4-9c79-56847afe9799/runs/9553688a-5ac9-4a8a-b61d-ec9bb6bc3ee4'
Dec 24 18:48:03 mesos-slave[695]: I1224 712 slave.cpp:1316] Queuing task
'inky.6471bf78-8b9d-11e4-9c79-56847afe9799' for executor
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
'20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:03 mesos-slave[695]: I1224 696 docker.cpp:928] Starting container
'9553688a-5ac9-4a8a-b61d-ec9bb6bc3ee4' for task
'inky.6471bf78-8b9d-11e4-9c79-56847afe9799' (and executor
'inky.6471bf78-8b9d-11e4-9c79-56847afe9799') of framework
'20141224-001522-352986122-5050-5505-0000'
Dec 24 18:48:04 mesos-slave[695]: I1224 705 slave.cpp:2840] Monitoring executor
'inky.6471bf78-8b9d-11e4-9c79-56847afe9799' of framework
'20141224-001522-352986122-5050-5505-0000' in container
'9553688a-5ac9-4a8a-b61d-ec9bb6bc3ee4'
Dec 24 18:48:04 mesos-slave[695]: I1224 707 slave.cpp:1860] Got registration
for executor 'inky.6471bf78-8b9d-11e4-9c79-56847afe9799' of framework
20141224-001522-352986122-5050-5505-0000 from executor(1)@10.36.10.21:41950
Dec 24 18:48:04 mesos-slave[695]: I1224 707 slave.cpp:1979] Flushing queued
task inky.6471bf78-8b9d-11e4-9c79-56847afe9799 for executor
'inky.6471bf78-8b9d-11e4-9c79-56847afe9799' of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:04 mesos-slave[695]: I1224 715 slave.cpp:2215] Handling status
update TASK_RUNNING (UUID: cd77fa58-dea2-4ed5-947d-18fac6e6d562) for task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 from executor(1)@10.36.10.21:41950
Dec 24 18:48:04 mesos-slave[695]: I1224 719 status_update_manager.cpp:317]
Received status update TASK_RUNNING (UUID:
cd77fa58-dea2-4ed5-947d-18fac6e6d562) for task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:04 mesos-slave[695]: I1224 707 slave.cpp:2458] Forwarding the
update TASK_RUNNING (UUID: cd77fa58-dea2-4ed5-947d-18fac6e6d562) for task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 to [email protected]:5050
Dec 24 18:48:04 mesos-slave[695]: I1224 707 slave.cpp:2391] Sending
acknowledgement for status update TASK_RUNNING (UUID:
cd77fa58-dea2-4ed5-947d-18fac6e6d562) for task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 to executor(1)@10.36.10.21:41950
Dec 24 18:48:04 mesos-slave[695]: I1224 714 status_update_manager.cpp:389]
Received status update acknowledgement (UUID:
cd77fa58-dea2-4ed5-947d-18fac6e6d562) for task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:04 mesos-slave[695]: I1224 709 docker.cpp:1680] Executor for
container '5b26c52f-0e12-403f-aff8-0a08b4a3ee36' has exited
Dec 24 18:48:04 mesos-slave[695]: I1224 699 slave.cpp:2898] Executor
'inky.63d8dad7-8b9d-11e4-9c79-56847afe9799' of framework
20141224-001522-352986122-5050-5505-0000 exited with status 0
Dec 24 18:48:04 mesos-slave[695]: I1224 699 slave.cpp:3007] Cleaning up
executor 'inky.63d8dad7-8b9d-11e4-9c79-56847afe9799' of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:04 mesos-slave[695]: I1224 719 gc.cpp:56] Scheduling
'/state/var/lib/mesos/slaves/20141224-001522-352986122-5050-5505-S2/frameworks/20141224-001522-352986122-5050-5505-0000/executors/inky.63d8dad7-8b9d-11e4-9c79-56847afe9799/runs/5b26c52f-0e12-403f-aff8-0a08b4a3ee36'
for gc 6.99999225899259days in the future
Dec 24 18:48:04 mesos-slave[695]: I1224 719 gc.cpp:56] Scheduling
'/state/var/lib/mesos/slaves/20141224-001522-352986122-5050-5505-S2/frameworks/20141224-001522-352986122-5050-5505-0000/executors/inky.63d8dad7-8b9d-11e4-9c79-56847afe9799'
for gc 6.9999922546637days in the future
Dec 24 18:48:05 mesos-slave[695]: I1224 718 slave.cpp:2215] Handling status
update TASK_FINISHED (UUID: 61912f9a-4e10-4c9a-b1f0-d5d74ee33b96) for task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 from executor(1)@10.36.10.21:41950
Dec 24 18:48:05 mesos-slave[695]: E1224 698 slave.cpp:2344] Failed to update
resources for container 9553688a-5ac9-4a8a-b61d-ec9bb6bc3ee4 of executor
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 running task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 on status update for terminal task,
destroying container: Failed to determine cgroup for the 'cpu' subsystem:
Failed to read /proc/9134/cgroup: Failed to open file '/proc/9134/cgroup': No
such file or directory
Dec 24 18:48:05 mesos-slave[695]: I1224 715 docker.cpp:1502] Destroying
container '9553688a-5ac9-4a8a-b61d-ec9bb6bc3ee4'
Dec 24 18:48:05 mesos-slave[695]: I1224 715 docker.cpp:1597] Running docker
kill on container '9553688a-5ac9-4a8a-b61d-ec9bb6bc3ee4'
Dec 24 18:48:05 mesos-slave[695]: I1224 702 status_update_manager.cpp:317]
Received status update TASK_FINISHED (UUID:
61912f9a-4e10-4c9a-b1f0-d5d74ee33b96) for task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:05 mesos-slave[695]: I1224 700 slave.cpp:2458] Forwarding the
update TASK_FINISHED (UUID: 61912f9a-4e10-4c9a-b1f0-d5d74ee33b96) for task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 to [email protected]:5050
Dec 24 18:48:05 mesos-slave[695]: I1224 700 slave.cpp:2391] Sending
acknowledgement for status update TASK_FINISHED (UUID:
61912f9a-4e10-4c9a-b1f0-d5d74ee33b96) for task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 to executor(1)@10.36.10.21:41950
Dec 24 18:48:05 mesos-slave[695]: I1224 718 status_update_manager.cpp:389]
Received status update acknowledgement (UUID:
61912f9a-4e10-4c9a-b1f0-d5d74ee33b96) for task
inky.6471bf78-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:05 mesos-slave[695]: I1224 710 slave.cpp:1083] Got assigned task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 for framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:05 mesos-slave[695]: I1224 710 slave.cpp:1193] Launching task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 for framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:05 mesos-slave[695]: I1224 710 slave.cpp:3997] Launching executor
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 in work directory
'/state/var/lib/mesos/slaves/20141224-001522-352986122-5050-5505-S2/frameworks/20141224-001522-352986122-5050-5505-0000/executors/inky.65a31389-8b9d-11e4-9c79-56847afe9799/runs/cec4b48d-4d4e-4792-85e8-8beab6a07e18'
Dec 24 18:48:05 mesos-slave[695]: I1224 710 slave.cpp:1316] Queuing task
'inky.65a31389-8b9d-11e4-9c79-56847afe9799' for executor
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
'20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:05 mesos-slave[695]: I1224 702 docker.cpp:928] Starting container
'cec4b48d-4d4e-4792-85e8-8beab6a07e18' for task
'inky.65a31389-8b9d-11e4-9c79-56847afe9799' (and executor
'inky.65a31389-8b9d-11e4-9c79-56847afe9799') of framework
'20141224-001522-352986122-5050-5505-0000'
Dec 24 18:48:06 mesos-slave[695]: I1224 702 slave.cpp:2840] Monitoring executor
'inky.65a31389-8b9d-11e4-9c79-56847afe9799' of framework
'20141224-001522-352986122-5050-5505-0000' in container
'cec4b48d-4d4e-4792-85e8-8beab6a07e18'
Dec 24 18:48:06 mesos-slave[695]: I1224 710 slave.cpp:1860] Got registration
for executor 'inky.65a31389-8b9d-11e4-9c79-56847afe9799' of framework
20141224-001522-352986122-5050-5505-0000 from executor(1)@10.36.10.21:56457
Dec 24 18:48:06 mesos-slave[695]: I1224 710 slave.cpp:1979] Flushing queued
task inky.65a31389-8b9d-11e4-9c79-56847afe9799 for executor
'inky.65a31389-8b9d-11e4-9c79-56847afe9799' of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:06 mesos-slave[695]: I1224 702 slave.cpp:2215] Handling status
update TASK_RUNNING (UUID: c228b462-96d7-4a71-b5be-711a0edb234d) for task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 from executor(1)@10.36.10.21:56457
Dec 24 18:48:06 mesos-slave[695]: I1224 708 status_update_manager.cpp:317]
Received status update TASK_RUNNING (UUID:
c228b462-96d7-4a71-b5be-711a0edb234d) for task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:06 mesos-slave[695]: I1224 698 slave.cpp:2458] Forwarding the
update TASK_RUNNING (UUID: c228b462-96d7-4a71-b5be-711a0edb234d) for task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 to [email protected]:5050
Dec 24 18:48:06 mesos-slave[695]: I1224 698 slave.cpp:2391] Sending
acknowledgement for status update TASK_RUNNING (UUID:
c228b462-96d7-4a71-b5be-711a0edb234d) for task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 to executor(1)@10.36.10.21:56457
Dec 24 18:48:06 mesos-slave[695]: I1224 698 status_update_manager.cpp:389]
Received status update acknowledgement (UUID:
c228b462-96d7-4a71-b5be-711a0edb234d) for task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:06 mesos-slave[695]: I1224 704 docker.cpp:1299] Updated
'cpu.shares' to 1126 at
/sys/fs/cgroup/cpu,cpuacct/system.slice/docker-8da5a9137370f00914dda3d06afeb1a7a3deb85ffbc0b4ce1c354ffdb2630941.scope
for container cec4b48d-4d4e-4792-85e8-8beab6a07e18
Dec 24 18:48:06 mesos-slave[695]: I1224 704 docker.cpp:1334] Updated
'memory.soft_limit_in_bytes' to 8224MB for container
cec4b48d-4d4e-4792-85e8-8beab6a07e18
Dec 24 18:48:06 mesos-slave[695]: I1224 704 docker.cpp:1360] Updated
'memory.limit_in_bytes' to 8224MB at
/sys/fs/cgroup/memory/system.slice/docker-8da5a9137370f00914dda3d06afeb1a7a3deb85ffbc0b4ce1c354ffdb2630941.scope
for container cec4b48d-4d4e-4792-85e8-8beab6a07e18
Dec 24 18:48:06 mesos-slave[695]: I1224 707 slave.cpp:2215] Handling status
update TASK_FINISHED (UUID: fda78d82-1af1-48ee-8727-c3759df58b0c) for task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 from executor(1)@10.36.10.21:56457
Dec 24 18:48:06 mesos-slave[695]: E1224 715 slave.cpp:2344] Failed to update
resources for container cec4b48d-4d4e-4792-85e8-8beab6a07e18 of executor
inky.65a31389-8b9d-11e4-9c79-56847afe9799 running task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 on status update for terminal task,
destroying container: Failed to determine cgroup for the 'cpu' subsystem:
Failed to read /proc/9250/cgroup: Failed to open file '/proc/9250/cgroup': No
such file or directory
Dec 24 18:48:06 mesos-slave[695]: I1224 701 docker.cpp:1502] Destroying
container 'cec4b48d-4d4e-4792-85e8-8beab6a07e18'
Dec 24 18:48:06 mesos-slave[695]: I1224 702 status_update_manager.cpp:317]
Received status update TASK_FINISHED (UUID:
fda78d82-1af1-48ee-8727-c3759df58b0c) for task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:06 mesos-slave[695]: I1224 701 docker.cpp:1597] Running docker
kill on container 'cec4b48d-4d4e-4792-85e8-8beab6a07e18'
Dec 24 18:48:06 mesos-slave[695]: I1224 718 slave.cpp:2458] Forwarding the
update TASK_FINISHED (UUID: fda78d82-1af1-48ee-8727-c3759df58b0c) for task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 to [email protected]:5050
Dec 24 18:48:06 mesos-slave[695]: I1224 718 slave.cpp:2391] Sending
acknowledgement for status update TASK_FINISHED (UUID:
fda78d82-1af1-48ee-8727-c3759df58b0c) for task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 to executor(1)@10.36.10.21:56457
Dec 24 18:48:06 mesos-slave[695]: I1224 709 status_update_manager.cpp:389]
Received status update acknowledgement (UUID:
fda78d82-1af1-48ee-8727-c3759df58b0c) for task
inky.65a31389-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:06 mesos-slave[695]: I1224 698 docker.cpp:1680] Executor for
container '9553688a-5ac9-4a8a-b61d-ec9bb6bc3ee4' has exited
Dec 24 18:48:06 mesos-slave[695]: I1224 705 slave.cpp:2898] Executor
'inky.6471bf78-8b9d-11e4-9c79-56847afe9799' of framework
20141224-001522-352986122-5050-5505-0000 exited with status 0
Dec 24 18:48:06 mesos-slave[695]: I1224 705 slave.cpp:3007] Cleaning up
executor 'inky.6471bf78-8b9d-11e4-9c79-56847afe9799' of framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:06 mesos-slave[695]: I1224 704 gc.cpp:56] Scheduling
'/state/var/lib/mesos/slaves/20141224-001522-352986122-5050-5505-S2/frameworks/20141224-001522-352986122-5050-5505-0000/executors/inky.6471bf78-8b9d-11e4-9c79-56847afe9799/runs/9553688a-5ac9-4a8a-b61d-ec9bb6bc3ee4'
for gc 6.99999322163259days in the future
Dec 24 18:48:06 mesos-slave[695]: I1224 704 gc.cpp:56] Scheduling
'/state/var/lib/mesos/slaves/20141224-001522-352986122-5050-5505-S2/frameworks/20141224-001522-352986122-5050-5505-0000/executors/inky.6471bf78-8b9d-11e4-9c79-56847afe9799'
for gc 6.99999321907556days in the future
Dec 24 18:48:06 mesos-slave[695]: I1224 707 slave.cpp:1083] Got assigned task
inky.663bf82a-8b9d-11e4-9c79-56847afe9799 for framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:06 mesos-slave[695]: I1224 707 slave.cpp:1193] Launching task
inky.663bf82a-8b9d-11e4-9c79-56847afe9799 for framework
20141224-001522-352986122-5050-5505-0000
Dec 24 18:48:06 mesos-slave[695]: I1224 707 slave.cpp:3997] Launching executor
inky.663bf82a-8b9d-11e4-9c79-56847afe9799 of framework
20141224-001522-352986122-5050-5505-0000 in work directory
'/state/var/lib/mesos/slaves/20141224-001522-352986122-5050-5505-S2/frameworks/20141224-001522-352986122-5050-5505-0000/executors/inky.663bf82a-8b9d-11e4-9c79-56847afe9799/runs/79958273-0858-48a0-b0df-725405705eb9'
Dec 24 18:48:06 mesos-slave[695]: I1224 707 slave.cpp:1316] Queuing task
'inky.663bf82a-8b9d-11e4-9c79-56847afe9799' for executor
inky.663bf82a-8b9d-11e4-9c79-56847afe9799 of framework
'20141224-001522-352986122-5050-5505-0000
{code}
> failed to determine cgroup for the 'cpu' subsystem
> --------------------------------------------------
>
> Key: MESOS-1837
> URL: https://issues.apache.org/jira/browse/MESOS-1837
> Project: Mesos
> Issue Type: Bug
> Components: general
> Affects Versions: 0.20.1
> Environment: Ubuntu 14.04
> Reporter: Chris Fortier
>
> Attempting to launch Docker container with Marathon. Container is launched
> then fails.
> A search of /var/log/syslog reveals:
> Sep 27 03:01:43 vagrant-ubuntu-trusty-64 mesos-slave[1409]: E0927
> 03:01:43.546957 1463 slave.cpp:2205] Failed to update resources for
> container 8c2429d9-f090-4443-8108-0206ca37f3fd of executor
> hello-world.970dbe74-45f2-11e4-8b1d-56847afe9799 running task
> hello-world.970dbe74-45f2-11e4-8b1d-56847afe9799 on status update for
> terminal task, destroying container: Failed to determine cgroup for the 'cpu'
> subsystem: Failed to read /proc/9792/cgroup: Failed to open file
> '/proc/9792/cgroup': No such file or directory
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
