[
https://issues.apache.org/jira/browse/MESOS-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308593#comment-14308593
]
Matt Christiansen commented on MESOS-1837:
------------------------------------------
Here is a run with GLOG_v=1
{noformat}
I0205 20:17:56.035176 10114 slave.cpp:1083] Got assigned task
80025a2d-e93b-4415-9b86-6a52d744165f for framework
20150205-122729-654050058-5050-31913-0000
I0205 20:17:56.035425 10114 slave.cpp:3942] Checkpointing FrameworkInfo to
'/tmp/mesos/meta/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/framework.info'
I0205 20:17:56.035660 10114 slave.cpp:3949] Checkpointing framework pid
'scheduler(1)@10.3.0.100:21690' to
'/tmp/mesos/meta/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/framework.pid'
I0205 20:17:56.035943 10114 gc.cpp:84] Unscheduling
'/tmp/mesos/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000'
from gc
I0205 20:17:56.036052 10114 gc.cpp:84] Unscheduling
'/tmp/mesos/meta/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000'
from gc
I0205 20:17:56.036178 10116 slave.cpp:1193] Launching task
80025a2d-e93b-4415-9b86-6a52d744165f for framework
20150205-122729-654050058-5050-31913-0000
I0205 20:17:56.037063 10116 slave.cpp:4365] Checkpointing ExecutorInfo to
'/tmp/mesos/meta/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/executors/80025a2d-e93b-4415-9b86-6a52d744165f/executor.info'
I0205 20:17:56.037256 10116 slave.cpp:3997] Launching executor
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000 in work directory
'/tmp/mesos/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/executors/80025a2d-e93b-4415-9b86-6a52d744165f/runs/5caf4b12-1bde-4c41-a127-0890001bba70'
I0205 20:17:56.037475 10116 slave.cpp:4388] Checkpointing TaskInfo to
'/tmp/mesos/meta/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/executors/80025a2d-e9
3b-4415-9b86-6a52d744165f/runs/5caf4b12-1bde-4c41-a127-0890001bba70/tasks/80025a2d-e93b-4415-9b86-6a52d744165f/task.info'
I0205 20:17:56.038048 10116 slave.cpp:1316] Queuing task
'80025a2d-e93b-4415-9b86-6a52d744165f' for executor
80025a2d-e93b-4415-9b86-6a52d744165f of framework
'20150205-122729-654050058-5050-31913-0000
I0205 20:17:56.038278 10116 slave.cpp:555] Successfully attached file
'/tmp/mesos/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/executors/80025a2d-e93b-4415-9b86-6a52d744165f/runs/5caf4b12-1bde-4c41-a127-0890001bba70'
I0205 20:17:56.039800 10115 docker.cpp:927] Starting container
'5caf4b12-1bde-4c41-a127-0890001bba70' for task
'80025a2d-e93b-4415-9b86-6a52d744165f' (and executor
'80025a2d-e93b-4415-9b86-6a52d744165f') of framework
'20150205-122729-654050058-5050-31913-0000'
I0205 20:17:56.039957 10115 docker.cpp:802] Running docker inspect
docker-repo:5000/task_app:qa-SNAPSHOT
I0205 20:17:56.105748 10111 docker.cpp:611] Docker pull
docker-repo:5000/task_app:qa-SNAPSHOT completed
I0205 20:17:56.105984 10111 docker.cpp:462] Running docker run -d -c 1024 -m
4731174912 -e mesos_task_id=80025a2d-e93b-4415-9b86-6a52d744165f -e ENV=qa -e
MEM=4000 -e MESOS_SANDBOX=/mnt/mesos/sandbox -v
/data/mesos/80025a2d-e93b-4415-9b86-6a52d744165f:/data:rw -v
/tmp/mesos/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/executors/80025a2d-e93b-4415-9b86-6a52d744165f/runs/5caf4b12-1bde-4c41-a127-0890001bba70:/mnt/mesos/sandbox
--net host --entrypoint /bin/sh --name
mesos-5caf4b12-1bde-4c41-a127-0890001bba70
docker-repo:5000/task_app:qa-SNAPSHOT -c /usr/local/bin/compact_logs_app
I0205 20:17:59.815831 10117 docker.cpp:633] Checkpointing pid 10207 to
'/tmp/mesos/meta/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/executors/80025a2d-e
93b-4415-9b86-6a52d744165f/runs/5caf4b12-1bde-4c41-a127-0890001bba70/pids/forked.pid'
I0205 20:17:59.816103 10117 docker.cpp:674] Running logs() {
docker logs --follow $1 &
pid=$!
docker wait $1 >/dev/null 2>&1
sleep 10
kill -TERM $pid >/dev/null 2>&1 &
}
logs mesos-5caf4b12-1bde-4c41-a127-0890001bba70
I0205 20:17:59.816797 10114 slave.cpp:2840] Monitoring executor
'80025a2d-e93b-4415-9b86-6a52d744165f' of framework
'20150205-122729-654050058-5050-31913-0000' in container
'5caf4b12-1bde-4c41-a127-0890001bba
70'
I0205 20:17:59.834761 10115 slave.cpp:1860] Got registration for executor
'80025a2d-e93b-4415-9b86-6a52d744165f' of framework
20150205-122729-654050058-5050-31913-0000 from executor(1)@10.3.0.115:54470
I0205 20:17:59.840859 10115 slave.cpp:1946] Checkpointing executor pid
'executor(1)@10.3.0.115:54470' to
'/tmp/mesos/meta/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/executors/80025a2d-e93b-4415-9b86-6a52d744165f/runs/5caf4b12-1bde-4c41-a127-0890001bba70/pids/libprocess.pid'
I0205 20:17:59.841169 10112 docker.cpp:568] Running docker inspect
mesos-5caf4b12-1bde-4c41-a127-0890001bba70
I0205 20:17:59.841234 10115 slave.cpp:1979] Flushing queued task
80025a2d-e93b-4415-9b86-6a52d744165f for executor
'80025a2d-e93b-4415-9b86-6a52d744165f' of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:17:59.843946 10114 slave.cpp:2215] Handling status update TASK_RUNNING
(UUID: 28c5d3a7-72d3-49f2-904a-15290644779c) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000 from executor(1)@10.3.0.115:54470
I0205 20:17:59.844101 10116 status_update_manager.cpp:317] Received status
update TASK_RUNNING (UUID: 28c5d3a7-72d3-49f2-904a-15290644779c) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:17:59.844138 10116 status_update_manager.cpp:494] Creating
StatusUpdate stream for task 80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:17:59.844301 10116 status_update_manager.hpp:346] Checkpointing UPDATE
for status update TASK_RUNNING (UUID: 28c5d3a7-72d3-49f2-904a-15290644779c) for
task 80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:17:59.918642 10110 docker.cpp:1298] Updated 'cpu.shares' to 1126 at
/cgroup/cpu/docker/99b809a513c5989b46f2505200cb4adef270eee4b17c38f8dbdebe5be1d8ed8b
for container 5caf4b12-1bde-4c41-a127-0890001bba70
I0205 20:17:59.918958 10110 docker.cpp:1333] Updated
'memory.soft_limit_in_bytes' to 4544MB for container
5caf4b12-1bde-4c41-a127-0890001bba70
I0205 20:17:59.919420 10110 docker.cpp:1359] Updated 'memory.limit_in_bytes' to
4544MB at
/cgroup/memory/docker/99b809a513c5989b46f2505200cb4adef270eee4b17c38f8dbdebe5be1d8ed8b
for container 5caf4b12-1bde-4c41-a127-0890001bba70
I0205 20:18:00.083292 10116 status_update_manager.cpp:371] Forwarding update
TASK_RUNNING (UUID: 28c5d3a7-72d3-49f2-904a-15290644779c) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000 to the slave
I0205 20:18:00.083425 10117 slave.cpp:2458] Forwarding the update TASK_RUNNING
(UUID: 28c5d3a7-72d3-49f2-904a-15290644779c) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000 to [email protected]:5050
I0205 20:18:00.083562 10117 slave.cpp:2385] Status update manager successfully
handled status update TASK_RUNNING (UUID: 28c5d3a7-72d3-49f2-904a-15290644779c)
for task 80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:00.083624 10117 slave.cpp:2391] Sending acknowledgement for status
update TASK_RUNNING (UUID: 28c5d3a7-72d3-49f2-904a-15290644779c) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000 to executor(1)@10.3.0.115:54470
I0205 20:18:00.114864 10114 status_update_manager.cpp:389] Received status
update acknowledgement (UUID: 28c5d3a7-72d3-49f2-904a-15290644779c) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:00.114946 10114 status_update_manager.hpp:346] Checkpointing ACK
for status update TASK_RUNNING (UUID: 28c5d3a7-72d3-49f2-904a-15290644779c) for
task 80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:00.258815 10112 slave.cpp:1800] Status update manager successfully
handled status update acknowledgement (UUID:
28c5d3a7-72d3-49f2-904a-15290644779c) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:01.362367 10115 slave.cpp:2538] Received ping from
slave-observer(2)@10.3.252.38:5050
I0205 20:18:16.363010 10110 slave.cpp:2538] Received ping from
slave-observer(2)@10.3.252.38:5050
I0205 20:18:30.059305 10112 monitor.cpp:140] Failed to collect resource usage
for container '5caf4b12-1bde-4c41-a127-0890001bba70' for executor
'80025a2d-e93b-4415-9b86-6a52d744165f' of framework
'20150205-122729-654050058-5050-31913-0000': Failed to get usage: No process
found at 10186
I0205 20:18:30.634464 10114 slave.cpp:3321] Current usage 16.52%. Max allowed
age: 5.143297473037627days
I0205 20:18:31.134436 10114 monitor.cpp:140] Failed to collect resource usage
for container '5caf4b12-1bde-4c41-a127-0890001bba70' for executor
'80025a2d-e93b-4415-9b86-6a52d744165f' of framework
'20150205-122729-654050058-5050-31913-0000': Failed to get usage: No process
found at 10186
I0205 20:18:31.363132 10113 slave.cpp:2538] Received ping from
slave-observer(2)@10.3.252.38:5050
I0205 20:18:31.732449 10116 slave.cpp:2215] Handling status update
TASK_FINISHED (UUID: 366dc316-4f3c-4066-9014-119542079add) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000 from executor(1)@10.3.0.115:54470
I0205 20:18:31.732532 10116 slave.cpp:4305] Terminating task
80025a2d-e93b-4415-9b86-6a52d744165f
E0205 20:18:31.732729 10112 slave.cpp:2344] Failed to update resources for
container 5caf4b12-1bde-4c41-a127-0890001bba70 of executor
80025a2d-e93b-4415-9b86-6a52d744165f running task
80025a2d-e93b-4415-9b86-6a52d744165f on status update for terminal task,
destroying container: Failed to determine cgroup for the 'cpu' subsystem:
Failed to read /proc/10186/cgroup: Failed to open file '/proc/10186/cgroup': No
such file or directory
I0205 20:18:31.732857 10116 docker.cpp:1501] Destroying container
'5caf4b12-1bde-4c41-a127-0890001bba70'
I0205 20:18:31.732914 10111 status_update_manager.cpp:317] Received status
update TASK_FINISHED (UUID: 366dc316-4f3c-4066-9014-119542079add) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:31.732941 10116 docker.cpp:1593] Running docker stop on container
'5caf4b12-1bde-4c41-a127-0890001bba70'
I0205 20:18:31.732986 10111 status_update_manager.hpp:346] Checkpointing UPDATE
for status update TASK_FINISHED (UUID: 366dc316-4f3c-4066-9014-119542079add)
for task 80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:31.733029 10116 docker.cpp:502] Running docker stop -t 0
mesos-5caf4b12-1bde-4c41-a127-0890001bba70
I0205 20:18:31.842528 10111 status_update_manager.cpp:371] Forwarding update
TASK_FINISHED (UUID: 366dc316-4f3c-4066-9014-119542079add) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000 to the slave
I0205 20:18:31.842747 10113 slave.cpp:2458] Forwarding the update TASK_FINISHED
(UUID: 366dc316-4f3c-4066-9014-119542079add) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000 to [email protected]:5050
I0205 20:18:31.842913 10113 slave.cpp:2385] Status update manager successfully
handled status update TASK_FINISHED (UUID:
366dc316-4f3c-4066-9014-119542079add) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:31.843026 10113 slave.cpp:2391] Sending acknowledgement for status
update TASK_FINISHED (UUID: 366dc316-4f3c-4066-9014-119542079add) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000 to executor(1)@10.3.0.115:54470
I0205 20:18:31.865635 10111 status_update_manager.cpp:389] Received status
update acknowledgement (UUID: 366dc316-4f3c-4066-9014-119542079add) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:31.880321 10111 status_update_manager.hpp:346] Checkpointing ACK
for status update TASK_FINISHED (UUID: 366dc316-4f3c-4066-9014-119542079add)
for task 80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:32.076510 10111 status_update_manager.cpp:525] Cleaning up status
update stream for task 80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:32.076659 10111 slave.cpp:1800] Status update manager successfully
handled status update acknowledgement (UUID:
366dc316-4f3c-4066-9014-119542079add) for task
80025a2d-e93b-4415-9b86-6a52d744165f of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:32.076742 10111 slave.cpp:4344] Completing task
80025a2d-e93b-4415-9b86-6a52d744165f
I0205 20:18:32.135555 10110 monitor.cpp:140] Failed to collect resource usage
for container '5caf4b12-1bde-4c41-a127-0890001bba70' for executor
'80025a2d-e93b-4415-9b86-6a52d744165f' of framework
'20150205-122729-654050058-5050-31913-0000': Container
'5caf4b12-1bde-4c41-a127-0890001bba70' not found
I0205 20:18:32.810014 10117 docker.cpp:1678] Executor for container
'5caf4b12-1bde-4c41-a127-0890001bba70' has exited
I0205 20:18:32.810206 10114 slave.cpp:2898] Executor
'80025a2d-e93b-4415-9b86-6a52d744165f' of framework
20150205-122729-654050058-5050-31913-0000 exited with status 0
I0205 20:18:32.810349 10114 slave.cpp:3007] Cleaning up executor
'80025a2d-e93b-4415-9b86-6a52d744165f' of framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:32.810535 10113 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/executors/80025a2d-e93b-4415-9b86-6a52d744165f/runs/5caf4b12-1bde-4c41-a127-0890001bba70'
for gc 6.9999906190163days in the future
I0205 20:18:32.810627 10114 slave.cpp:3084] Cleaning up framework
20150205-122729-654050058-5050-31913-0000
I0205 20:18:32.810672 10113 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/executors/80025a2d-e93b-4415-9b86-6a52d744165f'
for gc 6.99999061865778days in the future
I0205 20:18:32.810726 10111 status_update_manager.cpp:279] Closing status
update streams for framework 20150205-122729-654050058-5050-31913-0000
I0205 20:18:32.810782 10113 gc.cpp:56] Scheduling
'/tmp/mesos/meta/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/executors/80025a2d-e93b-4415-9b86-6a52d744165f/runs/5caf4b12-1bde-4c41-a127-0890001bba70'
for gc 6.99999061833481days in the future
I0205 20:18:32.810847 10113 gc.cpp:56] Scheduling
'/tmp/mesos/meta/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000/executors/80025a2d-e93b-4415-9b86-6a52d744165f'
for gc 6.99999061813926days in the future
I0205 20:18:32.810881 10113 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000'
for gc 6.99999061653926days in the future
I0205 20:18:32.810914 10113 gc.cpp:56] Scheduling
'/tmp/mesos/meta/slaves/20150205-122729-654050058-5050-31913-S0/frameworks/20150205-122729-654050058-5050-31913-0000'
for gc 6.9999906163763days in the future
{noformat}
> failed to determine cgroup for the 'cpu' subsystem
> --------------------------------------------------
>
> Key: MESOS-1837
> URL: https://issues.apache.org/jira/browse/MESOS-1837
> Project: Mesos
> Issue Type: Bug
> Components: docker
> Affects Versions: 0.20.1
> Environment: Ubuntu 14.04
> Reporter: Chris Fortier
> Assignee: Timothy Chen
>
> Attempting to launch Docker container with Marathon. Container is launched
> then fails.
> A search of /var/log/syslog reveals:
> Sep 27 03:01:43 vagrant-ubuntu-trusty-64 mesos-slave[1409]: E0927
> 03:01:43.546957 1463 slave.cpp:2205] Failed to update resources for
> container 8c2429d9-f090-4443-8108-0206ca37f3fd of executor
> hello-world.970dbe74-45f2-11e4-8b1d-56847afe9799 running task
> hello-world.970dbe74-45f2-11e4-8b1d-56847afe9799 on status update for
> terminal task, destroying container: Failed to determine cgroup for the 'cpu'
> subsystem: Failed to read /proc/9792/cgroup: Failed to open file
> '/proc/9792/cgroup': No such file or directory
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
