Daniel Hall created MESOS-1915:
----------------------------------
Summary: Docker containers that fail to launch are not killed
Key: MESOS-1915
URL: https://issues.apache.org/jira/browse/MESOS-1915
Project: Mesos
Issue Type: Bug
Components: slave
Affects Versions: 0.20.1
Environment: Mesos 0.20.1 using the docker executor with a private
docker repository. Images often take up to 5 minutes to launch.
/etc/mesos-slave/executor_registration_timeout is set to '10mins'
Reporter: Daniel Hall
When we launch docker containers on our Mesos cluster using marathon we have
noticed that we end up with several docker containers running, with only one of
them actually being tracked my Mesos. When inspected the containers both have
the same start time.
This seems to be because Mesos gives up on trying to start the container after
1min, but fails to clean up the docker container because it is is not yet
running. Eventually the container starts alongside all the other attempts mesos
has made and we end up with several containers running with only one being
tracked by Mesos.
I've pasted some logs from the slave below filter for that particular task, but
it is pretty easy to replicate in our environment so I'm happy to provide
further logs, details and analysis as required. This is becoming a bit problem
for us so we are happy to help as much as possible.
{{ noformat }}
Oct 13 04:47:42 mesosslave-1 mesos-slave[16647]: I1013 04:47:42.776945 16661
docker.cpp:743] Starting container 'dd113461-4d18-4170-8e3f-9527e6d7f598' for
task 'docker-test.11588a48-5294-11e4-adea-42010af0f51e' (and executor
'docker-test.11588a48-5294-11e4-adea-42010af0f51e') of framework
'20140918-022627-519434250-5050-6171-0000'
Oct 13 04:48:42 mesosslave-1 mesos-slave[16647]: E1013 04:48:42.819563 16664
slave.cpp:2205] Failed to update resources for container
dd113461-4d18-4170-8e3f-9527e6d7f598 of executor
docker-test.11588a48-5294-11e4-adea-42010af0f51e running task
docker-test.11588a48-5294-11e4-adea-42010af0f51e on status update for terminal
task, destroying container: No container found
Oct 13 04:49:29 mesosslave-1 mesos-slave[16647]: I1013 04:49:29.916460 16665
slave.cpp:2538] Monitoring executor
'docker-test.11588a48-5294-11e4-adea-42010af0f51e' of framework
'20140918-022627-519434250-5050-6171-0000' in container
'dd113461-4d18-4170-8e3f-9527e6d7f598'
Oct 13 04:49:31 mesosslave-1 mesos-slave[16647]: I1013 04:49:31.103175 16663
docker.cpp:1286] Updated 'cpu.shares' to 102 at
/cgroup/cpu/docker/6a581f5c2174dc76bcfb2e5b89fd9a4310732c384d93901a8b37da8aeb700468
for container dd113461-4d18-4170-8e3f-9527e6d7f598
Oct 13 04:49:31 mesosslave-1 mesos-slave[16647]: I1013 04:49:31.105036 16663
docker.cpp:1321] Updated 'memory.soft_limit_in_bytes' to 32MB for container
dd113461-4d18-4170-8e3f-9527e6d7f598
{{ noformat }}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
