Qian Zhang commented on MESOS-8488:

commit a7714536fad1140fd0c07c47e32b40e9ed00a3c3
Author: Qian Zhang 
Date: Mon Feb 5 20:42:07 2018 +0800

Reaped the container process directly in Docker executor.
 Due to a Docker issue (https://github.com/moby/moby/issues/33820),
 Docker daemon can fail to catch a container exit, i.e., the container
 process has already exited but the command `docker ps` shows the
 container still running, this will lead to the "docker run" command
 that we execute in Docker executor never returning, and it will also
 cause the `docker stop` command takes no effect, i.e., it will return
 without error but `docker ps` shows the container still running, so
 the task will stuck in `TASK_KILLING` state.
 To workaround this Docker issue, in this patch we made Docker executor
 reaps the container process directly so Docker executor will be notified
 once the container process exits.
 Review: https://reviews.apache.org/r/65518

> Docker bug can cause unkillable tasks
> -------------------------------------
>                 Key: MESOS-8488
>                 URL: https://issues.apache.org/jira/browse/MESOS-8488
>             Project: Mesos
>          Issue Type: Improvement
>          Components: containerization
>    Affects Versions: 1.5.0
>            Reporter: Greg Mann
>            Assignee: Qian Zhang
>            Priority: Major
>              Labels: mesosphere
>             Fix For: 1.6.0
> Due to an [issue on the Moby 
> project|https://github.com/moby/moby/issues/33820], it's possible for Docker 
> versions 1.13 and later to fail to catch a container exit, so that the 
> {{docker run}} command which was used to launch the container will never 
> return. This can lead to the Docker executor becoming stuck in a state where 
> it believes the container is still running and cannot be killed.
> We should update the Docker executor to ensure that containers stuck in such 
> a state cannot cause unkillable Docker executors/tasks.
> One way to do this would be a timeout, after which the Docker executor will 
> commit suicide if a kill task attempt has not succeeded. However, if we do 
> this we should also ensure that in the case that the container was actually 
> still running, either the Docker daemon or the DockerContainerizer would 
> clean up the container when it does exit.
> Another option might be for the Docker executor to directly {{wait()}} on the 
> container's Linux PID, in order to notice when the container exits.

This message was sent by Atlassian JIRA

Reply via email to