[
https://issues.apache.org/jira/browse/MESOS-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471382#comment-16471382
]
Qian Zhang edited comment on MESOS-8877 at 5/11/18 2:05 AM:
------------------------------------------------------------
We may need to set the resources of the container when recovering it, but the
problem is, in `DockerContainerizerProcess::_recover` we only have the
resources of the Docker executor (0.1 cpus and 32 MB memory) but not the
resources of the Docker container, however when the Docker executor reregisters
with agent after agent recovery, `DockerContainerizerProcess::update` will be
called by `Slave::reregisterExecutor`, and the resources passed into
`DockerContainerizerProcess::update` include both the Docker executor's
resources and the Docker container's resources, so the result of [this
check|https://github.com/apache/mesos/blob/1.5.0/src/slave/containerizer/docker.cpp#L1652]
will still be false, and we will still do the update to enlarge the Docker
container's resources.
So to fix this issue, I think we may consider to leverage the field
[Container::launchesExecutorContainer|https://github.com/apache/mesos/blob/1.5.0/src/slave/containerizer/docker.hpp#L503:L505]
which will be:
* true: If we launch the executor in a Docker container.
* false: If we launch the executor in a normal process, i.e., the command task
case where the task will be launched by Docker executor as a Docker container.
In `DockerContainerizerProcess::update`, we can check if
`container->launchesExecutorContainer` is false, if it is, then we just ignore
updating container's resources, since it is the case that the command task is
launched as a Docker container whose resources has already been set when it is
launched by Docker executor, we do not need to update it again.
was (Author: qianzhang):
We may need to set the resources of the container when recovering it, but the
problem is, in `DockerContainerizerProcess::_recover` we only have the
resources of the Docker executor (0.1 cpus and 32 MB memory) but not the
resources of the Docker container, however when the Docker executor reregisters
with agent after agent recovery, `DockerContainerizerProcess::update` will be
called by `Slave::reregisterExecutor`, and the resources passed into
`DockerContainerizerProcess::update` include both the Docker executor's
resources and the Docker container's resources, so the result of [this
check|https://github.com/apache/mesos/blob/1.5.0/src/slave/containerizer/docker.cpp#L1652]
will still be false, and we will still do the update to enlarge the Docker
container's resources.
So to fix this issue, I think we may consider to leverage the field
[Container::launchesExecutorContainer|https://github.com/apache/mesos/blob/1.5.0/src/slave/containerizer/docker.hpp#L503:L505]
which will be:
* true: If we launch the executor in a Docker container or will be
* false: If we launch the executor in a normal process (i.e., the command task
case, the task will be launched by Docker executor as a Docker container)
In `DockerContainerizerProcess::update`, we can check if
`container->launchesExecutorContainer` is false, if it is, then we just ignore
updating container's resources, since it is the case that the command task is
launched as a Docker container whose resources has already been set when it is
launched by Docker executor, we do not need to update it again.
> Docker container's resources will be wrongly enlarged in cgroups after agent
> recovery
> -------------------------------------------------------------------------------------
>
> Key: MESOS-8877
> URL: https://issues.apache.org/jira/browse/MESOS-8877
> Project: Mesos
> Issue Type: Bug
> Components: docker
> Reporter: Qian Zhang
> Priority: Major
>
> Reproduce steps:
> 1. Run `mesos-execute --master=10.0.49.2:5050
> --task=[file:///home/qzhang/workspace/config/task_docker.json]
> --checkpoint=true` to launch a Docker container.
> {code:java}
> # cat task_docker.json
> {
> "name": "test",
> "task_id": {"value" : "test"},
> "agent_id": {"value" : ""},
> "resources": [
> {"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}},
> {"name": "mem", "type": "SCALAR", "scalar": {"value": 32}}
> ],
> "command": {
> "value": "sleep 55555"
> },
> "container": {
> "type": "DOCKER",
> "docker": {
> "image": "alpine"
> }
> }
> }
> {code}
> 2. When the Docker container is running, we can see its resources in cgroups
> are correctly set, so far so good.
> {code:java}
> # cat
> /sys/fs/cgroup/cpu,cpuacct/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106/cpu.cfs_quota_us
>
> 10000
> # cat
> /sys/fs/cgroup/memory/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106/memory.limit_in_bytes
>
> 33554432
> {code}
> 3. Restart Mesos agent, and then we will see the resources of the Docker
> container will be wrongly enlarged.
> {code}
> I0503 02:06:17.268340 29512 docker.cpp:1855] Updated 'cpu.shares' to 204 at
> /sys/fs/cgroup/cpu,cpuacct/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106
> for container 1b21295b-2f49-4d08-84c7-43b9ae15ad88
> I0503 02:06:17.271390 29512 docker.cpp:1882] Updated 'cpu.cfs_period_us' to
> 100ms and 'cpu.cfs_quota_us' to 20ms (cpus 0.2) for container
> 1b21295b-2f49-4d08-84c7-43b9ae15ad88
> I0503 02:06:17.273082 29512 docker.cpp:1924] Updated
> 'memory.soft_limit_in_bytes' to 64MB for container
> 1b21295b-2f49-4d08-84c7-43b9ae15ad88
> I0503 02:06:17.275908 29512 docker.cpp:1950] Updated 'memory.limit_in_bytes'
> to 64MB at
> /sys/fs/cgroup/memory/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106
> for container 1b21295b-2f49-4d08-84c7-43b9ae15ad88
> # cat
> /sys/fs/cgroup/cpu,cpuacct/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106/cpu.cfs_quota_us
> 20000
> # cat
> /sys/fs/cgroup/memory/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106/memory.limit_in_bytes
> 67108864
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
