[jira] [Commented] (MESOS-9950) memory cgroup gone before isolator cleaning up
[
https://issues.apache.org/jira/browse/MESOS-9950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912997#comment-16912997
]
longfei commented on MESOS-9950:
version: 1.7.3
> memory cgroup gone before isolator cleaning up
> --
>
> Key: MESOS-9950
> URL: https://issues.apache.org/jira/browse/MESOS-9950
> Project: Mesos
> Issue Type: Bug
>Reporter: longfei
>Priority: Major
>
> The memcg created by mesos may have been deleted before cgroup/memory
> isolator cleaning up.
> This would let the termination fail and lose information in the old
> termination(before fail).
> {code:java}
> I0821 15:16:03.025796 3354800 paths.cpp:745] Creating sandbox
> '/opt/tiger/mesos_deploy_videoarch/mesos_zeus/slave/slaves/fb5c1a5b-e106-47c1-9fe3-6ebd311b30ee-S628/frameworks/8e4967e5-736e-4a22-90c3-7b32d526914d-/executors/mt:z03584687:1/runs/a0706ca0-fe2c-4477-8161-329b26ea5d89'
> for user 'tiger'
> I0821 15:16:03.026199 3354800 paths.cpp:748] Creating sandbox
> '/opt/tiger/mesos_deploy_videoarch/mesos_zeus/slave/meta/slaves/fb5c1a5b-e106-47c1-9fe3-6ebd311b30ee-S628/frameworks/8e4967e5-736e-4a22-90c3-7b32d526914d-/executors/mt:z03584687:1/runs/a0706ca0-fe2c-4477-8161-329b26ea5d89'
> I0821 15:16:03.026304 3354800 slave.cpp:9064] Launching executor
> 'mt:z03584687:1' of framework
> 8e4967e5-736e-4a22-90c3-7b32d526914d- with resources
> [{"allocation_info":{"role":"*"},"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"mem","scalar":{"value":32.0},"type":"SCALAR"}]
> in work directory
> '/opt/tiger/mesos_deploy_videoarch/mesos_zeus/slave/slaves/fb5c1a5b-e106-47c1-9fe3-6ebd311b30ee-S628/frameworks/8e4967e5-736e-4a22-90c3-7b32d526914d-/executors/mt:z03584687:1/runs/a0706ca0-fe2c-4477-8161-329b26ea5d89'
> I0821 15:16:03.051795 3354800 slave.cpp:3520] Launching container
> a0706ca0-fe2c-4477-8161-329b26ea5d89 for executor
> 'mt:z03584687:1' of framework
> 8e4967e5-736e-4a22-90c3-7b32d526914d-
> I0821 15:16:03.076608 3354807 containerizer.cpp:1325] Starting container
> a0706ca0-fe2c-4477-8161-329b26ea5d89
> I0821 15:16:03.076911 3354807 containerizer.cpp:3185] Transitioning the state
> of container a0706ca0-fe2c-4477-8161-329b26ea5d89 from PROVISIONING to
> PREPARING
> I0821 15:16:03.077906 3354802 memory.cpp:478] Started listening for OOM
> events for container a0706ca0-fe2c-4477-8161-329b26ea5d89
> I0821 15:16:03.079540 3354804 memory.cpp:198] Updated
> 'memory.soft_limit_in_bytes' to 4032MB for container
> a0706ca0-fe2c-4477-8161-329b26ea5d89
> I0821 15:16:03.079587 3354820 cpu.cpp:92] Updated 'cpu.shares' to 1126 (cpus
> 1.1) for container a0706ca0-fe2c-4477-8161-329b26ea5d89
> I0821 15:16:03.079589 3354804 memory.cpp:227] Updated 'memory.limit_in_bytes'
> to 4032MB for container a0706ca0-fe2c-4477-8161-329b26ea5d89
> I0821 15:16:03.080901 3354802 switchboard.cpp:316] Container logger module
> finished preparing container a0706ca0-fe2c-4477-8161-329b26ea5d89;
> IOSwitchboard server is not required
> I0821 15:16:03.081593 3354801 linux_launcher.cpp:492] Launching container
> a0706ca0-fe2c-4477-8161-329b26ea5d89 and cloning with namespaces
> I0821 15:16:03.083823 3354808 containerizer.cpp:2107] Checkpointing
> container's forked pid 1857418 to
> '/opt/tiger/mesos_deploy_videoarch/mesos_zeus/slave/meta/slaves/fb5c1a5b-e106-47c1-9fe3-6ebd311b30ee-S628/frameworks/8e4967e5-736e-4a22-90c3-7b32d526914d-/executors/mt:z03584687:1/runs/a0706ca0-fe2c-4477-8161-329b26ea5d89/pids/forked.pid'
> I0821 15:16:03.084156 3354808 containerizer.cpp:3185] Transitioning the state
> of container a0706ca0-fe2c-4477-8161-329b26ea5d89 from PREPARING to ISOLATING
> I0821 15:16:03.091468 3354808 containerizer.cpp:3185] Transitioning the state
> of container a0706ca0-fe2c-4477-8161-329b26ea5d89 from ISOLATING to FETCHING
> I0821 15:16:03.094933 3354808 containerizer.cpp:3185] Transitioning the state
> of container a0706ca0-fe2c-4477-8161-329b26ea5d89 from FETCHING to RUNNING
> I0821 15:16:03.197753 3354808 memory.cpp:198] Updated
> 'memory.soft_limit_in_bytes' to 4032MB for container
> a0706ca0-fe2c-4477-8161-329b26ea5d89
> I0821 15:16:03.197757 3354801 cpu.cpp:92] Updated 'cpu.shares' to 1126 (cpus
> 1.1) for container a0706ca0-fe2c-4477-8161-329b26ea5d89
> I0821 15:21:39.692978 3354814 memory.cpp:515] OOM detected for container
> a0706ca0-fe2c-4477-8161-329b26ea5d89
> I0821 15:21:39.693182 3354805 containerizer.cpp:3044] Container
> a0706ca0-fe2c-4477-8161-329b26ea5d89 has reached its limit for resource []
> and will be terminated
> I0821 15:21:39.693192 3354805 containerizer.cpp:2518] Destroying container
> a0706ca0-fe2c-4477-8161-329b26ea5d89 in RUNNING state
> I0821 15:21:39.693197 3354805
[jira] [Created] (MESOS-9950) memory cgroup gone before isolator cleaning up
longfei created MESOS-9950:
--
Summary: memory cgroup gone before isolator cleaning up
Key: MESOS-9950
URL: https://issues.apache.org/jira/browse/MESOS-9950
Project: Mesos
Issue Type: Bug
Reporter: longfei
The memcg created by mesos may have been deleted before cgroup/memory isolator
cleaning up.
This would let the termination fail and lose information in the old
termination(before fail).
{code:java}
I0821 15:16:03.025796 3354800 paths.cpp:745] Creating sandbox
'/opt/tiger/mesos_deploy_videoarch/mesos_zeus/slave/slaves/fb5c1a5b-e106-47c1-9fe3-6ebd311b30ee-S628/frameworks/8e4967e5-736e-4a22-90c3-7b32d526914d-/executors/mt:z03584687:1/runs/a0706ca0-fe2c-4477-8161-329b26ea5d89'
for user 'tiger'
I0821 15:16:03.026199 3354800 paths.cpp:748] Creating sandbox
'/opt/tiger/mesos_deploy_videoarch/mesos_zeus/slave/meta/slaves/fb5c1a5b-e106-47c1-9fe3-6ebd311b30ee-S628/frameworks/8e4967e5-736e-4a22-90c3-7b32d526914d-/executors/mt:z03584687:1/runs/a0706ca0-fe2c-4477-8161-329b26ea5d89'
I0821 15:16:03.026304 3354800 slave.cpp:9064] Launching executor
'mt:z03584687:1' of framework
8e4967e5-736e-4a22-90c3-7b32d526914d- with resources
[{"allocation_info":{"role":"*"},"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"mem","scalar":{"value":32.0},"type":"SCALAR"}]
in work directory
'/opt/tiger/mesos_deploy_videoarch/mesos_zeus/slave/slaves/fb5c1a5b-e106-47c1-9fe3-6ebd311b30ee-S628/frameworks/8e4967e5-736e-4a22-90c3-7b32d526914d-/executors/mt:z03584687:1/runs/a0706ca0-fe2c-4477-8161-329b26ea5d89'
I0821 15:16:03.051795 3354800 slave.cpp:3520] Launching container
a0706ca0-fe2c-4477-8161-329b26ea5d89 for executor 'mt:z03584687:1'
of framework 8e4967e5-736e-4a22-90c3-7b32d526914d-
I0821 15:16:03.076608 3354807 containerizer.cpp:1325] Starting container
a0706ca0-fe2c-4477-8161-329b26ea5d89
I0821 15:16:03.076911 3354807 containerizer.cpp:3185] Transitioning the state
of container a0706ca0-fe2c-4477-8161-329b26ea5d89 from PROVISIONING to PREPARING
I0821 15:16:03.077906 3354802 memory.cpp:478] Started listening for OOM events
for container a0706ca0-fe2c-4477-8161-329b26ea5d89
I0821 15:16:03.079540 3354804 memory.cpp:198] Updated
'memory.soft_limit_in_bytes' to 4032MB for container
a0706ca0-fe2c-4477-8161-329b26ea5d89
I0821 15:16:03.079587 3354820 cpu.cpp:92] Updated 'cpu.shares' to 1126 (cpus
1.1) for container a0706ca0-fe2c-4477-8161-329b26ea5d89
I0821 15:16:03.079589 3354804 memory.cpp:227] Updated 'memory.limit_in_bytes'
to 4032MB for container a0706ca0-fe2c-4477-8161-329b26ea5d89
I0821 15:16:03.080901 3354802 switchboard.cpp:316] Container logger module
finished preparing container a0706ca0-fe2c-4477-8161-329b26ea5d89;
IOSwitchboard server is not required
I0821 15:16:03.081593 3354801 linux_launcher.cpp:492] Launching container
a0706ca0-fe2c-4477-8161-329b26ea5d89 and cloning with namespaces
I0821 15:16:03.083823 3354808 containerizer.cpp:2107] Checkpointing container's
forked pid 1857418 to
'/opt/tiger/mesos_deploy_videoarch/mesos_zeus/slave/meta/slaves/fb5c1a5b-e106-47c1-9fe3-6ebd311b30ee-S628/frameworks/8e4967e5-736e-4a22-90c3-7b32d526914d-/executors/mt:z03584687:1/runs/a0706ca0-fe2c-4477-8161-329b26ea5d89/pids/forked.pid'
I0821 15:16:03.084156 3354808 containerizer.cpp:3185] Transitioning the state
of container a0706ca0-fe2c-4477-8161-329b26ea5d89 from PREPARING to ISOLATING
I0821 15:16:03.091468 3354808 containerizer.cpp:3185] Transitioning the state
of container a0706ca0-fe2c-4477-8161-329b26ea5d89 from ISOLATING to FETCHING
I0821 15:16:03.094933 3354808 containerizer.cpp:3185] Transitioning the state
of container a0706ca0-fe2c-4477-8161-329b26ea5d89 from FETCHING to RUNNING
I0821 15:16:03.197753 3354808 memory.cpp:198] Updated
'memory.soft_limit_in_bytes' to 4032MB for container
a0706ca0-fe2c-4477-8161-329b26ea5d89
I0821 15:16:03.197757 3354801 cpu.cpp:92] Updated 'cpu.shares' to 1126 (cpus
1.1) for container a0706ca0-fe2c-4477-8161-329b26ea5d89
I0821 15:21:39.692978 3354814 memory.cpp:515] OOM detected for container
a0706ca0-fe2c-4477-8161-329b26ea5d89
I0821 15:21:39.693182 3354805 containerizer.cpp:3044] Container
a0706ca0-fe2c-4477-8161-329b26ea5d89 has reached its limit for resource [] and
will be terminated
I0821 15:21:39.693192 3354805 containerizer.cpp:2518] Destroying container
a0706ca0-fe2c-4477-8161-329b26ea5d89 in RUNNING state
I0821 15:21:39.693197 3354805 containerizer.cpp:3185] Transitioning the state
of container a0706ca0-fe2c-4477-8161-329b26ea5d89 from RUNNING to DESTROYING
I0821 15:21:39.693542 3354815 linux_launcher.cpp:576] Asked to destroy
container a0706ca0-fe2c-4477-8161-329b26ea5d89
I0821 15:21:39.693563 3354815 linux_launcher.cpp:618] Destroying cgroup
[jira] [Commented] (MESOS-9806) Address allocator performance regression due to the removal of quota role sorter.
[
https://issues.apache.org/jira/browse/MESOS-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912823#comment-16912823
]
Benjamin Mahler commented on MESOS-9806:
https://reviews.apache.org/r/71345/
https://reviews.apache.org/r/71346/
https://reviews.apache.org/r/71347/
Master branch:
{noformat}
HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 36.1185929secs
Made 0 allocation in 32.62218602secs
{noformat}
After all patches in review chain:
{noformat}
HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 21.389381617secs
Made 0 allocation in 18.593000222secs
{noformat}
> Address allocator performance regression due to the removal of quota role
> sorter.
> -
>
> Key: MESOS-9806
> URL: https://issues.apache.org/jira/browse/MESOS-9806
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Meng Zhu
>Assignee: Meng Zhu
>Priority: Critical
> Labels: resource-management
>
> In MESOS-9802, we removed the quota role sorter which is tech debt.
> However, this slows down the allocator. The problem is that in the first
> stage, even though a cluster might have no active roles with non-default
> quota, the allocator will now have to sort and go through each and every role
> in the cluster. Benchmark result shows that for 1k roles with 2k frameworks,
> the allocator could experience ~50% performance degradation.
> There are a couple of ways to address this issue. For example, we could make
> the sorter aware of quota. And add a method, say `sortQuotaRoles`, to return
> all the roles with non-default quota. Alternatively, an even better approach
> would be to deprecate the sorter concept and just have two standalone
> functions e.g. sortRoles() and sortQuotaRoles() that takes in the role tree
> structure (not yet exist in the allocator) and return the sorted roles.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (MESOS-8400) Handle plugin crashes gracefully in SLRP recovery.
[
https://issues.apache.org/jira/browse/MESOS-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Bannier reassigned MESOS-8400:
---
Sprint: Resource Mgmt: RI-17 Sprint 53
Assignee: Benjamin Bannier
> Handle plugin crashes gracefully in SLRP recovery.
> --
>
> Key: MESOS-8400
> URL: https://issues.apache.org/jira/browse/MESOS-8400
> Project: Mesos
> Issue Type: Improvement
>Reporter: Chun-Hung Hsiao
>Assignee: Benjamin Bannier
>Priority: Blocker
> Labels: mesosphere, mesosphere-dss-post-ga, storage
>
> When a CSI plugin crashes, the container daemon in SLRP will reset its
> corresponding {{csi::Client}} service future. However, if a CSI call races
> with a plugin crash, the call may be issued before the service future is
> reset, resulting in a failure for that CSI call. MESOS-9517 partly addresses
> this for {{CreateVolume}} and {{DeleteVolume}} calls, but calls in the SLRP
> recovery path, e.g., {{ListVolume}}, {{GetCapacity}}, {{Probe}}, could make
> the SLRP unrecoverable.
> There are two main issues:
> 1. For {{Probe}}, we should investigate if it is needed to make a few retry
> attempts, then after that, we should recover from failed attempts (e.g., kill
> the plugin container), then make the container daemon relaunch the plugin
> instead of failing the daemon.
> 2. For other calls in the recovery path, we should either retry the call, or
> make the local resource provider daemon be able to restart the SLRP after it
> fails.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (MESOS-9949) Track allocated/offered in the allocator's role tree.
Benjamin Mahler created MESOS-9949: -- Summary: Track allocated/offered in the allocator's role tree. Key: MESOS-9949 URL: https://issues.apache.org/jira/browse/MESOS-9949 Project: Mesos Issue Type: Task Components: allocation, master Reporter: Benjamin Mahler Currently the allocator's role tree only tracks the reserved resources for each role subtree. For metrics purposes, it would be ideal to track offered / allocated as well. This requires augmenting the allocator's structs and recoverResources to hold the two categories independently and transition from offered -> allocated as applicable when recovering resources. This might require a slight change to the recoverResources interface. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MESOS-9887) Race condition between two terminal task status updates for Docker executor.
[
https://issues.apache.org/jira/browse/MESOS-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912558#comment-16912558
]
Andrei Budnik commented on MESOS-9887:
--
https://reviews.apache.org/r/71343/
> Race condition between two terminal task status updates for Docker executor.
>
>
> Key: MESOS-9887
> URL: https://issues.apache.org/jira/browse/MESOS-9887
> Project: Mesos
> Issue Type: Bug
> Components: agent, containerization
>Reporter: Andrei Budnik
>Assignee: Andrei Budnik
>Priority: Blocker
> Labels: agent, containerization
> Attachments: race_example.txt
>
>
> h2. Overview
> Expected behavior:
> Task successfully finishes and sends TASK_FINISHED status update.
> Observed behavior:
> Task successfully finishes, but the agent sends TASK_FAILED with the reason
> "REASON_EXECUTOR_TERMINATED".
> In normal circumstances, Docker executor
> [sends|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/docker/executor.cpp#L758]
> final status update TASK_FINISHED to the agent, which then [gets
> processed|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L5543]
> by the agent before termination of the executor's process.
> However, if the processing of the initial TASK_FINISHED gets delayed, then
> there is a chance that Docker executor terminates and the agent
> [triggers|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L6662]
> TASK_FAILED which will [be
> handled|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L5816-L5826]
> prior to the TASK_FINISHED status update.
> See attached logs which contain an example of the race condition.
> h2. Reproducing bug
> 1. Add the following code:
> {code:java}
> static int c = 0;
> if (++c == 3) { // to skip TASK_STARTING and TASK_RUNNING status updates.
> ::sleep(2);
> }
> {code}
> to the
> [`ComposingContainerizerProcess::status`|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/containerizer/composing.cpp#L578]
> and to the
> [`DockerContainerizerProcess::status`|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/containerizer/docker.cpp#L2167].
> 2. Recompile mesos
> 3. Launch mesos master and agent locally
> 4. Launch a simple Docker task via `mesos-execute`:
> {code}
> # cd build
> ./src/mesos-execute --master="`hostname`:5050" --name="a"
> --containerizer=docker --docker_image=alpine --resources="cpus:1;mem:32"
> --command="ls"
> {code}
> h2. Race condition - description
> 1. Mesos agent receives TASK_FINISHED status update and then subscribes on
> [`containerizer->status()`|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L5754-L5761].
> 2. `containerizer->status()` operation for TASK_FINISHED status update gets
> delayed in the composing containerizer (e.g. due to switch of the worker
> thread that executes `status` method).
> 3. Docker executor terminates and the agent
> [triggers|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L6662]
> TASK_FAILED.
> 4. Docker containerizer destroys the container. A registered callback for the
> `containerizer->wait` call in the composing containerizer dispatches [lambda
> function|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/containerizer/composing.cpp#L368-L373]
> that will clean up `containers_` map.
> 5. Composing c'zer resumes and dispatches
> `[status()|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/containerizer/composing.cpp#L579]`
> method to the Docker containerizer for TASK_FINISHED, which in turn hangs
> for a few seconds.
> 6. Corresponding `containerId` gets removed from the `containers_` map of the
> composing c'zer.
> 7. Mesos agent subscribes on
> [`containerizer->status()`|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L5754-L5761]
> for the TASK_FAILED status update.
> 8. Composing c'zer returns ["Container not
> found"|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/containerizer/composing.cpp#L576]
> for TASK_FAILED.
> 9.
> `[Slave::_statusUpdate|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L5826]`
> stores TASK_FAILED terminal status update in the executor's data structure.
> 10. Docker containerizer resumes and finishes processing of `status()` method
> for TASK_FINISHED. Finally, it returns control to the
[jira] [Created] (MESOS-9948) master::Slave::hasExecutor occupies 37% of a 150 second perf sample from a user.
Benjamin Mahler created MESOS-9948: -- Summary: master::Slave::hasExecutor occupies 37% of a 150 second perf sample from a user. Key: MESOS-9948 URL: https://issues.apache.org/jira/browse/MESOS-9948 Project: Mesos Issue Type: Improvement Components: master Reporter: Benjamin Mahler Attachments: long-fei-enable-debug-slow-master.gz If you drop the attached perf stacks into flamescope, you can see that mesos::internal::master::Slave::hasExecutor occupies 37% of the overall samples! This function does 3 hashmap lookups, 1 can be eliminated for a quick win. However, the larger improvement here will come from eliminating many of the calls to this function. This was reported by [~carlone]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MESOS-9887) Race condition between two terminal task status updates for Docker executor.
[
https://issues.apache.org/jira/browse/MESOS-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912400#comment-16912400
]
Andrei Budnik commented on MESOS-9887:
--
Discarding these patches ^^ since multiple consecutive requests to the
underlying containerizer might finish in a different order than they were sent.
Hence, the agent should not rely on the order of completion of requests sent to
the containerizer.
> Race condition between two terminal task status updates for Docker executor.
>
>
> Key: MESOS-9887
> URL: https://issues.apache.org/jira/browse/MESOS-9887
> Project: Mesos
> Issue Type: Bug
> Components: agent, containerization
>Reporter: Andrei Budnik
>Assignee: Andrei Budnik
>Priority: Blocker
> Labels: agent, containerization
> Attachments: race_example.txt
>
>
> h2. Overview
> Expected behavior:
> Task successfully finishes and sends TASK_FINISHED status update.
> Observed behavior:
> Task successfully finishes, but the agent sends TASK_FAILED with the reason
> "REASON_EXECUTOR_TERMINATED".
> In normal circumstances, Docker executor
> [sends|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/docker/executor.cpp#L758]
> final status update TASK_FINISHED to the agent, which then [gets
> processed|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L5543]
> by the agent before termination of the executor's process.
> However, if the processing of the initial TASK_FINISHED gets delayed, then
> there is a chance that Docker executor terminates and the agent
> [triggers|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L6662]
> TASK_FAILED which will [be
> handled|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L5816-L5826]
> prior to the TASK_FINISHED status update.
> See attached logs which contain an example of the race condition.
> h2. Reproducing bug
> 1. Add the following code:
> {code:java}
> static int c = 0;
> if (++c == 3) { // to skip TASK_STARTING and TASK_RUNNING status updates.
> ::sleep(2);
> }
> {code}
> to the
> [`ComposingContainerizerProcess::status`|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/containerizer/composing.cpp#L578]
> and to the
> [`DockerContainerizerProcess::status`|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/containerizer/docker.cpp#L2167].
> 2. Recompile mesos
> 3. Launch mesos master and agent locally
> 4. Launch a simple Docker task via `mesos-execute`:
> {code}
> # cd build
> ./src/mesos-execute --master="`hostname`:5050" --name="a"
> --containerizer=docker --docker_image=alpine --resources="cpus:1;mem:32"
> --command="ls"
> {code}
> h2. Race condition - description
> 1. Mesos agent receives TASK_FINISHED status update and then subscribes on
> [`containerizer->status()`|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L5754-L5761].
> 2. `containerizer->status()` operation for TASK_FINISHED status update gets
> delayed in the composing containerizer (e.g. due to switch of the worker
> thread that executes `status` method).
> 3. Docker executor terminates and the agent
> [triggers|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L6662]
> TASK_FAILED.
> 4. Docker containerizer destroys the container. A registered callback for the
> `containerizer->wait` call in the composing containerizer dispatches [lambda
> function|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/containerizer/composing.cpp#L368-L373]
> that will clean up `containers_` map.
> 5. Composing c'zer resumes and dispatches
> `[status()|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/containerizer/composing.cpp#L579]`
> method to the Docker containerizer for TASK_FINISHED, which in turn hangs
> for a few seconds.
> 6. Corresponding `containerId` gets removed from the `containers_` map of the
> composing c'zer.
> 7. Mesos agent subscribes on
> [`containerizer->status()`|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L5754-L5761]
> for the TASK_FAILED status update.
> 8. Composing c'zer returns ["Container not
> found"|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/containerizer/composing.cpp#L576]
> for TASK_FAILED.
> 9.
> `[Slave::_statusUpdate|https://github.com/apache/mesos/blob/0026ea46dc35cbba1f442b8e425c6cbaf81ee8f8/src/slave/slave.cpp#L5826]`
> stores
[jira] [Commented] (MESOS-9947) Java bindings sporadically fail to build
[
https://issues.apache.org/jira/browse/MESOS-9947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912277#comment-16912277
]
Benno Evers commented on MESOS-9947:
Diagnostics review: https://reviews.apache.org/r/71337/
> Java bindings sporadically fail to build
>
>
> Key: MESOS-9947
> URL: https://issues.apache.org/jira/browse/MESOS-9947
> Project: Mesos
> Issue Type: Bug
>Reporter: Benno Evers
>Priority: Major
> Labels: maven
>
> We sporadically (maybe once a month?) observe build failures in the java
> bindings in our internal CI. They look like this:
> {noformat}
> 14:32:18 [ERROR]
> /home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/java/generated/org/apache/mesos/Protos.java:[14594,45]
> error: cannot access StringBuilder
> 14:32:18 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile
> (default-compile) on project mesos: Compilation failure
> 14:32:18 [ERROR]
> /home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/java/generated/org/apache/mesos/Protos.java:[14594,45]
> error: cannot access StringBuilder
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (MESOS-9947) Java bindings sporadically fail to build
Benno Evers created MESOS-9947:
--
Summary: Java bindings sporadically fail to build
Key: MESOS-9947
URL: https://issues.apache.org/jira/browse/MESOS-9947
Project: Mesos
Issue Type: Bug
Reporter: Benno Evers
We sporadically (maybe once a month?) observe build failures in the java
bindings in our internal CI. They look like this:
{noformat}
14:32:18 [ERROR]
/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/java/generated/org/apache/mesos/Protos.java:[14594,45]
error: cannot access StringBuilder
14:32:18 [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile)
on project mesos: Compilation failure
14:32:18 [ERROR]
/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/java/generated/org/apache/mesos/Protos.java:[14594,45]
error: cannot access StringBuilder
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (MESOS-9836) Docker containerizer overwrites `/mesos/slave` cgroups.
[
https://issues.apache.org/jira/browse/MESOS-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qian Zhang reassigned MESOS-9836:
-
Shepherd: Gilbert Song
Assignee: Qian Zhang
RR: [https://reviews.apache.org/r/71335/]
> Docker containerizer overwrites `/mesos/slave` cgroups.
> ---
>
> Key: MESOS-9836
> URL: https://issues.apache.org/jira/browse/MESOS-9836
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Chun-Hung Hsiao
>Assignee: Qian Zhang
>Priority: Critical
> Labels: docker, mesosphere
>
> The following bug was observed on our internal testing cluster.
> The docker containerizer launched a container on an agent:
> {noformat}
> I0523 06:00:53.888579 21815 docker.cpp:1195] Starting container
> 'f69c8a8c-eba4-4494-a305-0956a44a6ad2' for task
> 'apps_docker-sleep-app.1fda5b8e-7d20-11e9-9717-7aa030269ee1' (and executor
> 'apps_docker-sleep-app.1fda5b8e-7d20-11e9-9717-7aa030269ee1') of framework
> 415284b7-2967-407d-b66f-f445e93f064e-0011
> I0523 06:00:54.524171 21815 docker.cpp:783] Checkpointing pid 13716 to
> '/var/lib/mesos/slave/meta/slaves/60c42ab7-eb1a-4cec-b03d-ea06bff00c3f-S2/frameworks/415284b7-2967-407d-b66f-f445e93f064e-0011/executors/apps_docker-sleep-app.1fda5b8e-7d20-11e9-9717-7aa030269ee1/runs/f69c8a8c-eba4-4494-a305-0956a44a6ad2/pids/forked.pid'
> {noformat}
> After the container was launched, the docker containerizer did a {{docker
> inspect}} on the container and cached the pid:
>
> [https://github.com/apache/mesos/blob/0c431dd60ae39138cc7e8b099d41ad794c02c9a9/src/slave/containerizer/docker.cpp#L1764]
> The pid should be slightly greater than 13716.
> The docker executor sent a {{TASK_FINISHED}} status update around 16 minutes
> later:
> {noformat}
> I0523 06:16:17.287595 21809 slave.cpp:5566] Handling status update
> TASK_FINISHED (Status UUID: 4e00b786-b773-46cd-8327-c7deb08f1de9) for task
> apps_docker-sleep-app.1fda5b8e-7d20-11e9-9717-7aa030269ee1 of framework
> 415284b7-2967-407d-b66f-f445e93f064e-0011 from executor(1)@172.31.1.7:36244
> {noformat}
> After receiving the terminal status update, the agent asked the docker
> containerizer to update {{cpu.cfs_period_us}}, {{cpu.cfs_quota_us}} and
> {{memory.soft_limit_in_bytes}} of the container through the cached pid:
>
> [https://github.com/apache/mesos/blob/0c431dd60ae39138cc7e8b099d41ad794c02c9a9/src/slave/containerizer/docker.cpp#L1696]
> {noformat}
> I0523 06:16:17.290447 21815 docker.cpp:1868] Updated 'cpu.shares' to 102 at
> /sys/fs/cgroup/cpu,cpuacct/mesos/slave for container
> f69c8a8c-eba4-4494-a305-0956a44a6ad2
> I0523 06:16:17.290660 21815 docker.cpp:1895] Updated 'cpu.cfs_period_us' to
> 100ms and 'cpu.cfs_quota_us' to 10ms (cpus 0.1) for container
> f69c8a8c-eba4-4494-a305-0956a44a6ad2
> I0523 06:16:17.889816 21815 docker.cpp:1937] Updated
> 'memory.soft_limit_in_bytes' to 32MB for container
> f69c8a8c-eba4-4494-a305-0956a44a6ad2
> {noformat}
> Note that the cgroup of {{cpu.shares}} was {{/mesos/slave}}. This was
> possibly because that over the 16 minutes the pid got reused:
> {noformat}
> # zgrep 'systemd.cpp:98\]' /var/log/mesos/archive/mesos-agent.log.12.gz
> ...
> I0523 06:00:54.525178 21815 systemd.cpp:98] Assigned child process '13716' to
> 'mesos_executors.slice'
> I0523 06:00:55.078546 21808 systemd.cpp:98] Assigned child process '13798' to
> 'mesos_executors.slice'
> I0523 06:00:55.134096 21808 systemd.cpp:98] Assigned child process '13799' to
> 'mesos_executors.slice'
> ...
> I0523 06:06:30.997439 21808 systemd.cpp:98] Assigned child process '32689' to
> 'mesos_executors.slice'
> I0523 06:06:31.050976 21808 systemd.cpp:98] Assigned child process '32690' to
> 'mesos_executors.slice'
> I0523 06:06:31.110514 21815 systemd.cpp:98] Assigned child process '32692' to
> 'mesos_executors.slice'
> I0523 06:06:33.143726 21818 systemd.cpp:98] Assigned child process '446' to
> 'mesos_executors.slice'
> I0523 06:06:33.196251 21818 systemd.cpp:98] Assigned child process '447' to
> 'mesos_executors.slice'
> I0523 06:06:33.266332 21816 systemd.cpp:98] Assigned child process '449' to
> 'mesos_executors.slice'
> ...
> I0523 06:09:34.870056 21808 systemd.cpp:98] Assigned child process '13717' to
> 'mesos_executors.slice'
> I0523 06:09:34.937762 21813 systemd.cpp:98] Assigned child process '13744' to
> 'mesos_executors.slice'
> I0523 06:09:35.073971 21817 systemd.cpp:98] Assigned child process '13754' to
> 'mesos_executors.slice'
> ...
> {noformat}
> It was highly likely that the container itself exited around 06:09:35, way
> before the docker executor detected and reported the terminal status update,
> and then its pid was reused by another forked child of the agent, and thus
> {{cpu.cfs_period_us}}, {{cpu.quota_us}} and
[jira] [Commented] (MESOS-9836) Docker containerizer overwrites `/mesos/slave` cgroups.
[
https://issues.apache.org/jira/browse/MESOS-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912257#comment-16912257
]
Qian Zhang commented on MESOS-9836:
---
On second thought, I would prefer Chun's solution: After obtaining the pid
through `docker inspect`, we look into the proc filesystem and get the cgroup
right away and cache it. And we do not need to checkpoint the cached cgroup
since after agent restarts we can still get it through `docker inspect` and
proc filesystem.
> Docker containerizer overwrites `/mesos/slave` cgroups.
> ---
>
> Key: MESOS-9836
> URL: https://issues.apache.org/jira/browse/MESOS-9836
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Chun-Hung Hsiao
>Priority: Critical
> Labels: docker, mesosphere
>
> The following bug was observed on our internal testing cluster.
> The docker containerizer launched a container on an agent:
> {noformat}
> I0523 06:00:53.888579 21815 docker.cpp:1195] Starting container
> 'f69c8a8c-eba4-4494-a305-0956a44a6ad2' for task
> 'apps_docker-sleep-app.1fda5b8e-7d20-11e9-9717-7aa030269ee1' (and executor
> 'apps_docker-sleep-app.1fda5b8e-7d20-11e9-9717-7aa030269ee1') of framework
> 415284b7-2967-407d-b66f-f445e93f064e-0011
> I0523 06:00:54.524171 21815 docker.cpp:783] Checkpointing pid 13716 to
> '/var/lib/mesos/slave/meta/slaves/60c42ab7-eb1a-4cec-b03d-ea06bff00c3f-S2/frameworks/415284b7-2967-407d-b66f-f445e93f064e-0011/executors/apps_docker-sleep-app.1fda5b8e-7d20-11e9-9717-7aa030269ee1/runs/f69c8a8c-eba4-4494-a305-0956a44a6ad2/pids/forked.pid'
> {noformat}
> After the container was launched, the docker containerizer did a {{docker
> inspect}} on the container and cached the pid:
>
> [https://github.com/apache/mesos/blob/0c431dd60ae39138cc7e8b099d41ad794c02c9a9/src/slave/containerizer/docker.cpp#L1764]
> The pid should be slightly greater than 13716.
> The docker executor sent a {{TASK_FINISHED}} status update around 16 minutes
> later:
> {noformat}
> I0523 06:16:17.287595 21809 slave.cpp:5566] Handling status update
> TASK_FINISHED (Status UUID: 4e00b786-b773-46cd-8327-c7deb08f1de9) for task
> apps_docker-sleep-app.1fda5b8e-7d20-11e9-9717-7aa030269ee1 of framework
> 415284b7-2967-407d-b66f-f445e93f064e-0011 from executor(1)@172.31.1.7:36244
> {noformat}
> After receiving the terminal status update, the agent asked the docker
> containerizer to update {{cpu.cfs_period_us}}, {{cpu.cfs_quota_us}} and
> {{memory.soft_limit_in_bytes}} of the container through the cached pid:
>
> [https://github.com/apache/mesos/blob/0c431dd60ae39138cc7e8b099d41ad794c02c9a9/src/slave/containerizer/docker.cpp#L1696]
> {noformat}
> I0523 06:16:17.290447 21815 docker.cpp:1868] Updated 'cpu.shares' to 102 at
> /sys/fs/cgroup/cpu,cpuacct/mesos/slave for container
> f69c8a8c-eba4-4494-a305-0956a44a6ad2
> I0523 06:16:17.290660 21815 docker.cpp:1895] Updated 'cpu.cfs_period_us' to
> 100ms and 'cpu.cfs_quota_us' to 10ms (cpus 0.1) for container
> f69c8a8c-eba4-4494-a305-0956a44a6ad2
> I0523 06:16:17.889816 21815 docker.cpp:1937] Updated
> 'memory.soft_limit_in_bytes' to 32MB for container
> f69c8a8c-eba4-4494-a305-0956a44a6ad2
> {noformat}
> Note that the cgroup of {{cpu.shares}} was {{/mesos/slave}}. This was
> possibly because that over the 16 minutes the pid got reused:
> {noformat}
> # zgrep 'systemd.cpp:98\]' /var/log/mesos/archive/mesos-agent.log.12.gz
> ...
> I0523 06:00:54.525178 21815 systemd.cpp:98] Assigned child process '13716' to
> 'mesos_executors.slice'
> I0523 06:00:55.078546 21808 systemd.cpp:98] Assigned child process '13798' to
> 'mesos_executors.slice'
> I0523 06:00:55.134096 21808 systemd.cpp:98] Assigned child process '13799' to
> 'mesos_executors.slice'
> ...
> I0523 06:06:30.997439 21808 systemd.cpp:98] Assigned child process '32689' to
> 'mesos_executors.slice'
> I0523 06:06:31.050976 21808 systemd.cpp:98] Assigned child process '32690' to
> 'mesos_executors.slice'
> I0523 06:06:31.110514 21815 systemd.cpp:98] Assigned child process '32692' to
> 'mesos_executors.slice'
> I0523 06:06:33.143726 21818 systemd.cpp:98] Assigned child process '446' to
> 'mesos_executors.slice'
> I0523 06:06:33.196251 21818 systemd.cpp:98] Assigned child process '447' to
> 'mesos_executors.slice'
> I0523 06:06:33.266332 21816 systemd.cpp:98] Assigned child process '449' to
> 'mesos_executors.slice'
> ...
> I0523 06:09:34.870056 21808 systemd.cpp:98] Assigned child process '13717' to
> 'mesos_executors.slice'
> I0523 06:09:34.937762 21813 systemd.cpp:98] Assigned child process '13744' to
> 'mesos_executors.slice'
> I0523 06:09:35.073971 21817 systemd.cpp:98] Assigned child process '13754' to
> 'mesos_executors.slice'
> ...
> {noformat}
> It was highly likely that the container itself exited around
[jira] [Assigned] (MESOS-9353) libprocess triggers deprecation warnings when built against openssl 1.1.
[
https://issues.apache.org/jira/browse/MESOS-9353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Toenshoff reassigned MESOS-9353:
-
Assignee: Till Toenshoff
> libprocess triggers deprecation warnings when built against openssl 1.1.
> -
>
> Key: MESOS-9353
> URL: https://issues.apache.org/jira/browse/MESOS-9353
> Project: Mesos
> Issue Type: Bug
> Components: build, libprocess
>Affects Versions: 1.8.0
>Reporter: Till Toenshoff
>Assignee: Till Toenshoff
>Priority: Minor
> Labels: OpenSSL, build, deprecation, libprocess, warnings
>
> When building libprocess against libevent and openssl 1.1, the results are
> deprecation warnings;
> {noformat}
> ../3rdparty/libprocess/src/openssl.cpp:764:39: warning: 'ASN1_STRING_data' is
> deprecated [-Wdeprecated-declarations]
> reinterpret_cast(ASN1_STRING_data(
> ^
> /usr/include/openssl/asn1.h:553:1: note: 'ASN1_STRING_data' has been
> explicitly marked deprecated here
> DEPRECATEDIN_1_1_0(unsigned char *ASN1_STRING_data(ASN1_STRING *x))
> ^
> /usr/include/openssl/opensslconf-x86_64.h:122:34: note: expanded from macro
> 'DEPRECATEDIN_1_1_0'
> # define DEPRECATEDIN_1_1_0(f) DECLARE_DEPRECATED(f)
> ^
> /usr/include/openssl/opensslconf-x86_64.h:96:56: note: expanded from macro
> 'DECLARE_DEPRECATED'
> #define DECLARE_DEPRECATED(f)f __attribute__ ((deprecated));
>^
> {noformat}
> These warnings are benign and can get ignored for now, as long as no warning
> to error escalation is active. For successful builds we need to activate
> {{disable-werror}} for autotools - cmake builds do not escalate warnings to
> errors by default.
> We should still fix those warnings, likely by adapting towards openssl1.1 via
> preprocessor/define gates like;
> {noformat}
> #if OPENSSL_VERSION_NUMBER < 0x1010L
> [leave code as is]
> #else
> [add variant that does not rely on deprecated ASN1_STRING_data]
> #endif
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
