[
https://issues.apache.org/jira/browse/MESOS-9868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892555#comment-16892555
]
Qian Zhang commented on MESOS-9868:
-----------------------------------
Root cause of this issue:
Every time when executor sends a status update for a running task to Mesos
agent (e.g., when the state of the task health check is changed from healthy to
unhealthy), agent will call `containerizer->status(containerId)` to get the
container status for the task (see [this
code|https://github.com/apache/mesos/blob/1.8.1/src/slave/slave.cpp#L5754] for
details), internally containerizer will call CNI isolator's `status` method to
get the network info of the container, if we cannot get the container's network
info from CNI isolator somehow, agent will just fill in the container IP with
the its own IP (see [this
code|https://github.com/apache/mesos/blob/1.8.1/src/slave/slave.cpp#L5783:L5803]
for details), and then it will use the status update to update the task's
state in its memory (see [this
code|https://github.com/apache/mesos/blob/1.8.1/src/slave/slave.cpp#L5826] for
details), that's why we see agent IP for the task in the output of the agent's
/state endpoint.
So the key is why CNI isolator cannot find the network info for the nested
container (i.e, the task in the task group). I think it is caused by this
patch: [https://reviews.apache.org/r/65987/]. Before that patch, the code of
CNI isolator's `status` method is like the following, i.e., for nested
container we will always get its status from its parent container since nested
container always shares its parent container's network.
{code:java}
if (containerId.has_parent()) {
return status(containerId.parent());
}
if (!infos.contains(containerId)) {
return ContainerStatus();
}
{code}
But after that patch, the code was changes to:
{code:java}
if (!infos.contains(containerId)) {
return ContainerStatus();
}
const bool isNestedContainer = containerId.has_parent();
if (isNestedContainer && infos[containerId]->joinsParentsNetwork) {
return status(containerId.parent());
}
{code}
So we will check if `infos` contains the nested container ID first, if not we
will just return an empty container status (i.e., no network info). For the
nested container which joins its parent's network, CNI isolator will NOT
persist anything about it under runtime directory, so when agent is restarted,
CNI isolator will no recover anything for the nested container, i.e., it will
not exist in the `info`, that's why CNI isolator's `status` method just return
an empty container status for it.
> NetworkInfo from the agent /state endpoint is not correct.
> ----------------------------------------------------------
>
> Key: MESOS-9868
> URL: https://issues.apache.org/jira/browse/MESOS-9868
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 1.8.0
> Reporter: Gilbert Song
> Assignee: Qian Zhang
> Priority: Blocker
> Labels: containerization
>
> NetworkInfo from the agent /state endpoint is not correct, which is also
> different from the networkInfo of /containers endpoint. Some frameworks rely
> on the state endpoint to get the ip address for other containers to run.
> agent's state endpoint
> {noformat}
> {
> "state": "TASK_RUNNING",
> "timestamp": 1561574343.1521769,
> "container_status": {
> "container_id": {
> "value": "9a2633be-d2e5-4636-9ad4-7b2fc669da99",
> "parent": {
> "value": "45ebab16-9b4b-416e-a7f2-4833fd4ed8ff"
> }
> },
> "network_infos": [
> {
> "ip_addresses": [
> {
> "protocol": "IPv4",
> "ip_address": "172.31.10.35"
> }
> ]
> }
> ]
> },
> "healthy": true
> }
> {noformat}
> agent's /containers endpoint
> {noformat}
> "status": {
> "container_id": {
> "value": "45ebab16-9b4b-416e-a7f2-4833fd4ed8ff"
> },
> "executor_pid": 1723,
> "network_infos": [
> {
> "ip_addresses": [
> {
> "ip_address": "9.0.73.65",
> "protocol": "IPv4"
> }
> ],
> "name": "dcos"
> }
> ]
> }
> {noformat}
> The ip addresses are different^^.
> The container is in RUNNING state and is running correctly. Just the state
> endpoint is not correct. One thing to notice is that the state endpoint used
> to show the correct IP. After there was an agent restart and master leader
> re-election, the IP address in the state endpoint was changed.
> Here is the checkpoint CNI network information
> {noformat}
> OK-23:37:48-root@int-mountvolumeagent2-soak113s:/var/lib/mesos/slave/meta/slaves/60c42ab7-eb1a-4cec-b03d-ea06bff00c3f-S4/frameworks/26ffb84c-81ba-4b3b-989b-9c6560e51fa1-0171/executors/k8s-clusters.kc02__etcd__b50dc403-30d1-4b54-a367-332fb3621030/runs/latest/tasks/k8s-clusters.kc02__etcd-2-peer__5b6aa5fc-e113-4021-9db8-b63e0c8d1f6c
> # cat
> /var/run/mesos/isolators/network/cni/45ebab16-9b4b-416e-a7f2-4833fd4ed8ff/dcos/network.conf
>
> {"args":{"org.apache.mesos":{"network_info":{"name":"dcos"}}},"chain":"M-DCOS","delegate":{"bridge":"m-dcos","hairpinMode":true,"ipMasq":false,"ipam":{"dataDir":"/var/run/dcos/cni/networks","routes":[{"dst":"0.0.0.0/0"}],"subnet":"9.0.73.0/25","type":"host-local"},"isGateway":true,"mtu":1420,"type":"bridge"},"excludeDevices":["m-dcos"],"name":"dcos","type":"mesos-cni-port-mapper"}
> {noformat}
> {noformat}
> OK-01:30:05-root@int-mountvolumeagent2-soak113s:/var/lib/mesos/slave/meta/slaves/60c42ab7-eb1a-4cec-b03d-ea06bff00c3f-S4/frameworks/26ffb84c-81ba-4b3b-989b-9c6560e51fa1-0171/executors/k8s-clusters.kc02__etcd__b50dc403-30d1-4b54-a367-332fb3621030/runs/latest/tasks/k8s-clusters.kc02__etcd-2-peer__5b6aa5fc-e113-4021-9db8-b63e0c8d1f6c
> # cat
> /var/run/mesos/isolators/network/cni/45ebab16-9b4b-416e-a7f2-4833fd4ed8ff/dcos/eth0/network.info
> {"dns":{},"ip4":{"gateway":"9.0.73.1","ip":"9.0.73.65/25","routes":[{"dst":"0.0.0.0/0","gw":"9.0.73.1"}]}}
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
