Jie Yu created MESOS-9142:
-----------------------------
Summary: CNI detach might fail due to missing network config file.
Key: MESOS-9142
URL: https://issues.apache.org/jira/browse/MESOS-9142
Project: Mesos
Issue Type: Bug
Affects Versions: 1.6.1, 1.5.1, 1.4.1, 1.7.0
Reporter: Jie Yu
Observed this in one of the scale testing:
If the container is destroyed while in PREPARING state, the network config file
won't be there. The CNI detach in that case should return success because
attach is not called yet.
{code}
Aug 08 23:52:03 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:52:03.762208 4045 slave.cpp:3562] Launching container
278d88a8-2bd1-4359-8c70-25dc90879cdc for executor
'testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a' of framework
4397555f-b8fd-478c-907a-7532e2f5e32e-0001
Aug 08 23:53:03 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:53:03.232110 4047 containerizer.cpp:1211] Starting container
278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:53:03 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:53:03.249105 4049 provisioner.cpp:546] Provisioning image rootfs
'/var/lib/mesos/slave/provisioner/containers/278d88a8-2bd1-4359-8c70-25dc90879cdc/backends/overlay/rootfses/25dc9ff3-a4ad-4b59-bdf7-02f7e741c729'
for container 278d88a8-2bd1-4359-8c70-25dc90879cdc using overlay backend
Aug 08 23:53:31 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:53:31.367316 4047 containerizer.cpp:2981] Transitioning the state of
container 278d88a8-2bd1-4359-8c70-25dc90879cdc from PROVISIONING to PREPARING
Aug 08 23:55:22 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:55:22.988091 4047 memory.cpp:479] Started listening for OOM events
for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:55:23 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:55:23.196844 4047 memory.cpp:591] Started listening on 'low' memory
pressure events for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:55:23 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:55:23.277029 4047 memory.cpp:591] Started listening on 'medium'
memory pressure events for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:55:23 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:55:23.407688 4047 memory.cpp:591] Started listening on 'critical'
memory pressure events for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:57:31 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:57:31.760859 4043 memory.cpp:199] Updated
'memory.soft_limit_in_bytes' to 64MB for container
278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:57:31 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:57:31.878435 4043 memory.cpp:228] Updated 'memory.limit_in_bytes' to
64MB for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:57:35 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:57:35.892071 4044 cpu.cpp:101] Updated 'cpu.shares' to 112 (cpus
0.11) for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:57:36 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0808 23:57:36.132098 4044 cpu.cpp:121] Updated 'cpu.cfs_period_us' to 100ms
and 'cpu.cfs_quota_us' to 11ms (cpus 0.11) for container
278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 09 00:02:01 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:02:01.791821 4047 isolator_module.cpp:68] Container prepare:
container_id[value: "278d88a8-2bd1-4359-8c70-25dc90879cdc"]
container_config[directory:
"/var/lib/mesos/slave/slaves/4397555f-b8fd-478c-907a-7532e2f5e32e-S0/frameworks/4397555f-b8fd-478c-907a-7532e2f5e32e-0001/executors/testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a/runs/278d88a8-2bd1-4359-8c70-25dc90879cdc"
user: "root" rootfs:
"/var/lib/mesos/slave/provisioner/containers/278d88a8-2bd1-4359-8c70-25dc90879cdc/backends/overlay/rootfses/25dc9ff3-a4ad-4b59-bdf7-02f7e741c729"
docker { manifest { id:
"d7066268059eb2499922833bd058e9e0b9da8365bcae4b85ee22296b02fa0489" parent:
"f18ee96f0b1656cab52554b270f19e8df5046d307296d2146539c04565d67747" created:
"2018-07-06T14:14:06.393355914Z" container:
"b449414436d7bc63cc85dcd24abde14bcec1d92f54f2316212059ee2ed7f3a65"
container_config { Hostname: "b449414436d7" Env:
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" User: ""
Cmd: "/bin/sh" Cmd: "-c" Cmd: "#(nop) " Cmd: "CMD [\"/bin/sh\"]" WorkingDir: ""
Image:
"sha256:9025f4d6338c3425933e4734e8a27fb615d56ca8862e78bb8c4e2426f2db78bd" }
docker_version: "17.06.2-ce" config { Hostname: "" Env:
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" User: ""
Cmd: "/bin/sh" WorkingDir: "" Image:
"sha256:9025f4d6338c3425933e4734e8a27fb615d56ca8862e78bb8c4e2426f2db78bd" }
architecture: "amd64" os: "linux" } } executor_info { executor_id { value:
"testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" } resources { name: "cpus"
type: SCALAR scalar { value: 0.11 } allocation_info { role: "slave_public" } }
resources { name: "mem" type: SCALAR scalar { value: 64 } allocation_info {
role: "slave_public" } } resources { name: "ports" type: RANGES ranges { range
{ begin: 19471 end: 19471 } } allocation_info { role: "slave_public" } }
command { environment { variables { name: "MARATHON_APP_VERSION" value:
"2018-08-08T23:45:27.052Z" } variables { name: "HOST" value: "10.0.3.181" }
variables { name: "MARATHON_APP_RESOURCE_CPUS" value: "0.01" } variables {
name: "MARATHON_APP_RESOURCE_GPUS" value: "0" } variables { name:
"MESOS_TASK_ID" value: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" }
variables { name: "PORT" value: "19471" } variables { name:
"MARATHON_APP_RESOURCE_MEM" value: "32.0" } variables { name: "PORTS" value:
"19471" } variables { name: "MARATHON_APP_RESOURCE_DISK" value: "0.0" }
variables { name: "PORT_80" value: "19471" } variables { name:
"MARATHON_APP_LABELS" value: "" } variables { name: "MARATHON_APP_ID" value:
"/testcni2" } variables { name: "PORT0" value: "19471" } } value:
"/opt/mesosphere/packages/mesos--39041edf914496bb882949ade1e78043d53dadd0/libexec/mesos/mesos-executor"
shell: false arguments: "mesos-executor" arguments:
"--launcher_dir=/opt/mesosphere/active/mesos/libexec/mesos" arguments:
"--sandbox_directory=/mnt/mesos/sandbox" arguments: "--user=root" }
framework_id { value: "4397555f-b8fd-478c-907a-7532e2f5e32e-0001" } name:
"Command Executor (Task: testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a)
(Command: sh -c \'sleep 10000\')" source:
"testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" container { type: MESOS mesos {
image { type: DOCKER docker { name: "alpine" } cached: true } } network_infos {
labels { labels { key: "DCOS_SPACE" value: "/testcni2" } } ip_addresses { }
name: "mesos-bridge" port_mappings { host_port: 19471 container_port: 80
protocol: "tcp" } } } discovery { visibility: FRAMEWORK name: "testcni2" ports
{ ports { number: 19471 protocol: "tcp" labels { labels { key: "network-scope"
value: "host" } } } } } labels { labels { key: "DCOS_SPACE" value: "/testcni2"
} } } task_info { name: "testcni2" task_id { value:
"testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" } slave_id { value:
"4397555f-b8fd-478c-907a-7532e2f5e32e-S0" } resources { name: "cpus" type:
SCALAR scalar { value: 0.01 } allocation_info { role: "slave_public" } }
resources { name: "mem" type: SCALAR scalar { value: 32 } allocation_info {
role: "slave_public" } } resources { name: "ports" type: RANGES ranges { range
{ begin: 19471 end: 19471 } } allocation_info { role: "slave_public" } }
command { environment { variables { name: "MARATHON_APP_VERSION" value:
"2018-08-08T23:45:27.052Z" } variables { name: "HOST" value: "10.0.3.181" }
variables { name: "MARATHON_APP_RESOURCE_CPUS" value: "0.01" } variables {
name: "MARATHON_APP_RESOURCE_GPUS" value: "0" } variables { name:
"MESOS_TASK_ID" value: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" }
variables { name: "PORT" value: "19471" } variables { name:
"MARATHON_APP_RESOURCE_MEM" value: "32.0" } variables { name: "PORTS" value:
"19471" } variables { name: "MARATHON_APP_RESOURCE_DISK" value: "0.0" }
variables { name: "PORT_80" value: "19471" } variables { name:
"MARATHON_APP_LABELS" value: "" } variables { name: "MARATHON_APP_ID" value:
"/testcni2" } variables { name: "PORT0" value: "19471" } } value: "sleep 10000"
} container { type: MESOS mesos { image { type: DOCKER docker { name: "alpine"
} cached: true } } network_infos { labels { labels { key: "DCOS_SPACE" value:
"/testcni2" } } ip_addresses { } name: "mesos-bridge" port_mappings {
host_port: 19471 container_port: 80 protocol: "tcp" } } } labels { labels {
key: "DCOS_SPACE" value: "/testcni2" } } discovery { visibility: FRAMEWORK
name: "testcni2" ports { ports { number: 19471 protocol: "tcp" labels { labels
{ key: "network-scope" value: "host" } } } } } } command_info { environment {
variables { name: "MARATHON_APP_VERSION" value: "2018-08-08T23:45:27.052Z" }
variables { name: "HOST" value: "10.0.3.181" } variables { name:
"MARATHON_APP_RESOURCE_CPUS" value: "0.01" } variables { name:
"MARATHON_APP_RESOURCE_GPUS" value: "0" } variables { name: "MESOS_TASK_ID"
value: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" } variables { name:
"PORT" value: "19471" } variables { name: "MARATHON_APP_RESOURCE_MEM" value:
"32.0" } variables { name: "PORTS" value: "19471" } variables { name:
"MARATHON_APP_RESOURCE_DISK" value: "0.0" } variables { name: "PORT_80" value:
"19471" } variables { name: "MARATHON_APP_LABELS" value: "" } variables { name:
"MARATHON_APP_ID" value: "/testcni2" } variables { name: "PORT0" value: "19471"
} } value:
"/opt/mesosphere/packages/mesos--39041edf914496bb882949ade1e78043d53dadd0/libexec/mesos/mesos-executor"
shell: false arguments: "mesos-executor" arguments:
"--launcher_dir=/opt/mesosphere/active/mesos/libexec/mesos" arguments:
"--sandbox_directory=/mnt/mesos/sandbox" arguments: "--user=root" }
container_info { type: MESOS mesos { image { type: DOCKER docker { name:
"alpine" } cached: true } } network_infos { labels { labels { key: "DCOS_SPACE"
value: "/testcni2" } } ip_addresses { } name: "mesos-bridge" port_mappings {
host_port: 19471 container_port: 80 protocol: "tcp" } } } resources { name:
"cpus" type: SCALAR scalar { value: 0.11 } allocation_info { role:
"slave_public" } } resources { name: "mem" type: SCALAR scalar { value: 64 }
allocation_info { role: "slave_public" } } resources { name: "ports" type:
RANGES ranges { range { begin: 19471 end: 19471 } } allocation_info { role:
"slave_public" } }]
Aug 09 00:02:01 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:02:01.792201 4047 container_assigner.cpp:64] Registering and
retrieving endpoint for container_id[value:
"278d88a8-2bd1-4359-8c70-25dc90879cdc"] executor_info[executor_id { value:
"testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" } resources { name: "cpus"
type: SCALAR scalar { value: 0.11 } allocation_info { role: "slave_public" } }
resources { name: "mem" type: SCALAR scalar { value: 64 } allocation_info {
role: "slave_public" } } resources { name: "ports" type: RANGES ranges { range
{ begin: 19471 end: 19471 } } allocation_info { role: "slave_public" } }
command { environment { variables { name: "MARATHON_APP_VERSION" value:
"2018-08-08T23:45:27.052Z" } variables { name: "HOST" value: "10.0.3.181" }
variables { name: "MARATHON_APP_RESOURCE_CPUS" value: "0.01" } variables {
name: "MARATHON_APP_RESOURCE_GPUS" value: "0" } variables { name:
"MESOS_TASK_ID" value: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" }
variables { name: "PORT" value: "19471" } variables { name:
"MARATHON_APP_RESOURCE_MEM" value: "32.0" } variables { name: "PORTS" value:
"19471" } variables { name: "MARATHON_APP_RESOURCE_DISK" value: "0.0" }
variables { name: "PORT_80" value: "19471" } variables { name:
"MARATHON_APP_LABELS" value: "" } variables { name: "MARATHON_APP_ID" value:
"/testcni2" } variables { name: "PORT0" value: "19471" } } value:
"/opt/mesosphere/packages/mesos--39041edf914496bb882949ade1e78043d53dadd0/libexec/mesos/mesos-executor"
shell: false arguments: "mesos-executor" arguments:
"--launcher_dir=/opt/mesosphere/active/mesos/libexec/mesos" arguments:
"--sandbox_directory=/mnt/mesos/sandbox" arguments: "--user=root" }
framework_id { value: "4397555f-b8fd-478c-907a-7532e2f5e32e-0001" } name:
"Command Executor (Task: testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a)
(Command: sh -c \'sleep 10000\')" source:
"testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" container { type: MESOS mesos {
image { type: DOCKER docker { name: "alpine" } cached: true } } network_infos {
labels { labels { key: "DCOS_SPACE" value: "/testcni2" } } ip_addresses { }
name: "mesos-bridge" port_mappings { host_port: 19471 container_port: 80
protocol: "tcp" } } } discovery { visibility: FRAMEWORK name: "testcni2" ports
{ ports { number: 19471 protocol: "tcp" labels { labels { key: "network-scope"
value: "host" } } } } } labels { labels { key: "DCOS_SPACE" value: "/testcni2"
} }].
Aug 09 00:02:01 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:02:01.822054 4195 container_assigner_strategy.cpp:156] New
ephemeral-port reader for container[value:
"278d88a8-2bd1-4359-8c70-25dc90879cdc"] created at endpoint[198.51.100.1:47454].
Aug 09 00:02:01 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:02:01.822118 4195 container_state_cache_impl.cpp:134] Writing
container
file[/run/dcos/mesos/isolators/com_mesosphere_MetricsIsolatorModule/containers/278d88a8-2bd1-4359-8c70-25dc90879cdc]
with endpoint[198.51.100.1:47454]
Aug 09 00:02:01 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:02:01.828858 4045 secret.cpp:296] 0 secrets have been resolved for
container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 09 00:02:03 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:02:03.230397 4047 switchboard.cpp:317] Container logger module
finished preparing container 278d88a8-2bd1-4359-8c70-25dc90879cdc;
IOSwitchboard server is not required
Aug 09 00:02:04 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:02:04.510732 4045 containerizer.cpp:2367] Destroying container
278d88a8-2bd1-4359-8c70-25dc90879cdc in PREPARING state
Aug 09 00:02:04 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:02:04.511010 4045 containerizer.cpp:2981] Transitioning the state of
container 278d88a8-2bd1-4359-8c70-25dc90879cdc from PREPARING to DESTROYING
Aug 09 00:02:04 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:02:04.745018 4042 container_assigner.cpp:101] Unregistering
container_id[value: "278d88a8-2bd1-4359-8c70-25dc90879cdc"].
Aug 09 00:02:04 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:02:04.746515 4195 container_assigner_strategy.cpp:202] Closing
ephemeral-port reader for container[value:
"278d88a8-2bd1-4359-8c70-25dc90879cdc"] at endpoint[198.51.100.1:47454].
Aug 09 00:02:04 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:02:04.749131 4195 container_state_cache_impl.cpp:145] Removing
container
file[/run/dcos/mesos/isolators/com_mesosphere_MetricsIsolatorModule/containers/278d88a8-2bd1-4359-8c70-25dc90879cdc]
Aug 09 00:02:05 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
E0809 00:02:05.153556 4048 slave.cpp:6201] Container
'278d88a8-2bd1-4359-8c70-25dc90879cdc' for executor
'testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a' of framework
4397555f-b8fd-478c-907a-7532e2f5e32e-0001 failed to start: Container is being
destroyed during preparing
Aug 09 00:07:08 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
E0809 00:07:08.151012 4045 slave.cpp:6308] Termination of executor
'testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a' of framework
4397555f-b8fd-478c-907a-7532e2f5e32e-0001 failed: Failed to clean up an
isolator when destroying container: Failed to parse CNI network configuration
file:
'/run/mesos/isolators/network/cni/278d88a8-2bd1-4359-8c70-25dc90879cdc/mesos-bridge/network.conf':
Failed to read CNI network configuration file:
'/run/mesos/isolators/network/cni/278d88a8-2bd1-4359-8c70-25dc90879cdc/mesos-bridge/network.conf':
Failed to open file: No such file or directory
Aug 09 00:07:08 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
E0809 00:07:08.151798 4046 slave.cpp:5639] Failed to update resources for
container 278d88a8-2bd1-4359-8c70-25dc90879cdc of executor
'testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a' running task
testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a on status update for terminal
task, destroying container: Container not found
Aug 09 00:07:08 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:08.152240 4043 composing.cpp:600] Attempted to destroy unknown
container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 09 00:07:08 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:07:08.164916 4047 gc.cpp:95] Scheduling
'/var/lib/mesos/slave/slaves/4397555f-b8fd-478c-907a-7532e2f5e32e-S0/frameworks/4397555f-b8fd-478c-907a-7532e2f5e32e-0001/executors/testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a/runs/278d88a8-2bd1-4359-8c70-25dc90879cdc'
for gc 1.99999809405037days in the future
Aug 09 00:07:08 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
I0809 00:07:08.165079 4047 gc.cpp:95] Scheduling
'/var/lib/mesos/slave/meta/slaves/4397555f-b8fd-478c-907a-7532e2f5e32e-S0/frameworks/4397555f-b8fd-478c-907a-7532e2f5e32e-0001/executors/testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a/runs/278d88a8-2bd1-4359-8c70-25dc90879cdc'
for gc 1.99999809211852days in the future
Aug 09 00:07:26 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:26.571424 4047 cgroups.cpp:737] Skipping resource statistic for
container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to get usage for
subsystem 'memory': Unknown container
Aug 09 00:07:26 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:26.571480 4047 cgroups.cpp:737] Skipping resource statistic for
container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to read
'cpu.stat': 'mesos/278d88a8-2bd1-4359-8c70-25dc90879cdc' is not a valid cgroup
Aug 09 00:07:26 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:26.571648 4047 containerizer.cpp:2221] Skipping resource statistic
for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Unknown container
Aug 09 00:07:27 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:27.801781 4043 cgroups.cpp:737] Skipping resource statistic for
container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to read from
'blkio.time': 'mesos/278d88a8-2bd1-4359-8c70-25dc90879cdc' is not a valid cgroup
Aug 09 00:07:27 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:27.801839 4043 cgroups.cpp:737] Skipping resource statistic for
container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to get usage for
subsystem 'memory': Unknown container
Aug 09 00:07:27 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:27.801862 4043 cgroups.cpp:737] Skipping resource statistic for
container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to read
'cpu.stat': 'mesos/278d88a8-2bd1-4359-8c70-25dc90879cdc' is not a valid cgroup
Aug 09 00:07:27 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:27.801961 4043 containerizer.cpp:2221] Skipping resource statistic
for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Unknown container
Aug 09 00:07:29 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:29.182669 4048 cgroups.cpp:737] Skipping resource statistic for
container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to read from
'blkio.time': 'mesos/278d88a8-2bd1-4359-8c70-25dc90879cdc' is not a valid cgroup
Aug 09 00:07:29 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:29.182729 4048 cgroups.cpp:737] Skipping resource statistic for
container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to get usage for
subsystem 'memory': Unknown container
Aug 09 00:07:29 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:29.182752 4048 cgroups.cpp:737] Skipping resource statistic for
container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to read
'cpu.stat': 'mesos/278d88a8-2bd1-4359-8c70-25dc90879cdc' is not a valid cgroup
Aug 09 00:07:29 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]:
W0809 00:07:29.182870 4046 containerizer.cpp:2221] Skipping resource statistic
for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Unknown container
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
