[jira] [Commented] (MESOS-8418) mesos-agent high cpu usage because of numerous /proc/mounts reads
[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585412#comment-16585412
]
Jie Yu commented on MESOS-8418:
---
I posted another patch to eliminate the mount table read for cgroups creation
and write
https://reviews.apache.org/r/68426/
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
>Affects Versions: 1.4.1, 1.5.1, 1.6.1
>Reporter: Stéphane Cottin
>Assignee: Benjamin Mahler
>Priority: Critical
> Labels: containerizer, performance
> Fix For: 1.4.2, 1.5.2, 1.6.2, 1.7.0
>
> Attachments: image-2018-08-06-13-49-03-241.png,
> image-2018-08-06-13-49-03-317.png, mesos-agent-flamegraph.png,
> mesos-agent.stacks.gz
>
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8418) mesos-agent high cpu usage because of numerous /proc/mounts reads
[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570101#comment-16570101
]
Stephan Erb commented on MESOS-8418:
The first graph gives a rough impression of how many tasks per agents we have.
The second one shows the improvement of the end-to-end monitoring latency once
we have applied the cgroup performance patch.
We use the Prometheus Mesos exporter to scrape metrics (i.e.
[https://github.com/mesosphere/mesos_exporter]) and it scrapes multiple agent
endpoints for this.
!image-2018-08-06-13-49-03-317.png!!image-2018-08-06-13-49-03-241.png!
So in our usecase the improvement has been significant. Thanks again!
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
>Affects Versions: 1.4.1, 1.5.1, 1.6.1
>Reporter: Stéphane Cottin
>Assignee: Benjamin Mahler
>Priority: Critical
> Labels: containerizer, performance
> Fix For: 1.4.2, 1.5.2, 1.7.0, 1.6.2
>
> Attachments: image-2018-08-06-13-49-03-241.png,
> image-2018-08-06-13-49-03-317.png, mesos-agent-flamegraph.png,
> mesos-agent.stacks.gz
>
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8418) mesos-agent high cpu usage because of numerous /proc/mounts reads
[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546413#comment-16546413
]
Stephan Erb commented on MESOS-8418:
Thanks a lot for the super quick resolution [~bmahler]!
We are currently busy with prepping the 1.6.x deployment inhouse and will throw
this patch on top as well. I will report back with my findings. I am confident
that this will have solved our current performance issues.
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
>Affects Versions: 1.4.1, 1.5.1, 1.6.1
>Reporter: Stéphane Cottin
>Assignee: Benjamin Mahler
>Priority: Critical
> Labels: containerizer, performance
> Fix For: 1.4.2, 1.5.2, 1.7.0, 1.6.2
>
> Attachments: mesos-agent-flamegraph.png, mesos-agent.stacks.gz
>
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8418) mesos-agent high cpu usage because of numerous /proc/mounts reads
[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545686#comment-16545686
]
Benjamin Mahler commented on MESOS-8418:
[~kaalh] ok, that's a different issue that is not in the existing perf data on
this ticket so please do include perf data for what you're seeing, here's how
to do it: http://mesos.apache.org/documentation/latest/performance-profiling/
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
>Affects Versions: 1.4.1, 1.5.1, 1.6.1
>Reporter: Stéphane Cottin
>Priority: Critical
> Labels: containerizer, performance
> Attachments: mesos-agent-flamegraph.png, mesos-agent.stacks.gz
>
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8418) mesos-agent high cpu usage because of numerous /proc/mounts reads
[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544916#comment-16544916
]
Stéphane Cottin commented on MESOS-8418:
after applying [https://reviews.apache.org/r/67923/] on our test node (1.6
branch), the agent average cpu usage dropped from ~230% to ~120%.
Well done, this is a great improvement, but the average cpu usage remains quite
high, a quick perf report shows the statfs usage is still significant.
@[~StephanErb] if possible, could you please update the profile data /
flamegraph with this patch applied ?
2.5.0.0
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
>Affects Versions: 1.4.1, 1.5.1, 1.6.1
>Reporter: Stéphane Cottin
>Priority: Critical
> Labels: containerizer, performance
> Attachments: mesos-agent-flamegraph.png, mesos-agent.stacks.gz
>
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8418) mesos-agent high cpu usage because of numerous /proc/mounts reads
[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544737#comment-16544737
]
Benjamin Mahler commented on MESOS-8418:
Thanks [~kaalh] for filing this ticket and [~StephanErb] for providing the
profiling data. Sorry that this ticket has been open for so long. We'll make
sure to address this soon and backport the improvements (which version are you
running?). I'll make sure we scan for new tickets with the 'performance' label
in them before/during each performance working group meeting to see if any
severe issues (like this one) arise. But as always, please feel free to reach
out directly on the dev@ list or in the #performance slack channel if something
important isn't getting any attention; JIRA is not so flexible when it comes to
setting email alerts based on ticket labels, so sometimes we accidentally miss
important issues.
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
>Reporter: Stéphane Cottin
>Priority: Major
> Labels: containerizer, performance
> Attachments: mesos-agent-flamegraph.png, mesos-agent.stacks.gz
>
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8418) mesos-agent high cpu usage because of numerous /proc/mounts reads
[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544728#comment-16544728
]
Benjamin Mahler commented on MESOS-8418:
[~jieyu] [~gilbert] as a short term fix, I wrote a patch to read first (for
performance), then only if the read fails call verify (to provide a more
meaningful error message). This doesn't address verify performance problems in
the other helpers, which I suspect will be necessary if we do some
containerization benchmarking, but it at least addresses this steady state high
cpu consumption issue (assuming the reads succeed). Let me know what you think:
https://reviews.apache.org/r/67923/
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
>Reporter: Stéphane Cottin
>Priority: Major
> Labels: containerizer, performance
> Attachments: mesos-agent-flamegraph.png, mesos-agent.stacks.gz
>
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8418) mesos-agent high cpu usage because of numerous /proc/mounts reads
[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544399#comment-16544399
]
Jie Yu commented on MESOS-8418:
---
Yeah, I feel that we should let caller decide whether to call `verify` or not,
instead of doing that for all cgroup related operations (e.g.,
create/read/write/mount/unmount/etc.)
cc [~gilbert]
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
>Reporter: Stéphane Cottin
>Priority: Major
> Labels: containerizer, performance
> Attachments: mesos-agent-flamegraph.png, mesos-agent.stacks.gz
>
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8418) mesos-agent high cpu usage because of numerous /proc/mounts reads
[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544376#comment-16544376
]
Benjamin Mahler commented on MESOS-8418:
The source of this issue is that all of the cgroup reads go through a verify
function:
https://github.com/apache/mesos/blob/1.6.0/src/linux/cgroups.cpp#L922
It looks like the vast majority of the cpu time wasted here is inside this
function, when checking to see if the cgroup is mounted in the hierarchy:
https://github.com/apache/mesos/blob/1.6.0/src/linux/cgroups.cpp#L472
I'm not sure why the cgroup reads need to go through this. It provides a
clearer error message but at an extreme performance cost. The fix here seems to
be to let the reads go directly through to os::read without going through the
verify function, and letting it fail with a more generic read error message.
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
>Reporter: Stéphane Cottin
>Priority: Major
> Labels: containerizer, performance
> Attachments: mesos-agent-flamegraph.png, mesos-agent.stacks.gz
>
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8418) mesos-agent high cpu usage because of numerous /proc/mounts reads
[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543743#comment-16543743
]
Stephan Erb commented on MESOS-8418:
I have attached a profile [^mesos-agent.stacks.gz] gathered on a host with:
* ~140 tasks, all running docker images via the Mesos containerizer
* relevant isolators
{{cgroups/cpu,cgroups/mem,filesystem/linux,docker/runtime,...'}}
* agent monitoring endpoints {{/slave(1)/monitor/statistic}} and
{{/slave(1)/state}} are scraped every 15s. In total, a scrape takes between
5-13sec
* potentially related agent settings:
** --oversubscribed_resources_interval=45secs
** --qos_correction_interval_min=45secs
** --cgroups_cpu_enable_pids_and_tids_count
The profile confirms that mount table reads take a significant amount of time
!mesos-agent-flamegraph.png|width=800,height=450!
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
>Reporter: Stéphane Cottin
>Priority: Major
> Labels: containerizer, performance
> Attachments: mesos-agent-flamegraph.png, mesos-agent.stacks.gz
>
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8418) mesos-agent high cpu usage because of numerous /proc/mounts reads
[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16538309#comment-16538309
]
Stephan Erb commented on MESOS-8418:
As a workaround bumping the following options seems to help reduce the
{{mesos-agent}} load:
* container_disk_watch_interval
* disk_watch_interval
* oversubscribed_resources_interval
* qos_correction_interval_min
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
>Reporter: Stéphane Cottin
>Priority: Major
> Labels: containerizer, performance
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
