[
https://issues.apache.org/jira/browse/MESOS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544916#comment-16544916
]
Stéphane Cottin edited comment on MESOS-8418 at 7/16/18 7:56 AM:
-----------------------------------------------------------------
after applying [https://reviews.apache.org/r/67923/] on our test node (1.6
branch), the agent average cpu usage dropped from ~230% to ~120%.
Well done, this is a great improvement, but the average cpu usage remains quite
high, a quick perf report shows the statfs usage is still significant.
@[~StephanErb] if possible, could you please update the profile data /
flamegraph with this patch applied ?
2.5.0.0
was (Author: kaalh):
after applying [https://reviews.apache.org/r/67923/] on our test node (1.6
branch), the agent average cpu usage dropped from ~230% to ~120%.
Well done, this is a great improvement, but the average cpu usage remains quite
high, a quick perf report shows the statfs usage is still significant.
@[~StephanErb] if possible, could you please update the profile data /
flamegraph with this patch applied ?
2.5.0.0
> mesos-agent high cpu usage because of numerous /proc/mounts reads
> -----------------------------------------------------------------
>
> Key: MESOS-8418
> URL: https://issues.apache.org/jira/browse/MESOS-8418
> Project: Mesos
> Issue Type: Improvement
> Components: agent, containerization
> Affects Versions: 1.4.1, 1.5.1, 1.6.1
> Reporter: Stéphane Cottin
> Priority: Critical
> Labels: containerizer, performance
> Attachments: mesos-agent-flamegraph.png, mesos-agent.stacks.gz
>
>
> /proc/mounts is read many, many times from
> src/(linux/fs|linux/cgroups|slave/slave).cpp.
> When using overlayfs, the /proc/mounts contents can become quite large.
> As an example, one of our Q/A single node running ~150 tasks, have a 361
> lines/ 201299 chars /proc/mounts file.
> This 200kB file is read on this node about 25 to 150 times per second. This
> is a (huge) waste of cpu and I/O time.
> Most of these calls are related to cgroups.
> Please consider these proposals :
> 1/ Is /proc/mounts mandatory for cgroups ?
> We already have cgroup subsystems list from /proc/cgroups.
> The only compelling information from /proc/mounts seems to be the root mount
> point,
> /sys/fs/cgroup/, which could be obtained by a unique read on agent start.
> 2/ use /proc/self/mountstats
> {noformat}
> wc /proc/self/mounts /proc/self/mountstats
> 361 2166 201299 /proc/self/mounts
> 361 2888 50200 /proc/self/mountstats
> {noformat}
> {noformat}
> grep cgroup /proc/self/mounts
> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
> cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
> cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
> cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
> {noformat}
> {noformat}
> grep cgroup /proc/self/mountstats
> device cgroup mounted on /sys/fs/cgroup with fstype tmpfs
> device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpu with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/cpuacct with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/devices with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_cls with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/net_prio with fstype cgroup
> device cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup
> {noformat}
> This file contains all the required information, and is 4x smaller
> 3/ microcaching
> Caching cgroups data for just 1 second would be a huge perfomance
> improvement, but i'm not aware of the possible side effects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
