[jira] [Commented] (MESOS-9848) Blkio cgroup statistics files missing in Linux 5.1
[
https://issues.apache.org/jira/browse/MESOS-9848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17001944#comment-17001944
]
MohammadAli Baghershemirani commented on MESOS-9848:
This problem is related to ["IO
Scheduler"|https://wiki.ubuntu.com/Kernel/Reference/IOSchedulers]. Because
*CFQ* scheduler is being deprecated in newer kernel versions, these files no
longer exist, But similar files exist when using *BFQ* scheduler as you can see
in [this
code|https://github.com/torvalds/linux/blob/7ada90eb9c7ae4a8eb066f8e9b4d80122f0363db/block/bfq-cgroup.c#L1298],
but they're being renamed. (from blkio.io_merged to blkio.bfq.io_merged)
You can see my answer [here|https://superuser.com/a/1511865/1123573] for more
info.
I'm not using MESOS but if you can't use older versions of kernel, until it's
being fixed in MESOS a workaround could be using *BFQ* scheduler. In addition
to enabling CONFIG_BFQ_CGROUP_DEBUG option you need to change
[block/bfq-cgroup.c|https://github.com/torvalds/linux/blob/7ada90eb9c7ae4a8eb066f8e9b4d80122f0363db/block/bfq-cgroup.c#L1298]
and remove "bfq." from metrics files' names ( to have same filenames as
expected by MESOS ) and rebuild kernel.
> Blkio cgroup statistics files missing in Linux 5.1
> --
>
> Key: MESOS-9848
> URL: https://issues.apache.org/jira/browse/MESOS-9848
> Project: Mesos
> Issue Type: Improvement
> Components: containerization
>Reporter: James Peach
>Priority: Major
>
> In recent Fedora release, the Linux blkio cgroup no longer publishes certain
> stats files that the Mesos isolator expects should exist.
> In {{BlkioSubsystemProcess::usage}}, the isolator looks for
> * {{blkio.time}}
> * {{blkio.sectors}}
> * {{blkio.io_merged}}
> * {{blkio.io_queued}}
> Here's the actual cgroup:
> {noformat}
> $ uname -r
> 5.1.8-300.fc30.x86_64
> ...
> [root@jpeach 184cf411-e73f-4c6e-bd54-8181222801af]# pwd
> /sys/fs/cgroup/blkio/mesos_test_c83596ce-76ff-47c8-b23d-1276c16e93ae/184cf411-e73f-4c6e-bd54-8181222801af
> [root@jpeach 184cf411-e73f-4c6e-bd54-8181222801af]# ls -l
> total 0
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_service_bytes
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_service_bytes_recursive
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_serviced
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_serviced_recursive
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.weight
> --w--- 1 root root 0 Jun 16 18:07 blkio.reset_stats
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_service_bytes
> -r--r--r-- 1 root root 0 Jun 16 18:07
> blkio.throttle.io_service_bytes_recursive
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_serviced
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_serviced_recursive
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.read_bps_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.read_iops_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.write_bps_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.write_iops_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 cgroup.clone_children
> -rw-r--r-- 1 root root 0 Jun 16 18:06 cgroup.procs
> -rw-r--r-- 1 root root 0 Jun 16 18:07 notify_on_release
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (MESOS-9848) Blkio cgroup statistics files missing in Linux 5.1
[
https://issues.apache.org/jira/browse/MESOS-9848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954310#comment-16954310
]
Stéphane Cottin commented on MESOS-9848:
Same with latest debian linux kernels.
This is not some specific kernel build config of some linux distributions,they
were removed from the mainline a year ago.
https://github.com/torvalds/linux/commit/f382fb0bcef4c37dc049e9f6963e3baf204d815c#diff-a2ece9e4900b64104c546cf00b162828
> Blkio cgroup statistics files missing in Linux 5.1
> --
>
> Key: MESOS-9848
> URL: https://issues.apache.org/jira/browse/MESOS-9848
> Project: Mesos
> Issue Type: Improvement
> Components: containerization
>Reporter: James Peach
>Priority: Major
>
> In recent Fedora release, the Linux blkio cgroup no longer publishes certain
> stats files that the Mesos isolator expects should exist.
> In {{BlkioSubsystemProcess::usage}}, the isolator looks for
> * {{blkio.time}}
> * {{blkio.sectors}}
> * {{blkio.io_merged}}
> * {{blkio.io_queued}}
> Here's the actual cgroup:
> {noformat}
> $ uname -r
> 5.1.8-300.fc30.x86_64
> ...
> [root@jpeach 184cf411-e73f-4c6e-bd54-8181222801af]# pwd
> /sys/fs/cgroup/blkio/mesos_test_c83596ce-76ff-47c8-b23d-1276c16e93ae/184cf411-e73f-4c6e-bd54-8181222801af
> [root@jpeach 184cf411-e73f-4c6e-bd54-8181222801af]# ls -l
> total 0
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_service_bytes
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_service_bytes_recursive
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_serviced
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_serviced_recursive
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.weight
> --w--- 1 root root 0 Jun 16 18:07 blkio.reset_stats
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_service_bytes
> -r--r--r-- 1 root root 0 Jun 16 18:07
> blkio.throttle.io_service_bytes_recursive
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_serviced
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_serviced_recursive
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.read_bps_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.read_iops_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.write_bps_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.write_iops_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 cgroup.clone_children
> -rw-r--r-- 1 root root 0 Jun 16 18:06 cgroup.procs
> -rw-r--r-- 1 root root 0 Jun 16 18:07 notify_on_release
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (MESOS-9848) Blkio cgroup statistics files missing in Linux 5.1
[
https://issues.apache.org/jira/browse/MESOS-9848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865359#comment-16865359
]
Qian Zhang commented on MESOS-9848:
---
I found the similar result in an AWS instance created from the image
`ubuntu-minimal/images-testing/hvm-ssd/ubuntu-disco-daily-amd64-minimal-20190613`:
{code:java}
ubuntu@ip-172-31-44-196:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=19.04
DISTRIB_CODENAME=disco
DISTRIB_DESCRIPTION="Ubuntu 19.04"
ubuntu@ip-172-31-44-196:~$ uname -an
Linux ip-172-31-44-196 5.0.0-1007-aws #7-Ubuntu SMP Thu May 16 15:18:21 UTC
2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ip-172-31-44-196:~$ ll /sys/fs/cgroup/blkio/
total 0
dr-xr-xr-x 2 root root 0 Jun 17 03:17 ./
drwxr-xr-x 15 root root 380 Jun 17 03:14 ../
--w--- 1 root root 0 Jun 17 03:17 blkio.reset_stats
-r--r--r-- 1 root root 0 Jun 17 03:17 blkio.throttle.io_service_bytes
-r--r--r-- 1 root root 0 Jun 17 03:17 blkio.throttle.io_service_bytes_recursive
-r--r--r-- 1 root root 0 Jun 17 03:17 blkio.throttle.io_serviced
-r--r--r-- 1 root root 0 Jun 17 03:17 blkio.throttle.io_serviced_recursive
-rw-r--r-- 1 root root 0 Jun 17 03:17 blkio.throttle.read_bps_device
-rw-r--r-- 1 root root 0 Jun 17 03:17 blkio.throttle.read_iops_device
-rw-r--r-- 1 root root 0 Jun 17 03:17 blkio.throttle.write_bps_device
-rw-r--r-- 1 root root 0 Jun 17 03:17 blkio.throttle.write_iops_device
-rw-r--r-- 1 root root 0 Jun 17 03:17 cgroup.clone_children
-rw-r--r-- 1 root root 0 Jun 17 03:17 cgroup.procs
-r--r--r-- 1 root root 0 Jun 17 03:17 cgroup.sane_behavior
-rw-r--r-- 1 root root 0 Jun 17 03:17 notify_on_release
-rw-r--r-- 1 root root 0 Jun 17 03:17 release_agent
-rw-r--r-- 1 root root 0 Jun 17 03:17 tasks
{code}
> Blkio cgroup statistics files missing in Linux 5.1
> --
>
> Key: MESOS-9848
> URL: https://issues.apache.org/jira/browse/MESOS-9848
> Project: Mesos
> Issue Type: Improvement
> Components: containerization
>Reporter: James Peach
>Priority: Major
>
> In recent Fedora release, the Linux blkio cgroup no longer publishes certain
> stats files that the Mesos isolator expects should exist.
> In {{BlkioSubsystemProcess::usage}}, the isolator looks for
> * {{blkio.time}}
> * {{blkio.sectors}}
> * {{blkio.io_merged}}
> * {{blkio.io_queued}}
> Here's the actual cgroup:
> {noformat}
> $ uname -r
> 5.1.8-300.fc30.x86_64
> ...
> [root@jpeach 184cf411-e73f-4c6e-bd54-8181222801af]# pwd
> /sys/fs/cgroup/blkio/mesos_test_c83596ce-76ff-47c8-b23d-1276c16e93ae/184cf411-e73f-4c6e-bd54-8181222801af
> [root@jpeach 184cf411-e73f-4c6e-bd54-8181222801af]# ls -l
> total 0
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_service_bytes
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_service_bytes_recursive
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_serviced
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_serviced_recursive
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.weight
> --w--- 1 root root 0 Jun 16 18:07 blkio.reset_stats
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_service_bytes
> -r--r--r-- 1 root root 0 Jun 16 18:07
> blkio.throttle.io_service_bytes_recursive
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_serviced
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_serviced_recursive
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.read_bps_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.read_iops_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.write_bps_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.write_iops_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 cgroup.clone_children
> -rw-r--r-- 1 root root 0 Jun 16 18:06 cgroup.procs
> -rw-r--r-- 1 root root 0 Jun 16 18:07 notify_on_release
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9848) Blkio cgroup statistics files missing in Linux 5.1
[
https://issues.apache.org/jira/browse/MESOS-9848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865230#comment-16865230
]
James Peach commented on MESOS-9848:
/cc [~jieyu] [~gilbert] [~qianzhang]
> Blkio cgroup statistics files missing in Linux 5.1
> --
>
> Key: MESOS-9848
> URL: https://issues.apache.org/jira/browse/MESOS-9848
> Project: Mesos
> Issue Type: Improvement
> Components: containerization
>Reporter: James Peach
>Priority: Major
>
> In recent Fedora release, the Linux blkio cgroup no longer publishes certain
> stats files that the Mesos isolator expects should exist.
> In {{BlkioSubsystemProcess::usage}}, the isolator looks for
> * {{blkio.time}}
> * {{blkio.sectors}}
> * {{blkio.io_merged}}
> * {{blkio.io_queued}}
> Here's the actual cgroup:
> {noformat}
> $ uname -r
> 5.1.8-300.fc30.x86_64
> ...
> [root@jpeach 184cf411-e73f-4c6e-bd54-8181222801af]# pwd
> /sys/fs/cgroup/blkio/mesos_test_c83596ce-76ff-47c8-b23d-1276c16e93ae/184cf411-e73f-4c6e-bd54-8181222801af
> [root@jpeach 184cf411-e73f-4c6e-bd54-8181222801af]# ls -l
> total 0
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_service_bytes
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_service_bytes_recursive
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_serviced
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.io_serviced_recursive
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.bfq.weight
> --w--- 1 root root 0 Jun 16 18:07 blkio.reset_stats
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_service_bytes
> -r--r--r-- 1 root root 0 Jun 16 18:07
> blkio.throttle.io_service_bytes_recursive
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_serviced
> -r--r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.io_serviced_recursive
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.read_bps_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.read_iops_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.write_bps_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 blkio.throttle.write_iops_device
> -rw-r--r-- 1 root root 0 Jun 16 18:07 cgroup.clone_children
> -rw-r--r-- 1 root root 0 Jun 16 18:06 cgroup.procs
> -rw-r--r-- 1 root root 0 Jun 16 18:07 notify_on_release
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
