We believe we have identified the root cause of this regression. Setting
read_ahead_kb=128 before starting the workload fully resolves the issue
on 6.17.
The default read_ahead_kb changed between 6.14 and 6.17 in Azure:
Azure VM - Azure Kernel:
6.14.0-1017-azure: read_ahead_kb = 128
6.17.0-1008-azure: read_ahead_kb = 4096
Azure VM - Generic Kernel:
6.14.0-37-generic: read_ahead_kb = 128
6.17.0-14-generic: read_ahead_kb = 4096
AWS VM - AWS Kernel (NVMe devices):
6.14.0-1018-aws: read_ahead_kb = 128
6.17.0-1008-aws: read_ahead_kb = 128
Checked with `cat /sys/block/sd*/queue/read_ahead_kb` on Azure and `cat
/sys/block/nvme*/queue/read_ahead_kb` on AWS.
The Azure VMs use Premium SSD V1 managed disks.
The change affects all 6.17 kernels on Azure but not on AWS. This
explains why we could reproduce on Azure (also on generic kernel) but
not on AWS.
Our workload uses mmap for random-access reads across many files (we
have ~130GB of memory mapped files in total).
I can confirm that setting the below before starting RavenDB server
solved the issue on nodes running on 6.17.0-1008-azure and
6.19.0-1001-azure kernels:
sudo sh -c 'echo 128 > /sys/block/sda/queue/read_ahead_kb'
sudo sh -c 'echo 128 > /sys/block/sdb/queue/read_ahead_kb'
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2143713
Title:
Performance regression between 6.14.0-1017 and 6.17.0-1008.8
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/2143713/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
