4k randrw on 6.14: --- 0$ sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=$PWD/testfile --bs=4kB --iodepth=64 --size=1G --readwrite=randrw Run status group 0 (all jobs): READ: bw=381KiB/s (390kB/s), 381KiB/s-381KiB/s (390kB/s-390kB/s), io=241MiB (253MB), run=648627-648627msec WRITE: bw=379KiB/s (389kB/s), 379KiB/s-379KiB/s (389kB/s-389kB/s), io=240MiB (252MB), run=648627-648627msec
4k randrw on 6.8: Run status group 0 (all jobs): READ: bw=373MiB/s (391MB/s), 373MiB/s-373MiB/s (391MB/s-391MB/s), io=512MiB (537MB), run=1374-1374msec WRITE: bw=373MiB/s (391MB/s), 373MiB/s-373MiB/s (391MB/s-391MB/s), io=512MiB (537MB), run=1374-1374msec ** Description changed: After switching to the 6.14 kernel (as shipped with linux-generic- hwe-24.04 recently), ZFS read and write performance has seen a dramatic drop. Distributor ID: Ubuntu Description: Ubuntu 24.04.2 LTS Release: 24.04 Codename: noble I've observed this on two separate ZFS baremetal hosts hosting different disks and pool types (mirrored/striped and raidz2). FIO Tests ------- uname -r 6.14.0-27-generic sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=$PWD/testfile --bs=1m --iodepth=64 --size=1G --readwrite=read test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64 fio-3.36 Starting 1 process Jobs: 1 (f=1): [R(1)][93.3%][r=76.9MiB/s][r=76 IOPS][eta 00m:01s] test: (groupid=0, jobs=1): err= 0: pid=38786: Mon Aug 4 12:04:23 2025 read: IOPS=74, BW=74.7MiB/s (78.4MB/s)(1024MiB/13704msec) bw ( KiB/s): min=18432, max=96256, per=100.00%, avg=77250.56, stdev=17058.34, samples=25 iops : min= 18, max= 94, avg=75.44, stdev=16.66, samples=25 cpu : usr=0.18%, sys=7.70%, ctx=1028, majf=0, minf=16393 IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.1%, >=64=93.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=74.7MiB/s (78.4MB/s), 74.7MiB/s-74.7MiB/s (78.4MB/s-78.4MB/s), io=1024MiB (1074MB), run=13704-13704msec uname -r 6.8.0-71-generic After switching back to 6.8 kernel sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=$PWD/testfile --bs=1m --iodepth=64 --size=1G --readwrite=read test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64 fio-3.36 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=5541: Mon Aug 4 12:14:13 2025 read: IOPS=3710, BW=3710MiB/s (3890MB/s)(1024MiB/276msec) cpu : usr=0.00%, sys=99.64%, ctx=5, majf=0, minf=16393 IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.1%, >=64=93.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=3710MiB/s (3890MB/s), 3710MiB/s-3710MiB/s (3890MB/s-3890MB/s), io=1024MiB (1074MB), run=276-276msec + + --------------------------------------- + + + 4k randrw on 6.14: + --- + 0$ sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=$PWD/testfile --bs=4kB --iodepth=64 --size=1G --readwrite=randrw + Run status group 0 (all jobs): + READ: bw=381KiB/s (390kB/s), 381KiB/s-381KiB/s (390kB/s-390kB/s), io=241MiB (253MB), run=648627-648627msec + WRITE: bw=379KiB/s (389kB/s), 379KiB/s-379KiB/s (389kB/s-389kB/s), io=240MiB (252MB), run=648627-648627msec + + 4k randrw on 6.8: + Run status group 0 (all jobs): + READ: bw=373MiB/s (391MB/s), 373MiB/s-373MiB/s (391MB/s-391MB/s), io=512MiB (537MB), run=1374-1374msec + WRITE: bw=373MiB/s (391MB/s), 373MiB/s-373MiB/s (391MB/s-391MB/s), io=512MiB (537MB), run=1374-1374msec + + + zpool status: https://paste.ubuntu.com/p/wmZPRcTmCY/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2119470 Title: ZFS I/O Performance degredation on 6.14 kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2119470/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
