Hi all,
I'm experiencing some unexpected disk I/O latency on a newly built server.
Here are some of the details:
- Supermicro motherboard with Intel PCH RAID [1]
- Debian 13.6 installed
- / is mounted on /dev/nvme0n1p2, which is an NVMe SSD, formatted with ext4
- /home is mounted on /dev/md125, which is a RAID 5 consisting of three disks,
formatted with ext4
$ sudo mdadm -D /dev/md125
/dev/md125:
Container : /dev/md/imsm1, member 0
Raid Level : raid5
Array Size : 9279152128 (8.64 TiB 9.50 TB)
Used Dev Size : 4639576064 (4.32 TiB 4.75 TB)
Raid Devices : 3
Total Devices : 3
State : active
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-asymmetric
Chunk Size : 64K
Consistency Policy : resync
UUID : 2fbd736b:9b750e2a:e723b80c:df2f2c33
Number Major Minor RaidDevice State
2 8 0 0 active sync /dev/sda
1 8 16 1 active sync /dev/sdb
0 8 32 2 active sync /dev/sdc
I'm getting some weird latency issues with fsync on the RAID disk (/home on
/dev/md125):
(Caveat: I'm doing a file transfer that is writing to /home, so I might expect
some latency, but nothing like this.)
$ for i in {1..5}; do time sync; done 2>&1 | grep real
real 0m21.647s
real 0m11.770s
real 0m5.234s
real 0m3.023s
real 0m0.462s
Even when I turn off the file transfer, I still get something like this:
$ for i in {1..10}; do time sync; done 2>&1 | grep real
real 0m0.147s
real 0m0.077s
real 0m0.003s
real 0m0.004s
real 0m0.003s
I've tried turning on and off smart monitoring with smartctl to see if that
would help, and it didn't seem to help.
I'm not really sure what else I might try, and I'm reaching out to the
community to see if anyone has any suggestions for identifying and fixing the
problem.
Thanks,
Casey
[1] https://www.supermicro.com/manuals/other/Intel_PCH_RAID_Config.pdf
