Hi, we are noticing some strange block device IO performance and our investigations are leading us away from the hardware and towards the kernel. This could be a simple tuning problem or a known issue, so before taking a deep dive down kernel sources and debian kernel patches we would like to rule out the simple things first.
The context is a 16 GB 32-bit intel debian workstation, using an ext4 filesystem with journalling, on a lvm SATA3 SSD disk, with relatively recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual machines. The host system (so not the virual machines) shows sporadic extremely slow write performance (around 4 megabytes per second). However, if we use the debian 3.2.0 kernel this problem does not manifest itself. We've created a simple IO performance test script to investigate this. It is basically a smart wrapper around dd, copying data between the block device under test and a ramdisk filesystem, using flags to select the usage of cache, sync and direct io, clearing caches before the test and running the write and read tests three times to account for transient performance. The results of these tests are unexpected: we see the expected normal write performance when using uncached, unsynced, direct IO and very slow write performance using the regular cached IO. The difference is huge: it is sometimes two orders of magnitude! This seems to rule out the block device and IO controller being at fault. In fact, other tests showed the same performance discrepancy with an NFS mounted filesystem and a platter disk. We also noticed that when the performance is slow, if we shut down the KVM virtual machines the performance returns to normal. Maybe there is something going wrong with cache/buffer handling? Thanks for your insights. I've kept this mail intentionally free from too many technical details but I'll be happy to provide additional relevant info as required. Regards, Erik Cumps -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
