Hi jacobmartin, Testing on: 7.1.2-070102-generic (installed via mainline cli)
After facing the same issue on 7.1.2 (using exactly same host where I tested 7.0 and also tried mainline 7.1.1) the same sequential read test with --rw=read --bs=1M --size=1G --numjobs=1 --iodepth=64 was getting stalled the same way, all evidence was showing the same symptoms and then I decided to test with a smaller block size and tried different smaller sizes until I saw up to 900KB blocks it would NOT get stalled... then I thought I could give the iscsi_tcp thread a bit more buffer to deal with slower processing, so decided to increase the tcp buffers like "net.ipv4.tcp_rmem=4194304 4194304 67108864" and with the increased buffer tests run smoothly and no longer get stall. As I dont want to increase the global tcp buffers and we already allow big enough net.core.rmem_max I moved the configuration to be iscsi specific by adding "node.conn[0].tcp.window_size = 4194304" to /etc/iscsi/iscsid.conf (and kept default node.session.cmds_max = 128 and node.session.queue_depth = 32) and moved back to "/proc/sys/net/ipv4/tcp_rmem=4096 87380 67108864" In that setup, only increasing the node.conn[0].tcp.window_size to 4MB, it works (1MB stalls and 2MB is not as performant) on kernel 7.0.0 and 7.1.2. I also tried the same in ubuntu 24 with kernel 6.8.0-88 but in that case is not helping. I believe the iscsi_tcp reading is really the bottleneck, seems like increasing the buffer from 512KB to 4MB brings the performance back... but I would argue it looks like there is room to improve the iscsi_tcp reading. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2157924 Title: iscsi_tcp 40-50% sequential read performance regression (5.15 vs 6.8/7.0) due to release_sock serialization bottleneck To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2157924/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
