Re: Low IOPs for certain block sizes
On Sun, Apr 12, 2009 at 8:09 PM, Bart Van Assche wrote: > While running iSCSI performance tests I noticed that the performance > for certain block sizes deviated significantly (more than ten times) > from the performance for other block sizes, both larger and smaller. > This surprised me. > > The test I ran was as follows: > * A file of 1 GB residing on a tmpfs filesystem was exported via iSCSI > target software. The test has been repeated with both SCST and STGT. > * On the initiator system open-iscsi version 2.0.870 was used for > performing reads and writes with dd via direct I/O. Read-ahead was set > to zero. > * Both systems were running kernel 2.6.29.1 in run level 3 (no X > server) and the 1 GbE interfaces in the two systems were connected via > a crossed cable. The MTU has been left to its default value, 1500 > bytes. Netperf reported a throughput of 600 Mbit/s = 75 MB/s for the > TCP/IP stream test on this setup. > * 128 MB of data has been transferred during each test. > * Each measurement has been repeated three times. > * All caches were flushed before each test. > * The ratio of standard deviation to average was 2% or lower for all > measurements. > * The measurement result are as follows (transfer speeds in MB/s): > > Block SCST STGT SCST STGT > size writing writing reading reading > -- --- --- --- --- > 64 MB 71.7 63.3 62.1 58.4 > 32 MB 71.9 63.4 61.7 58.1 > 16 MB 72.4 63.0 61.7 57.1 > 8 MB 72.7 63.3 61.7 56.9 > 4 MB 72.9 63.5 61.3 57.0 > 2 MB 72.8 59.5 60.3 56.9 > 1 MB 72.1 38.7 59.4 56.0 > 512 KB 67.3 21.4 58.0 54.4 > 256 KB 67.4 22.8 55.5 53.4 > 128 KB 60.9 22.6 53.3 51.7 > 64 KB 53.2 22.2 53.0 45.7 > 32 KB 48.9 21.6 40.0 40.0 > 16 KB 40.0 20.8 0.6 1.3 > 8 KB 20.0 19.9 19.9 20.0 > 4 KB 0.6 1.6 18.9 10.3 > > All results look normal to me, except the write throughput for a block > size of 4 KB and the read throughput for a block size of 16 KB. > > Regarding CPU load: during the 4 KB write test, the CPU load was 0.9 > on the initiator system and 0.1 on the target. By this time I have repeated the above test on the same two servers but this time using a DDR IB network using IPoIB in connected mode instead of a GbE network (netperf reports 1375 MB/s). The results for the IB network look normal to me. But it's still not clear to me why for certain block sizes the number of IOPs was so low on the GbE network. Block SCST STGT SCST STGT size writing writing reading reading -- --- --- --- --- 64 MB 307 251 658 439 32 MB 460 258 630 431 16 MB 580 258 598 414 8 MB 585 262 566 413 4 MB 665 273 613 428 2 MB 737 240 645 476 1 MB 727 172 631 457 512 KB 623 115 583 326 256 KB 619 120 559 399 128 KB 467 118 499 306 64 KB 309 115 333 251 32 KB 233 107 248 198 16 KB 157 95 162 133 8 KB 97 79 95 78 4 KB 51 42 52 42 Bart. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Low IOPs for certain block sizes
On Tue, Apr 14, 2009 at 7:24 PM, Vladislav Bolkhovitin wrote: > I would suggest you to make sure you have any hardware coalescing disabled > on both hosts. You can do that by using "ethtool -c" The output of ethtool is as follows: (target) $ ethtool -i eth0 driver: sky2 version: 1.22 firmware-version: N/A bus-info: :02:00.0 $ ethtool -c eth0 Coalesce parameters for eth0: Adaptive RX: off TX: off stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0 rx-usecs: 100 rx-frames: 16 rx-usecs-irq: 20 rx-frames-irq: 16 tx-usecs: 1000 tx-frames: 10 tx-usecs-irq: 0 tx-frames-irq: 0 rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0 rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0 (initiator) $ ethtool -i eth0 driver: atl1 version: 2.1.3 firmware-version: N/A bus-info: :01:00.0 $ ethtool -c eth0 Coalesce parameters for eth0: Cannot get device coalesce settings: Operation not supported Bart. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Low IOPs for certain block sizes
Hello, Bart Van Assche, on 04/12/2009 10:09 PM wrote: > Hello, > > While running iSCSI performance tests I noticed that the performance > for certain block sizes deviated significantly (more than ten times) > from the performance for other block sizes, both larger and smaller. > This surprised me. > > The test I ran was as follows: > * A file of 1 GB residing on a tmpfs filesystem was exported via iSCSI > target software. The test has been repeated with both SCST and STGT. > * On the initiator system open-iscsi version 2.0.870 was used for > performing reads and writes with dd via direct I/O. Read-ahead was set > to zero. > * Both systems were running kernel 2.6.29.1 in run level 3 (no X > server) and the 1 GbE interfaces in the two systems were connected via > a crossed cable. The MTU has been left to its default value, 1500 > bytes. Netperf reported a throughput of 600 Mbit/s = 75 MB/s for the > TCP/IP stream test on this setup. > * 128 MB of data has been transferred during each test. > * Each measurement has been repeated three times. > * All caches were flushed before each test. > * The ratio of standard deviation to average was 2% or lower for all > measurements. > * The measurement result are as follows (transfer speeds in MB/s): > > Block SCSTSTGTSCSTSTGT > size writing writing reading reading > -- --- --- --- --- > 64 MB 71.763.362.158.4 > 32 MB 71.963.461.758.1 > 16 MB 72.463.061.757.1 > 8 MB 72.763.361.756.9 > 4 MB 72.963.561.357.0 > 2 MB 72.859.560.356.9 > 1 MB 72.138.759.456.0 > 512 KB 67.321.458.054.4 > 256 KB 67.422.855.553.4 > 128 KB 60.922.653.351.7 > 64 KB 53.222.253.045.7 > 32 KB 48.921.640.040.0 > 16 KB 40.020.8 0.6 1.3 > 8 KB 20.019.919.920.0 > 4 KB 0.6 1.618.910.3 > > All results look normal to me, except the write throughput for a block > size of 4 KB and the read throughput for a block size of 16 KB. > > Regarding CPU load: during the 4 KB write test, the CPU load was 0.9 > on the initiator system and 0.1 on the target. I would suggest you to make sure you have any hardware coalescing disabled on both hosts. You can do that by using "ethtool -c" > Has anyone observed similar behavior before ? > > Bart. > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
