Re: [storage-discuss] Seriously degraded SAS multipathing performance.
[email protected] said: > Very glad to hear you've made progress. I'm going to try the logical-block > load balance setting on my workstation at home. It'll be an interesting > experiment I'm sure! Multipath on your home workstation? My teenager might say, "Dude!". Well, if he was the least bit interested anyway (:-). I've managed to throw together a little summary of my test sessions from yesterday. I found it was quite worthwhile to make a tiny little filebench profile instead of messing around with "dd" and "time" on the command-line, and trying to keep track of all the results. http://acc.ohsu.edu/~hakansom/md1200_loadbal_bench.html Regards, Marion ___ storage-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/storage-discuss
Re: [storage-discuss] Seriously degraded SAS multipathing performance.
On 3/07/10 02:48 AM, Marion Hakanson wrote: [email protected] said: different driver --> different bug. Josh's bug is 6963122. You should file a new one - bugs.opensolaris.org, Product Solaris, Category kernel, Subcategory driver-mpt-x86. I suggest looking at the public comments field for Josh's bug first, though - the RE for that bug has some D you can run to help narrow down the performance problem point. If I interpret the bug report correctly, it's been closed, having been diagnosed as a function of HBA or drive firmware, not a driver issue. Yes, that's correct. I went looking for an easier workaround than manually disabling every drive's second path using "mpathadm disable path ...", and stumbled across "load-balance=logical-block" as a possibility. http://sunsolve.sun.com/search/document.do?assetkey=1-61-206894-1 http://sunsolve.sun.com/search/document.do?assetkey=1-61-202693-1 I gather that mpxio with "load-balance=round-robin" makes these drives go slow because the drive waits for one revolution before it can switch ports/paths to handle next command from the host. That's correct. Probably does away with the drives command-queue, too. Pretty sure that's not the case. Spent a good chunk of yesterday testing a number of different region-size settings with "load-balance=logical-block", finding that basically anything is much better than the default "load-balance=round-robin", but nothing is as good as "load-balance=none". That's really good to know, thankyou. Best performance of all was after manually assigning half the drives to use one path, and half to use the other path. I now find myself wishing there was a "load-balance=" setting which would do this automatically. If you log an RFE for it (solaris/kernel/io-multipath) then there's a decent chance it could get implemented :-) I speculate that this is what the LSI RAID-controllers must do to provide their redundant-path load-balance functionality. No idea on this, sorry. Very glad to hear you've made progress. I'm going to try the logical-block load balance setting on my workstation at home. It'll be an interesting experiment I'm sure! best regards, James -- Senior Software Engineer, Solaris Oracle http://www.jmcp.homeunix.com/blog ___ storage-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/storage-discuss
Re: [storage-discuss] Seriously degraded SAS multipathing performance.
[email protected] said: > different driver --> different bug. > > Josh's bug is 6963122. > > You should file a new one - bugs.opensolaris.org, Product Solaris, Category > kernel, Subcategory driver-mpt-x86. > > I suggest looking at the public comments field for Josh's bug first, though - > the RE for that bug has some D you can run to help narrow down the > performance problem point. If I interpret the bug report correctly, it's been closed, having been diagnosed as a function of HBA or drive firmware, not a driver issue. I went looking for an easier workaround than manually disabling every drive's second path using "mpathadm disable path ...", and stumbled across "load-balance=logical-block" as a possibility. http://sunsolve.sun.com/search/document.do?assetkey=1-61-206894-1 http://sunsolve.sun.com/search/document.do?assetkey=1-61-202693-1 I gather that mpxio with "load-balance=round-robin" makes these drives go slow because the drive waits for one revolution before it can switch ports/paths to handle next command from the host. Probably does away with the drives command-queue, too. Spent a good chunk of yesterday testing a number of different region-size settings with "load-balance=logical-block", finding that basically anything is much better than the default "load-balance=round-robin", but nothing is as good as "load-balance=none". Best performance of all was after manually assigning half the drives to use one path, and half to use the other path. I now find myself wishing there was a "load-balance=" setting which would do this automatically. I speculate that this is what the LSI RAID-controllers must do to provide their redundant-path load-balance functionality. Regards, Marion ___ storage-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/storage-discuss
Re: [storage-discuss] Seriously degraded SAS multipathing performance.
On 2/07/10 06:49 AM, Marion Hakanson wrote: On 06/21/2010 10:06 AM, Josh Simon wrote: I'm seeing seriously degraded performance with round-robin SAS multipathing. I'm hoping you guys can help me achieve full throughput across both paths... [email protected] said: Hi Josh, I suggest logging a bug on bugs.opensolaris.org, using solaris / driver / mptsas. Please include the info you sent in the earlier message too. Has this bug turned up in the database yet? I want to file a "me too". Except for us, this is Solaris-10U8 (not snv_134), and it's the mpt driver (not mpt_sas). Hi Marion, different driver --> different bug. Josh's bug is 6963122. You should file a new one - bugs.opensolaris.org, Product Solaris, Category kernel, Subcategory driver-mpt-x86. I suggest looking at the public comments field for Josh's bug first, though - the RE for that bug has some D you can run to help narrow down the performance problem point. cheers, James After seeing this problem report, I tried turning off the 2nd path on each of the 2TB "Near-Line SAS" drives in our Dell MD1200 array. Sequential write performance went up by 4.25 times. Making the same change to an identical server with a J4400 and 1TB SATA drives resulted in only an incremental improvement in bandwidth: filebench fileio, multistreamwrite-1M: MD1200, 2TB SAS, mpxio round-robin:131 MB/sec MD1200, 2TB SAS, mpxio static single: 557 MB/sec J4400, 1TB SATA, mpxio round-robin:427 MB/sec J4400, 1TB SATA, mpxio static single: 498 MB/sec Note that on the MD1200, with round-robin in effect, the drives were showing 100%-busy with service-times in the 300 msec range. With the static single-path in effect, service-times dropped to the 30 msec range. The above numbers were observed on this type of pool layout (both systems): NAME STATE READ WRITE CKSUM f1b_zp1ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c5t5000C50014D641DDd0 ONLINE 0 0 0 c5t5000C50014D55B13d0 ONLINE 0 0 0 c5t5000C50014D5B9B4d0 ONLINE 0 0 0 c5t5000C50014D6EF76d0 ONLINE 0 0 0 c5t5000C50014D07BC2d0 ONLINE 0 0 0 c5t5000C50014D6DBA5d0 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c5t5000C50014D6CCB2d0 ONLINE 0 0 0 c5t5000C50014D6A126d0 ONLINE 0 0 0 c5t5000C50014D641BBd0 ONLINE 0 0 0 c5t5000C50014D65228d0 ONLINE 0 0 0 c5t5000C50014D6C530d0 ONLINE 0 0 0 c5t5000C50014D6E35Ed0 ONLINE 0 0 0 Regards, Marion -- James C. McPherson -- Senior Software Engineer, Solaris Oracle http://www.jmcp.homeunix.com/blog ___ storage-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/storage-discuss
Re: [storage-discuss] Seriously degraded SAS multipathing performance.
> On 06/21/2010 10:06 AM, Josh Simon wrote: >> I'm seeing seriously degraded performance with round-robin SAS >> multipathing. I'm hoping you guys can help me achieve full throughput >> across both paths... [email protected] said: > Hi Josh, I suggest logging a bug on bugs.opensolaris.org, using solaris / > driver / mptsas. Please include the info you sent in the earlier message too. Has this bug turned up in the database yet? I want to file a "me too". Except for us, this is Solaris-10U8 (not snv_134), and it's the mpt driver (not mpt_sas). After seeing this problem report, I tried turning off the 2nd path on each of the 2TB "Near-Line SAS" drives in our Dell MD1200 array. Sequential write performance went up by 4.25 times. Making the same change to an identical server with a J4400 and 1TB SATA drives resulted in only an incremental improvement in bandwidth: filebench fileio, multistreamwrite-1M: MD1200, 2TB SAS, mpxio round-robin:131 MB/sec MD1200, 2TB SAS, mpxio static single: 557 MB/sec J4400, 1TB SATA, mpxio round-robin:427 MB/sec J4400, 1TB SATA, mpxio static single: 498 MB/sec Note that on the MD1200, with round-robin in effect, the drives were showing 100%-busy with service-times in the 300 msec range. With the static single-path in effect, service-times dropped to the 30 msec range. The above numbers were observed on this type of pool layout (both systems): NAME STATE READ WRITE CKSUM f1b_zp1ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c5t5000C50014D641DDd0 ONLINE 0 0 0 c5t5000C50014D55B13d0 ONLINE 0 0 0 c5t5000C50014D5B9B4d0 ONLINE 0 0 0 c5t5000C50014D6EF76d0 ONLINE 0 0 0 c5t5000C50014D07BC2d0 ONLINE 0 0 0 c5t5000C50014D6DBA5d0 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c5t5000C50014D6CCB2d0 ONLINE 0 0 0 c5t5000C50014D6A126d0 ONLINE 0 0 0 c5t5000C50014D641BBd0 ONLINE 0 0 0 c5t5000C50014D65228d0 ONLINE 0 0 0 c5t5000C50014D6C530d0 ONLINE 0 0 0 c5t5000C50014D6E35Ed0 ONLINE 0 0 0 Regards, Marion ___ storage-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/storage-discuss
Re: [storage-discuss] Seriously degraded SAS multipathing performance.
On 22/06/10 12:55 AM, Josh Simon wrote: I should also mention that I am running snv_134. Thanks! Josh Simon On 06/21/2010 10:06 AM, Josh Simon wrote: I'm seeing seriously degraded performance with round-robin SAS multipathing. I'm hoping you guys can help me achieve full throughput across both paths... Hi Josh, I suggest logging a bug on bugs.opensolaris.org, using solaris / driver / mptsas. Please include the info you sent in the earlier message too. thankyou, James C. McPherson -- Senior Software Engineer, Solaris Oracle http://www.jmcp.homeunix.com/blog ___ storage-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/storage-discuss
Re: [storage-discuss] Seriously degraded SAS multipathing performance.
I should also mention that I am running snv_134. Thanks! Josh Simon On 06/21/2010 10:06 AM, Josh Simon wrote: I'm seeing seriously degraded performance with round-robin SAS multipathing. I'm hoping you guys can help me achieve full throughput across both paths... ___ storage-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/storage-discuss
