> Greetings, > > We've attached a J4200 array via a Sun PCI-E SAS HBA > (SG-XPCIE8SAS-E-Z) > to a Sun V245. A small amount of write traffic to a > ZFS pool on the J4200 > is causing one of the V245's CPU's to be 100% busy > handling mpt interrupts > for the HBA, according to "intrstat". By "small > amount", I mean we're able > to get only as much as 2-4MB/second throughput from a > simple: > > dd if=/dev/zero of=/testpool/testfile bs=1024k > count=1000 & > device | cpu0 %tim cpu1 %tim > -------+------------------------------ > bge#0 | 5 0.0 0 0.0 > hci#0 | 0 0.0 1 0.0 > mpt#0 | 0 0.0 47 99.7 > hci#0 | 0 0.0 0 0.0 > ohci#1 | 0 0.0 0 0.0 > system is running Solaris-10U7, patches are current > as of a couple weeks > ago, and all firmware (V245, HBA, J4200, drives) is > up to date. > > This same server got pretty reasonable throughput > using a Sun/Qlogic QLE2462 > FC HBA connected to an old HDS 9520V array, and > without the heavy interrupt > load, so something seems seriously awry with the > current situation. > > The LSI documentation for the HBA claims it is > capable of interrupt > coalescing, which would seem beneficial for this > setup. I've not found > much in the way of mpt driver, msi/msix, or other > performance tuning > available for this HBA, so I'm hoping I'm missing > something obvious. The mpt hba hardware only provides one msi/msi-x interrupt, therefore, only one CPU can service the interrupt, so all the IOs' callback will be processed by the interrupt CPU.
When you run 'dd' on the JBOD as the pool, I think all the harddisks in the JBOD are doing IO, that is what you saw high interrupt load on mpt driver. A putback about interrupt fanout feature had been made to U8. with it, the load on the CPU where the interrupt is registered can ben fanout to other CPUs. 3 properties can be edited in mpt.conf to tune the performance: mpt_doneq_thread_n_prop - default: 0 (SunOS 5.10), 8 (SunOS 5.11) This property defines the number of helper threads created per HBA (mpt instance). The value zero disables the I/O completion fanout feature, making the value of the other two properties irrelevant. mpt_doneq_length_threshold_prop - default 8 This property is the tolerable backlog of completed IO operations at the interrupt CPU for which I/O completion processing has not been performed. While the backlog remains below this level, I/O completion callbacks will be made in the traditional manner on the interrupt CPU. When the backlog exceeds this level, the entire backlog will be transferred to one of the helper threads. mpt_doneq_thread_threshold_prop - default 10 This property is the tolerable backlog for the helper threads. When selecting a helper thread, the interrupt CPU will avoid using a helper thread if that thread already has a backlog of this level or higher. May this help. Brian > > Suggestions? > > Thanks and regards, > > Marion > > > _______________________________________________ > storage-discuss mailing list > [email protected] > http://mail.opensolaris.org/mailman/listinfo/storage-d > iscuss -- This message posted from opensolaris.org _______________________________________________ storage-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/storage-discuss
