Hi Roland, IOMMU folks, So we've noted that when configuring the kernel && booting with intel iommu set to on on a physical node (non VM, and without enabling SRIOV by the HW device driver) raw performance of the iSER (iSCSI RDMA) SAN initiator is reduced notably, e.g in the testbed we looked today we had ~260K 1KB random IOPS and 5.5GBs BW for 128KB IOs with iommu turned off for single LUN, and ~150K IOPS and 4GBs BW with iommu turned on. No change on the target node between runs.
This was done over kernel 3.5.x, I will re-run tomorrow with latest upstream and send the top perf hits for both cases, but does this rings some bells? basically it makes sense for some extra latency, but I didn't expect the IOPS and BW drop to be so notable. Is it possible that at these rates the IOMMU is so much trashed that it can't keep the perf we have without it being on? We're a SCSI LLD which is called through the queuecommand API by the SCSI midlayer, get a scatter-gather list pointing to set of pages, issue dma_map_sg on the SG list, and then conduct some registration with the HCA of this set of DMA addresses which produced a token (rkey) we send to the target node with the request. The target node does RDMA using that token, without the CPU of the initiator being involved (the two HCAs talk) later the target sends respose which means the transaction is done, after we get the response we undo the registration at the HCA and do dma_unmap_sg. Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
