BTW, does Mellanox (or anyone else) have any numbers showing that
using FMRs makes any difference in performance on a semi-realistic benchmark?
I'm using xdd to test the performance
www.ioperformance.com/products.htm
The target is Mellanox srp target reference implemenation
with 14 SATA spindles
I can get ~780 MB/s max without FMRs and ~920 MB/s with FMRs
(using 256 KB sequential read direct IO request)
Wow, that's awesome Vu!
So what's the consensus on the reason for the improvement?
- Fewer WR to send the same amount of data because the memory is
virtually contiguous?
Fewer wqe, fewer interrupts & context switches, big I/O
request directly sent to back-end storage/spindles (all are
applied for target)
For write command (RDMA read), it help to utilize the max
outstanding rdma read operations (more data transfer per
outstanding rdma read operations)
- Fewer PDU due to larger writes and better packing?
This is not much.
- Something else?
Are these huge reads, ... or even better ... what are the parameters to
xdd?
-op read -reqsize 128 -block 2048 -dio -mbytes 8192
-targetdir /dev/ -targets 14 sdb sdc sdd ....
Did you quantify the FMR registration cost per MB of registered memory
space? This would be a very good number to have for figuring out where
the sweet spot is...
I don't have this number of FMR registration cost per MB.
Dror, do we have any number on how many cycles cost per FMR
registration?
On the side note the FMR registration hit rate is ~100% with
direct I/O
Vu
_______________________________________________
openib-general mailing list
[email protected]
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
