Michael S. Tsirkin wrote:
Quoting r. Arlin Davis <[EMAIL PROTECTED]>:
Subject: [PATCH] perftest/rdma_bw; add support for RDMA read and starting PSN
Michael,
The patch adds command line options for RDMA reads and starting PSN. I
used these modifications to
help isolate the RDMA read performance degradation with 4.6.2 firmware.
-arlin
Thanks Arlin. I plan to look into integrating this.
One question: for which psn values do you see performance drop on 4.6.0 FW?
A quick run at 1 and then 0x100000 dropped from 682MB/s to 49MB/s for
32KB buffers.
What is really strange is that it takes a couple runs to start seeing
the drop in performance.
PSN=1 no problems...
[EMAIL PROTECTED] perftest]$ ./rdma_bw -P 0x1 -s 32768 -r iclust-19
local address: LID 0x02, QPN 0x20406, PSN 0x0001 RKey 0x0c0032 VAddr
0x00000000514000 RDMA_READ
remote address: LID 0x05, QPN 0x20406, PSN 0x0001 RKey 0x0c0032 VAddr
0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #999): 682.504 MB/sec
Bandwidth average: 682.501 MB/sec
Service Demand peak (#0 to #999): 5138 cycles/KB
Service Demand Avg : 5138 cycles/KB
[EMAIL PROTECTED] perftest]$ ./rdma_bw -P 0x1 -s 32768 -r iclust-19
local address: LID 0x02, QPN 0x30406, PSN 0x0001 RKey 0x120032 VAddr
0x00000000514000 RDMA_READ
remote address: LID 0x05, QPN 0x30406, PSN 0x0001 RKey 0x120032 VAddr
0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #990): 682.496 MB/sec
Bandwidth average: 682.496 MB/sec
Service Demand peak (#0 to #990): 5138 cycles/KB
Service Demand Avg : 5138 cycles/KB
[EMAIL PROTECTED] perftest]$ ./rdma_bw -P 0x1 -s 32768 -r iclust-19
local address: LID 0x02, QPN 0x40406, PSN 0x0001 RKey 0x180032 VAddr
0x00000000514000 RDMA_READ
remote address: LID 0x05, QPN 0x40406, PSN 0x0001 RKey 0x180032 VAddr
0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #990): 682.5 MB/sec
Bandwidth average: 682.499 MB/sec
Service Demand peak (#0 to #990): 5138 cycles/KB
Service Demand Avg : 5138 cycles/KB
PSN=0x100000 (start to see problems after first run)
[EMAIL PROTECTED] perftest]$ ./rdma_bw -P 0x100000 -s 32768 -r iclust-19
local address: LID 0x02, QPN 0xb0406, PSN 0x100000 RKey 0x420032
VAddr 0x00000000514000 RDMA_READ
remote address: LID 0x05, QPN 0x90406, PSN 0x100000 RKey 0x360032
VAddr 0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #996): 682.5 MB/sec
Bandwidth average: 682.499 MB/sec
Service Demand peak (#0 to #996): 5138 cycles/KB
Service Demand Avg : 5138 cycles/KB
[EMAIL PROTECTED] perftest]$ ./rdma_bw -P 0x100000 -s 32768 -r iclust-19
local address: LID 0x02, QPN 0xc0406, PSN 0x100000 RKey 0x480032
VAddr 0x00000000514000 RDMA_READ
remote address: LID 0x05, QPN 0xa0406, PSN 0x100000 RKey 0x3c0032
VAddr 0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #0): 48.5441 MB/sec
Bandwidth average: 47.4502 MB/sec
Service Demand peak (#0 to #0): 72244 cycles/KB
Service Demand Avg : 73909 cycles/KB
[EMAIL PROTECTED] perftest]$ ./rdma_bw -P 0x100000 -s 32768 -r iclust-19
local address: LID 0x02, QPN 0xd0406, PSN 0x100000 RKey 0x4e0032
VAddr 0x00000000514000 RDMA_READ
remote address: LID 0x05, QPN 0xb0406, PSN 0x100000 RKey 0x420032
VAddr 0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #0): 48.4803 MB/sec
Bandwidth average: 47.4501 MB/sec
Service Demand peak (#0 to #0): 72339 cycles/KB
Service Demand Avg : 73909 cycles/KB
PSN = 1 (first run is bad, and then it is back to normal)
[EMAIL PROTECTED] perftest]$ ./rdma_bw -P 0x1 -s 32768 -r iclust-19
local address: LID 0x02, QPN 0xe0406, PSN 0x0001 RKey 0x540032 VAddr
0x00000000514000 RDMA_READ
remote address: LID 0x05, QPN 0xc0406, PSN 0x0001 RKey 0x480032 VAddr
0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #0): 48.5798 MB/sec
Bandwidth average: 47.4502 MB/sec
Service Demand peak (#0 to #0): 72190 cycles/KB
Service Demand Avg : 73909 cycles/KB
[EMAIL PROTECTED] perftest]$ ./rdma_bw -P 0x1 -s 32768 -r iclust-19
local address: LID 0x02, QPN 0xf0406, PSN 0x0001 RKey 0x5a0032 VAddr
0x00000000514000 RDMA_READ
remote address: LID 0x05, QPN 0xd0406, PSN 0x0001 RKey 0x4e0032 VAddr
0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #990): 682.492 MB/sec
Bandwidth average: 682.49 MB/sec
Service Demand peak (#0 to #990): 5138 cycles/KB
Service Demand Avg : 5138 cycles/KB
-arlin
_______________________________________________
openib-general mailing list
[email protected]
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
