Chuck Hartley wrote:
We are doing performance measurements on an application that is using
uDAPL RDMA reads for some large transfers and the BW is less than we
expected. The transfers are 4MB and we are seeing BW of 930MiB/sec
(DDR). When we do the same transfer size using ib_read_bw we get 1475
MB/sec. On a pair of machines with SDR interfaces, we get 697MiB/sec
and 918MB/sec respectively.
We expected some overhead from uDAPL, but this much seems excessive. Or
is this typical? I looked at the code for dat_ep_post_rdma_read() and
compared it to the ib_read_bw() code. The difference is that in uDAPL
there is some cookie management and some WR struct setup before it gets
around to calling the IB verbs function ibv_post_send(). It doesn't
look like something that would take much time - about 1.5msec given the
numbers above.
You performance should be very close to the verbs rdma
bandwidth tests, especially at 4MB.
Can you try dapltest bw test suite to compare?
server:
dapltest
client:
dapltest -T P -m p -d -s cst-50-ib0 -i 100 RR 4096000
Here are my numbers (Intel x86_64 and DDR):
Total Time : 0.31 sec
Total Data Exchanged : 390.62 MB
CPU Utilization : 25.80
Operation Throughput : 319.65 ops/sec
Bandwidth : 1248.66 MB/sec
-arlin
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
