Hello,
I'm interested in experimenting with Qpid Rdma in hopes of further
lowering message latency. I've noticed that the Qpid Rdma implementation
uses a (nonblocking) completion channel with the InfiniBand verbs
completion queue (cq). I'd like to see if not using the completion
channel and polling the cq only would yield benefits.
As evidence of the performance potential I ran a test with
ibv_rc_pingpong, the raw IB verbs latency test that comes with the OFED
distribution. This code uses the same reliable connection, send/recv
QueuePair modes that Qpid Rdma uses. I've attached a file of my results.
I ran ibv_rc_pingpong with and without a completion channel. In
non-completion channel mode, message latency reduced by 10 usec for all
sizes tested. For message sizes <= 16K this means ~25% or greater
improvement. For message sizes <= 512K this yields ~50% improvement. It
would be interesting to see how this would translate into Qpid.
I've modified Rdma::AsynchIO.processCompletions and Rdma::QueuePair to
work without a completion channel. However I'm at a loss as to how to
modify the Poller which epolls the file descriptor underlying the
completion channel. Here is how I see the current event chain of
dependency in Qpid:
Dispatcher.run
-> Poller.wait (EpollPoller)
-> epoll_wait (I want to omit Rdma::QueuePair.cchannel's fd from the
epollfd set)
-> DispatchHandle.ProcessEvent
-> Rdma::AsynchIO.DataEvent
-> Rdma::AsynchIO.ProcessCompletions (I want to omit
ibv_get_cq_event on Rdma::QueuePair.cchannel)
-> Rdma::QueuePair.GetNextEvent
-> ibv_poll_cq IB verb until no more events
My main question is how (where?) would I best substitute ibv_poll_cq() in
place of epoll_wait() to drive event polling for Rdma::AsynchIO?
I also have some miscellaneous questions regarding the Poller:
I'd still need to maintain the epoll framework as it seems there are other
objects registering (startWatch) file descriptors with epoll. I've
noticed that regular AsynchIO and Rdma::ConnectionManager are. What else
is registering its file descriptor with the EpollPoller?
How many Poller objects/threads are instantiated? I ran with gdb and
noticed that different threads were calling Poller.wait().
Thanks in advance for any direction/advice,
Greg Marsh
Network-Based Computing Lab
Ohio State University
Table below shows test run of ibv_rc_pingpong ("raw" IB verbs test pgm) between
2 nodes in an InfiniBand cluster. 50000 iterations were used for each msg
size. "With Event Channel" column shows usec/iteration ("round trip time")
using completion event channel (ibv_rc_pingpong -n 50000 -s $msg_size -e) in
conjunction with the completion queue. "ibv_poll_cq only" column shows
usec/iteration WITHOUT using the completion event channel (ibv_rc_pingpong -n
50000 -s $msg_size) and polling the completion queue only.
"Msg Bytes" "With Event Channel" "ibv_poll_cq only"
1 19.67 9.33
2 19.53 9.34
4 19.62 9.34
8 19.56 9.37
16 19.46 9.09
32 19.7 9.52
64 19.48 9.58
128 19.81 9.74
256 20.24 10.1
512 20.74 10.72
1024 21.74 12.37
2048 22.68 13.61
4096 25.13 16.13
8192 30.27 20.87
16384 41.5 31.01
32768 61.89 51.78
65536 102.89 93.47
131072 186.36 176.77
262144 352.05 343.5
524288 693.28 680.01
1048576 1361.72 1351.98
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
