On Thu, Oct 14, 2010 at 6:28 AM, Andrew Stitcher <astitc...@redhat.com> wrote: > On Wed, 2010-10-13 at 21:59 -0700, Aaron Fabbri wrote: >> On Tue, Oct 12, 2010 at 10:22 AM, Andrew Stitcher <astitc...@redhat.com> >> wrote: >> > For those interested in the Rdma implementation: >> > >> > I've been doing a lot of stability work, stressing the rdma code in odd >> > corner cases (unexpected disconnects mostly). While on this trail I >> > reailised I could simplify the Rdma::AsynchIO state machine drastically >> > by ensuring that all callbacks generated by this layer happen in the >> > "thread context" of the connection. >> >> Thanks for the heads up. I'm taking a quick look at the diffs. By >> "thread context of the connection", do you mean always having these >> callbacks happen from the poller threads? > > yes. > >> >> Can you give some hints on how this simplified things? > > Look at the code, and you will see ;-)
Fair enough. I did look at the diffs and the new stuff is much cleaner. I was fishing for some color or background on the previous complex state machine, and why restricting thread contexts simplifies it so much. I'm sure I could figure it out with more than 15 minutes staring at 1500 lines of diffs. > In a little more detail - the Rdma::AsynchIO code is a lot easier to > understand IMO. being sure that all the callbacks happens from an IO > thread with the connection properly serialised makes it easier to not > screw up changes too. > >> >> > >> > After an iteration to improve the performance which added a simple >> > version of the state machine back, we have a version that has very >> > similar throughput, but a little better latency as measured by me on my >> > development boxes. >> >> What sort of latency improvement are you seeing? > > Irrelevant I think given I did no real tuning and this seems to be > important to get good and reliable results. I was only testing in order > to be sure there was no obvious regression. It is curious that forcing a context switch in the write path (notifyPendingWrite now wakes up a poller which does the idle callback which enqueues the write) is OK performance-wise. A major motivation of verbs/RDMA is to avoid context switches. That's the feedback that came to mind. I know sending a patch would be more constructive. ;-) I will give this latest stuff a go and let you know if anything interesting happens. Thanks, Aaron --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
