I'm not sure I follow the rationale as to why acking is needed. You're requiring a solution to a problem that most apps and most devices do not have.
But anyway, if you *do* have an acked cq reap, you should take advantage of it by having the cq_poll return a *pointer* to the work completion rather than copying it to a supplied buffer. This avoids an unecessary copy whenever the work completion will be processed immediately or transformed into another format (which, between them, I believe account for most applications). The only problem with a cq_poll routine that returns a pointer is that it *requires* an ack call -- but if you're going to do it anyway you might as well get the benefit of a two stage call. > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Roland Dreier > Sent: Tuesday, September 06, 2005 2:23 PM > To: Sean Hefty > Cc: openib > Subject: [openib-general] Re: ibv_get_async_event > > I thought about this some more and I came to the conclusion > that Sean is right. We should come up with something > race-free, even if an app is perverse enough to use multiple > threads to read CQ events. > > I think the only way to do that is for the app to acknowledge > completion events, since a completion event could be read by > a thread that loses the CPU before returning to the app and > then delayed for arbitrarily long before the app sees the > event. However, it is possible to amortize the locking cost > of acknowledging events by allowing the app to acknowledge > multiple events in a single call. > > The API I came up with is the following: > > /** > * ibv_ack_cq_events - Free an async event > * @cq: CQ to acknowledge events for > * @nevents: Number of events to acknowledge. > * > * All completion events which are returned by > ibv_get_cq_event() must > * be acknowledged. ibv_destroy_cq() will wait for all > completion > * events to be acknowledged, so there should be a one-to-one > * correspondence between acks and successful gets. An > application > * may accumulate multiple completion events and > acknowledge them in a > * single call by passing the number of events to ack > in @nevents. > */ > extern void ibv_ack_cq_events(struct ibv_cq *cq, > unsigned int nevents); > > (I also renamed ibv_put_async_event() to ibv_ack_async_event() for > symmetry) > > I coded this up and did some unscientific measurements using > ibv_rc_pingpong (using CQ events with --size=1). Even with a call to > ibv_async_event() every time a CQ event is read, the cost is > too small to measure. In other words, the variability from > run to run of my test drowns out the cost of the call to > ibv_ack_cq_events(). > > Patches to follow... > > - R. > _______________________________________________ > openib-general mailing list > [email protected] > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
