Michael> If thats what we are trying to do, I'd like to propose
Michael> another idea: when cq is destroyed, and after all cq
Michael> events are queued to user, put a special "cq destroyed"
Michael> event into the event queue. When a user calls
Michael> destroy_cq, and after releasing kernel/hardware
Michael> resources, move the cq structure to a special cleanup
Michael> list, userspace library will only destroy the userspace
Michael> cq structure when it gets this event.
Makes sense. I coded this up with the API as below:
/**
* ibv_destroy_cq - Destroy a completion queue
* @cq: CQ to destroy.
*
* If ibv_req_notify_cq() has ever been called for @cq, then the
* CQ structure will not be freed until ibv_get_cq_event() has
* returned a "dead CQ" event (ie is_dead != 0) for this CQ.
*
* Calling ibv_destroy_cq() from one thread while another thread calls
* ibv_req_notify_cq() for the same CQ will lead to unpredictable
* results. Don't do that.
*/
extern int ibv_destroy_cq(struct ibv_cq *cq);
/**
* ibv_get_cq_event - Read next CQ event
* @context: Context to get CQ event for
* @comp_num: Index of completion event to check. Must be >= 0 and
* <= context->num_comp.
* @cq: Used to return pointer to CQ.
* @cq_context: Used to return consumer-supplied CQ context.
* @is_dead: If non-zero, then the event indicates that the CQ has
* been destroyed and no more completion events will be generated
* for the CQ.
*/
extern int ibv_get_cq_event(struct ibv_context *context, int comp_num,
struct ibv_cq **cq, void **cq_context,
int *is_dead);
I'll post the full kernel patches and userspace changes in separate
emails shortly.
Michael> Instead of reference counting, we require the user to
Michael> poll for this event after destroying the cq. Since all
Michael> events are polled from a single thread, when the library
Michael> sees this event it knows the user does not keep a pointer
Michael> to the cq any more.
Actually there's no rule that all events need to be read from a single
thread. But if an application wants to use multiple threads and ends
up getting things out of order, then I think that's the app's fault.
- R.
_______________________________________________
openib-general mailing list
[email protected]
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
