Sean,
Thanks for your reply. It turns out the problem is the file created by the
ibv_create_comp_channel() call. I do make sure to call the destroy call for
each create call, the problem is that it is failing with error 16 (device or
resource busy) and I missed that fact.
So this brings me to another newbie question which I haven't been able to
completely solve and that is how to cleanly and successfully destroy all IB
resources. Since this is a new subject I have changed the thread subject
appropriately.
I init IB as follows:
- ibv_create_comp_channel()
- make ccc_fd non-blocking
- ibv_create_cq()
- ibv_req_notify_cq()
- ibv_alloc_pd()
- ibv_create_qp
I shutdown in the reverse order
- drain_cq()
- ibv_destroy_qp()
- ibv_dealloc_pd()
- ibv_destroy_cq()
- ibv_destroy_comp_channel()
my drain_cq() function is:
loop:
- ibv_get_cq_event()
- ibv_ack_cq_events() all unacknowledged events pending, if any
- ibv_req_notify_cq()
- ibv_poll_cq()
until either ibv_get_cq_event() returns an error, ibv_poll_cq() returns 0
completions or I have looped the depth of the cq.
It works, in a fashion. Without the drain function ibv_destroy_cq() hangs.
However now, ibv_get_cq_event() returns EAGAIN continuonsly so I exit after I
have looped the depth of cq and then ibv_dealloc_pd(), ibv_destroy_cq() and
ibv_destroy_comp_channel() all return error EBUSY. This leaves the file open in
the system.
I guess my question is, what's the best way to destroy IB resources? (Perhaps
even, what's the best way to init them in the first place).
Thanks,
Nitin
----- Original Message -----
From: "Sean Hefty" <[email protected]>
To: "Nitin Mehrotra" <[email protected]>, [email protected]
Sent: Friday, August 14, 2009 12:34:15 PM GMT -05:00 US/Canada Eastern
Subject: RE: [ofa-general] Help - RDMA event files remain open after
acknowledging them
>What am I doing wrong? Is there something more I need to do than calling
>rdma_ack_cm_event after every rdma_ack_cm_event to get these event files to be
>closed? As an fyi, I have even tried closing the rdma_id and destroying the
>event channel when the connection fails to force the event files to be closed
>without success.
The following calls result in opening files to the kernel:
ibv_create_comp_channel() - used to report cq events
rdma_create_event_channel() - used to report rdma cm events
Be sure that there are corresponding calls to:
ibv_destroy_comp_channel()
rdma_destroy_event_channel()
These are the calls that close the opened files.
- Sean
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
