On Tue, 2 Aug 2005, Guy German wrote:
Hi Muli,
Muli Ben-Yehuda <mailto:[EMAIL PROTECTED]> wrote:
On Tue, Aug 02, 2005 at 06:24:49PM +0300, Guy German wrote:
There are some places where kmalloc might not be enough :
in dapl_evd_event_alloc there is an allocation:
event = kmalloc(evd->qlen * sizeof *event);
whereas evd->qlen can be 128k (depends on max_cqe of the hca) and
kmalloc would fail.
The same goes to dapl_rbuf_alloc.
Is it legit to replace those kmallocs with vmallocs ?
We should only add calls to vmalloc() as a last resort. As Muli points
out, they are discouraged.
Why do we need such a large allocation?
kDAPL creates two large pools of memory.
One is for events. When the kDAPL consumer creates an EVD, it
specifies a queue size (the number of events the EVD can hold). The
implementation pre-allocates a pool of events equal to the size of the
queue. These events are used when an IB upcall is made (e.g.
connection request, connection established, aysnc. error, etc.) or the
kDAPL consumer posts a "software event" via dat_evd_post_se().
The other memory pool is for cookies. A kDAPL event contains certain
fields that the IB work completion (ib_wc) does not provide (like the
EVD, EP, etc.). For that reason, the kDAPL provider sticks the missing
information in a dapl_cookie structure and sets it as the work
request's context value. When the work completion comes back, the
kDAPL provider pulls the cookie out and uses it to populate the
missing event fields. These cookies are also pre-allocated in a pool
equal to the EVD size.
To answer your question, vmalloc has a performance overhead and can
and will fail when vmalloc-space is exhausted (as can kmalloc, for
different reasons). Can this allocation be cut down so that it
becomes a non-issue?
The size of the event pool seems much larger than necessary. I would
expect most consumers only use a few events from this pool (with no
errors or software events, a client will use 2 and a server will use
3).
We may be able to eliminate the cookie pool entirely. There are only a
few values we need from the cookie. I'll look into that.
evd_min_qlen defines the size of the event queue that the Consumer requested.
sizeof *event = 184 - that leaves ~712 pending events, which is not much.
ISER target is trying to support about 5000 (by their calculations), but other
consumers
might want to support even more and there is no reason for dapl to limit what
the ib can provide.
Note that iser dequeues the events itself (only the first event is accepted
from a callback), hence the
need for a normal size queue.
_______________________________________________
openib-general mailing list
[email protected]
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
