On Sun, 7 Aug 2005, Guy German wrote:
Hi,
James Lentini <mailto:[EMAIL PROTECTED]> wrote:
On Thu, 4 Aug 2005, Guy German wrote:
James,
I see what you mean.
The allocation of the event vector is derived from evd->qlen.
In DTO ev'd, however, qlen is also the parameter passed to
ib_create_cq. Since we don't want to limit DAPL consumers to an
unnecessary small completion queue size, maybe we
could differentiate between DTO supporting evd's and
CONN evd's, when allocating the events vector.
if evd supports CONN only, leave it :
event = kmalloc(evd->qlen * sizeof *event)
(Relying on the consumer he knows what he is doing)
if evd is DTO only :
don't allocate an event buffer, at all
if evd supports both :
event = kmalloc(DEFAULT_4_CONN * sizeof *event)
And dynamically add additional events up to qlen as needed?
if DEFAULT_4_CONN=256, that's a 3 pages allocation.
How does that sound to you ?
I'd prefer that the EVDs were uniform. I would worry about bugs
otherwise.
But if it's the right thing to do (allocating less, or not
allocating at all), wouldn't it be best to try and fix those bugs ?
I think that the dynamic allocation strategy could be used for all
types of EVDs.
EVDs that only handled DTO events (events generated by an IB CQ) would
have an initial allocation of 0 DAT events. These EVDs would never
grow the event pool.
EVDs that handled "software" events might grow this event pool. For
these EVDs we could allocate some number of initial events.
The advantage of making the EVDs uniform is that the de-allocation
code would be the same.
The eventual solution has to support qlen generated events (connection
request, connection, disconnect, software) if those event types are
supported by the EVD (even if the EVD is being used for both generated
events and DTOs).
I think that there might be a mix up in the qlen meaning, for the consumer.
ia_quey returns cq_len parameter, even if the evd in mind is used
for connection only. In that case the result of qlen (128k), from the hca
has nothing to do with the real resources availability.
I assume you are referring to the dat_ia_attr's max_evd_qlen value. You
are right that it is not entirely accurate to report the CQ max qlen
as this value. As you point out, there are really two limits: the
maximum number of DTO and RMR events and the maximum number of
"software" events. The former are limited by the capabilities of the
underlying CQ while the later are constrained by the systems
resources.
You mentioned before that "allocating an event pool equal to
the queue length seems like overkill", and I agree.
If you want to support pending events list of more than ~2700, just
for CONN events, and not use vmalloc, we need some way of doing a
few kmallocs and managing that. Is it really necessary ? wouldn't
256 *pending* events for CONN. purposes be enough ?
I don't know what the consumer requirements are this value. Remember
that there are more "software" events than just pending connection
events. All of these event types correspond to what I have been
terming "software" events:
DAT_CONNECTION_REQUEST_EVENT = 0x02001,
DAT_CONNECTION_EVENT_ESTABLISHED = 0x04001,
DAT_CONNECTION_EVENT_PEER_REJECTED = 0x04002,
DAT_CONNECTION_EVENT_NON_PEER_REJECTED = 0x04003,
DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR = 0x04004,
DAT_CONNECTION_EVENT_DISCONNECTED = 0x04005,
DAT_CONNECTION_EVENT_BROKEN = 0x04006,
DAT_CONNECTION_EVENT_TIMED_OUT = 0x04007,
DAT_CONNECTION_EVENT_UNREACHABLE = 0x04008,
DAT_ASYNC_ERROR_EVD_OVERFLOW = 0x08001,
DAT_ASYNC_ERROR_IA_CATASTROPHIC = 0x08002,
DAT_ASYNC_ERROR_EP_BROKEN = 0x08003,
DAT_ASYNC_ERROR_TIMED_OUT = 0x08004,
DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR = 0x08005,
DAT_SOFTWARE_EVENT = 0x10001
If place a limit (be it 256 or some other value) on the number of
events of the types above, there will be a consumer that will want
more.
My goal is that the queue length of an EVD supporting all events types
(DTO and software) only be limited by the capabilities of the
underlying CQ.
_______________________________________________
openib-general mailing list
[email protected]
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
