On Mon, 23 May 2005, Michael S. Tsirkin wrote:
Quoting r. James Lentini <[EMAIL PROTECTED]>:
Subject: Re: Re: libibverbs and max inline data
On Mon, 23 May 2005, Michael S. Tsirkin wrote:
Quoting r. James Lentini <[EMAIL PROTECTED]>:
Subject: Re: Re: libibverbs and max inline data
On Sun, 22 May 2005, Roland Dreier wrote:
Michael> Hi! I tried to figure out how much inline data a qp can
Michael> support, to know what max_inline_data value to put in the
Michael> qp. However, there doesnt seem to exist a way to do
Michael> that, currently.
I think you're right.
Michael> Maybe the driver should be changed to trim
Michael> max_inline_data to the maximum possible value?
Yes, that would probably make sense. In a sense that's what the
kernel driver already does -- it always clips the max_inline_data
request from the consumer down to 0.
ib[v]_create_qp should return an error when an unsupported inline data
size is requested.
Why is it important? The user could always check the max_inline_data
returned and close the qp if thats too small, but I dont see
many users doing this - inline data is mainly a tuning opportunity.
Once there is a way to programmatically determine the max_inline_data
value, your proposal would be feasible.
There is a way to do it - the actual values are returned in the attr that you
pass to create_qp.
See my previous email to Roland. I believe that mthca_create_qp does
not update the attr if an error occurs.
Even if it were possible, I
still believe that the driver should validate this value.
Why have the user specify the ib[v]_qp_cap values if they are going to
be ignored?
max_inline_data is special in that its a tuning opportunity. So its
mostly harmless for the driver to ignore.
The driver doesn't ignore it though. Take a look at mthca_set_qp_size.
If you set max_inline_data to a value such that qp->sq.max_gs will be
dev->limits.max_sg, then mthca_set_qp_size will fail with error
EINVAL. This will result in the failure of ib_create_qp.
As a rule, the driver should return an error when the user requests a
capability/resource that the hardware cannot support.
kDAPL was bitten by this last week. The call to ib_create_qp succeed
but the value kDAPL was placing in the ib_qp_init_attr's
max_inline_data field was too large.
It was just a bug in the mthca kernel driver.
You shouldnt work around such bugs in an application.
I'm sorry. My memory was wrong here. ib_create_qp did fail when the
max_inline_data field was too large.
As a result, a subsequent qp call failed.
If ib_create_qp had returned an error to begin with, it would
have been easier to determine the problem.
It seems wrong for the application to hard-code the max_inline_data
value that suits a particular hardware. Better for the app to tell
the driver what it needs and the driver will give it what hardware
can support.
Independent of that we could add a field to the device properties
struct (and implement ibv_query_device() for userspace).
ibv_query_device would be useful for other things as well (e.g.
getting the max MTU).
--
MST - Michael S. Tsirkin
_______________________________________________
openib-general mailing list
[email protected]
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
