> 1) ib_create_qp() fails with max_sge > If you use ib_query_device() to return the device specific > attribute max_sge, it seems reasonable to expect you can create > a QP with max_send_sge=max_sge. The problem is that this often > fails. > > The reason is that depending on the QP type (RC, UD, etc.) and > how the QP will be used (send, RDMA, atomic, etc.), there can be > extra segments required in the WQE that eat up SGE entries. So > while some send WQE might have max_sge available SGEs, many will > not.
> This issue may need API extensions to definitively resolve. In > the short term, it would be very nice if max_sge reported by > ib_query_device() could always return a value that ib_create_qp() > could use. Think of it as the minimum max_send_sge value that > will work for all QP types. The intention is that any attempt to create a QP with the maximum number of S/G entries as reported by query device should succeed. However, as you note there may be issues that make this fail, but I would consider them as bugs to be fixed. You mention API extensions to handle this -- do you have any concrete ideas? In the past we've talked a little about this, but I don't think anyone has suggested any changes that would help matters while still keeping the API no more complex than it already is. > The recent patch to support shrinking WQEs introduces a > behavior that creates a big difference between the mlx4 > supported send SGEs (checked against 61, should be 59 or 60, > and reported in ib_query_device as 32 to equal receive side > max_rq_sg value). I'm not sure I understand this. What's the new behavior? Are you trying to take advantage of the fact that using non-power-of-2 size send WQEs would let you have a send queue with more than 32 S/G entries? I think doing that actually would require a change in the API to allow different values for max_sge_rq and max_sge_sq to be reported from ib_query_device(). Which in turn would break the userspace ABI, etc, etc. and leaves me wondering if it's really worth it. (BTW I hate the "shrinking WQE" terminology for this, although obviously you weren't the one to introduce it) - R. _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
