[ofa-general] device attributes

2008-06-04 Thread Steve Wise 

Roland/All,

Should the device attributes (for instance max_send_wr) be the max 
supported by the HW or the max supported by the OS or something else?


For instance:  Chelsio's HW can handle very large work queues, but since 
Linux limits the size of contiguous dma coherent memory allocations, the 
actual limits are much smaller.  Which should I be using for the device 
attributes?


Also, the chelsio device uses a single work queue to implement the SQ 
and RQ abstractions.  So the max SQ depth depends on the RQ depth and 
vice versa.  This leads to device max attributes that aren't that useful.


I'm wondering what application writes should glean from these attributes...

Steve

___
general mailing list
general@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [ofa-general] device attributes

2008-06-04 Thread Tom Tucker 

On Wed, 2008-06-04 at 09:28 -0500, Steve Wise wrote:
 Roland/All,
 
 Should the device attributes (for instance max_send_wr) be the max 
 supported by the HW or the max supported by the OS or something else?
 

Something else. 

 For instance:  Chelsio's HW can handle very large work queues, but since 
 Linux limits the size of contiguous dma coherent memory allocations, the 
 actual limits are much smaller.

Basing the limit on an OS resource seems arbitrary and dangerous.
Applications using advertised adapter resource limits will unnecessarily
consume the maximum.

   Which should I be using for the device 
 attributes?
 

Arbitrary knee jerk == 512. 

However, surveying the current app usage as well as the other
manufacturers advertised limits will make it less arbitrary.

 Also, the chelsio device uses a single work queue to implement the SQ 
 and RQ abstractions.  So the max SQ depth depends on the RQ depth and 
 vice versa.  This leads to device max attributes that aren't that useful.
 

So the real limit is the HW WQ max and therefore max SQ = HW WQ max -
RQ max?  Setting RQ and SQ to 512 solves this problem. 


 I'm wondering what application writes should glean from these attributes...
 

I guess you mean application writer? 

Here's what I suggest:

- Set the RQ and SQ max to some reasonable default limit (e.g. 512).
- Add an escape hatch by providing module options to override
  the default max.

Tom

 Steve
 
 ___
 general mailing list
 general@lists.openfabrics.org
 http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
 
 To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

___
general mailing list
general@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [ofa-general] device attributes

2008-06-04 Thread Steve Wise 

Here's what I suggest:

- Set the RQ and SQ max to some reasonable default limit (e.g. 512).
- Add an escape hatch by providing module options to override
  the default max.



Or don't actually enforce the 512 max.  IE set the attrs to what we 
think apps should use or max out at, but allow larger ones if the OS can 
handle it.


Stevo.
___
general mailing list
general@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general