My point was not the total storage used for the array (it ends up more than the linked list, as you noted).
I'm concerned that an allocation of a 4K buffer may fail in a situation where lots of small allocations of around 256 bytes would succeed. Is your point that if we fail to allocate a 4K buffer, we're in deep trouble already? Note that I've only considered a 1000 host cluster. What about scalability (e.g., 10,000 nodes -- we then need a 40K buffer) -- the linked list has no scalability problem (no need to push RMPP handling to user space). Regarding the list-walk, if we track the "last-sent segment" in the list, there is no need to do the list walk (we simply get the next segment in the list). We'll only have a short list walk when the "ack" pointer gets updated (need to walk forward only <current-RMPP-ack-window-size> items in the linked list from the previously ack'ed item). -- What is the reason you are thinking about 64-byte boundary support? Jack -----Original Message----- From: Sean Hefty [mailto:[EMAIL PROTECTED] Sent: Wednesday, February 08, 2006 7:13 PM To: Jack Morgenstein; [email protected] Subject: RE: [openib-general] FW: [PATCH 1 of 3] mad: large RMPP support >For example, a 1000 host cluster, with 2 ports per HCA will have at >least 4000 records in a SubnAdmGetTableResp for all PortInfo records on >the network (2000 for HCAs, and at least 2000 for the switch ports). >Such a query response will generate an RMPP of size 256K -- 1000 >segments, or a 4K buffer on an X86 machine just for the array (assuming >one allocation per RMPP segment -- N=1). I think that this is a good reason to use an array. Walking a 1000 entry list 1000 times is a substantial performance hit. Lost MADs and retries will make this worse. A 4K buffer for the array is less than the 8K total needed for the 1000 list items. We're already talking about allocating over 256K of memory just for the data payload. An additional contiguous 4k buffer seems like a minor issue. I'm not convinced that there's a real issue here. To support ridiculously large transfers from userspace, we may need to push the RMPP handling up into userspace. - Sean _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
