Hi Brian, With one 4x QDR IB port, you can achieve 2 GB/Sec on single client, multi-threaded workload provided that you have right storage (with enough bandwidth) at other end. We have tested this multiple times at DDN.
I have seen sites that do IB-bonding across 2 ports but mostly in failover configuration. To get 10GB/Sec to a single node requires aggregating 5 QDR IB ports. You will need to confirm from your IB vendor (Mellanox? ), OS vendor (SGI/RedHat/Novell) and Lustre vendor whether they support aggregating so many links. I think the challenge you will have is to find a Lustre client node that has enough x8 PCIe slots to sustain 3 dual-port Infiniband adapters at full rate (think multiple such nodes in a typical Lustre filesystem, not so economical). Other alternative is to find a server that can support 8X or 12X QDR IB port on the motherboard to get more bandwidth. With a typical Lustre client memory of 24-64GB and memory to CPU bandwidth of 10GB/Sec (with standard DDR3-1333MHz DIMMS), it is not possible to fit dataset larger than 2/3rd of memory. If you still want to achieve 10GB/Sec of bandwidth between storage and memory, there are clever alternatives. You will have to stage your data into memory beforehand and keep memory pages locked and continue feeding data as these pages are consumed. It is lot harder than it seems on the paper. Cheers, -Atul From: [email protected] [mailto:[email protected]] On Behalf Of Brian O'Connor Sent: Monday, 21 March 2011 9:23 AM To: [email protected] Subject: [Lustre-discuss] Multiple IB ports Hi, Any body actually using multiple IB ports on a client for an aggregated connection? Ie. Many oss with one qdr IB each. Clients with 4 qdr IB ports. Assuming the normal issues with bus bandwidth etc, what sort of perf can I expect qdr ~ 3-4Gbytes/Sec I'm trying to size a cluster and clients to get ~10GBytes/Sec on *one* client node. If I can aggregate IB linearly the next step will be to try and figure out How to get 10Gigabytes/s to local storage :( Some times customers are crazy....... Brian O'Connor ------------------------------------------------- SGI Consulting Email: [email protected]<mailto:[email protected]>, Mobile +61 417 746 452 Phone: +61 3 9963 1900, Fax: +61 3 9963 1902 357 Camberwell Road, Camberwell, Victoria, 3124 AUSTRALIA http://www.sgi.com/support/services -------------------------------------------------
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
