On 11/12/2010 10:58 PM, Edward Ned Harvey wrote: >> From: [email protected] [mailto:[email protected]] >> On Behalf Of Edward Ned Harvey >> >> 1G Ether is so cheap, but not as fast as desired. 10G ether is fast > enough, but > > What exactly is infiniband, or brocade switches, etc? Anyone know how these > things differ from Ethernet? I mean ... they're all carrying IP traffic, > right? So if there's an architectural difference of some kind ... > > I suppose I could just go read something on Wikipedia. > What is infiniband? Well, the short of it is it's a Layer1/2 specification for a high speed low latency, scalable, redundant switching fabric.
The major vendor for Infiniband hardware is Mellanox. They make the majority of the chips out there, and they are pretty amazing in their technological acumen. Their roadmap is pretty incredible and they introduce new (reliable!) features in business cycles that others can't touch. The second major hardware vendor is Qlogic who bought SIlverStorm some years ago. Infiniband software vendors included Voltaire, who also make some interesting and relatively inexpensive high-density 10G switching fabrics. What abbreviations will you see? QDR - quad data rate or 40Gbits (8/10 encoding, so actually 32 Gbits/sec) DDR - double data rate or 20Gbits (16) SDR - single data rate or 10 Gbits (8) The next version will be EDR at 80Gbits sometime next year. What sets infiniband apart? 40Gbit chips that work as either 40Gb infiniband, or 10gbit ethernet, and soon 40gbit ethernet on the same chip.. They are multiple personality on-the-chip.If you have a 2 port card, you can even run one port as 40Gb IB and the other as 10Gb ethernet. Also, they are extremely low latency. If this is important to you (see financials and HPC), then 10Gbit ethernet can't touch it. We're talking about 1/2 a microsecond in latency end to end. The 40Gbit chips Some of the switches have this ConnectX technology letting you mix Infiniband, 10/40Gbit ethernet, ISCI, and SAN all on the same chip. It will let any source talk to any destination. What higher level protocols are available on IB? RDMA - remote direct memory access. Applications can directly write/read buffers on a remote machine over Infiniband, saving overhead. DirectGPU - when combined with infiniband, it lets applications eliminate an extra memory copy operation. SRP - SCSI RDMA protocol. Running SCSI commands on an infiniband fabric. iSER - iSCSI extensions for RDMA. http://en.wikipedia.org/wiki/SCSI_RDMA_Protocol What else does it do? Infiniband has something called a 'Subnet Manager'. The subnet manager (can be a switch or software running on a host) is responsible for determining all routes in the fabric and recomputing the topology when changes happen. This can happen dynamically or statically. Infiniband failover *just works*. In a fully connected fabric (see also CLOS: http://en.wikipedia.org/wiki/Clos_network) all hosts can exchange data with any other host at full bandwidth. In QDR and some DDR fabrics, dynamic reconfiguration will happen without jobs aborting or getting reset on the network. on DDR and SDR, you can issue a command to rebalance the fabric when links are repaired. Most of the time there is no noticeable difference for HPC applications. The only reason you might notice is is you have an application that requires extremely low latency (<10micro seconds). In practice, we have very few issues with this on a 3 year old DDR fabric. link failure is not noticeable for running applications. Newer switches can support automatic failback, and they have support built-in for VM migrations that are totally transparent. I have not used this, but it looks pretty impressive. (the switching fabric has direct support for transparent re-routing and migration) For you, it seems like a DDR setup would be pretty inexpensive (older tech, but still rock solid) without the cost of latest-and-greatest QDR. SDR probably wouldn't have the features you'd want in terms of migration. QDR, DDR, and SDR are fully compatible for fabric upgrades, and cabling options are similar in price to 10G ethernet. The one downside to DDR is that most of the switches are the inferior CX4 technology, but there are some vendors that now make this easier. (Cinch, for one) _______________________________________________ Tech mailing list [email protected] https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
