It seems the latency of DDR infiniband to do a blocked read from remote memory (RDMA) is between that of SDR and quadrics, with quadrics being a lot faster.
http://www.google.nl/url?sa=t&rct=j&q=rdma%20latency%20ddr% 20infiniband&source=web&cd=9&ved=0CF8QFjAI&url=http%3A%2F% 2Fwww.cse.scitech.ac.uk%2Fdisco%2Fmew18%2FPresentations%2FDay2% 2F5th_Session% 2FMarkLehrer.pdf&ei=tjW4ToWjOY2dOoD69esB&usg=AFQjCNEzRhG5ljCxmm1r0SMXVob nAbZUAQ&cad=rja If i click there i get to a MarkLehrer.pdf www.cse.scitech.ac.uk/disco/mew18/Presentations/.../MarkLehrer.pdf It claims a RDMA read has latency of 1.91 us. However i'll have to see that in my own benchmark first before i believe it when we hammer with many different processes at that card at the same time. You get problems like switch latencies and other nasty stuff then. This is a presentation slide and i need something that works in reality. HP 4X DDR InfiniBand Mezzanine HCA 410533-B21 SFF-8470 they're $75 but just 2 of them available on ebay. The next 'ddr' one is QLE7104 QLOGIC INFINIBAND 8X DDR SINGLE PORT HBA So that's a qlogic one, $108 just 3 of them available, but we already get at a dangerous price level. Remember i want well over a million reads getting done a second and i didn't count the pollution by writes even yet. HP 4X DDR InfiniBand Mezzanine HCA - 2 Ports 448262-B21 They're $121 and again just 2 available. This seems a problem with infiniband on ebay. Even if you search 16 cards, you can each time buy 2 or so max. As if sometimes a scientist takes 2 back home and puts 'em on ebay. No big 'old' masses get posted there. The first one to offer 10, that's http://www.ebay.com/itm/HP- INFINIBAND-4X-DDR-PCI-E-DUAL-PORT-HCA-448397B21-/110649801200? pt=COMP_EN_Hubs&hash=item19c33df9f0 That 's at $192.11 a piece. It seems DDR infiniband still isn't in my pricerange Prentice. The QM500-B's from quadrics go for between $30 and $50 however. On Nov 7, 2011, at 7:16 PM, Prentice Bisbal wrote: > Vincent, > > Don't forget that between SDR and QDR, there is DDR. If SDR is too > slow, and QDR is too expensive, DDR might be just right. > > -- > Goldilocks > > > On 11/07/2011 11:58 AM, Vincent Diepeveen wrote: >> hi Prentice, >> >> I had noticed the diff between SDR up to QDR, >> the SDR cards are affordable, the QDR isn't. >> >> The SDR's are all $50-$75 on ebay now. The QDR's i didn't find cheap >> prices in that pricerange yet. >> >> If i would want to build a network that's low latency and had a >> budget >> of $800 or so a node of course i would >> build a dolphin SCI network, as that's probably the fastest latency >> card sold for a $675 or so a piece. >> >> I do not really see a rival latency wise to Dolphin there. I bet most >> manufacturers selling clusters don't use >> it as they can make $100 more profit or so selling other networking >> stuff, and universities usually swallow that. >> >> So price total dominates the network. As it seems now infiniband >> 4x is >> not going to offer enough performance. >> The one-way pingpong latencies over a switch that i see of it, are >> not >> very convincing. I see remote writes to RAM >> are like nearly 10 microseconds for 4x infiniband and that card is >> the >> only one affordable. >> >> The old QM400's i have here are one-way pingpong 2.1 us or so, and >> QM500-B's are plentyful on the net (of course big disadvantage: needs >> pci-x), >> which are a 1.3 us or so there and have SHMEM. Not seeing a cheap >> switch for the QM500's though nor cables. >> >> You see price really dominates everything here. Small cheap nodes you >> cannot build if the port price, thanks to expensive network card, >> more than doubles. >> >> Power is not the real concern for now - if a factory already burns a >> couple of hundreds of megawatts, a small cluster somewhere on the >> attick eating >> a few kilowatts is not really a problem :) >> >> >> >> On Nov 7, 2011, at 5:36 PM, Prentice Bisbal wrote: >> >>> >>> On 11/06/2011 06:01 PM, Vincent Diepeveen wrote: >>>> hi, >>>> >>>> There is a lot of infiniband 4x stuff on ebay now. >>> >>> Vincent, >>> >>> Do you mean 4x, or QDR? They refer to different parts of the IB >>> architecture. 4x refers to the number of lanes for the data to >>> travel >>> down and QDR refers to the data signalling rate. >>> >>> It's probably irrelevant for this conversation, but if you are just >>> learning about IB, It's good to understand that difference. >>> >>> Prentice >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin >>> Computing >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >> > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin > Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
