Sebastien, ORNL is assembling a fairly large configuration that will rely heavily on routers. We have a small configuration running similar to yours (IB to IB). Being a Cray shop, we also have IB to Portals running.
We are trying to reach 1.25 GB/s per router. We haven't achieved that yet. The best so far is about 1.1 GB/s sustained. We do see burst up to 1.25-1.3, but not sustained. This was measured on an IB to portals setup with the XT having PCI-e riser (not GA yet). I don't have good numbers on the IB-to-IB routers. If I get some, I will post to the list. Depending on the scale of the cluster, memory can be a driver. You will need to allocate enough buffers on the routers to sustain traffic flow. The main driver for memory consumption is the number of 1M buffers that the routers will run with. For IB we have not seen a huge CPU load, but I don't have good numbers to back that up. So CPU may still be a factor. We hope to repeat the above test but on a Dual core box (versus the single core used thus far). I will update the list as things evolve. --Shane -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Sébastien Buisson Sent: Thursday, April 10, 2008 4:07 AM To: [email protected] Subject: Re: [Lustre-discuss] Lustre routers capabilities Let's consider that the internal bus of the machine is bigger enough so that it will not be saturated. In that case, what will be the limiting factor? memory? CPU? I know that it depends on how many I/B cards are plugged in the machine, but generally speaking, is the routing activity CPU or memory hungry? By the way, are there people on that list that have feedback about Lustre routers sizing? For instance, I know that Lustre routers have been set up at the LLNL. What is the throughput obtained via the routers, compared to the raw bandwidth of the interconnect? Thanks, Sebastien. Brian J. Murrell a écrit : > On Wed, 2008-04-09 at 19:07 +0200, Sébastien Buisson wrote: >> I mean, if I >> have an available bandwith of 100 on each side of a router, what will be >> the max reachable bandwith from clients on one side of the router to >> servers on the other side of the router? Is it 50? 80? 99? Is the >> routing process CPU or memory hungry? > > While I can't answer these things specifically another important > consideration is the bus architecture involved. How many I/B cards can > you put on a bus before you saturate the bus? > > b. > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-discuss mailing list > [email protected] > http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
