The best way to evaluate an interconnect and its applicability to a given problem is simply to run the application and see where time is spent. If the majority of the time is not spent in the interconnect, then focus the performance analysis on where the majority of time is spent and tune accordingly. Yes, there are many inefficiencies in the various protocols used - IB, PCIe, coherency, etc. - we comprehend these since protocol creation is often termed a house-of-cards design where trade-offs must occur to meet the overall objectives. When we designed the IB and PCIe protocols, we were very conscious of these trade-offs and our analysis showed that people can build solutions that deliver excellent performance even they choose wisely in the transactions used. To date, the bandwidth and latency achieved in various benchmarks over these interconnects shows that people can get this right. Hence, I recommend you focus your performance analysis on where time is spent in the applications you care about and apply the 80/20 rule for subsequent analysis and tuning. The PCIe and IB protocols are largely set and at best will evolve over time based on demonstrable market needs but some of the fundamentals will be unlikely to change, e.g. IB's 8b/10b encoding which is used from 2.5-10GT/s signaling.
Mike
At 01:43 AM 10/2/2008, Ramiro Alba Queipo wrote:
On Wed, 2008-10-01 at 14:21 -0700, Michael Krause wrote:
> At 09:08 AM 10/1/2008, Ramiro Alba Queipo wrote:
> > Hi all,
> >
> > We have an infiniband cluster of 22 nodes witch 20 Gbps Mellanox
> > MHGS18-XTC cards and I tried to make performance net tests both to
> > check
> > hardware as to clarify concepts.
> >
> > Starting from the theoretic pick according to the infiniband card
> > (in my
> > case 4X DDR => 20 Gbits/s => 2.5 Gbytes/s) we have some limits:
> >
> > 1) Bus type: PCIe 8x => 250 Mbytes/lane => 250 * 8 = 2 Gbytes/s
> >
> > 2) According to a thread an users openmpi mail-list (???):
> >
> > The 16 Gbit/s number is the theoretical peak, IB is coded 8/10 so
> > out of the 20 Gbit/s 16 is what you get. On SDR this number is
> > (of course) 8 Gbit/s achievable (which is ~1000 MB/s) and I've
> > seen well above 900 on MPI (this on 8x PCIe, 2x margin)
> >
> > Is this true?
>
> IB uses 8b/10 encoding. This results in a 20% overhead on every
> frame. Further, IB protocol - header, CRC, flow control credits, etc.
> will consume additional bandwidth - the amount will vary with workload
> and traffic patters. Also, any fabric can experience congestion which
> may reduce throughput for any given data flow.
>
> PCIe uses 8b/10b encoding for both 2.5GT/s and 5.0 GT/s signaling (the
> next generation signaling is scrambled based so provides 2x the data
> bandwidth with significantly less encoding overhead). It also has
> protocol overheads conceptually similar to IB which will consume
> additional bandwidth (keep in mind many volume chipsets only support a
> 256B transaction size so a single IB frame may require 8-16 PCIe
> transactions to process. There will also be application / device
> driver control messages between the host and the I/O device which will
> consume additional bandwidth.
>
> Also keep in mind that the actual application bandwidth may be further
> gated by the memory subsystem, the I/O-to-memory latency, etc. so
> while the theoretical bandwidths may be quite high, they will be
> constrained by the interactions and the limitations within the overall
> hardware and software stacks.
>
>
> > 3) According to other comment in the same thread:
> >
> > The data throughput limit for 8x PCIe is ~12 Gb/s. The theoretical
> > limit is 16 Gb/s, but each PCIe packet has a whopping 20 byte
> > overhead. If the adapter uses 64 byte packets, then you see 1/3 of
> > the throughput go to overhead.
> >
> > Could someone explain me that?
>
> DMA Read completions are often returned one cache line at a time while
> DMA Writes are often transmitted at the Max_Payload_Size of 256B (some
> chipsets do coalesce completions allowing up to the Max_Payload_Size
> to be returned). Depending upon the mix of transactions required to
> move an IB frame, the overheads may seem excessive.
>
> PCIe overheads vary with the transaction type, the flow control credit
> exchanges, CRC, etc. It is important to keep these in mind when
> evaluating the solution.
>
> > Then I got another comment about the matter:
> >
> > The best uni-directional performance I have heard of for PCIe 8x IB
> > DDR is ~1,400 MB/s (11.2 Gb/s) with Lustre, which is about 55% of
> > the
> > theoretical 20 Gb/s advertised speed.
> >
> >
> > ---------------------------------------------------------------------
> >
> >
> > Now, I did some tests (mpi used is OpenMPI) with the following
> > results:
> >
> > a) Using "Performance tests" from OFED 1.31
> >
> > ib_write_bw -a server -> 1347 MB/s
> >
> > b) Using hpcc (2 cores at diferent nodes) -> 1157 MB/s (--mca
> > mpi_leave_pinned 1)
> >
> > c) Using "OSU Micro-Benchmarks" in "MPItests" from OFED 1.3.1
> >
> > 1) 2 cores from different nodes
> >
> > - mpirun -np 2 --hostfile pool osu_bibw -> 2001.29 MB/s
> > (bidirectional)
> > - mpirun -np 2 --hostfile pool osu_bw -> 1311.31 MB/s
> >
> > 2) 2 cores from the same node
> >
> > - mpirun -np 2 osu_bibw -> 2232 MB/s (bidirectional)
> > - mpirun -np 2 osu_bw -> 2058 MB/s
> >
> > The questions are:
> >
> > - Are those results coherent with what it should be?
> > - Why tests with the two core in the same node are better?
> > - Should not the bidirectional test be a bit higher?
> > - Why hpcc is so low?
>
> You would need to provide more information about the system hardware,
> the fabrics, etc. to make any rational response. There are many
Whe have DELL PowerEdge SC1435 nodes with two AMD 2350 processors (2.0
GHz processor frequency and 1.8 GHz of Integrated Memory Controller
Speed). The fabrics is built from 20 Gbps Mellanox MHGS18-XTC cards and
a Flextrics 24 port 4X DDR switch, with 3 meter cables from Mellanox
(MCC4L30-003 4X microGiGaCN latch, 30 AWG).
> variables here and as I noted above, one cannot just derate the
> hardware by a fixed percentage and conclude there is a real problem in
> the solution stack. He is more complex. The question you should
> ask is whether the micro-benchmarks you are executing are a realistic
> reflection of the real workload. If not, then do any of these numbers
No I don't think they are. My main intention is to understand what I
really have and why, and to check for link degradations. Keep in mind
that this is my first contact with infiniband problematics and before
the end of this year we will have 76 nodes (608 cores) with an
infiniband net that will be used both for calculations and data, using
NFS-RDMA.
Apart from our own test, what tests you would use to check for a ready
cluster?
> matter at the end of the day especially if the total time spent
> within the interconnect stacks are relatively small or bursty.
>
> Mike
--
Aquest missatge ha estat analitzat per MailScanner
a la cerca de virus i d'altres continguts perillosos,
i es considera que està net.
For all your IT requirements visit: http://www.transtec.co.uk
_______________________________________________ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
