I'm not sure... I didn't investigate much further once I figured out it'd work fine for our application. This was sending data through a switch, so it could even have been due to the switch fabric. I didn't try a direct cable connection. But it might be there's overhead in our 10GbE core's CPU/fabric arbiter, for example. As I say, I didn't look too closely.
Jason On 14 Mar 2014, at 11:55, Jack Hickish <[email protected]> wrote: > Thanks Jason. That info is really helpful. > > It sounds like if I boost my packets to ~1% header overhead, I > shouldn't reckon on being able to throw away less than 5% of the > bandwidth. > > Out of interest, do you know what limit you're hitting at 96% data > rate (I interpreted raw data rate to mean payload + headers(?))? -- is > it something inherent to the tx/rx firmware or the switch, or is it > ARP packets and the like floating round? > > Thanks again, > > Jack > > On 14 March 2014 06:20, Jason Manley <[email protected]> wrote: >> You can push them pretty hard. We're reliably running at an average of >> around 90% wire speed on all four links for MeerKAT. But be sure to >> factor-in the overheads of the core... >> >> In our case: >> L1 Ethernet HDR & FTR 44B >> IPv4 header 20B >> UDP header 8B >> Application header 32B >> Data payload 5120B >> ====== >> 5224B >> >> We normally run each link at 8.733Gbps raw data rate, plus M&C data, ARP and >> all the rest. But I've done tests showing it working well up to about 96%. >> Beyond that I started losing lots of packets. >> >> Jason Manley >> CBF Manager >> SKA-SA >> >> Cell: +27 82 662 7726 >> Work: +27 21 506 7300 >> >> On 13 Mar 2014, at 20:08, Jack Hickish <[email protected]> wrote: >> >>> Hi all, >>> >>> I have an application that generates 40 Gb/s of data and I'd like to >>> output it over 4 x 10GbE links (i.e. one roach2 mezzanine card). >>> >>> I'm going to have to throw some data away to make room for packetising >>> overhead, and right now I'm trying to work out how much, to see >>> whether I should just go to an dual-mezzanine card setup. Is anyone >>> with experience operating ROACH2 10GbE networks at near to peak >>> capacity able to give me an idea of what throughput I might be able to >>> get once I factor in overheads? For what it's worth, I'm using packet >>> sizes of ~1kB in size in a network of 10 ROACH2s connected to a 48 >>> port switch. >>> >>> Thanks, >>> >>> Jack >>> >>
