PAPER do this very thing with a 48-port Netgear GS748TAU. I usually use packets with payload of 4096 or 8192 bytes, along with a small header. Up to 9kB should be fine on just about any 1GbE switch these days.
Remember that you can't have jumbo packets on 100mbps Ethernet. So if any of the traffic routes through a 100mbps link, there will be fragmentation and it'll slaughter your throughput. Remember that some ROACH boards' PHYs don't work at 1000mbps. Remember to tune your Linux network stack for lossless UDP packets. Assuming you have the RAM to spare... net.core.netdev_max_backlog = 2000 net.core.wmem_max = 838860800 net.core.rmem_max = 838860800 net.core.rmem_default = 819260800 net.ipv4.udp_rmem_min = 8192000 Jason On 20 Jun 2011, at 19:51, John Ford wrote: > Hi Tom. > > Most newer switches support jumbo frames, and most support ~9000 byte > packets. I say most because you just have to look at the feature set in > the data sheet. Some of the inexpensive Netgear 5,8, and 16 port switches > support a packet size of from 9000 to 10240. Another problem might be the > amount of buffer space. Large packets need more buffer space, so you have > to make sure that there is enough to avoid dropped packets, particularly > since your ROACH boards might be operating in synchrony, with packets from > all 16 coming in at once to be forwarded to one host. The netgear $160.00 > GS116 has 512K bytes per port, which seems like plenty... > > If you try to toss these packets over the internet or through routers, you > may be sunk. But you should be able to do fine if you control the > switches and host adapters. > > We're interested in doing this same thing with some of our lower-bandwidth > spectrometer projects. I'd like to hear more about your project as it > gets finished! > > John > >> Casper folks: >> >> I've finally started working in earnest on a PPC-side C server that >> reads out FPGA/PPC shared memory and sends out UDP packets over the >> 1Gbe interface. The idea being that our data rate is so much below >> 10GBe (about 1 megabyte per second per board) that it did not merit >> investing in the 10Gbe switches necessary to readout 10-20 boards at a >> time. Also, the experiment is expected to be running 5 years from now >> and these switches are already hard to find and expensive. >> >> I've got this scheme basically working and want to ask what I hope are >> a couple reasonably simple questions that people out there may have >> experience with. What is the typical largest UDP packet size that will >> be passed by standard network hardware? Does anyone out there have >> experience buying a 1Gbe switch with ~20 ports that has a good >> configuration interface to basically ensure 100% passage of ~6kB UDP >> packets? >> >> My impression from CASPER Memo 21 is that a typical router will pass >> UDP packets up to 9kb (maybe 8192?). My experience while writing the C >> server is that packets above this size sometimes get through the LAN >> but most times not. I have managed to divide our data into packets of >> 6208 bytes in size and they seem to get through every time. But I want >> to make sure that the UDP packets really, truly get through every >> time, not 99.9%. >> >> I understand that the UDP standard is not designed around ensuring >> 100% delivery of information. I am mostly concerned that routers can >> prioritize certain kinds of traffic over others. I don't want the >> wrong choice of router to introduce problems. >> >> The data will be passing through a private LAN comprised of a standard >> linux machine receiving UDP packets from approximately 16 ROACH >> boards. So we can buy a router that is very appropriate to this task >> rather than generic network conditions. i.e. turn off anti-virus >> scanning, firewall, etc, etc. >> >> My testing thus far has been across a LAN shared with many computers at >> Caltech. >> >> Tom >> > > >
