Hi Tom. Most newer switches support jumbo frames, and most support ~9000 byte packets. I say most because you just have to look at the feature set in the data sheet. Some of the inexpensive Netgear 5,8, and 16 port switches support a packet size of from 9000 to 10240. Another problem might be the amount of buffer space. Large packets need more buffer space, so you have to make sure that there is enough to avoid dropped packets, particularly since your ROACH boards might be operating in synchrony, with packets from all 16 coming in at once to be forwarded to one host. The netgear $160.00 GS116 has 512K bytes per port, which seems like plenty...
If you try to toss these packets over the internet or through routers, you may be sunk. But you should be able to do fine if you control the switches and host adapters. We're interested in doing this same thing with some of our lower-bandwidth spectrometer projects. I'd like to hear more about your project as it gets finished! John > Casper folks: > > I've finally started working in earnest on a PPC-side C server that > reads out FPGA/PPC shared memory and sends out UDP packets over the > 1Gbe interface. The idea being that our data rate is so much below > 10GBe (about 1 megabyte per second per board) that it did not merit > investing in the 10Gbe switches necessary to readout 10-20 boards at a > time. Also, the experiment is expected to be running 5 years from now > and these switches are already hard to find and expensive. > > I've got this scheme basically working and want to ask what I hope are > a couple reasonably simple questions that people out there may have > experience with. What is the typical largest UDP packet size that will > be passed by standard network hardware? Does anyone out there have > experience buying a 1Gbe switch with ~20 ports that has a good > configuration interface to basically ensure 100% passage of ~6kB UDP > packets? > > My impression from CASPER Memo 21 is that a typical router will pass > UDP packets up to 9kb (maybe 8192?). My experience while writing the C > server is that packets above this size sometimes get through the LAN > but most times not. I have managed to divide our data into packets of > 6208 bytes in size and they seem to get through every time. But I want > to make sure that the UDP packets really, truly get through every > time, not 99.9%. > > I understand that the UDP standard is not designed around ensuring > 100% delivery of information. I am mostly concerned that routers can > prioritize certain kinds of traffic over others. I don't want the > wrong choice of router to introduce problems. > > The data will be passing through a private LAN comprised of a standard > linux machine receiving UDP packets from approximately 16 ROACH > boards. So we can buy a router that is very appropriate to this task > rather than generic network conditions. i.e. turn off anti-virus > scanning, firewall, etc, etc. > > My testing thus far has been across a LAN shared with many computers at > Caltech. > > Tom >
