John / etc: The data rate is about 75kbytes/sec per board. The previous number I (mis-)quoted was the total data rate: about 1.2mbytes/sec.
So, what I'm taking away is that I want a buffer of the order of several hundred milliseconds so that the port connected to my linux box has enough buffer. It looks like your typical switch has a buffer shared between all ports of a couple to a few MB. However, the port you point out has a per-port buffer of 0.5MB. Is shared versus per-port an issue to be concerned about? Would it be better to have shared memory that is much larger? Essentially all data traffic will flow from 16 ROACH ports to one PC port if that matters. It looks like the Netgear GS724T is a comparable to the GS116 in that it has 0.5 mbyte buffer per port, but with 24 ports? This meets the buffer criteria stated above. The GS748 looks like it has 4MB shared. http://www.netgear.com/business/products/switches/smart-switches/gs724t.aspx http://www.netgear.com/business/products/switches/smart-switches/gs748t.aspx I'll share the code and readout scheme with you when it's a bit more ready to go. Shouldn't be too long. Tom On Mon, Jun 20, 2011 at 11:51 AM, John Ford <[email protected]> wrote: > Hi Tom. > > Most newer switches support jumbo frames, and most support ~9000 byte > packets. I say most because you just have to look at the feature set in > the data sheet. Some of the inexpensive Netgear 5,8, and 16 port switches > support a packet size of from 9000 to 10240. Another problem might be the > amount of buffer space. Large packets need more buffer space, so you have > to make sure that there is enough to avoid dropped packets, particularly > since your ROACH boards might be operating in synchrony, with packets from > all 16 coming in at once to be forwarded to one host. The netgear $160.00 > GS116 has 512K bytes per port, which seems like plenty... > > If you try to toss these packets over the internet or through routers, you > may be sunk. But you should be able to do fine if you control the > switches and host adapters. > > We're interested in doing this same thing with some of our lower-bandwidth > spectrometer projects. I'd like to hear more about your project as it > gets finished! > > John > >> Casper folks: >> >> I've finally started working in earnest on a PPC-side C server that >> reads out FPGA/PPC shared memory and sends out UDP packets over the >> 1Gbe interface. The idea being that our data rate is so much below >> 10GBe (about 1 megabyte per second per board) that it did not merit >> investing in the 10Gbe switches necessary to readout 10-20 boards at a >> time. Also, the experiment is expected to be running 5 years from now >> and these switches are already hard to find and expensive. >> >> I've got this scheme basically working and want to ask what I hope are >> a couple reasonably simple questions that people out there may have >> experience with. What is the typical largest UDP packet size that will >> be passed by standard network hardware? Does anyone out there have >> experience buying a 1Gbe switch with ~20 ports that has a good >> configuration interface to basically ensure 100% passage of ~6kB UDP >> packets? >> >> My impression from CASPER Memo 21 is that a typical router will pass >> UDP packets up to 9kb (maybe 8192?). My experience while writing the C >> server is that packets above this size sometimes get through the LAN >> but most times not. I have managed to divide our data into packets of >> 6208 bytes in size and they seem to get through every time. But I want >> to make sure that the UDP packets really, truly get through every >> time, not 99.9%. >> >> I understand that the UDP standard is not designed around ensuring >> 100% delivery of information. I am mostly concerned that routers can >> prioritize certain kinds of traffic over others. I don't want the >> wrong choice of router to introduce problems. >> >> The data will be passing through a private LAN comprised of a standard >> linux machine receiving UDP packets from approximately 16 ROACH >> boards. So we can buy a router that is very appropriate to this task >> rather than generic network conditions. i.e. turn off anti-virus >> scanning, firewall, etc, etc. >> >> My testing thus far has been across a LAN shared with many computers at >> Caltech. >> >> Tom >> > > >
