Jason: This all looks familiar to this old hand, but it's been a while. So let me ask some dumb questions and get the solution explicit. Your suggestion is to put those configuration options in, say, /etc/rc.local with the caveats below?
1. sysctl -w before each option 2. tuned to the actual amount of RAM free and data rate we expect Since this is a kernel-wide option, does this also do the job of increasing the UDP receive RAM buffer size in a Python client? Or is there something I should also be doing in my actual data receiving program, i.e. socket.setsockopt()? Tom On Mon, Jun 20, 2011 at 12:01 PM, Jason Manley <[email protected]> wrote: > PAPER do this very thing with a 48-port Netgear GS748TAU. I usually use > packets with payload of 4096 or 8192 bytes, along with a small header. Up to > 9kB should be fine on just about any 1GbE switch these days. > > Remember that you can't have jumbo packets on 100mbps Ethernet. So if any of > the traffic routes through a 100mbps link, there will be fragmentation and > it'll slaughter your throughput. > > Remember that some ROACH boards' PHYs don't work at 1000mbps. > > Remember to tune your Linux network stack for lossless UDP packets. Assuming > you have the RAM to spare... > net.core.netdev_max_backlog = 2000 > net.core.wmem_max = 838860800 > net.core.rmem_max = 838860800 > net.core.rmem_default = 819260800 > net.ipv4.udp_rmem_min = 8192000 > > > Jason > > On 20 Jun 2011, at 19:51, John Ford wrote: > >> Hi Tom. >> >> Most newer switches support jumbo frames, and most support ~9000 byte >> packets. I say most because you just have to look at the feature set in >> the data sheet. Some of the inexpensive Netgear 5,8, and 16 port switches >> support a packet size of from 9000 to 10240. Another problem might be the >> amount of buffer space. Large packets need more buffer space, so you have >> to make sure that there is enough to avoid dropped packets, particularly >> since your ROACH boards might be operating in synchrony, with packets from >> all 16 coming in at once to be forwarded to one host. The netgear $160.00 >> GS116 has 512K bytes per port, which seems like plenty... >> >> If you try to toss these packets over the internet or through routers, you >> may be sunk. But you should be able to do fine if you control the >> switches and host adapters. >> >> We're interested in doing this same thing with some of our lower-bandwidth >> spectrometer projects. I'd like to hear more about your project as it >> gets finished! >> >> John >> >>> Casper folks: >>> >>> I've finally started working in earnest on a PPC-side C server that >>> reads out FPGA/PPC shared memory and sends out UDP packets over the >>> 1Gbe interface. The idea being that our data rate is so much below >>> 10GBe (about 1 megabyte per second per board) that it did not merit >>> investing in the 10Gbe switches necessary to readout 10-20 boards at a >>> time. Also, the experiment is expected to be running 5 years from now >>> and these switches are already hard to find and expensive. >>> >>> I've got this scheme basically working and want to ask what I hope are >>> a couple reasonably simple questions that people out there may have >>> experience with. What is the typical largest UDP packet size that will >>> be passed by standard network hardware? Does anyone out there have >>> experience buying a 1Gbe switch with ~20 ports that has a good >>> configuration interface to basically ensure 100% passage of ~6kB UDP >>> packets? >>> >>> My impression from CASPER Memo 21 is that a typical router will pass >>> UDP packets up to 9kb (maybe 8192?). My experience while writing the C >>> server is that packets above this size sometimes get through the LAN >>> but most times not. I have managed to divide our data into packets of >>> 6208 bytes in size and they seem to get through every time. But I want >>> to make sure that the UDP packets really, truly get through every >>> time, not 99.9%. >>> >>> I understand that the UDP standard is not designed around ensuring >>> 100% delivery of information. I am mostly concerned that routers can >>> prioritize certain kinds of traffic over others. I don't want the >>> wrong choice of router to introduce problems. >>> >>> The data will be passing through a private LAN comprised of a standard >>> linux machine receiving UDP packets from approximately 16 ROACH >>> boards. So we can buy a router that is very appropriate to this task >>> rather than generic network conditions. i.e. turn off anti-virus >>> scanning, firewall, etc, etc. >>> >>> My testing thus far has been across a LAN shared with many computers at >>> Caltech. >>> >>> Tom >>> >> >> >> > >
