So, here's the answer to the puzzle... I did as "Per" suggested -- set the so_sndbuf and so_rcvbuf to 65535 each, as well as read size and max xmit to the same. And low and behold, the throughput between my Windows XP box and my Linux Box's Firewire RAID 5 arrays went from around 20 MB/sec to around 30 MB/sec. And when I raised all those values by another factor of 4 (up to 262140 for each one) my throughput increased to around 36-37 MB/sec. Raising those values further didn't seem to make any difference.
And, and, and, ... at the 262140 level, now I was able to set the MTUs to 4088 on the Windows side and 4074 on the Linux side (4K Jumbo Frames). While throughput essentially remained the same as what I was getting with a standard 1514/1500 MTU, I was happy that it didn't DROP by 50 percent, as it did before when I was trying to use 4K and 9K Jumbo Frames with the buffers set to 8092. And although throughput didn't go UP with the 4K frames, the CPU usage in each machine was pretty much cut in half. That made my Windows XP video editing applications happy. Incidentally, I still get a serious DROP in throughput, about 50 percent, if I try to use 9K Jumbo Frames -- even with those big buffers (I even tried up to 1 MB buffers). Perhaps that could be due to the fact that both my Windows and Linux boxes only have 32-bit PCI slots. I should add a few other notes. First, I am using Intel Gigabit Server NICs because they seem to have good Linux drivers -- and the Intel Tech Support folks were totally blown away by the fact that a few variables in a Samba configuration file could affect network throughput so dramatically. Alas, Samba is a mystery to many folks in this world -- myself included. Second, after hours of playing with TCP/IP settings on both the Windows and Linux sides -- TcpWindowSize in Windows, and a bunch of wmem, rmem, and mem values in Linux -- I came to the conclusion that none of my changes gave me any improvement over the default configurations of both operating systems (at least the way Linux is configured in Mandrake 9.2). Before I started tweaking, I had tried all sorts of Window sizes using the program "Iperf" on both ends -- and found pretty much that I got to the maximum throughput (a pure bandwidth of about 780 Mbits) at about 64K TCPWindows. But both OS's must do that by default, so tweaking was a waste of time in my case, where I have a fast LAN with almost zero latency. Finally, I have to say that I am impressed with the Intel Gigabit Server Ethernet adapters. The Intel e1000 drivers (not the eepro1000 driver that came with my Mandrake 9.2 distribution) give you fantastic flexibility in both Linux and Windows. For instance, the driver has a number of methods to let you choose whether the NIC interrupts the CPU each time it receives or transmits a packet, or whether the NIC stores up lots of packets and interrupts less frequently. Playing around with that and other settings in the driver allowed me to trade a 15 percent reduction in maximum throughput (and a slight increase in latency) for a 75 percent reduction in CPU usage! Which let my Windows video editing application flawlessly playback uncompressed video through some hardware that's also vying for attention on the PCI bus. Without the ability to do this kind of tweaking, my video editing application was getting interrupted too often to work properly. Surely there are other real time applications that can benefit from such control. I certainly had a happier experience with the Intel NIC than with either Linksys or SysKonnect NICs -- although I hear the SysKonnect NICs are a little faster than most. Hope all this information is useful to somebody out there. As a last word, of course my switch supports Jumbo Frames. I'm trying out a fairly new SMC switch, the SMC8508T. It's fast, cheap and out of control... No just kidding. It's fast and cheap (around $ 140 US) and as far as I can see it's the only switch any where near that price that supports Jumbo Frames. Says so right on the box. It's an unmanaged switch. But I know it's not the switch that's caused my original problems with Jumbo Frames, because I had the same problems when I took the switch out of the system and connected the computers directly. And that's still the case with the 9K Jumbo Frames. Finally, I put "use sendfile=yes" in my samba configuration file. But what does it do??? I suppose I should at least try taking it out and seeing what it does. Thanks for your suggestions. It's because of people like you that Linux keeps getting better, and Linux users keep getting better results. Andy In a message dated 12/17/2003 10:10:10 AM Eastern Standard Time, [EMAIL PROTECTED] writes: I believe samba just does setsockopt or ioctl on the sockets. Do you get any errors on the interfaces in jumbo? Does your switch support jumbo? Setting "use sendfile=yes" will help alot on read speeds from samba. On the windows side check the settings. I think the e1000 has some adaptive spacing setting that kills throughput. Also some things to check on the linux side. e1000 module options like rxIntDelay, etc. You will not get much more performance out of jumbo unless your CPUs are maxed but you should not get less. What kind of numbers are you seeing? hope this helps. Greg On Wednesday 17 December 2003 08:23 am, [EMAIL PROTECTED] wrote: > Thanks for the reply. Do you know (and if so, caan you tell me) what the > relationship is between these Samba settings and Linux settings such as > net.core.rmem_default (or _max), net.core.wmem_default (or _max), > net.ipv4.tcp_rmem and net.ipv4.tcp_wmem. Do the Samba options override the > Linux socket options, or do they act as another layer of limits and > buffers? > > Perhaps your TCP window is too small > You should try the following global settings: > > read size = 65535 > max xmit = 65535 > socket options = TCP_NODELAY SO_SNDBUF=65535 SO_RCVBUF=65535 > > > Rgds Per > > [EMAIL PROTECTED] wrote: > > Hi, > > > > I am trying to optimize my gigabit network. I have two Intel 1000 MT > > Gigabit Server Adapters, which support Jumbo Frames -- as well as a > > Switch that supports Jumbo Frames. However, I am observing some strange > > behavior in my > > file > > > transfers from Windows XP to Linux and I am wondering if it has anything > > to > > do > > > with the way the Samba variables are set on my Linux box? > > > > The "strange behavior" is that when I set both NICs to use Jumbo Frames > > [MTU=9014 on the Windows side (includes IP headers) , 9000 on the Linux > > side (doesn't include the headers], I am getting about half the > > throughput that > > I get > > > when I set both NICs to use the standard MTU of 1514/1500. I see the same > > behavior even if I take the switch out of the system and connect the > > Windows XP and > > > Linux machines directly to each other (crossover cable not required for > > computer-to-computer connection with these NICs -- and by the way all of > > my > > cables > > > are CAT6). > > > > On the Linux side, I am using Samba 3.0.0 on Mandrake Linux 9.2 with all > > of Mandrake's current updates -- kernel = 2.4.22-21enterprisemdk. The > > Linux machine is a P4-3.06 Ghz with 1 GB of RAM -- running in > > hyperthreading mode. > > > > I am wondering if any of the Samba socket options settings like > > tcp_nodelay, so_sndbuf=8192 or so_rcvbuf=8192 are affecting my throughput > > -- particularly when I am using Jumbo Frames? And are there any other > > Samba settings that > > might > > > be interacting in a negative way with my TCP/IP and NIC driver settings > > that are causing me to get lower throughput with Jumbo Frames instead of > > higher throughput (which is what I am told I should be getting). > > > > Any guidance would be appreciated. I have purchased "The Official Samba 3 > > HOW-TO and Reference Guide" but it really isn't very helpful when it > > comes > > to > > > understanding how to tune these options and how various socket options > > settings > > > interact with other network settings and hardware. > > > > Andy Liebman > > Resolute Films > > 119 Braintree Street, Suite 410 > > Boston, MA 02134 > > > > Tel: 617-782-0479 > > Cell: 617-308-0488 > > Fax: 617-782-1071 > > -- > > To unsubscribe from this list go to the following URL and read the > > instructions: http://lists.samba.org/mailman/listinfo/samba -- To unsubscribe from this list go to the following URL and read the instructions: http://lists.samba.org/mailman/listinfo/samba
