Doug VanLeuven wrote:
OK, I'll top post.
I can't let this stand unanswered.
I ran a LOT of tests with gigabit copper and windows machines. I never
did better than 40 seconds per gig. That was with the Intel cards
configured for maximum cpu utilization. 80-90% cpu for 40 sec per gig.
On windows. Uploads went half as fast. Asymetric. Of course I only
had 32 bit PCI, 2.5Gig processor motherboards with 45MBps drives.
Which leads me to my point. One can't rationally compare performance of
gigabit ethernet without talking about hardware on the platforms. I
wouldn't think you'd have overlooked this, but one can bump up against
the speed of the disk drive. Raid has overhead. Have you tried
something like iostat? Serial ATA? I seem to recall the folks at
Enterasys indicating 300Gbps as a practical upper limit on copper gig.
Are you using fiber? 64 bit PCI? Who made which model of the network
card? Is it a network card that's well supported in Linux? Can you
change the interrupt utilization of the card? What's the CPU
utilization on the Redhat machine during transfers?
>
> I don't have specific answers for your questions, but one can't just say
> this software product is slower on gigabit than the other one without
> talking hardware at the same time.
You have a very good point: I never indicated what my hardware
situation was.
Server: Rackable UltraDense. It's an Opteron 250, 2GB RAM, a 3Ware RAID
controller and 12x 500GB SATA disks (about 460GB formatted) in 2x 6 disk
RAID5 arrays (a little space wasted due to a 2TB limit somewhere.)
Ethernet is a BroadCom BCM85702A20 gigabit (two of them, actually, but
we're only using one.)
I've used a number of different clients, ranging from a Dell 850 copying
to /dev/null, to a Dell OptiPlex GX620 copying to a local SATA drive, to
another Rackable UltraDense. Both Linux and WinXP. (Not so
surprisingly, the Linux client is slower than the WinXP client.
Although, using smbclient (as Jeremy suggested) was just as fast as the
WinXP client, our famous 45 second 1GB transfer.)
Reasons I didn't list hardware in my first email:
- iPerf shows that I can saturate the Ethernet interfaces, TCP/IP stack,
and switching fabric to 120MBps, 960Mbps.
- Copying the same file to/from the same machines using HTTP (Apache2)
transfers at about 60MBps, 480Mbps. This uses the same disk and network
subsystems.
- Copying a 1GB file from a RAM disk on the server to /dev/null on the
client (eliminating disk performance from the equation entirely) does
_NOT_ speed things up at all, still stuck at about 45 seconds, about
23MBps, 182Mbps.
- Copying locally from the disk to /dev/null (using dd, no network at
all) takes about 17 seconds for a 1GB file, which matches up nicely with
the 60MBps, 480Mbps seen with HTTP.
Given these tests, I would expect to see transfer rates of up to 60MBps
in the best case. Admittedly, that is a _BEST_ case. I know I can't
avoid that bottle neck, and honestly, that would be totally sufficient
for our use.
The question is, what bottle neck am I hitting now? The only thing that
changes between the HTTP and SMB tests are the transport mechanisms (and
their interactions with other systems, eg: kernel), so naturally I
suspect those. For the time being, at least, I need to use the SMB
protocol. So I'm trying to figure out what I can tweak, if anything, to
make this go faster.
As a data point, I'm going to try a newer version of Samba. (RHEL4 uses
3.0.10-RedHat-Heavily-Modified-Of-Course) If that makes a difference,
then I have to decide whether it's worth it to me to keep RedHat support
or not. (And when I say "I," I really mean "my management.")
I have lots of memory. I use these configurations in sysctl.conf to up
the performance of send/recieve windows on my systems. There's articles
out there. I don't have historical references handy.
YMMV.
net.core.wmem_max = 1048576
net.core.rmem_max = 1048576
net.ipv4.tcp_wmem = 4096 65536 1048575
net.ipv4.tcp_rmem = 4096 524288 1048575
net.ipv4.tcp_window_scaling = 1
I have not tried tweaking the TCP stack in the OS. I'll give that a shot.
Thank you very much, Doug.
-Mark
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/listinfo/samba
