Re: [zfs-discuss] diagnosing read performance problem

2008-10-31 Thread Matt Harrison
Ok, I have recieved a new set of NICs and a new switch and the problem still remains. Just for something to do I ran some tests: Copying a 200Mb file over scp from the main problem workstation to a totally unrelated gentoo linux box. Absolutely no problems. So I thought it was down to the zfs

Re: [zfs-discuss] diagnosing read performance problem

2008-10-31 Thread Matt Harrison
Well, somehow it's fixed: Since putting in the new Intel card, the transfer from the box dropped so badly, I couldn't even copy a snoop from it. So I removed the dohwchksum line from /etc/system and rebooted. Then just to clean up a bit I disabled the onboard NICs in the bios. Now I'm still

Re: [zfs-discuss] diagnosing read performance problem

2008-10-30 Thread Nigel Smith
Hi Matt Well this time you have filtered out any SSH traffic on port 22 successfully. But I'm still only seeing half of the conversation! I see packets sent from client to server. That is from source: 10.194.217.12 to destination: 10.194.217.3 So a different client IP this time And the Duplicate

Re: [zfs-discuss] diagnosing read performance problem

2008-10-30 Thread Matt Harrison
Nigel Smith wrote: Hi Matt Well this time you have filtered out any SSH traffic on port 22 successfully. But I'm still only seeing half of the conversation! Grr this is my day, I think I know what the problem was...user error as I'm not used to snoop. I see packets sent from client to

Re: [zfs-discuss] diagnosing read performance problem

2008-10-29 Thread Matt Harrison
On Tue, Oct 28, 2008 at 05:30:55PM -0700, Nigel Smith wrote: Hi Matt. Ok, got the capture and successfully 'unzipped' it. (Sorry, I guess I'm using old software to do this!) I see 12840 packets. The capture is a TCP conversation between two hosts using the SMB aka CIFS protocol.

Re: [zfs-discuss] diagnosing read performance problem

2008-10-29 Thread Matt Harrison
On Tue, Oct 28, 2008 at 05:45:48PM -0700, Richard Elling wrote: I replied to Matt directly, but didn't hear back. It may be a driver issue with checksum offloading. Certainly the symptoms are consistent. To test with a workaround see http://bugs.opensolaris.org/view_bug.do?bug_id=6686415

Re: [zfs-discuss] diagnosing read performance problem

2008-10-29 Thread Richard Elling
Matt Harrison wrote: On Tue, Oct 28, 2008 at 05:45:48PM -0700, Richard Elling wrote: I replied to Matt directly, but didn't hear back. It may be a driver issue with checksum offloading. Certainly the symptoms are consistent. To test with a workaround see

Re: [zfs-discuss] diagnosing read performance problem

2008-10-29 Thread Nigel Smith
Hi Matt Can you just confirm if that Ethernet capture file, that you made available, was done on the client, or on the server. I'm beginning to suspect you did it on the client. You can get a capture file on the server (OpenSolaris) using the 'snoop' command, as per one of my previous emails.

Re: [zfs-discuss] diagnosing read performance problem

2008-10-29 Thread Matt Harrison
On Wed, Oct 29, 2008 at 10:01:09AM -0700, Nigel Smith wrote: Hi Matt Can you just confirm if that Ethernet capture file, that you made available, was done on the client, or on the server. I'm beginning to suspect you did it on the client. That capture was done from the client You can get a

Re: [zfs-discuss] diagnosing read performance problem

2008-10-29 Thread Nigel Smith
Hi Matt In your previous capture, (which you have now confirmed was done on the Windows client), all those 'Bad TCP checksum' packets sent by the client, are explained, because you must be doing hardware TCP checksum offloading on the client network adaptor. WireShark will capture the packets

Re: [zfs-discuss] diagnosing read performance problem

2008-10-29 Thread Matt Harrison
On Wed, Oct 29, 2008 at 05:32:39PM -0700, Nigel Smith wrote: Hi Matt In your previous capture, (which you have now confirmed was done on the Windows client), all those 'Bad TCP checksum' packets sent by the client, are explained, because you must be doing hardware TCP checksum offloading

Re: [zfs-discuss] diagnosing read performance problem

2008-10-28 Thread Matt Harrison
On Mon, Oct 27, 2008 at 06:18:59PM -0700, Nigel Smith wrote: Hi Matt Unfortunately, I'm having problems un-compressing that zip file. I tried with 7-zip and WinZip reports this: skipping _1_20081027010354.cap: this file was compressed using an unknown compression method. Please

Re: [zfs-discuss] diagnosing read performance problem

2008-10-28 Thread Nigel Smith
Hi Matt. Ok, got the capture and successfully 'unzipped' it. (Sorry, I guess I'm using old software to do this!) I see 12840 packets. The capture is a TCP conversation between two hosts using the SMB aka CIFS protocol. 10.194.217.10 is the client - Presumably Windows? 10.194.217.3 is the server

Re: [zfs-discuss] diagnosing read performance problem

2008-10-28 Thread Richard Elling
I replied to Matt directly, but didn't hear back. It may be a driver issue with checksum offloading. Certainly the symptoms are consistent. To test with a workaround see http://bugs.opensolaris.org/view_bug.do?bug_id=6686415 -- richard Nigel Smith wrote: Hi Matt. Ok, got the capture and

Re: [zfs-discuss] diagnosing read performance problem

2008-10-27 Thread Nigel Smith
Hi Matt Unfortunately, I'm having problems un-compressing that zip file. I tried with 7-zip and WinZip reports this: skipping _1_20081027010354.cap: this file was compressed using an unknown compression method. Please visit www.winzip.com/wz54.htm for more information. The compression

Re: [zfs-discuss] diagnosing read performance problem

2008-10-26 Thread Matt Harrison
On Sat, Oct 25, 2008 at 06:50:46PM -0700, Nigel Smith wrote: Hi Matt What chipset is your PCI network card? (obviously, it not Intel, but what is it?) Do you know which driver the card is using? I believe it's some sort of Realtek (8139 probably). It's coming up as rtls0 You say '..The

Re: [zfs-discuss] diagnosing read performance problem

2008-10-26 Thread Nigel Smith
Ok on the answers to all my questions. There's nothing that really stands out as being obviously wrong. Just out of interest, what build of OpenSolaris are you using? One thing you could try on the Ethernet capture file, is to set the WireShark 'Time' column like this: View Time Display Format

Re: [zfs-discuss] diagnosing read performance problem

2008-10-26 Thread Matt Harrison
Nigel Smith wrote: Ok on the answers to all my questions. There's nothing that really stands out as being obviously wrong. Just out of interest, what build of OpenSolaris are you using? One thing you could try on the Ethernet capture file, is to set the WireShark 'Time' column like this:

Re: [zfs-discuss] diagnosing read performance problem

2008-10-26 Thread Matt Harrison
Nigel Smith wrote: Ok on the answers to all my questions. There's nothing that really stands out as being obviously wrong. Just out of interest, what build of OpenSolaris are you using? Damn forgot to add that, I'm running SXCE snv_97. Thanks Matt

Re: [zfs-discuss] diagnosing read performance problem

2008-10-25 Thread Matt Harrison
Bob Friesenhahn wrote: Other people on this list who experienced the exact same problem ultimately determined that the problem was with the network card. I recall that Intel NICs were the recommended solution. Note that 100MBit is now considered to be a slow link and PCI is also considered

Re: [zfs-discuss] diagnosing read performance problem

2008-10-25 Thread Bob Friesenhahn
On Sat, 25 Oct 2008, Matt Harrison wrote: The onboard ones haven't so much died (they still allow me to use them from the OS) but they just won't start up or accept there is a cable plugged in. The PCI nic does seem to be working and transfers to/from the server seem ok except when there's

Re: [zfs-discuss] diagnosing read performance problem

2008-10-25 Thread Matt Harrison
On Sat, Oct 25, 2008 at 11:10:42AM -0500, Bob Friesenhahn wrote: Hmmm, this may indicate that there is an ethernet cable problem. Use 'netstat -I interface' (where interface is the interface name shown by 'ifconfig -a') to see if the interface error count is increasing. If you are using a

Re: [zfs-discuss] diagnosing read performance problem

2008-10-25 Thread Nigel Smith
Hi Matt What chipset is your PCI network card? (obviously, it not Intel, but what is it?) Do you know which driver the card is using? You say '..The system was fine for a couple of weeks..'. At that point did you change any software - do any updates or upgrades? For instance, did you upgrade to a

[zfs-discuss] diagnosing read performance problem

2008-10-24 Thread Matt Harrison
Hi all, I've got a lot of video files on a zfs/cifs fileserver running SXCE. A little while ago the dual onboard NICs died and I had to replace them with a PCI 10/100 NIC. The system was fine for a couple of weeks but now the performance when viewing a video file from the cifs share is appauling.

Re: [zfs-discuss] diagnosing read performance problem

2008-10-24 Thread Bob Friesenhahn
On Sat, 25 Oct 2008, Matt Harrison wrote: I've got a lot of video files on a zfs/cifs fileserver running SXCE. A little while ago the dual onboard NICs died and I had to replace them with a PCI 10/100 NIC. The system was fine for a couple of weeks but now the performance when viewing a video