Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
From: Giovanni Tirloni [mailto:gtirl...@sysdroid.com] We have hundreds of servers using LACP and so far have not noticed any increase in the error rate. If the error rate is not zero, you have an increased error rate. In linux, you just do this: sudo /sbin/ifconfig -a | grep errors | grep -v 'errors:0 ' All of my servers right now have error counter = 0 cumulative since last reboot. In solaris, I had to figure out the command once before, but I forget it now. I remember being misled ... Seeing one error counter stay at zero, while another error counter increased. While we were trying to enable LACP, we'd get one error packet per ... I guess every 1Gb or so that traversed the LACP link. This was fatal to ssh connections. So unless you see a nonzero error counter, I'm not sure how to tell you which error counter to check. Sorry I couldn't be more informative on that ... Could you share what implementations (OS, switch) have you tested and how it was done ? I would like to try to simulate these issues. I've done this on Dell switches, almost exclusively. I've done it with Linux, Solaris 10u6 x86, and I think one Netapp. So no, I cannot exclude Dell switches as the possible cause. I revisit and reattempt once every few years, always with the same result. I simply do not use LACP anywhere. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
On Tue, Jul 20, 2010 at 12:59 AM, Edward Ned Harvey sh...@nedharvey.com wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Richard Jahnel I'vw also tried mbuffer, but I get broken pipe errors part way through the transfer. The standard answer is mbuffer. I think you should ask yourself what's going wrong with mbuffer. You're not, by any chance, sending across a LACP aggregated link, are you? Most people don't notice it, but I sure do, that usually LACP introduces packet errors. Just watch your error counter, and start cramming data through there. So far I've never seen a single implementation that passed this test... Although I'm sure I've just had bad luck. If you're having some packet loss, that might explain the poor performance of ssh too. Although ssh is known to slow things down in the best of cases ... I don't know if the speed you're seeing is reasonable considering. We have hundreds of servers using LACP and so far have not noticed any increase in the error rate. Could you share what implementations (OS, switch) have you tested and how it was done ? I would like to try to simulate these issues. -- Giovanni Tirloni gtirl...@sysdroid.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
On the receiver /opt/csw/bin/mbuffer -m 1G -I Ostor-1:8000 | zfs recv -F e...@sunday in @ 0.0 kB/s, out @ 0.0 kB/s, 43.7 GB total, buffer 100% fullcannot receive new filesystem stream: invalid backup stream mbuffer: error: outputThread: error writing to stdout at offset 0xaedf6a000: Broken pipe summary: 43.7 GByte in 3 min 25.5 sec - average of 218 MB/s mbuffer: warning: error during output to stdout: Broken pipe On the sender /sbin/zfs send e...@sunday | /opt/csw/bin/mbuffer -m 1G -O Ostor-2:8000 in @ 0.0 kB/s, out @ 0.0 kB/s, 44.7 GB total, buffer 100% fullmbuffer: error: outputThread: error writing to Ostor-2:8000 at offset 0xb2e0e6000: Broken pipe summary: 44.7 GByte in 3 min 25.6 sec - average of 223 MB/s mbuffer: warning: error during output to Ostor-2:8000: Broken pipe -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
I'll try an export/import and scrub of the receiving pool and see what that does. I can't take the sending pool offline to try that stuff though. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
On 07/21/10 03:12 AM, Richard Jahnel wrote: On the receiver /opt/csw/bin/mbuffer -m 1G -I Ostor-1:8000 | zfs recv -F e...@sunday in @ 0.0 kB/s, out @ 0.0 kB/s, 43.7 GB total, buffer 100% fullcannot receive new filesystem stream: invalid backup stream mbuffer: error: outputThread: error writing tostdout at offset 0xaedf6a000: Broken pipe It looks like zfs receive found an error in the received stream and broke the connection, hence the SIGPIPE on the sender. Try streaming the receive data to a file rather than zfs receive and see if the full stream is received. You could do the same on the sander and compare (md5sum) the two files. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
I've tried ssh blowfish and scp arcfour. both are CPU limited long before the 10g link is. I'vw also tried mbuffer, but I get broken pipe errors part way through the transfer. I'm open to ideas for faster ways to to either zfs send directly or through a compressed file of the zfs send output. For the moment I; zfs send pigz scp arcfour the file gz file to the remote host gunzip to zfs receive This takes a very long time for 3 TB of data, and barely makes use the 10g connection between the machines due to the CPU limiting on the scp and gunzip processes. Thank you for your thoughts Richard J. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
Richard Jahnel wrote: I've tried ssh blowfish and scp arcfour. both are CPU limited long before the 10g link is. I'vw also tried mbuffer, but I get broken pipe errors part way through the transfer. Any idea why? Does the zfs send or zfs receive bomb out part way through? Might be worth trying it over rsh if security isn't an issue, and then you lose the encryption overhead. Trouble is that then you've got almost no buffering, which can do bad things to the performance, which is why mbuffer would be ideal if it worked for you. I'm open to ideas for faster ways to to either zfs send directly or through a compressed file of the zfs send output. For the moment I; zfs send pigz scp arcfour the file gz file to the remote host gunzip to zfs receive This takes a very long time for 3 TB of data, and barely makes use the 10g connection between the machines due to the CPU limiting on the scp and gunzip processes. Also, if you have multiple datasets to send, might be worth seeing if sending them in parallel helps. -- Andrew Gabriel ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
If this is across a trusted link, have a look at the HPN patches to ZFS. There are three main benefits to these patches: - increased (and dynamic) buffers internal to SSH - adds a multi-threaded aes cipher - adds the NONE cipher for non-encrypted data transfers (authentication is still encrypted) Yes I've looked at that site before. While I'm comfortable with the zfs storage aspects of osol, I'm still quite leary of compiling from source code. I really really don't want to break these machines. If it's a package I can figure out how to get and install it, but code patches scare me. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
Hi, If you can share those scripts that make use of mbuffer, please feel free to do so ;) Bruno On 19-7-2010 20:02, Brent Jones wrote: On Mon, Jul 19, 2010 at 9:06 AM, Richard Jahnel rich...@ellipseinc.com wrote: I've tried ssh blowfish and scp arcfour. both are CPU limited long before the 10g link is. I'vw also tried mbuffer, but I get broken pipe errors part way through the transfer. I'm open to ideas for faster ways to to either zfs send directly or through a compressed file of the zfs send output. For the moment I; zfs send pigz scp arcfour the file gz file to the remote host gunzip to zfs receive This takes a very long time for 3 TB of data, and barely makes use the 10g connection between the machines due to the CPU limiting on the scp and gunzip processes. Thank you for your thoughts Richard J. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss I found builds 130 had issues with TCP. I could reproduce TCP timeouts/socket errors up until I got on 132. I have stayed on 132 so far since I haven't found any other show stoppers. Mbuffer is probably your best bet, I rolled mbuffer into my replication scripts, which I could share if anyone's interested. Older versions of my script are on www.brentrjones.com but I have a new one which uses mbuffer -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
Richard, On 19 Jul 2010, at 18:49, Richard Jahnel wrote: I heard of some folks using netcat. I haven't figured out where to get netcat nor the syntax for using it yet. I also did a bit of research into using netcat and found this... http://www.mail-archive.com/storage-disc...@opensolaris.org/msg05920.html sourcehost: zfs send | netcat $remotehost $remoteport desthost: netcat -l -p $myport | zfs receive Hope that helps, - Daniel ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
On Mon, Jul 19, 2010 at 11:14 AM, Bruno Sousa bso...@epinfante.com wrote: Hi, If you can share those scripts that make use of mbuffer, please feel free to do so ;) Bruno On 19-7-2010 20:02, Brent Jones wrote: On Mon, Jul 19, 2010 at 9:06 AM, Richard Jahnel rich...@ellipseinc.com wrote: I've tried ssh blowfish and scp arcfour. both are CPU limited long before the 10g link is. I'vw also tried mbuffer, but I get broken pipe errors part way through the transfer. I'm open to ideas for faster ways to to either zfs send directly or through a compressed file of the zfs send output. For the moment I; zfs send pigz scp arcfour the file gz file to the remote host gunzip to zfs receive This takes a very long time for 3 TB of data, and barely makes use the 10g connection between the machines due to the CPU limiting on the scp and gunzip processes. Thank you for your thoughts Richard J. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss I found builds 130 had issues with TCP. I could reproduce TCP timeouts/socket errors up until I got on 132. I have stayed on 132 so far since I haven't found any other show stoppers. Mbuffer is probably your best bet, I rolled mbuffer into my replication scripts, which I could share if anyone's interested. Older versions of my script are on www.brentrjones.com but I have a new one which uses mbuffer I can't seem to upload files to my Wordpress site any longer, so I put it up on Pastebin for now: http://pastebin.com/2feviTCy Hope it helps others -- Brent Jones br...@servuhome.net ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
On 19-7-2010 20:36, Brent Jones wrote: On Mon, Jul 19, 2010 at 11:14 AM, Bruno Sousa bso...@epinfante.com wrote: Hi, If you can share those scripts that make use of mbuffer, please feel free to do so ;) Bruno On 19-7-2010 20:02, Brent Jones wrote: On Mon, Jul 19, 2010 at 9:06 AM, Richard Jahnel rich...@ellipseinc.com wrote: I've tried ssh blowfish and scp arcfour. both are CPU limited long before the 10g link is. I'vw also tried mbuffer, but I get broken pipe errors part way through the transfer. I'm open to ideas for faster ways to to either zfs send directly or through a compressed file of the zfs send output. For the moment I; zfs send pigz scp arcfour the file gz file to the remote host gunzip to zfs receive This takes a very long time for 3 TB of data, and barely makes use the 10g connection between the machines due to the CPU limiting on the scp and gunzip processes. Thank you for your thoughts Richard J. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss I found builds 130 had issues with TCP. I could reproduce TCP timeouts/socket errors up until I got on 132. I have stayed on 132 so far since I haven't found any other show stoppers. Mbuffer is probably your best bet, I rolled mbuffer into my replication scripts, which I could share if anyone's interested. Older versions of my script are on www.brentrjones.com but I have a new one which uses mbuffer I can't seem to upload files to my Wordpress site any longer, so I put it up on Pastebin for now: http://pastebin.com/2feviTCy Hope it helps others Thanks mate! Bruno -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
On Jul 19, 2010, at 10:49 AM, Richard Jahnel wrote: Any idea why? Does the zfs send or zfs receive bomb out part way through? I have no idea why mbuffer fails. Changing the -s from 128 to 1536 made it take longer to occur and slowed it down bu about 20% but didn't resolve the issue. It just ment I might get as far as 2.5gb before mbuffer bombed with broken pipe. Trying -r and -R with various values had no effect. I've used mbuffer to transfer hundreds of TB without a problem in mbuffer itself. You will get disconnected if the send or receive prematurely ends, though. Might be worth trying it over rsh if security isn't an issue, and then you lose the encryption overhead. Trouble is that then you've got almost no buffering, which can do bad things to the performance, which is why mbuffer would be ideal if it worked for you. I seem to remember reading that rsh was remapped to ssh in Solaris. I heard of some folks using netcat. I haven't figured out where to get netcat nor the syntax for using it yet. On Nexenta systems, netcat (and nc) are in /usr/bin and the man pages are in the default MANPATH. -- richard -- Richard Elling rich...@nexenta.com +1-760-896-4422 Enterprise class storage for everyone www.nexenta.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
I've used mbuffer to transfer hundreds of TB without a problem in mbuffer itself. You will get disconnected if the send or receive prematurely ends, though. mbuffer itself very specifically ends with a broken pipe error. Very quickly with s set to 128 or after sometime with s set over 1024. My only thought at this point is it maybe something with either the myri10ge driver or the fact the machines are directly connected with out the use of an intervening switch. Perhaps someone over in networking my have a thought in that direction. Richard J. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
Using SunOS X 5.11 snv_133 i86pc i386 i86pc. So the network thing that was fixed in 129 shouldn't be the issue. -Original Message- From: Brent Jones [mailto:br...@servuhome.net] Sent: Monday, July 19, 2010 1:02 PM To: Richard Jahnel Cc: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh? On Mon, Jul 19, 2010 at 9:06 AM, Richard Jahnel rich...@ellipseinc.com wrote: I've tried ssh blowfish and scp arcfour. both are CPU limited long before the 10g link is. I'vw also tried mbuffer, but I get broken pipe errors part way through the transfer. I'm open to ideas for faster ways to to either zfs send directly or through a compressed file of the zfs send output. For the moment I; zfs send pigz scp arcfour the file gz file to the remote host gunzip to zfs receive This takes a very long time for 3 TB of data, and barely makes use the 10g connection between the machines due to the CPU limiting on the scp and gunzip processes. Thank you for your thoughts Richard J. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss I found builds 130 had issues with TCP. I could reproduce TCP timeouts/socket errors up until I got on 132. I have stayed on 132 so far since I haven't found any other show stoppers. Mbuffer is probably your best bet, I rolled mbuffer into my replication scripts, which I could share if anyone's interested. Older versions of my script are on www.brentrjones.com but I have a new one which uses mbuffer -- Brent Jones br...@servuhome.net ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
FWIW I found netcat over at CSW. http://www.opencsw.org/packages/CSWnetcat/ -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
On 07/20/10 08:20 AM, Richard Jahnel wrote: I've used mbuffer to transfer hundreds of TB without a problem in mbuffer itself. You will get disconnected if the send or receive prematurely ends, though. mbuffer itself very specifically ends with a broken pipe error. Very quickly with s set to 128 or after sometime with s set over 1024. My only thought at this point is it maybe something with either the myri10ge driver or the fact the machines are directly connected with out the use of an intervening switch. I assume the error comes from the sending side? That may mean the receiving process has died. Are there any errors (do you use -v for send and or receive?) reported? I don't use mbuffer, but do use large buffers to couple zfs sends to receives and I've never seen a broken pipe. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Richard Jahnel I'vw also tried mbuffer, but I get broken pipe errors part way through the transfer. The standard answer is mbuffer. I think you should ask yourself what's going wrong with mbuffer. You're not, by any chance, sending across a LACP aggregated link, are you? Most people don't notice it, but I sure do, that usually LACP introduces packet errors. Just watch your error counter, and start cramming data through there. So far I've never seen a single implementation that passed this test... Although I'm sure I've just had bad luck. If you're having some packet loss, that might explain the poor performance of ssh too. Although ssh is known to slow things down in the best of cases ... I don't know if the speed you're seeing is reasonable considering. zfs send pigz scp arcfour the file gz file to the remote host gunzip to zfs receive I would recommend xz -1 instead of pigz. It's roughly the same speed compression, with approx 2x smaller result. I know xz does multi-threading, but you might just want to top it or prstat it to be sure. I have a hunch that it can only multi-thread actual files, and maybe can't multi-thread compression of stdin. Maybe. This takes a very long time for 3 TB of data, and barely makes use the 10g connection between the machines due to the CPU limiting on the scp and gunzip processes. That's something ... I know pigz, pbzip2, and xz all compress multi-threaded. But what about uncompress? You say your gunzip is cpu-bound... If the above don't decompress multi-threaded, I know threadzip does multi-thread both compression decompression. For speed, I would recommend using the --fast option. If you consider threadzip, just ignore the comment about this project is considered dead, in favor of pigz, pbzip2, xz. I'm the guy behind threadzip, and if there's a reason for it to exist, I'll just put that on the page instead. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send to remote any ideas for a faster way than ssh?
On Mon, Jul 19, 2010 at 11:06 PM, Richard Jahnel rich...@ellipseinc.com wrote: I've tried ssh blowfish and scp arcfour. both are CPU limited long before the 10g link is. I'vw also tried mbuffer, but I get broken pipe errors part way through the transfer. I'm open to ideas for faster ways to to either zfs send directly or through a compressed file of the zfs send output. For the moment I; zfs send pigz scp arcfour the file gz file to the remote host gunzip to zfs receive This takes a very long time for 3 TB of data, and barely makes use the 10g connection between the machines due to the CPU limiting on the scp and gunzip processes. I usually do zfs send | lzop -c | ssh -c blowfish ip_of_remote_server lzop -dc | zfs receive lzop is much faster than gzip. I'd also check how fast your disks are, make sure it's not the bottleneck. -- Fajar ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
