Re: problems with large transfers form solaris 2.6
On Fri, Sep 28, 2001 at 08:42:36AM +1000, Tomasz Ciolek wrote: > On Thu, Sep 27, 2001 at 09:48:18AM -0500, Dave Dykstra wrote: > > On Wed, Sep 26, 2001 at 10:46:51AM +1000, Tomasz Ciolek wrote: > > > Hi all. > > > > > > I have a large data set on solaris 2.6 (approx 16gb and about 900K files > > > and rectory structures). Rsync is misbehaving in the most obnoxious way: > > > > > > it is launched as such: rsync -e ssh -aLv --bwlimit=512 --timeout=300 > > > > > > > A transfer starts, a list of files and directories is built, and data > > > starts flowing. then for no apparent reason data stops flowing form the > > > rsync on the solaris end the the connection times out after 5 minutes of no > > > IO activity. When the timeout parameter is not there, the rsync will just > > > stop and hang around for hours untill killed. > > > > > > The only other this worth mentioning is that the solaris box is hidden > > > behind a firewall and we use a ssh proxy (TIS plug proxy) to connect to the > > > box. > > > > > > Addibd many v's on the commandline produces a lot of debug but no real > > > information as to whats going on. The data flow just stops and there is no > > > error messages on either end in rsync output or logs... I am stumped... I > > > wander is there is no known issues with globbing functions (if rsync uses > > > them) on solaris or some such... > > > > > > Any isnights or pointers would be appreceated. > > > > What version of rsync is running on both sides, and what version of ssh are > > you using? There have been versions of both that have hanging problems, > > and there are even patches available for the current version of rsync > > although I'm not sure whether they're relevant over ssh. People have > > reported that the "-v" option makes things worse. Using "netstat" on both > > sides (and probably on your proxy machine too) may help to detect where > > packets are piling up in queues. > Hi Dave, > > Rsync 2.4.6 on both ends. > Openssh 2.9p2 on the NT end (Cygwin build, install early 09/2001) > SSH 1.2.22 on the Solaris end (don't ask me why, it not my box, I cannot > make upgrades as I please) > > As far as we can tell there is no packets in the queues on the > proxy or the solaris box. Thats why its so wierd > > Regards > Tomasz Ciolek I'm not very familiar with NT problems. There is a known hanging problem with the SSH 1.2.X series -- it needs to be compiled without the "USE_PIPES" option, which was the default at least on 1.2.27 and later. 1.2.22 is very old indeed, with more than one widely published security hole, so maybe you can pursuade the owner to upgrade to 1.2.27 and make a few modifications. There's also security patches for 1.2.27, have them check out http://www.core-sdi.com/advisories/ssh1_deattack.htm http://www.core-sdi.com/advisories/ssh1_sessionkey_recovery.htm Perhaps you're running into the problem fixed by Wayne Davison's patches: http://www.clari.net/~wayne/rsync-nohang.patch http://www.clari.net/~wayne/rsync-nohang2.patch - Dave Dykstra
Re: problems with large transfers form solaris 2.6
Hi Dave, Rsync 2.4.6 on both ends. Openssh 2.9p2 on the NT end (Cygwin build, install early 09/2001) SSH 1.2.22 on the Solaris end (don't ask me why, it not my box, I cannot make upgrades as I please) As far as we can tell there is no packets in the queues on the proxy or the solaris box. Thats why its so wierd Regards Tomasz Ciolek On Thu, Sep 27, 2001 at 09:48:18AM -0500, Dave Dykstra wrote: > On Wed, Sep 26, 2001 at 10:46:51AM +1000, Tomasz Ciolek wrote: > > Hi all. > > > > I have a large data set on solaris 2.6 (approx 16gb and about 900K files > > and rectory structures). Rsync is misbehaving in the most obnoxious way: > > > > it is launched as such: rsync -e ssh -aLv --bwlimit=512 --timeout=300 > > > > > A transfer starts, a list of files and directories is built, and data > > starts flowing. then for no apparent reason data stops flowing form the > > rsync on the solaris end the the connection times out after 5 minutes of no > > IO activity. When the timeout parameter is not there, the rsync will just > > stop and hang around for hours untill killed. > > > > The only other this worth mentioning is that the solaris box is hidden > > behind a firewall and we use a ssh proxy (TIS plug proxy) to connect to the > > box. > > > > Addibd many v's on the commandline produces a lot of debug but no real > > information as to whats going on. The data flow just stops and there is no > > error messages on either end in rsync output or logs... I am stumped... I > > wander is there is no known issues with globbing functions (if rsync uses > > them) on solaris or some such... > > > > Any isnights or pointers would be appreceated. > > What version of rsync is running on both sides, and what version of ssh are > you using? There have been versions of both that have hanging problems, > and there are even patches available for the current version of rsync > although I'm not sure whether they're relevant over ssh. People have > reported that the "-v" option makes things worse. Using "netstat" on both > sides (and probably on your proxy machine too) may help to detect where > packets are piling up in queues. > > - Dave Dykstra -- Tomasz M. Ciolek* * Everything falls under the law of change; <[EMAIL PROTECTED]> * Like a dream, a phantom, a bubble, a shadow, <[EMAIL PROTECTED]>* like dew of flash of lightning. * You should contemplate like this. *** GPG Key ID: 0x7A18E49D * Available on www.pgp.net *** msg02155/pgp0.pgp Description: PGP signature
Re: problems with large transfers form solaris 2.6
On Wed, Sep 26, 2001 at 10:46:51AM +1000, Tomasz Ciolek wrote: > Hi all. > > I have a large data set on solaris 2.6 (approx 16gb and about 900K files > and rectory structures). Rsync is misbehaving in the most obnoxious way: > > it is launched as such: rsync -e ssh -aLv --bwlimit=512 --timeout=300 > > A transfer starts, a list of files and directories is built, and data > starts flowing. then for no apparent reason data stops flowing form the > rsync on the solaris end the the connection times out after 5 minutes of no > IO activity. When the timeout parameter is not there, the rsync will just > stop and hang around for hours untill killed. > > The only other this worth mentioning is that the solaris box is hidden > behind a firewall and we use a ssh proxy (TIS plug proxy) to connect to the > box. > > Addibd many v's on the commandline produces a lot of debug but no real > information as to whats going on. The data flow just stops and there is no > error messages on either end in rsync output or logs... I am stumped... I > wander is there is no known issues with globbing functions (if rsync uses > them) on solaris or some such... > > Any isnights or pointers would be appreceated. What version of rsync is running on both sides, and what version of ssh are you using? There have been versions of both that have hanging problems, and there are even patches available for the current version of rsync although I'm not sure whether they're relevant over ssh. People have reported that the "-v" option makes things worse. Using "netstat" on both sides (and probably on your proxy machine too) may help to detect where packets are piling up in queues. - Dave Dykstra
problems with large transfers form solaris 2.6
Hi all. I have a large data set on solaris 2.6 (approx 16gb and about 900K files and rectory structures). Rsync is misbehaving in the most obnoxious way: it is launched as such: rsync -e ssh -aLv --bwlimit=512 --timeout=300 A transfer starts, a list of files and directories is built, and data starts flowing. then for no apparent reason data stops flowing form the rsync on the solaris end the the connection times out after 5 minutes of no IO activity. When the timeout parameter is not there, the rsync will just stop and hang around for hours untill killed. The only other this worth mentioning is that the solaris box is hidden behind a firewall and we use a ssh proxy (TIS plug proxy) to connect to the box. Addibd many v's on the commandline produces a lot of debug but no real information as to whats going on. The data flow just stops and there is no error messages on either end in rsync output or logs... I am stumped... I wander is there is no known issues with globbing functions (if rsync uses them) on solaris or some such... Any isnights or pointers would be appreceated. Regards Tomasz Ciolek -- Tomasz M. Ciolek* * Everything falls under the law of change; <[EMAIL PROTECTED]> * Like a dream, a phantom, a bubble, a shadow, <[EMAIL PROTECTED]>* like dew of flash of lightning. * You should contemplate like this. *** GPG Key ID: 0x7A18E49D * Available on www.pgp.net *** PGP signature
