Re: Rsync between 2 datacenters not working

2018-03-30 Thread Perry Hutchison via rsync 
Marc Roos via rsync  wrote:
> Hmmm, looks indeed like changing the mtu helps, strange thing is that 
> with the bigger mtu sometimes I had no problems and often TB are copied 
> without problems. 
> Problem is I can't test the connection that much because the connection 
> has been established by a third party and it is just some vlan without 
> any hops. 

VLANs are notorious for reducing the MTU by adding metadata to the
packets.

2 possible reasons why it sometimes works OK:

* Overall net performance is sometimes good enough that the
  fragmentation introduced by the VLAN has little impact.

* The 3rd party who sets it up may not always do it in
  exactly the same way.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: Rsync between 2 datacenters not working

2018-03-30 Thread Marc Roos via rsync 
 
Hmmm, looks indeed like changing the mtu helps, strange thing is that 
with the bigger mtu sometimes I had no problems and often TB are copied 
without problems. 
Problem is I can't test the connection that much because the connection 
has been established by a third party and it is just some vlan without 
any hops. 

But thanks for the link to the nfs-test that looks interesting! 




-Original Message-
From: Dan Stromberg [mailto:drsali...@gmail.com] 
Sent: donderdag 29 maart 2018 4:36
To: Marc Roos
Cc: rsync
Subject: Re: Rsync between 2 datacenters not working

If reducing the MTU is helping, you might look into turning on Path MTU 
Discovery.

NFS can be fast for large transfers if you tune it.
http://stromberg.dnsalias.org/~strombrg/nfs-test.html

NFS is not terribly secure though - at least v2 and v3 weren't. Not sure 
about v4.

On Wed, Mar 28, 2018 at 12:59 AM, Marc Roos via rsync 
<rsync@lists.samba.org> wrote:
>
> Kevin, Dan, Thanks for the pointers to work arounds, at the moment I 
> am testing with lower mtu size that seems to be working. Otherwise I 
> need to fall back on mounting the fs maybe even as nfs.
>
>
>
> -Original Message-
> From: Kevin Korb via rsync [mailto:rsync@lists.samba.org]
> Sent: zondag 25 maart 2018 22:42
> To: rsync@lists.samba.org
> Subject: Re: Rsync between 2 datacenters not working
>
> Note that if you do this you are stuck with --whole-file
>
> On 03/25/2018 04:36 PM, Dan Stromberg via rsync wrote:
>> You could try using an automounter, like autofs, in combination with 
>> sshfs.  It'll be slower, possibly a lot slower, but it should be more 

>> reliable over an unreliable connection.
>>
>> I've been using:
>> remote
>> -fstype=fuse,allow_other,nodev,noatime,reconnect,ServerAliveInterval=
>> 1 5,ServerAliveCountMax=40,uid=0,gid=0,ro,nodev,noatime
>> :sshfs\#r...@remote.host.com\:/
>>
>> BTW, I'm not sure it's necessary to escape the #.  I never tried it
> without.
>>
>> Also note that it flattens the remote host's mount tree into a single 

>> filesystem - so things like /proc look like they are in the same 
>> filesystem as /.  This can lead to backing up /proc's contents (many 
>> pseudofiles), if you don't exclude it, even if you use rsync's -x 
>> option.
>>
>> On Sun, Mar 25, 2018 at 6:43 AM, Marc Roos via rsync 
>> <rsync@lists.samba.org> wrote:
>>>
>>> I still stuck with these errors
>>>
>>> packet_write_wait: Connection to 192.168.10.43 port 22: Broken pipe
>>> rsync: connection unexpectedly closed (534132435 bytes received so
>>> far) [receiver] rsync error: error in rsync protocol data stream 
>>> (code 12) at io.c(605) [receiver=3.0.9]
>>> rsync: connection unexpectedly closed (27198 bytes received so far) 
>>> [generator] rsync error: unexplained error (code 255) at io.c(605) 
>>> [generator=3.0.9]
>>
>
> --
> 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> Kevin Korb  Phone:(407) 252-6853
> Systems Administrator   Internet:
> FutureQuest, Inc.   ke...@futurequest.net  (work)
> Orlando, Floridak...@sanitarium.net (personal)
> Web page:   http://www.sanitarium.net/
> PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._
> .,
>
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing 
list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: 
> http://www.catb.org/~esr/faqs/smart-questions.html



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync between 2 datacenters not working

2018-03-28 Thread Dan Stromberg via rsync 
If reducing the MTU is helping, you might look into turning on Path
MTU Discovery.

NFS can be fast for large transfers if you tune it.
http://stromberg.dnsalias.org/~strombrg/nfs-test.html

NFS is not terribly secure though - at least v2 and v3 weren't. Not
sure about v4.

On Wed, Mar 28, 2018 at 12:59 AM, Marc Roos via rsync
<rsync@lists.samba.org> wrote:
>
> Kevin, Dan, Thanks for the pointers to work arounds, at the moment I am
> testing with lower mtu size that seems to be working. Otherwise I need
> to fall back on mounting the fs maybe even as nfs.
>
>
>
> -Original Message-
> From: Kevin Korb via rsync [mailto:rsync@lists.samba.org]
> Sent: zondag 25 maart 2018 22:42
> To: rsync@lists.samba.org
> Subject: Re: Rsync between 2 datacenters not working
>
> Note that if you do this you are stuck with --whole-file
>
> On 03/25/2018 04:36 PM, Dan Stromberg via rsync wrote:
>> You could try using an automounter, like autofs, in combination with
>> sshfs.  It'll be slower, possibly a lot slower, but it should be more
>> reliable over an unreliable connection.
>>
>> I've been using:
>> remote
>> -fstype=fuse,allow_other,nodev,noatime,reconnect,ServerAliveInterval=1
>> 5,ServerAliveCountMax=40,uid=0,gid=0,ro,nodev,noatime
>> :sshfs\#r...@remote.host.com\:/
>>
>> BTW, I'm not sure it's necessary to escape the #.  I never tried it
> without.
>>
>> Also note that it flattens the remote host's mount tree into a single
>> filesystem - so things like /proc look like they are in the same
>> filesystem as /.  This can lead to backing up /proc's contents (many
>> pseudofiles), if you don't exclude it, even if you use rsync's -x
>> option.
>>
>> On Sun, Mar 25, 2018 at 6:43 AM, Marc Roos via rsync
>> <rsync@lists.samba.org> wrote:
>>>
>>> I still stuck with these errors
>>>
>>> packet_write_wait: Connection to 192.168.10.43 port 22: Broken pipe
>>> rsync: connection unexpectedly closed (534132435 bytes received so
>>> far) [receiver] rsync error: error in rsync protocol data stream
>>> (code 12) at io.c(605) [receiver=3.0.9]
>>> rsync: connection unexpectedly closed (27198 bytes received so far)
>>> [generator] rsync error: unexplained error (code 255) at io.c(605)
>>> [generator=3.0.9]
>>
>
> --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> Kevin Korb  Phone:(407) 252-6853
> Systems Administrator   Internet:
> FutureQuest, Inc.   ke...@futurequest.net  (work)
> Orlando, Floridak...@sanitarium.net (personal)
> Web page:   http://www.sanitarium.net/
> PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: Rsync between 2 datacenters not working

2018-03-28 Thread Marc Roos via rsync 
 
Kevin, Dan, Thanks for the pointers to work arounds, at the moment I am 
testing with lower mtu size that seems to be working. Otherwise I need 
to fall back on mounting the fs maybe even as nfs.



-Original Message-
From: Kevin Korb via rsync [mailto:rsync@lists.samba.org] 
Sent: zondag 25 maart 2018 22:42
To: rsync@lists.samba.org
Subject: Re: Rsync between 2 datacenters not working

Note that if you do this you are stuck with --whole-file

On 03/25/2018 04:36 PM, Dan Stromberg via rsync wrote:
> You could try using an automounter, like autofs, in combination with 
> sshfs.  It'll be slower, possibly a lot slower, but it should be more 
> reliable over an unreliable connection.
> 
> I've been using:
> remote 
> -fstype=fuse,allow_other,nodev,noatime,reconnect,ServerAliveInterval=1
> 5,ServerAliveCountMax=40,uid=0,gid=0,ro,nodev,noatime
> :sshfs\#r...@remote.host.com\:/
> 
> BTW, I'm not sure it's necessary to escape the #.  I never tried it 
without.
> 
> Also note that it flattens the remote host's mount tree into a single 
> filesystem - so things like /proc look like they are in the same 
> filesystem as /.  This can lead to backing up /proc's contents (many 
> pseudofiles), if you don't exclude it, even if you use rsync's -x 
> option.
> 
> On Sun, Mar 25, 2018 at 6:43 AM, Marc Roos via rsync 
> <rsync@lists.samba.org> wrote:
>>
>> I still stuck with these errors
>>
>> packet_write_wait: Connection to 192.168.10.43 port 22: Broken pipe
>> rsync: connection unexpectedly closed (534132435 bytes received so 
>> far) [receiver] rsync error: error in rsync protocol data stream 
>> (code 12) at io.c(605) [receiver=3.0.9]
>> rsync: connection unexpectedly closed (27198 bytes received so far) 
>> [generator] rsync error: unexplained error (code 255) at io.c(605) 
>> [generator=3.0.9]
> 

--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync between 2 datacenters not working

2018-03-25 Thread Kevin Korb via rsync 
Note that if you do this you are stuck with --whole-file

On 03/25/2018 04:36 PM, Dan Stromberg via rsync wrote:
> You could try using an automounter, like autofs, in combination with
> sshfs.  It'll be slower, possibly a lot slower, but it should be more
> reliable over an unreliable connection.
> 
> I've been using:
> remote 
> -fstype=fuse,allow_other,nodev,noatime,reconnect,ServerAliveInterval=15,ServerAliveCountMax=40,uid=0,gid=0,ro,nodev,noatime
> :sshfs\#r...@remote.host.com\:/
> 
> BTW, I'm not sure it's necessary to escape the #.  I never tried it without.
> 
> Also note that it flattens the remote host's mount tree into a single
> filesystem - so things like /proc look like they are in the same
> filesystem as /.  This can lead to backing up /proc's contents (many
> pseudofiles), if you don't exclude it, even if you use rsync's -x
> option.
> 
> On Sun, Mar 25, 2018 at 6:43 AM, Marc Roos via rsync
>  wrote:
>>
>> I still stuck with these errors
>>
>> packet_write_wait: Connection to 192.168.10.43 port 22: Broken pipe
>> rsync: connection unexpectedly closed (534132435 bytes received so far)
>> [receiver]
>> rsync error: error in rsync protocol data stream (code 12) at io.c(605)
>> [receiver=3.0.9]
>> rsync: connection unexpectedly closed (27198 bytes received so far)
>> [generator]
>> rsync error: unexplained error (code 255) at io.c(605) [generator=3.0.9]
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync between 2 datacenters not working

2018-03-25 Thread Dan Stromberg via rsync 
You could try using an automounter, like autofs, in combination with
sshfs.  It'll be slower, possibly a lot slower, but it should be more
reliable over an unreliable connection.

I've been using:
remote 
-fstype=fuse,allow_other,nodev,noatime,reconnect,ServerAliveInterval=15,ServerAliveCountMax=40,uid=0,gid=0,ro,nodev,noatime
:sshfs\#r...@remote.host.com\:/

BTW, I'm not sure it's necessary to escape the #.  I never tried it without.

Also note that it flattens the remote host's mount tree into a single
filesystem - so things like /proc look like they are in the same
filesystem as /.  This can lead to backing up /proc's contents (many
pseudofiles), if you don't exclude it, even if you use rsync's -x
option.

On Sun, Mar 25, 2018 at 6:43 AM, Marc Roos via rsync
 wrote:
>
> I still stuck with these errors
>
> packet_write_wait: Connection to 192.168.10.43 port 22: Broken pipe
> rsync: connection unexpectedly closed (534132435 bytes received so far)
> [receiver]
> rsync error: error in rsync protocol data stream (code 12) at io.c(605)
> [receiver=3.0.9]
> rsync: connection unexpectedly closed (27198 bytes received so far)
> [generator]
> rsync error: unexplained error (code 255) at io.c(605) [generator=3.0.9]

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync between 2 datacenters not working

2018-03-25 Thread Kevin Korb via rsync 
Since your problem is really that your ssh connection is dropping maybe
you need more verbosity/debug on the ssh part (possibly even on both
ends).  Also, if you are using --checksum don't.

On 03/25/2018 09:43 AM, Marc Roos via rsync wrote:
> 
> I still stuck with these errors
> 
> packet_write_wait: Connection to 192.168.10.43 port 22: Broken pipe
> rsync: connection unexpectedly closed (534132435 bytes received so far) 
> [receiver]
> rsync error: error in rsync protocol data stream (code 12) at io.c(605) 
> [receiver=3.0.9]
> rsync: connection unexpectedly closed (27198 bytes received so far) 
> [generator]
> rsync error: unexplained error (code 255) at io.c(605) [generator=3.0.9]
> 
> Is there any way to mitigate this? Is there some way to have rsync 
> continue or reestablish connection?
> 
> 
> I posted this here.
> https://www.mail-archive.com/rsync@lists.samba.org/msg32145.html
> 
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html