Re: rsync: connection unexpectedly closed

[Kip Warner]

On Thu, 2016-10-13 at 10:09 +0100, Dave Howorth wrote:
> Have you tried excluding the problem file from the transfer?

Hey Dave. All the other files appear to sync, up until it gets to that
one large file. Then it stalls, and finally times out. I could tell it
to exclude that important file, but that would defeat the purpose of my
backup.

> One possibility is that the problem is not caused directly by rsync
> but  because of some underlying filesystem glitch. What OS &
> filesystems are you using?

That could well be, but how to know?

Client side:

$ lsb_release -a
LSB Version:
security-9.20160110ubuntu5-amd64:security-9.20160110ubuntu5-noarch
Distributor ID: Ubuntu
Description:Ubuntu 16.10
Release:16.10
Codename:   yakkety

Server side:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:Debian GNU/Linux 7.11 (wheezy)
Release:7.11
Codename:   wheezy

-- 
Kip Warner -- Senior Software Engineer
OpenPGP encrypted/signed mail preferred
http://www.thevertigo.com

signature.asc
Description: This is a digitally signed message part
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync: connection unexpectedly closed

[Kevin Korb]

I don't remember whether or not you said you were running rsync over ssh
but if you are you can also debug the ssh layer.  You can even do it at
both ends

On the server run a debugging sshd on an alternate port with:
/usr/sbin/sshd -dDp222
(note that this will only accept 1 connection, debug to the terminal,
then exit)
then on the client use rsync -e "ssh -vp222" to go verbose and use the
debugging port.

On 10/13/2016 08:05 PM, Kip Warner wrote:
> On Thu, 2016-10-13 at 16:58 -0700, Wayne Davison wrote:
>> No, that's totally expected. While select() is waiting for I/O to
>> arrive, it returns to rsync every 60 seconds to allow it to decide if
>> it wants to continue waiting or do something else. You have to find
>> the process that is exiting first to discover why the connection is
>> closing. This assumes that it's not a network issue, where all of the
>> programs get a closed-connection
>> error.
> 
> Hey Wayne,
> 
> I took Paul's advice and tried monitoring with lsof. This is what I
> saw:
> 
> $ lsof -p 3104
> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
> rsync   3104  kip  cwdDIR   0,19 4096 214695958 
> /home/kip/Disk_Backups/kip-desktop/yakkety
> rsync   3104  kip  rtdDIR  179,2 4096 2 /
> rsync   3104  kip  txtREG  179,2   460584  7817 /usr/bin/rsync
> rsync   3104  kip  memREG  179,242692  3974 
> /lib/arm-linux-gnueabihf/libnss_files-2.13.so
> rsync   3104  kip  memREG  179,271624  3971 
> /lib/arm-linux-gnueabihf/libnsl-2.13.so
> rsync   3104  kip  memREG  179,238608  3976 
> /lib/arm-linux-gnueabihf/libnss_nis-2.13.so
> rsync   3104  kip  memREG  179,226484  3972 
> /lib/arm-linux-gnueabihf/libnss_compat-2.13.so
> rsync   3104  kip  memREG  179,2  2953776 18967 
> /usr/lib/locale/locale-archive
> rsync   3104  kip  memREG  179,2  1216624  3965 
> /lib/arm-linux-gnueabihf/libc-2.13.so
> rsync   3104  kip  memREG  179,2   130448 41964 
> /lib/arm-linux-gnueabihf/libgcc_s.so.1
> rsync   3104  kip  memREG  179,243024 10745 
> /lib/arm-linux-gnueabihf/libpopt.so.0.0.0
> rsync   3104  kip  memREG  179,226240  1840 
> /lib/arm-linux-gnueabihf/libacl.so.1.1.0
> rsync   3104  kip  memREG  179,217904  1854 
> /lib/arm-linux-gnueabihf/libattr.so.1.1.0
> rsync   3104  kip  memREG  179,210170 28719 
> /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so
> rsync   3104  kip  memREG  179,2   126236  3961 
> /lib/arm-linux-gnueabihf/ld-2.13.so
> rsync   3104  kip1w  FIFO   0,10  0t0  8455 pipe
> rsync   3104  kip2w  FIFO   0,10  0t0  8456 pipe
> rsync   3104  kip3u  unix 0xdbda86c0  0t0  8466 socket
> 
> And on the strace...
> 
> ...
> 3104  select(4, [3], [1], [3], {60, 0}) = 1 (out [1], left {59, 78})
> 3104  write(1, 
> "h\340\0\7\0y@\312\227\241\236\255\367\312\3637\323X\206\314\250\372\362P1\202\374\"i\265c\16"...,
>  57452) = 57452
> 3104  select(4, [3], [], [3], {60, 0})  = 0 (Timeout)
> (repeats above line until client (sender) reports connection timed out)
> 
> Sender reports the following until it times out...
> 
> ...
> match at 32949141504 last_match=32949141504 j=251382 len=131072 n=0
>  32.95G  96%   83.55MB/s0:00:12
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12378] why i cannot exclude dir/files if using option "--files-from"

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=12378

Wayne Davison  changed:

   What|Removed |Added

 Resolution|--- |WORKSFORME
 Status|NEW |RESOLVED

--- Comment #4 from Wayne Davison  ---
Command args (which includes names inside a files-from file) are never excluded
by an exclude directive. You told rsync to copy it, so it copies it. Excludes
only affect matching of files that rsync finds inside directories that you told
it to copy. It is expected that you already trimmed any unwanted files/dirs
from your args before you called rsync, since those names are under your
control.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12378] why i cannot exclude dir/files if using option "--files-from"

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=12378

--- Comment #3 from Kevin Korb  ---
I didn't say anything specific about a remote host.

You excluded .cache then you told it to copy specific files some of which are
in .cache.  If there was a file named .cache inside of a directory that you
told it to copy then it would be excluded.  If you excluded
userName/.cache/winetricks/msls31/InstMsiW.exe then it would be excluded. 
Exclude is not patern matched against parts of things you explicitly tell rsync
to copy...

kmk@dementia[1%]> cd /tmp
kmk@dementia[2%]> mkdir test
kmk@dementia[3%]> cd !$
cd test
kmk@dementia[4%]> mkdir src dst
kmk@dementia[5%]> cd src
kmk@dementia[6%]> mkdir -p a b c d e cache/c
kmk@dementia[7%]> cd ..
kmk@dementia[8%]> /bin/echo -ne "a\nb\nc\nd\ne\ncache/c\n" > list
kmk@dementia[9%]> cat list 
a
b
c
d
e
cache/c
kmk@dementia[10%]> rsync -vain --files-from=list --exclude=cache src/ dst/
building file list ... done
cd+ a/
cd+ b/
cd+ c/
cd+ cache/
cd+ cache/c/
cd+ d/
cd+ e/

sent 157 bytes  received 37 bytes  388.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)
kmk@dementia[11%]> rsync -vain --files-from=list --exclude=c src/ dst/
building file list ... done
cd+ a/
cd+ b/
cd+ d/
cd+ e/

sent 106 bytes  received 28 bytes  268.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)
kmk@dementia[12%]> rsync -vain --files-from=list --exclude=cache/c src/ dst/
building file list ... done
cd+ a/
cd+ b/
cd+ c/
cd+ d/
cd+ e/

sent 121 bytes  received 31 bytes  304.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

Cache isn't really being copied it is just being created to make a place for
cache/c to be stored when cache/c or c is not excluded.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12378] why i cannot exclude dir/files if using option "--files-from"

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=12378

--- Comment #2 from Alessio  ---
(In reply to Kevin Korb from comment #1)

Excuse me, but i don't get it, what's the difference between excluding
(matching) the pattern from a --files-from  and a remote host?

debugging the rsync transfer from the remote host I get:
[sender] hiding directory home/userName/.cache because of pattern .cache

in the other case nothing is being hidden instead  but .cache  is inside my
path:
userName/.cache/winetricks/msls31/InstMsiW.exe

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12378] why i cannot exclude dir/files if using option "--files-from"

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=12378

--- Comment #1 from Kevin Korb  ---
It did not copy the directory you excluded it copied the files within that
directory that you explicitly told it to copy and created the appropriate
directories to allow that to happen.

IOW, .cache is not relative to
userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12378] New: why i cannot exclude dir/files if using option "--files-from"

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=12378

Bug ID: 12378
   Summary: why i cannot exclude dir/files if using option
"--files-from"
   Product: rsync
   Version: 3.1.2
  Hardware: x64
OS: Linux
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: is...@sissa.it
QA Contact: rsync...@samba.org

I don't understand if this is intended or not, but when I use "--files-from"
and  "--exclude"  options together all files are synced anyhow.
eg.:
rsync -av -c --delete --relative --delete-excluded  --exclude=".cache"
--exclude="mydirToExclude" /home . 
it correctly doesn't copy the .cache folder.

but:
rsync -av -c --delete --relative --delete-excluded --files-from=myFileList.txt
--exclude=".cache" --exclude="mydirToExclude" /home .
does copy everything.

instead using a pattern such as  --exclude="**myDirToExclude**"  
doesn't copy all lines with that pattern

myFileList.txt is something like:
userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/D3BD7BB89AFC5E5C3921C55D38B7DB4F6A6A8C55
userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15
userName/.cache/mozilla/firefox/e9dwu0jm.default/directoryLinks.json
userName/.cache/winetricks/dotnet40/dotNetFx40_Full_x86_x64.exe
userName/.cache/winetricks/dotnet30/dotnetfx3.exe
userName/.cache/winetricks/dotnet20sp2/NetFx20SP2_x86.exe
userName/.cache/winetricks/msls31/InstMsiW.exe
userName/Downloads/Sentinel_LDK_Run-time_setup/HASPUserSetup.exe
userName/Downloads/Sentinel_LDK_Run-time_setup/readme.html


Thanks,
alessio

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync: connection unexpectedly closed

[Dave Howorth]

On 2016-10-10 17:24, Kip Warner wrote:

Note that I have temporarily disabled timeouts and added extra
verbosity. The transfer to the remote host via SSH works fine, up until
it gets to a 30+ GB file (a VM image). It gets about 90+ percent of the
way through, hangs, and then times out.


I have a similar but different problem. I make a regular download from a 
site that always errors out on a particular large file. However, my 
rsync error symptoms are different.


Unfortunately, the server admins seem to be the strong, silent types who 
have repeatedly changed their minds about what they think is wrong and 
who may or may not be attempting to solve the problem in isolation - 
I've failed to get any meaningful communication going with them. 
Fortunately, excluding the problem file is a reasonable workaround for me.


Have you tried excluding the problem file from the transfer?

One possibility is that the problem is not caused directly by rsync but 
because of some underlying filesystem glitch. What OS & filesystems are 
you using?


Cheers, Dave

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html