Re: -e escape rule

[Dave Howorth]

On 2016-10-20 10:24, Samuel Williams wrote:

Hello,

I'm using Ruby's Shellwords module, which generates a string from an
array, suitable for shell evaluation.

Ruby's implementation prefers escaping whitespace with a backslash
rather than quotes. However, this appears to cause some kind of issue
in Rsync when it computes argv from -e option.


The man page for rsync  
explicitly forbids the use of backslashes:


"Command-line arguments are permitted in COMMAND provided that COMMAND 
is presented to rsync as a single argument. You must use spaces (not 
tabs or other whitespace) to separate the command and args from each 
other, and you can use single- and/or double-quotes to preserve spaces 
in an argument (but not backslashes)."


So I think the rest of the question is moot ...

Cheers, Dave


Here is an example command generated by some Ruby code:

rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
../../latest/etc/ /etc/
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

We can check that something like this is valid:

files% echo foo\ bar\\\=baz
foo bar\=baz -- what Rsync should be receiving
files% echo foo bar\=baz
foo bar=baz  -- What Rsync should be executing

However this gives me an error

command-line: line 0: Bad configuration option: connecttimeout\\
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]

I think the problem here is the "ConnectTimeout\\\=60", in particular
how the equals symbol is escaped.

I'm looking in the function:

static pid_t do_cmd(char *cmd, char *machine, char *user, char
**remote_argv, int remote_argc,
   int *f_in_p, int *f_out_p)

This function splits based purely on whitespace:

 args[argc++] = t;
 while (*f != ' ' || in_quote) {
 // consume token...

I feel that this function should also handle backslash escapes.

I also checked using strace and it appears that this is the issue, but
I'm open to suggestions/ideas.

Kind regards,
Samuel



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync: connection unexpectedly closed

[Dave Howorth]

On 2016-10-10 17:24, Kip Warner wrote:

Note that I have temporarily disabled timeouts and added extra
verbosity. The transfer to the remote host via SSH works fine, up until
it gets to a 30+ GB file (a VM image). It gets about 90+ percent of the
way through, hangs, and then times out.


I have a similar but different problem. I make a regular download from a 
site that always errors out on a particular large file. However, my 
rsync error symptoms are different.


Unfortunately, the server admins seem to be the strong, silent types who 
have repeatedly changed their minds about what they think is wrong and 
who may or may not be attempting to solve the problem in isolation - 
I've failed to get any meaningful communication going with them. 
Fortunately, excluding the problem file is a reasonable workaround for me.


Have you tried excluding the problem file from the transfer?

One possibility is that the problem is not caused directly by rsync but 
because of some underlying filesystem glitch. What OS & filesystems are 
you using?


Cheers, Dave

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: size-related rsync bugs?

[Dave Howorth]

On 2016-06-29 13:40, Dave Howorth wrote:

I use rsync to download a directory from a remote daemon. This has
worked well for years but has recently started to give problems. The
rsync transfer is run by a program called dirvish that is run as a cron
job. The log looks like this:

[snip]

Sorry, I should also have included the rsync error messages that I see:


*** Execution cycle 0 ***

rsync: read error: Connection reset by peer (104)
rsync error: error in socket IO (code 10) at io.c(785) [receiver=3.1.2]
rsync: connection unexpectedly closed (14931 bytes received so far) 
[generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) 
[generator=3.1.2]



*** Execution cycle 1 ***

rsync: read error: Connection reset by peer (104)
rsync error: error in socket IO (code 10) at io.c(785) [receiver=3.1.2]
rsync: connection unexpectedly closed (17555 bytes received so far) 
[generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) 
[generator=3.1.2]



*** Execution cycle 2 ***

rsync: read error: Connection reset by peer (104)
rsync error: error in socket IO (code 10) at io.c(785) [receiver=3.1.2]
rsync: connection unexpectedly closed (16371 bytes received so far) 
[generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) 
[generator=3.1.2]



*** Execution cycle 3 ***

rsync: read error: Connection reset by peer (104)
rsync error: error in socket IO (code 10) at io.c(785) [receiver=3.1.2]
rsync: connection unexpectedly closed (16355 bytes received so far) 
[generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) 
[generator=3.1.2]



*** Execution cycle 4 ***

rsync: read error: Connection reset by peer (104)
rsync error: error in socket IO (code 10) at io.c(785) [receiver=3.1.2]
rsync: connection unexpectedly closed (16347 bytes received so far) 
[generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) 
[generator=3.1.2]



Cheers, Dave

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

size-related rsync bugs?

[Dave Howorth]

I use rsync to download a directory from a remote daemon. This has 
worked well for years but has recently started to give problems. The 
rsync transfer is run by a program called dirvish that is run as a cron 
job. The log looks like this:



ACTION: rsync -vrltH --delete --stats -D --numeric-ids 
--exclude-from=/nfs/scop5/data/msd/sifts-mirror/2016-06-28/exclude 
--link-dest=/nfs/scop5/data/msd/sifts-mirror/2016-06-07/tree 
rsync.ebi.ac.uk::pub/databases/msd/sifts/ 
/nfs/scop5/data/msd/sifts-mirror/2016-06-28/tree


receiving incremental file list
csv/pdb_chain_cath_uniprot.csv
csv/pdb_chain_enzyme.csv
csv/pdb_chain_go.csv
broken pipe
RESULTS: warnings = 0, errors = 1

ACTION: rsync -vrltH --delete --stats -D --numeric-ids 
--exclude-from=/nfs/scop5/data/msd/sifts-mirror/2016-06-28/exclude 
--link-dest=/nfs/scop5/data/msd/sifts-mirror/2016-06-07/tree 
rsync.ebi.ac.uk::pub/databases/msd/sifts/ 
/nfs/scop5/data/msd/sifts-mirror/2016-06-28/tree


receiving incremental file list
csv/pdb_chain_go.csv
broken pipe
RESULTS: warnings = 0, errors = 2

ACTION: rsync -vrltH --delete --stats -D --numeric-ids 
--exclude-from=/nfs/scop5/data/msd/sifts-mirror/2016-06-28/exclude 
--link-dest=/nfs/scop5/data/msd/sifts-mirror/2016-06-07/tree 
rsync.ebi.ac.uk::pub/databases/msd/sifts/ 
/nfs/scop5/data/msd/sifts-mirror/2016-06-28/tree


receiving incremental file list
csv/pdb_chain_go.csv
broken pipe
RESULTS: warnings = 0, errors = 3

ACTION: rsync -vrltH --delete --stats -D --numeric-ids 
--exclude-from=/nfs/scop5/data/msd/sifts-mirror/2016-06-28/exclude 
--link-dest=/nfs/scop5/data/msd/sifts-mirror/2016-06-07/tree 
rsync.ebi.ac.uk::pub/databases/msd/sifts/ 
/nfs/scop5/data/msd/sifts-mirror/2016-06-28/tree


receiving incremental file list
csv/pdb_chain_go.csv
broken pipe
RESULTS: warnings = 0, errors = 4

ACTION: rsync -vrltH --delete --stats -D --numeric-ids 
--exclude-from=/nfs/scop5/data/msd/sifts-mirror/2016-06-28/exclude 
--link-dest=/nfs/scop5/data/msd/sifts-mirror/2016-06-07/tree 
rsync.ebi.ac.uk::pub/databases/msd/sifts/ 
/nfs/scop5/data/msd/sifts-mirror/2016-06-28/tree


receiving incremental file list
csv/pdb_chain_go.csv
broken pipe
RESULTS: warnings = 0, errors = 5


It seems to me that the file pdb_chain_go.csv is triggering the problem. 
I notice that in the last transfer that succeeded on 2016-06-07, it was 
1107298708 bytes whilst on 2016-06-22 it had grown a bit to 629050 
so I suppose that one possibility is some size-related bug in rsync?


I downloaded the particular file using FTP without any problems and it 
is a text file that appears to be complete without any unusual features.


I was using rsync 3.0.6 but have just installed and used 3.1.2 with the 
same problem. I do not control the remote daemon and have not as yet 
been able to persuade its administrators to send me its logs. The 
protocol version in use is 30 regardless of which version of rsync I use.


I'd be grateful for any ideas that might get my downloads working again.

Thanks, Dave

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

writefd_unbuffered failed - Broken pipe on local rsync

[Dave Howorth]

I'm seeing an error that says there is a broken pipe, but the rsync
command is local to one machine:

/usr/bin/rsync -rltH --stats -D --delete --numeric-ids --whole-file
/data/dir /backup/suse1/suse1-data-dir/ /tmp/rsync-out 21

The complete contents of the rsync-out file are:

rsync: writefd_unbuffered failed to write 88 bytes [receiver]: Broken
pipe (32)
rsync error: error in rsync protocol data stream (code 12) at io.c(1122)
[receiver=2.6.9]


Q3 on http://samba.anu.edu.au/rsync/issues.html implies that the error
message necessarily involves a remote rsync. Does anybody know what the
message means for a local transfer, please?

Thanks, Dave
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html