Re: -e escape rule

2016-10-20 Thread Samuel Williams 
Hmm, so after reviewing this in a bit more detail.. it would be very
nice if at least backslash escapes would be supported. Quoted strings
are pretty ugly, especially when nested several levels. Anyway, just
my 2 cents.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

2016-10-20 Thread Samuel Williams 
Hi Dave, thanks for point that out. I didn't realise there was a
detailed explanation of that field in the man page, I only saw the
summary. Yes, that clearly explains how it's supposed to work.

On 21 October 2016 at 01:46, Dave Howorth  wrote:
> On 2016-10-20 10:24, Samuel Williams wrote:
>>
>> Hello,
>>
>> I'm using Ruby's Shellwords module, which generates a string from an
>> array, suitable for shell evaluation.
>>
>> Ruby's implementation prefers escaping whitespace with a backslash
>> rather than quotes. However, this appears to cause some kind of issue
>> in Rsync when it computes argv from -e option.
>
>
> The man page for rsync 
> explicitly forbids the use of backslashes:
>
> "Command-line arguments are permitted in COMMAND provided that COMMAND is
> presented to rsync as a single argument. You must use spaces (not tabs or
> other whitespace) to separate the command and args from each other, and you
> can use single- and/or double-quotes to preserve spaces in an argument (but
> not backslashes)."
>
> So I think the rest of the question is moot ...
>
> Cheers, Dave
>
>
>> Here is an example command generated by some Ruby code:
>>
>> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
>> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
>> ../../latest/etc/ /etc/
>> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>>
>> We can check that something like this is valid:
>>
>> files% echo foo\ bar\\\=baz
>> foo bar\=baz -- what Rsync should be receiving
>> files% echo foo bar\=baz
>> foo bar=baz  -- What Rsync should be executing
>>
>> However this gives me an error
>>
>> command-line: line 0: Bad configuration option: connecttimeout\\
>> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
>> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
>>
>> I think the problem here is the "ConnectTimeout\\\=60", in particular
>> how the equals symbol is escaped.
>>
>> I'm looking in the function:
>>
>> static pid_t do_cmd(char *cmd, char *machine, char *user, char
>> **remote_argv, int remote_argc,
>>int *f_in_p, int *f_out_p)
>>
>> This function splits based purely on whitespace:
>>
>>  args[argc++] = t;
>>  while (*f != ' ' || in_quote) {
>>  // consume token...
>>
>> I feel that this function should also handle backslash escapes.
>>
>> I also checked using strace and it appears that this is the issue, but
>> I'm open to suggestions/ideas.
>>
>> Kind regards,
>> Samuel
>>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync: connection unexpectedly closed

2016-10-20 Thread Bernd Hohmann 
On 20.10.2016 03:24, Kip Warner wrote:

> I ended up giving up.

Me too.

I'm copying all files via 'scp' now - takes 3 days but no aborts or errors.

So I am very sure the problem is somewhere in rsync.

Bernd

-- 
Bernd Hohmann
Organisationsprogrammierer
Höhenstrasse 2 * 61130 Nidderau
Telefon: 06187/900495 * Telefax: 06187/900493
Blog: http://blog.harddiskcafe.de


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

2016-10-20 Thread Kevin Korb 
The \ escapes are for the shell.  Rsync never sees them and therefore
can't honor them.

On 10/20/2016 05:24 AM, Samuel Williams wrote:
> Hello,
> 
> I'm using Ruby's Shellwords module, which generates a string from an
> array, suitable for shell evaluation.
> 
> Ruby's implementation prefers escaping whitespace with a backslash
> rather than quotes. However, this appears to cause some kind of issue
> in Rsync when it computes argv from -e option.
> 
> Here is an example command generated by some Ruby code:
> 
> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
> ../../latest/etc/ /etc/
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
> 
> We can check that something like this is valid:
> 
> files% echo foo\ bar\\\=baz
> foo bar\=baz -- what Rsync should be receiving
> files% echo foo bar\=baz
> foo bar=baz  -- What Rsync should be executing
> 
> However this gives me an error
> 
> command-line: line 0: Bad configuration option: connecttimeout\\
> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
> 
> I think the problem here is the "ConnectTimeout\\\=60", in particular
> how the equals symbol is escaped.
> 
> I'm looking in the function:
> 
> static pid_t do_cmd(char *cmd, char *machine, char *user, char
> **remote_argv, int remote_argc,
>int *f_in_p, int *f_out_p)
> 
> This function splits based purely on whitespace:
> 
>  args[argc++] = t;
>  while (*f != ' ' || in_quote) {
>  // consume token...
> 
> I feel that this function should also handle backslash escapes.
> 
> I also checked using strace and it appears that this is the issue, but
> I'm open to suggestions/ideas.
> 
> Kind regards,
> Samuel
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

2016-10-20 Thread Dave Howorth 

On 2016-10-20 10:24, Samuel Williams wrote:

Hello,

I'm using Ruby's Shellwords module, which generates a string from an
array, suitable for shell evaluation.

Ruby's implementation prefers escaping whitespace with a backslash
rather than quotes. However, this appears to cause some kind of issue
in Rsync when it computes argv from -e option.


The man page for rsync  
explicitly forbids the use of backslashes:


"Command-line arguments are permitted in COMMAND provided that COMMAND 
is presented to rsync as a single argument. You must use spaces (not 
tabs or other whitespace) to separate the command and args from each 
other, and you can use single- and/or double-quotes to preserve spaces 
in an argument (but not backslashes)."


So I think the rest of the question is moot ...

Cheers, Dave


Here is an example command generated by some Ruby code:

rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
../../latest/etc/ /etc/
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

We can check that something like this is valid:

files% echo foo\ bar\\\=baz
foo bar\=baz -- what Rsync should be receiving
files% echo foo bar\=baz
foo bar=baz  -- What Rsync should be executing

However this gives me an error

command-line: line 0: Bad configuration option: connecttimeout\\
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]

I think the problem here is the "ConnectTimeout\\\=60", in particular
how the equals symbol is escaped.

I'm looking in the function:

static pid_t do_cmd(char *cmd, char *machine, char *user, char
**remote_argv, int remote_argc,
   int *f_in_p, int *f_out_p)

This function splits based purely on whitespace:

 args[argc++] = t;
 while (*f != ' ' || in_quote) {
 // consume token...

I feel that this function should also handle backslash escapes.

I also checked using strace and it appears that this is the issue, but
I'm open to suggestions/ideas.

Kind regards,
Samuel



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

2016-10-20 Thread Samuel Williams 
> There's no reason to escape an "=" sign in the above command.

Okay, so at a certain level I agree with you and that's how I've fixed
the issue.

All the stuff about config files is not feasible for a variety of
reasons - I'm aware of those options. I appreciate that you took the
time to explain them.

I think the problem is the assumptions in do_cmd about the contents of
the -e argument.

In my opinion, the way this is split needs to be deferred back to the
user's shell to work correctly. I'm not sure if there is an easy way
to do this that works for any kind of shell.

Assuming that -e is stored into rsh_cmd, and is user supplied for
executing on the shell, the only valid thing you can really do with it
is:

execl("/bin/sh", "sh", "-c", rsh_cmd, (char *) NULL);

(assuming that sh is the user's current shell, might want to use
$SHELL or something else).

It might make more sense if rsh_cmd used string substitution rather
than argument splitting, e.g. -e 'ssh $HOST'  this way, no splitting
would be required. In lots of ways this is more flexible. For example,
appending the host might not be useful in some contexts.

Anyway, rsync is pretty awesome. the -e option is only a minor
annoyance, but it would be nice if this discussion could result in a
tangible improvement.


On 21 October 2016 at 00:03, Paul Slootman  wrote:
> On Thu 20 Oct 2016, Samuel Williams wrote:
>>
>> I'm using Ruby's Shellwords module, which generates a string from an
>> array, suitable for shell evaluation.
>>
>> Ruby's implementation prefers escaping whitespace with a backslash
>> rather than quotes. However, this appears to cause some kind of issue
>> in Rsync when it computes argv from -e option.
>>
>> Here is an example command generated by some Ruby code:
>>
>> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
>> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
>> ../../latest/etc/ /etc/
>> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
> There's no reason to escape an "=" sign in the above command. Try:
>
> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ 
> ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ 
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
> Or even:
>
> rsync --archive --stats -e ssh\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ 
> -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ 
> bac...@example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
> I'd probably create an entry for this host in ~/.ssh/config :
>
> Host example.backup.server.com
> User backup
> IdentityFile /etc/synco/id_rsa
> ConnectTimeout 60
> BatchMode=yes
>
> and then just use:
>
> rsync --archive --stats --link-dest ../../latest/etc/ /etc/ 
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
>
> If you're dynamically doing this, you can pass a config file with -F.
>
>
> Paul
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

2016-10-20 Thread Paul Slootman 
On Thu 20 Oct 2016, Samuel Williams wrote:
> 
> I'm using Ruby's Shellwords module, which generates a string from an
> array, suitable for shell evaluation.
> 
> Ruby's implementation prefers escaping whitespace with a backslash
> rather than quotes. However, this appears to cause some kind of issue
> in Rsync when it computes argv from -e option.
> 
> Here is an example command generated by some Ruby code:
> 
> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
> ../../latest/etc/ /etc/
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

There's no reason to escape an "=" sign in the above command. Try:

rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ 
ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ 
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

Or even:

rsync --archive --stats -e ssh\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ 
-o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ 
bac...@example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

I'd probably create an entry for this host in ~/.ssh/config :

Host example.backup.server.com
User backup
IdentityFile /etc/synco/id_rsa
ConnectTimeout 60
BatchMode=yes

and then just use:

rsync --archive --stats --link-dest ../../latest/etc/ /etc/ 
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/


If you're dynamically doing this, you can pass a config file with -F.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-e escape rule

2016-10-20 Thread Samuel Williams 
Hello,

I'm using Ruby's Shellwords module, which generates a string from an
array, suitable for shell evaluation.

Ruby's implementation prefers escaping whitespace with a backslash
rather than quotes. However, this appears to cause some kind of issue
in Rsync when it computes argv from -e option.

Here is an example command generated by some Ruby code:

rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
../../latest/etc/ /etc/
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

We can check that something like this is valid:

files% echo foo\ bar\\\=baz
foo bar\=baz -- what Rsync should be receiving
files% echo foo bar\=baz
foo bar=baz  -- What Rsync should be executing

However this gives me an error

command-line: line 0: Bad configuration option: connecttimeout\\
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]

I think the problem here is the "ConnectTimeout\\\=60", in particular
how the equals symbol is escaped.

I'm looking in the function:

static pid_t do_cmd(char *cmd, char *machine, char *user, char
**remote_argv, int remote_argc,
   int *f_in_p, int *f_out_p)

This function splits based purely on whitespace:

 args[argc++] = t;
 while (*f != ' ' || in_quote) {
 // consume token...

I feel that this function should also handle backslash escapes.

I also checked using strace and it appears that this is the issue, but
I'm open to suggestions/ideas.

Kind regards,
Samuel

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html