Re: -e escape rule

[Joe]

On 11/20/2016 02:55 PM, francis.montag...@inria.fr wrote:

Hi

On Sun, 30 Oct 2016 20:06:00 +1300 Samuel Williams wrote:


But it's also a surprise that backslash escape sequences don't work
according to intuition of how commands are normally executed. If you
supplied the string in -e to system, it would work as expected..
Unfortunately, this is the default when using Shellwords.join in Ruby.
So, I had to write a custom RSync "join" function to produce an
appropriate command for -e argument.

I guess you are building from ruby a single string for the whole rsync
command, and that you give it to the system ruby command.

If this is it, it would be simpler to build instead an array of
strings and to call system with this array as arguments.

This way, ruby will call directly rsync without calling an extra shell
and you will not have to re-quote the arguments, in particular the one
for -e.

Francis

A couple of years ago I was having somewhat similar problems building an 
rsync command in bash and someone on this list suggested exactly the 
same solution. It really makes things simpler - and easier to read/maintain.


Joe

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Francis . Montagnac]

Hi

On Sun, 30 Oct 2016 20:06:00 +1300 Samuel Williams wrote:

> But it's also a surprise that backslash escape sequences don't work
> according to intuition of how commands are normally executed. If you
> supplied the string in -e to system, it would work as expected..

> Unfortunately, this is the default when using Shellwords.join in Ruby.
> So, I had to write a custom RSync "join" function to produce an
> appropriate command for -e argument.

I guess you are building from ruby a single string for the whole rsync
command, and that you give it to the system ruby command.

If this is it, it would be simpler to build instead an array of
strings and to call system with this array as arguments.

This way, ruby will call directly rsync without calling an extra shell
and you will not have to re-quote the arguments, in particular the one
for -e.

Francis

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Samuel Williams]

> assuming that a shell is being used is invalid

I never made this assumption. I looked directly at the source code and
I stated that "I feel that this function should also handle backslash
escapes."

I think the assumption that splitting the command works the same way
as (all?) major shells, is not inappropriate given the circumstances,
and it seems like you agree.

> but probably non-zero

Well, I understand where you are coming from. You don't want to break
existing code.

The work-around is to use quotes to escape the white-space sequence. It's okay..

But it's also a surprise that backslash escape sequences don't work
according to intuition of how commands are normally executed. If you
supplied the string in -e to system, it would work as expected..

Unfortunately, this is the default when using Shellwords.join in Ruby.
So, I had to write a custom RSync "join" function to produce an
appropriate command for -e argument.


On 30 October 2016 at 04:49, Wayne Davison  wrote:
> On Sat, Oct 29, 2016 at 5:36 AM, Samuel Williams
>  wrote:
>>
>> The command
>>
>> ssh -l backup -i /etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes
>>
>> Is a correct and valid shell command.
>
>
> It is, but there is no shell involved, and assuming that a shell is being
> used is invalid. Adding backslash escaping now could potentially screw up
> anyone currently passing a backslash in their existing rsync scripts (a
> small group of people, but probably non-zero). It's tempting to go ahead and
> do this, but I think I'll just leave it as it is.
>
> If you want to work around that buggy escaping library, you could change
> "ssh" to "ssh-nobs" and put the following into that file:
>
> #!/usr/bin/perl
> exec 'ssh', map { s/\\(.)/$1/g; $_ } @ARGV;
>
> That adds backslash removal on the way to the ssh.  It does not allow you to
> backslash escape spaces, though, but someone could extend the script to
> support that.
>
> ..wayne..
>

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Wayne Davison]

On Sat, Oct 29, 2016 at 5:36 AM, Samuel Williams <
space.ship.travel...@gmail.com> wrote:

> The command
>
> ssh -l backup -i /etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes
>
> Is a correct and valid shell command.
>

It is, but there is no shell involved, and assuming that a shell is being
used is invalid. Adding backslash escaping now could potentially screw up
anyone currently passing a backslash in their existing rsync scripts (a
small group of people, but probably non-zero). It's tempting to go ahead
and do this, but I think I'll just leave it as it is.

If you want to work around that buggy escaping library, you could change
"ssh" to "ssh-nobs" and put the following into that file:

#!/usr/bin/perl
exec 'ssh', map { s/\\(.)/$1/g; $_ } @ARGV;

That adds backslash removal on the way to the ssh.  It does not allow you
to backslash escape spaces, though, but someone could extend the script to
support that.

..wayne..
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Samuel Williams]

> The point is that the original escaping DOUBLE escapes an equals sign:
> foo\\\=bar
> It shouldn't, there's no reason to.

If you paste into your command line:

rsync -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes

The list of arguments would be (i.e. the values in ARGV):

['rsync', '-e', 'ssh -l backup -i /etc/synco/id_rsa -o
ConnectTimeout\=60 -o BatchMode\=yes']

The command

ssh -l backup -i /etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes

Is a correct and valid shell command.

When RSync parses this in do_cmd, it should convert the '\=` sequence
into '=' but it doesn't.. This intuition is derived from the fact that
if you instead passed the string to `system('ssh -l backup -i
/etc/synco/id_rsa -o ConnectTimeout\=60 -o BatchMode\=yes')` that the
ARGV generated would be ['ssh', '-l', 'backup', '-i',
'/etc/synco/id_rsa', '-o', 'ConnectTimeout=60', '-o',
'BatchMode=yes'].

Basically, even if there WAS no reason to do so, doesn't mean it's
invalid or undesirable. In theory, even passing in -e \\s\\s\\h should
be valid.

On 30 October 2016 at 01:13, Paul Slootman  wrote:
> On Sat 29 Oct 2016, Samuel Williams wrote:
>
>> I'm not proposing some additional characters to split on, but quite
>> the opposite, to handle the backslash escaped spaces correctly and NOT
>> split. Rest assured, there is no bug with the original escaping. For
>> your edification:
>>
>> $ echo \I\'\m\ \a\ \s\t\r\i\n\g
>> I'm a string
>
> The point is that the original escaping DOUBLE escapes an equals sign:
>  foo\\\=bar
> It shouldn't, there's no reason to.
>
>
> Paul
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Paul Slootman]

On Sat 29 Oct 2016, Samuel Williams wrote:

> I'm not proposing some additional characters to split on, but quite
> the opposite, to handle the backslash escaped spaces correctly and NOT
> split. Rest assured, there is no bug with the original escaping. For
> your edification:
> 
> $ echo \I\'\m\ \a\ \s\t\r\i\n\g
> I'm a string

The point is that the original escaping DOUBLE escapes an equals sign:
 foo\\\=bar
It shouldn't, there's no reason to.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Samuel Williams]

> Yeah, it only does space-splitting and that's all it will ever do. It still 
> looks to me like there is a bug in the original escaping, since any command 
> receiving that string is receiving a backslash that is not supposed to be 
> there. It should only be escaping the string enough to get it to rsync, not 
> trying to guess what rsync is going to do with it after it gets it.

I'm not proposing some additional characters to split on, but quite
the opposite, to handle the backslash escaped spaces correctly and NOT
split. Rest assured, there is no bug with the original escaping. For
your edification:

$ echo \I\'\m\ \a\ \s\t\r\i\n\g
I'm a string

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Samuel Williams]

Kevin, I agree with what you are saying on some level, but I don't
think the code does what you think it does. rsync -e "foo\\ bar" will
be executed by the shell and yield a cmd string in the do_cmd function
of "foo\ bar". This will be split incorrectly into an argv of ["foo\",
"bar"]. I'm not sure in what reality this makes any sense. All I'm
proposing is that the do_cmd split function interprets the -e as a
string that would normally be executed on the command line, and if it
needs to be split, follow the normal rules of the command line for
handling whitespace.

On 29 October 2016 at 16:24, Samuel Williams
 wrote:
> Kevin, I agree with what you are saying on some level, but I don't
> think the code does what you think it does. rsync -e "foo\\ bar" will
> be executed by the shell and yield a cmd string in the do_cmd function
> of "foo\ bar". This will be split incorrectly into an argv of ["foo\",
> "bar"]. I'm not sure in what reality this makes any sense. All I'm
> proposing is that the do_cmd split function interprets the -e as a
> string that would normally be executed on the command line, and if it
> needs to be split, follow the normal rules of the command line for
> handling whitespace.
>
> On 29 October 2016 at 15:41, Kevin Korb  wrote:
>> A \  is a shell escape.  Every level of shell consumes one level of
>> escape.  For every session of a shell every example "\ " becomes " " and
>> every example of "\\" becomes "\".
>>
>> The raync command line is a mix of local shell commands + commands run
>> via a remote shell via ssh.
>>
>> The simple fact is that if only \ escapes are used, the local shell will
>> utilize those escapes and rsync will never see them.
>>
>> For every level of shell an escape \ does its job.
>>
>> If you want rsync to interpret a space char as other than whitespace
>> then a \ is one way to do so.
>>
>> But one we are talking about the other side of an rsync we are talking
>> about a local command line that consumes a \.  Then we are talking about
>> a remote command line that consumes another \.  So, within the -e
>> parameter, multiple stacked \ characters are required.
>>
>> Simply put, it is absolutely absurd to \ escape an rsync command line
>> while interpreting the subshell -e parameter and the remote shell
>> parameters the same way.
>>
>> At best you are insisting that ssh do something entirely nonsensical
>> while ensuring that the first s in "ssh" is interpreted as an "s".
>>
>>
>> On 10/28/2016 10:10 PM, Wayne Davison wrote:
>>>
>>> On Fri, Oct 28, 2016 at 5:39 AM, Samuel Williams
>>> >
>>> wrote:
>>>
>>> Rsync passed the backslashes through without dealing with them.
>>>
>>>
>>> Yeah, it only does space-splitting and that's all it will ever do. It
>>> still looks to me like there is a bug in the original escaping, since
>>> any command receiving that string is receiving a backslash that is not
>>> supposed to be there. It should only be escaping the string enough to
>>> get it to rsync, not trying to guess what rsync is going to do with it
>>> after it gets it.
>>>
>>> To work around that bug, you could consider using an ssh-calling shell
>>> script instead of manually specifying the ssh command and args to rsync.
>>>
>>> ..wayne..
>>>
>>>
>>
>> --
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>> Kevin Korb  Phone:(407) 252-6853
>> Systems Administrator   Internet:
>> FutureQuest, Inc.   ke...@futurequest.net  (work)
>> Orlando, Floridak...@sanitarium.net (personal)
>> Web page:   http://www.sanitarium.net/
>> PGP public key available on web site.
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>
>>
>> --
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options: 
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Kevin Korb]

A \  is a shell escape.  Every level of shell consumes one level of
escape.  For every session of a shell every example "\ " becomes " " and
every example of "\\" becomes "\".

The raync command line is a mix of local shell commands + commands run
via a remote shell via ssh.

The simple fact is that if only \ escapes are used, the local shell will
utilize those escapes and rsync will never see them.

For every level of shell an escape \ does its job.

If you want rsync to interpret a space char as other than whitespace
then a \ is one way to do so.

But one we are talking about the other side of an rsync we are talking
about a local command line that consumes a \.  Then we are talking about
a remote command line that consumes another \.  So, within the -e
parameter, multiple stacked \ characters are required.

Simply put, it is absolutely absurd to \ escape an rsync command line
while interpreting the subshell -e parameter and the remote shell
parameters the same way.

At best you are insisting that ssh do something entirely nonsensical
while ensuring that the first s in "ssh" is interpreted as an "s".


On 10/28/2016 10:10 PM, Wayne Davison wrote:
> 
> On Fri, Oct 28, 2016 at 5:39 AM, Samuel Williams
> >
> wrote:
> 
> Rsync passed the backslashes through without dealing with them.
> 
> 
> Yeah, it only does space-splitting and that's all it will ever do. It
> still looks to me like there is a bug in the original escaping, since
> any command receiving that string is receiving a backslash that is not
> supposed to be there. It should only be escaping the string enough to
> get it to rsync, not trying to guess what rsync is going to do with it
> after it gets it.
> 
> To work around that bug, you could consider using an ssh-calling shell
> script instead of manually specifying the ssh command and args to rsync.
> 
> ..wayne..
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Wayne Davison]

On Fri, Oct 28, 2016 at 5:39 AM, Samuel Williams <
space.ship.travel...@gmail.com> wrote:

> Rsync passed the backslashes through without dealing with them.
>

Yeah, it only does space-splitting and that's all it will ever do. It still
looks to me like there is a bug in the original escaping, since any command
receiving that string is receiving a backslash that is not supposed to be
there. It should only be escaping the string enough to get it to rsync, not
trying to guess what rsync is going to do with it after it gets it.

To work around that bug, you could consider using an ssh-calling shell
script instead of manually specifying the ssh command and args to rsync.

..wayne..
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Samuel Williams]

Wayne, I guess if you apply standard shell escaping logic you end up
with a string which isn't processed correctly, yes, ssh complains, but
only because Rsync passed the backslashes through without dealing with
them. My opinion is that if Rsync is splitting a string based on
whitespace it also needs to take into account backslashes.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Samuel Williams]

Hmm, so after reviewing this in a bit more detail.. it would be very
nice if at least backslash escapes would be supported. Quoted strings
are pretty ugly, especially when nested several levels. Anyway, just
my 2 cents.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Samuel Williams]

Hi Dave, thanks for point that out. I didn't realise there was a
detailed explanation of that field in the man page, I only saw the
summary. Yes, that clearly explains how it's supposed to work.

On 21 October 2016 at 01:46, Dave Howorth  wrote:
> On 2016-10-20 10:24, Samuel Williams wrote:
>>
>> Hello,
>>
>> I'm using Ruby's Shellwords module, which generates a string from an
>> array, suitable for shell evaluation.
>>
>> Ruby's implementation prefers escaping whitespace with a backslash
>> rather than quotes. However, this appears to cause some kind of issue
>> in Rsync when it computes argv from -e option.
>
>
> The man page for rsync 
> explicitly forbids the use of backslashes:
>
> "Command-line arguments are permitted in COMMAND provided that COMMAND is
> presented to rsync as a single argument. You must use spaces (not tabs or
> other whitespace) to separate the command and args from each other, and you
> can use single- and/or double-quotes to preserve spaces in an argument (but
> not backslashes)."
>
> So I think the rest of the question is moot ...
>
> Cheers, Dave
>
>
>> Here is an example command generated by some Ruby code:
>>
>> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
>> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
>> ../../latest/etc/ /etc/
>> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>>
>> We can check that something like this is valid:
>>
>> files% echo foo\ bar\\\=baz
>> foo bar\=baz -- what Rsync should be receiving
>> files% echo foo bar\=baz
>> foo bar=baz  -- What Rsync should be executing
>>
>> However this gives me an error
>>
>> command-line: line 0: Bad configuration option: connecttimeout\\
>> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
>> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
>>
>> I think the problem here is the "ConnectTimeout\\\=60", in particular
>> how the equals symbol is escaped.
>>
>> I'm looking in the function:
>>
>> static pid_t do_cmd(char *cmd, char *machine, char *user, char
>> **remote_argv, int remote_argc,
>>int *f_in_p, int *f_out_p)
>>
>> This function splits based purely on whitespace:
>>
>>  args[argc++] = t;
>>  while (*f != ' ' || in_quote) {
>>  // consume token...
>>
>> I feel that this function should also handle backslash escapes.
>>
>> I also checked using strace and it appears that this is the issue, but
>> I'm open to suggestions/ideas.
>>
>> Kind regards,
>> Samuel
>>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Kevin Korb]

The \ escapes are for the shell.  Rsync never sees them and therefore
can't honor them.

On 10/20/2016 05:24 AM, Samuel Williams wrote:
> Hello,
> 
> I'm using Ruby's Shellwords module, which generates a string from an
> array, suitable for shell evaluation.
> 
> Ruby's implementation prefers escaping whitespace with a backslash
> rather than quotes. However, this appears to cause some kind of issue
> in Rsync when it computes argv from -e option.
> 
> Here is an example command generated by some Ruby code:
> 
> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
> ../../latest/etc/ /etc/
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
> 
> We can check that something like this is valid:
> 
> files% echo foo\ bar\\\=baz
> foo bar\=baz -- what Rsync should be receiving
> files% echo foo bar\=baz
> foo bar=baz  -- What Rsync should be executing
> 
> However this gives me an error
> 
> command-line: line 0: Bad configuration option: connecttimeout\\
> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
> 
> I think the problem here is the "ConnectTimeout\\\=60", in particular
> how the equals symbol is escaped.
> 
> I'm looking in the function:
> 
> static pid_t do_cmd(char *cmd, char *machine, char *user, char
> **remote_argv, int remote_argc,
>int *f_in_p, int *f_out_p)
> 
> This function splits based purely on whitespace:
> 
>  args[argc++] = t;
>  while (*f != ' ' || in_quote) {
>  // consume token...
> 
> I feel that this function should also handle backslash escapes.
> 
> I also checked using strace and it appears that this is the issue, but
> I'm open to suggestions/ideas.
> 
> Kind regards,
> Samuel
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Dave Howorth]

On 2016-10-20 10:24, Samuel Williams wrote:

Hello,

I'm using Ruby's Shellwords module, which generates a string from an
array, suitable for shell evaluation.

Ruby's implementation prefers escaping whitespace with a backslash
rather than quotes. However, this appears to cause some kind of issue
in Rsync when it computes argv from -e option.


The man page for rsync  
explicitly forbids the use of backslashes:


"Command-line arguments are permitted in COMMAND provided that COMMAND 
is presented to rsync as a single argument. You must use spaces (not 
tabs or other whitespace) to separate the command and args from each 
other, and you can use single- and/or double-quotes to preserve spaces 
in an argument (but not backslashes)."


So I think the rest of the question is moot ...

Cheers, Dave


Here is an example command generated by some Ruby code:

rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
../../latest/etc/ /etc/
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

We can check that something like this is valid:

files% echo foo\ bar\\\=baz
foo bar\=baz -- what Rsync should be receiving
files% echo foo bar\=baz
foo bar=baz  -- What Rsync should be executing

However this gives me an error

command-line: line 0: Bad configuration option: connecttimeout\\
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]

I think the problem here is the "ConnectTimeout\\\=60", in particular
how the equals symbol is escaped.

I'm looking in the function:

static pid_t do_cmd(char *cmd, char *machine, char *user, char
**remote_argv, int remote_argc,
   int *f_in_p, int *f_out_p)

This function splits based purely on whitespace:

 args[argc++] = t;
 while (*f != ' ' || in_quote) {
 // consume token...

I feel that this function should also handle backslash escapes.

I also checked using strace and it appears that this is the issue, but
I'm open to suggestions/ideas.

Kind regards,
Samuel



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Samuel Williams]

> There's no reason to escape an "=" sign in the above command.

Okay, so at a certain level I agree with you and that's how I've fixed
the issue.

All the stuff about config files is not feasible for a variety of
reasons - I'm aware of those options. I appreciate that you took the
time to explain them.

I think the problem is the assumptions in do_cmd about the contents of
the -e argument.

In my opinion, the way this is split needs to be deferred back to the
user's shell to work correctly. I'm not sure if there is an easy way
to do this that works for any kind of shell.

Assuming that -e is stored into rsh_cmd, and is user supplied for
executing on the shell, the only valid thing you can really do with it
is:

execl("/bin/sh", "sh", "-c", rsh_cmd, (char *) NULL);

(assuming that sh is the user's current shell, might want to use
$SHELL or something else).

It might make more sense if rsh_cmd used string substitution rather
than argument splitting, e.g. -e 'ssh $HOST'  this way, no splitting
would be required. In lots of ways this is more flexible. For example,
appending the host might not be useful in some contexts.

Anyway, rsync is pretty awesome. the -e option is only a minor
annoyance, but it would be nice if this discussion could result in a
tangible improvement.


On 21 October 2016 at 00:03, Paul Slootman  wrote:
> On Thu 20 Oct 2016, Samuel Williams wrote:
>>
>> I'm using Ruby's Shellwords module, which generates a string from an
>> array, suitable for shell evaluation.
>>
>> Ruby's implementation prefers escaping whitespace with a backslash
>> rather than quotes. However, this appears to cause some kind of issue
>> in Rsync when it computes argv from -e option.
>>
>> Here is an example command generated by some Ruby code:
>>
>> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
>> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
>> ../../latest/etc/ /etc/
>> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
> There's no reason to escape an "=" sign in the above command. Try:
>
> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ 
> ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ 
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
> Or even:
>
> rsync --archive --stats -e ssh\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ 
> -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ 
> bac...@example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
> I'd probably create an entry for this host in ~/.ssh/config :
>
> Host example.backup.server.com
> User backup
> IdentityFile /etc/synco/id_rsa
> ConnectTimeout 60
> BatchMode=yes
>
> and then just use:
>
> rsync --archive --stats --link-dest ../../latest/etc/ /etc/ 
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
>
>
> If you're dynamically doing this, you can pass a config file with -F.
>
>
> Paul
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Paul Slootman]

On Thu 20 Oct 2016, Samuel Williams wrote:
> 
> I'm using Ruby's Shellwords module, which generates a string from an
> array, suitable for shell evaluation.
> 
> Ruby's implementation prefers escaping whitespace with a backslash
> rather than quotes. However, this appears to cause some kind of issue
> in Rsync when it computes argv from -e option.
> 
> Here is an example command generated by some Ruby code:
> 
> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
> ../../latest/etc/ /etc/
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

There's no reason to escape an "=" sign in the above command. Try:

rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ 
ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ 
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

Or even:

rsync --archive --stats -e ssh\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ 
-o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ 
bac...@example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

I'd probably create an entry for this host in ~/.ssh/config :

Host example.backup.server.com
User backup
IdentityFile /etc/synco/id_rsa
ConnectTimeout 60
BatchMode=yes

and then just use:

rsync --archive --stats --link-dest ../../latest/etc/ /etc/ 
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/


If you're dynamically doing this, you can pass a config file with -F.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html