help with rsync and Missing trailing-" in remote-shell command.

[dpich]

I have three servers.  nginx00 (master nginx server) and nginx01 and nginx02 
(slave nginx servers).

nginx00 is a staging environment that I do testing on.  Once it goes live I run 
a bash script that syncs the config files and restarts the nginx daemon.  It's 
working fine as root.  However, I am bound by rules in my company to run 
commands as users and not the root user.  

I have setup keys for ssh for all my sysadmin users, and it works fine if I 
just us it.

rsync -q -avz --delete -e 'ssh -i /home/dpich/.ssh/id_rsa' --rsync-path='sudo 
rsync' /etc/nginx/ dpich@balancer01:/etc/nginx/

If I run it in this bash script...

#!/bin/bash

currentuser=$(whoami)
OPTS=(-q -avz --delete -e \'ssh -i /home/$currentuser/.ssh/id_rsa\' 
--rsync-path=\'sudo rsync\')
SRC=(/etc/nginx/)
RSYNC_CMD=(rsync)

for h in nginx0{1..2} ; do
  DST=$currentuser'@'$h':/etc/nginx/'
  rsync ${OPTS[@]} ${SRC} ${DST}
done

I keep getting this error:
dpich@nginx00:~$ /usr/sbin/deploy2.sh
Missing trailing-" in remote-shell command.
rsync error: syntax or usage error (code 1) at main.c(430) [sender=3.1.1]
Missing trailing-" in remote-shell command.
rsync error: syntax or usage error (code 1) at main.c(430) [sender=3.1.1]
dpich@nginx00:~$

I'm 1000% convinced it has to do with quotes, but I haven't been able to solve 
this.

Any help would be appreciated.

Re: Backslashed space in variable/argument expansion does not prevent splitting.

[Greg Wooledge]

On Fri, Jun 17, 2016 at 11:21:42AM +0100, Dean Wakerley wrote:
>  I ran into this problem when trying to buiild a TCL extension. Building
>  the extension requires pulling in    with source tclConfig.sh which
>  provides various settings. TCL_DEFS contains space separated -D
>  arguments    to pass to gcc. Some of these arguements contains space
>  which are backslashed to prevent splitting.

I've never seen a Tcl extension that requires sourcing a .sh file.
Granted, the Tcl extension space is a royal mess, but *sane* ones
are supposed to follow a standard known as TEA which is basically GNU
autoconf.

I'm guessing you're stuck with an insane one.  It would help to know
which one it is.  Maybe you're simply doing something wrong.

> # TCL_DEFS is actually source-d from TCL config script so have no
> # control over it.
>  TCL_DEFS='-DPACKAGE_NAME=\"tcl\" -DPACKAGE_TARNAME=\"tcl\" -
>  DPACKAGE_VERSION=\"8.5\" -DPACKAGE_STRING=\"tcl\ 8.5\" -
>  DPACKAGE_BUGREPORT=\"\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -

What you have here is from a file that is installed as part of Tcl
itself.  I have no idea why your extension thinks you should be sourcing
this file, or what it wants you to do with it.

> ./echo_args.sh ${TCL_DEFS}

I have my own args program.  This is what mine shows, if I pass it an
unquoted $TCL_DEFS to perform the word-splitting that you seem to want:

wooledg@wooledg:~$ source /opt/ebase/lib/tclConfig.sh 
wooledg@wooledg:~$ args $TCL_DEFS
64 args: <-DPACKAGE_NAME=\"tcl\"> <-DPACKAGE_TARNAME=\"tcl\"> 
<-DPACKAGE_VERSION=\"8.6\"> <-DPACKAGE_STRING=\"tcl\> <8.6\"> [[...]]

I've snipped it, because it's quite long.

Now bear in mind, these are not the ACTUAL arguments you would want
to pass to gcc.  These are options that have been spelled out with extra
backslashes in them because they are meant to be dropped into a Makefile,
or some other shell environment.  The shell environment will strip out
the backslashes as it interprets the command.

See how -DPACKAGE_STRING (fourth and fifth args) has been split into
two words.  This is because TCL_DEFS is NOT actually an array of arguments
that you can pass directly to a program.  It's a single string that has to
be copied out, into a Makefile, or a shell script, and then interpreted.

In other words, this will not work:

$ gcc $TCL_DEFS

It needs to be this:

$ eval gcc "$TCL_DEFS"

Here's what that looks like, using my args prorgam:

wooledg@wooledg:~$ eval args gcc "$TCL_DEFS"
63 args:  <-DPACKAGE_NAME="tcl"> <-DPACKAGE_TARNAME="tcl"> 
<-DPACKAGE_VERSION="8.6"> <-DPACKAGE_STRING="tcl 8.6"> [[...]]

Again, I've snipped it.

Either your Tcl extension's documentation is incorrect, or you are doing
something wrong.  That's all I can come up with.  There is no bash bug
here.

You might be better served by asking for help on this extension's
mailing list, if it has one.  If it doesn't (wouldn't be a surprise),
then on a general Tcl mailing list or newsgroup.

Re: Backslashed space in variable/argument expansion does not prevent splitting.

[Chet Ramey]

On 6/17/16 6:21 AM, Dean Wakerley wrote:

> Bash Version: 4.3
> Patch Level: 42
> Release Status: release
> 
> Description:
>  The expansion of a variable containing spaces splits on all spaces
> including escaped/backslashed space.

This analysis isn't correct, and the description above describes the
correct behavior.  A backslash in the expansion of a variable does not
have any special meaning to word splitting.  Backslashes are special to
the shell parser; if you want to have them interpreted, you need to run
the text through the parser.  This is usually accomplished using `eval'.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Backslashed space in variable/argument expansion does not prevent splitting.

[Davide Brini]

On Fri, 17 Jun 2016 11:21:42 +0100, Dean Wakerley  wrote:

> Description: The expansion of a variable containing spaces splits on all
> spaces including escaped/backslashed space.

It's not a bug, you probably want

eval ./echo_args.sh ${TCL_DEFS}

but you're sitting on very thin ice here.

-- 
D.

Feature request: Support Linux packetized pipes

[sbaugh]

Hi bug-bash,

The Linux kernel supports creating pipes in a mode (enabled by passing
the O_DIRECT flag, and known sometimes as a "packetized pipe") that
respects message boundaries, to an extent.

An excerpt from the Linux manpage pipe(2):

O_DIRECT (since Linux 3.4)
Create a pipe that performs I/O in "packet" mode.  Each
write(2)  to  the pipe  is  dealt  with as  a  separate
packet, and read(2)s from the pipe will read one packet
at a time.  Note the following points:

*  Writes of greater than  PIPE_BUF bytes (see pipe(7))
   will be  split into multiple packets.   The constant
   PIPE_BUF is defined in .

*  If a read(2) specifies a buffer size that is smaller
   than the  next packet, then the  requested number of
   bytes are read,  and the excess bytes  in the packet
   are discarded.  Specifying a buffer size of PIPE_BUF
   will  be sufficient  to  read  the largest  possible
   packets (see the previous point).

*  Zero-length packets  are not supported.   (A read(2)
   that specifies a buffer size of zero is a no-op, and
   returns 0.)

Older  kernels  that  do  not support  this  flag  will
indicate this via an EINVAL error.

I would like to use a packetized pipe in a conventional shell pipeline
to support parallelism. Multiple processes can read from the same pipe
at the same time and not receive any partial lines, as long as the pipe
is being written to one-line-at-a-time, with lines under PIPE_BUF size.
(Parallelism in shell pipelines is an interest of mine; I wrote about
one technique here: https://catern.com/posts/pipes.html but bash support
of packetized pipes would make this much less painful)

So, it would be nice if it was possible in bash to somehow explicitly
request that a pipe between two processes be packetized (on systems that
support this behavior).

I can go into more detail about my use cases, and I can work on adding
such support. Before I do so, though, does this sound like a good
feature to add to bash?

Thanks!

Backslashed space in variable/argument expansion does not prevent splitting.

[Dean Wakerley]

From: deanw
 To: bug-bash@gnu.org
 Subject: Backslashed space in variable expansion does not prevent splitting.
Configuration Information [Automatically generated, do not change]:
Machine: i686 OS: cygwin Compiler: gcc Compilation CFLAGS:  -
DPROGRAM='bash.exe' -DCONF_HOSTTYPE='i686' -DCONF_OSTYPE='cygwin' 
-DCONF_MACHTYPE='i686-pc-
cygwin' -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -
DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -DRECYCLES_PIDS   -I.  
-I/usr/src/bash-4.3.42-4.i686/src/bash-
4.3 -I/usr/src/bash-4.3.42-4.i686/src/bash-4.3/include 
-I/usr/src/bash-4.3.42-4.i686/src/bash-
4.3/lib  -DWORDEXP_OPTION -ggdb -O2 -pipe -Wimplicit-function-
declaration -fdebug-prefix-map=/usr/src/bash-4.3.42-
4.i686/build=/usr/src/debug/bash-4.3.42-4 
-fdebug-prefix-map=/usr/src/bash-4.3.42-4.i686/src/bash-
4.3=/usr/src/debug/bash-4.3.42-4 uname output: CYGWIN_NT-6.1-WOW
DW03PC02 2.5.1(0.297/5/3) 2016-04-21 22:12 i686 Cygwin Machine Type:
i686-pc-cygwin
Bash Version: 4.3 Patch Level: 42 Release Status: release
Description:
  The expansion of a variable containing spaces splits on all spaces including 
escaped/backslashed space.

 I ran into this problem when trying to buiild a TCL extension. Building
 the extension requires pulling in    with source tclConfig.sh which
 provides various settings. TCL_DEFS contains space separated -D
 arguments    to pass to gcc. Some of these arguements contains space
 which are backslashed to prevent splitting.

 Argument expansion is not distinguishing between space and backslased
 space.    I have tried 3.00.16(1)-release on FedoraCore 4, 4.2.10(1)-
 release on FedoraCore 16, 4.3.42(4)-release on Cygwin 2.5.1    with the
 same outcome.
Repeat-By:
     Run the ./bad_exp.sh script. It explains the desired behaviour and 
demonstrates issue. I include below the contents of two script and my output 
log.
=Start File ./bad_exp.sh
#!/usr/bin/env bash
# Issue: Backslash space is not preventing argument splitting
echo $BASH_VERSION
# TCL_DEFS is actually source-d from TCL config script so have no
# control over it.
 TCL_DEFS='-DPACKAGE_NAME=\"tcl\" -DPACKAGE_TARNAME=\"tcl\" -
 DPACKAGE_VERSION=\"8.5\" -DPACKAGE_STRING=\"tcl\ 8.5\" -
 DPACKAGE_BUGREPORT=\"\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -
 DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -
 DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -
 DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_LIMITS_H=1 -
 DHAVE_SYS_PARAM_H=1 -DTCL_CFGVAL_ENCODING=\"iso8859-1\" -
 DMODULE_SCOPE=extern\ __attribute__\(\(__visibility__\(\"hidden\"\)\)\)
 -DTCL_CFG_DO64BIT=1 -DHAVE_CAST_TO_UNION=1 -DTCL_SHLIB_EXT=\".so\" -
 DNDEBUG=1 -DTCL_CFG_OPTIMIZED=1 -DTCL_TOMMATH=1 -DMP_PREC=4 -
 D_LARGEFILE64_SOURCE=1 -DTCL_WIDE_INT_IS_LONG=1 -DHAVE_GETCWD=1 -
 DHAVE_OPENDIR=1 -DHAVE_STRTOL=1 -DHAVE_WAITPID=1 -DHAVE_GETADDRINFO=1
 -DUSE_TERMIOS=1 -DHAVE_SYS_TIME_H=1 -DTIME_WITH_SYS_TIME=1 -
 DHAVE_GMTIME_R=1 -DHAVE_LOCALTIME_R=1 -DHAVE_MKTIME=1 -
 DHAVE_TM_GMTOFF=1 -DHAVE_TIMEZONE_VAR=1 -DHAVE_STRUCT_STAT_ST_BLOCKS=1
 -DHAVE_STRUCT_STAT_ST_BLKSIZE=1 -DHAVE_BLKCNT_T=1 -DHAVE_INTPTR_T=1 -
 DHAVE_UINTPTR_T=1 -DHAVE_SIGNED_CHAR=1 -DHAVE_LANGINFO=1 -
 DHAVE_SYS_IOCTL_H=1 -DTCL_UNLOAD_DLLS=1 -DHAVE_CPUID=1 '
./echo_args.sh ${TCL_DEFS}
echo 'Reduction of problem' echo 'The desired result is echo_args should
receive three arguments arg0, arg1 and "arg 2"' T='arg0 arg1 arg\ 2'
echo 'This receives four arguments, wrongly splitting on the backslashed
space' ./echo_args.sh $T
echo 'This correctly receives one argument but is not the desired
result' ./echo_args.sh "$T"
 =End File ./bad_exp.sh =Start File echo_args.sh
 #!/usr/bin/env bash
 # Script to display command line arguments
echo "${#} arguments" i=0 for arg in "${@}" do     echo "${i}:$arg"
((i++)) done
 =End File echo_args.sh =Start output log
 4.2.10(1)-release 53 arguments :-DPACKAGE_NAME=\"tcl\"
  1:-DPACKAGE_TARNAME=\"tcl\"
  2:-DPACKAGE_VERSION=\"8.5\"
  3:-DPACKAGE_STRING=\"tcl\
 4:8.5\"
  5:-DPACKAGE_BUGREPORT=\"\"
  6:-DSTDC_HEADERS=1
  7:-DHAVE_SYS_TYPES_H=1
  8:-DHAVE_SYS_STAT_H=1
  9:-DHAVE_STDLIB_H=1
 10:-DHAVE_STRING_H=1
 11:-DHAVE_MEMORY_H=1
 12:-DHAVE_STRINGS_H=1
 13:-DHAVE_INTTYPES_H=1
 14:-DHAVE_STDINT_H=1
 15:-DHAVE_UNISTD_H=1
 16:-DHAVE_LIMITS_H=1
 17:-DHAVE_SYS_PARAM_H=1
 18:-DTCL_CFGVAL_ENCODING=\"iso8859-1\"
 19:-DMODULE_SCOPE=extern\
 20:__attribute__\(\(__visibility__\(\"hidden\"\)\)\)
 21:-DTCL_CFG_DO64BIT=1
 22:-DHAVE_CAST_TO_UNION=1
 23:-DTCL_SHLIB_EXT=\".so\"
 24:-DNDEBUG=1
 25:-DTCL_CFG_OPTIMIZED=1
 26:-DTCL_TOMMATH=1
 27:-DMP_PREC=4
 28:-D_LARGEFILE64_SOURCE=1
 29:-DTCL_WIDE_INT_IS_LONG=1
 30:-DHAVE_GETCWD=1
 31:-DHAVE_OPENDIR=1
 32:-DHAVE_STRTOL=1
 33:-DHAVE_WAITPID=1
 34:-DHAVE_GETADDRINFO=1
 35:-DUSE_TERMIOS=1
 36:-DHAVE_SYS_TIME_H=1
 37:-DTIME_WITH_SYS_TIME=1
 38:-DHAVE_GMTIME_R=1
 39:-DHAVE_LOCALTIME_R=1
 40:-DHAVE_MKTIME=1
 41:-DHAVE_TM_GMTOFF=1
 42:-DHAVE_TIMEZONE_VAR=1
 43:-DHAVE_STRUCT_STAT_ST_BLOCKS=1
 44:-DHAVE_STRUCT_STAT_ST_BLKSIZE=1