bug#20733: [PATCH] doc: mention 'for' syntax issue on older shells

[Eric Blake]

On 06/04/2015 02:59 PM, Eric Blake wrote:

 +good:
 +   @@list='$(list)'; for arg in $$list; do echo $$arg; done
 [...]

 Another option is to use:

for arg in $${-+$(list)}; do echo $$arg; done

 Furthermore, your suggestion mishandles a list with a bare }, whereas
 mine does not.  make good list='} a' should output $'}\na\n', not $'a}\n'.

Of course, your version has the benefit of preserving a list that
contains shell quoting (such as list = a  b intended to produce a
single string $'a  b') while mine eats it early (producing $'a' and
$'b').  But remember that my example is written in a context of
makefiles, where makefile variables tend to be designed to be split at
all whitespace boundaries and do not have an easy mechanism for handling
text that must have specific embedded spacing, and as such are unlikely
to require shell quoting in the first place.

Once again going to show that you have to think about each particular
problem in context (what sort of input will it be processing), rather
than blindly assuming that one formulation will be a catch-all solution.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

bug#20733: [PATCH] doc: mention 'for' syntax issue on older shells

[Eric Blake]

On 06/04/2015 02:36 PM, Stephane Chazelas wrote:
 2015-06-04 14:06:03 -0600, Eric Blake:
 [...]
 +$ @kbd{cat Makefile}
 +list =
 +bad:
 +@@for arg in $(list); do echo $$arg; done
 +good:
 +@@list='$(list)'; for arg in $$list; do echo $$arg; done
 [...]
 
 Another option is to use:
 
for arg in $${-+$(list)}; do echo $$arg; done
 
 
 That's ${var+value} that expands to value if var is set. $- is
 always set. $0 could also be used instead.

Also could use $$, $? (except $? doesn't always start life set if probed
as the first thing in some shells), $# (except some shells confuse
${varOPvalue} vs. ${#var} when # is used as var). But from a readability
perspective, I don't think that playing golf for the shortest construct
is helpful, when compared to demonstrating something that is more
commonly seen in practice.

Furthermore, your suggestion mishandles a list with a bare }, whereas
mine does not.  make good list='} a' should output $'}\na\n', not $'a}\n'.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

bug#20733: [PATCH] doc: mention 'for' syntax issue on older shells

[Stephane Chazelas]

2015-06-04 14:06:03 -0600, Eric Blake:
[...]
 +$ @kbd{cat Makefile}
 +list =
 +bad:
 + @@for arg in $(list); do echo $$arg; done
 +good:
 + @@list='$(list)'; for arg in $$list; do echo $$arg; done
[...]

Another option is to use:

   for arg in $${-+$(list)}; do echo $$arg; done


That's ${var+value} that expands to value if var is set. $- is
always set. $0 could also be used instead.

-- 
Stephane

bug#20733: coreutils build problem

[Eric Blake]

On 06/04/2015 02:17 PM, Nick Bowler wrote:
 Do these problematic shells properly handle:
 
   for arg
   do
 ...
   done
 
 when $# is 0?

Yes; all shells do.

$ /bin/sh -c 'echo $#; for arg
do echo hi; done; echo bye'
0
bye

 
 If so, can we use the following as a workaround?
 
   set x words-that-might-expand-to-nothing; shift
   for arg
   do
 ...
   done

Not ideal, when there are shorter invocations that can do the same.  And
it's not the expand-to-nothing that is a problem, it is the actual omission:

$ /bin/sh -c 'for a in ; do :; done'
/bin/sh: syntax error at line 1: `;' unexpected
$ /bin/sh -c 'for a in $nothing; do :; done'
$

so anything that expands in shell to nothing (whether $nothing, ``, or
use of a shell variable rather than a make variable) is fine; the
problem is most common in Makefiles where make variables are expanded
before the shell sees anything.

 
 I suppose that might be hard to do in this /particular/ case, as it
 looks like the error is coming from a make rule.  The Autoconf manual
 quite emphatically says to avoid 'for arg; do ...' by using a newline
 instead of a semicolon, a feat which is not easily done in make rules.

The manual also has a workaround for getting a literal newline in make
rules:
 nlinit=`echo 'nl='; echo ''`; eval $$nlinit

although that only gives you $nl containing newline, and you'd still
need another layer of 'eval' if you wanted to actually write the
makefile fragment to interpret the newline as a separator between the
var-name and 'do'.  So yeah, it's not worth it.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

bug#20733: coreutils build problem

[Nick Bowler]

On 2015-06-04 13:34 -0600, Eric Blake wrote:
 [adding autoconf]
 
 On 06/04/2015 01:17 PM, Paul Eggert wrote:
  
  On 06/04/2015 09:41 AM, Michael Felt wrote:
GEN  src/coreutils.h
  /bin/sh: 0403-057 Syntax error at line 1 : `;' is not expected.
  
 
  Port to POSIX shell, which doesn't allow 'for i in ; do ...'.
 
 Actually, POSIX _does_ allow for missing words between 'in' and the
 terminator (; or newline) before 'do' (whether by a word that expands to
 nothing, or by omission of words), requiring that the body of the for
 statement is skipped in that case:
 
 http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_04
 
 But it is also true that older shells did not always follow this rule,
 so you are indeed better off always supplying at least one word that
 won't be expanded into nothingness.
 
 Hmmm, I thought that autoconf would document it as a portability
 pitfall, but I don't see it under 'for' in this link:
 
 https://www.gnu.org/software/autoconf/manual/autoconf.html#Limitations-of-Builtins

Yikes!

Some questions:

Do these problematic shells properly handle:

  for arg
  do
...
  done

when $# is 0?

If so, can we use the following as a workaround?

  set x words-that-might-expand-to-nothing; shift
  for arg
  do
...
  done

I suppose that might be hard to do in this /particular/ case, as it
looks like the error is coming from a make rule.  The Autoconf manual
quite emphatically says to avoid 'for arg; do ...' by using a newline
instead of a semicolon, a feat which is not easily done in make rules.

Cheers,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

bug#20733: coreutils build problem

[Nick Bowler]

On 2015-06-04 14:41 -0600, Eric Blake wrote:
 On 06/04/2015 02:17 PM, Nick Bowler wrote:
  Do these problematic shells properly handle:
  
for arg
do
  ...
done
  
  when $# is 0?
 
 Yes; all shells do.

OK, good to know.

[...]
 it's not the expand-to-nothing that is a problem, it is the actual
 omission:
 
 $ /bin/sh -c 'for a in ; do :; done'
 /bin/sh: syntax error at line 1: `;' unexpected
 $ /bin/sh -c 'for a in $nothing; do :; done'
 $

Right, I see that now in the doc patch you posted.  So in Autoconf this
might turn up if you generate the list with m4, but is highly unlikely
to be an issue for pure shell code.

Thanks,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

bug#20733: coreutils build problem

[Paul Eggert]

Eric Blake wrote:

Actually, POSIX_does_  allow for missing words between 'in' and the
terminator (; or newline) before 'do' (whether by a word that expands to
nothing, or by omission of words), requiring that the body of the for
statement is skipped in that case:


Ah, sorry, I was thinking of previous versions of POSIX, which required at least 
one word after the 'in'.  You're right, the current POSIX version doesn't 
require this any more.  So the Solaris sh in question is conforming to the old 
POSIX standard but not to the current one.


I liked the approach with ``; I hadn't thought of that.  I used the coreutils 
fix I did because other coreutils code already fixed similar for-loop problems 
that way.

bug#20733: [PATCH] doc: mention 'for' syntax issue on older shells

[Eric Blake]

Based on a report by Michael Felt, via Paul Eggert on the
coreutils list.

* doc/autoconf.texi (Limitations of Builtins) for: Document
problem with 'for var in ;'.

Signed-off-by: Eric Blake ebl...@redhat.com
---
 doc/autoconf.texi | 34 ++
 1 file changed, 34 insertions(+)

diff --git a/doc/autoconf.texi b/doc/autoconf.texi
index 8c4302d..bbc9fa3 100644
--- a/doc/autoconf.texi
+++ b/doc/autoconf.texi
@@ -17757,6 +17757,40 @@ Limitations of Builtins
 word splitting on @samp{$@{1+$@@@}}; see @ref{Shell Substitutions},
 item @samp{$@@}, for more.

+Posix requires support for a @command{for} loop with no list after
+@code{in}.  However, Solaris @command{/bin/sh} treats that as a syntax
+error.  It is possible to work around this by providing any shell word
+that expands to nothing, or by ignoring an obvious sentinel.
+
+@example
+$ @kbd{/bin/sh -c 'for a in $empty; do echo hi; done'}
+$ @kbd{/bin/sh -c 'for a in ; do echo hi; done'}
+/bin/sh: syntax error at line 1: `;' unexpected
+@end example
+
+The problem is most common in makefile fragments where a make variable
+is used as the source of a list, but sometimes results in no text.  In
+that context, another common workaround is to use a shell variable
+rather than a make variable as the source of the list.
+
+@example
+$ @kbd{cat Makefile}
+list =
+bad:
+   @@for arg in $(list); do echo $$arg; done
+good:
+   @@list='$(list)'; for arg in $$list; do echo $$arg; done
+$ @kbd{make bad list='a b'}
+a
+b
+$ @kbd{make bad 21 | head -n1}
+sh: syntax error at line 1: `;' unexpected
+$ @kbd{make good}
+$ @kbd{make good list='a b'}
+a
+b
+@end example
+
 In Solaris @command{/bin/sh}, when the list of arguments of a
 @command{for} loop starts with @emph{unquoted} tokens looking like
 variable assignments, the loop is not executed on those tokens:
-- 
2.4.2

bug#20733: coreutils build problem

[Andreas Schwab]

Paul Eggert egg...@cs.ucla.edu writes:

 --- a/Makefile.am
 +++ b/Makefile.am
 @@ -195,7 +195,8 @@ check-git-hook-script-sync:
  # the selected tools when installing.
  install-exec-hook:
   $(AM_V_at)ctrans=$$(printf coreutils | sed -e $(transform));  \
 - for p in $(single_binary_progs); do \
 + for p in x $(single_binary_progs); do   \
 +   test $$p = x  continue; \

A better way to solve that is to use a shell variable:

progs='$(single_binary_progs)'; \
for p in $$progs; do

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
And now for something completely different.

bug#20733: coreutils build problem

[Paul Eggert]

Let's focus on 8.23 as 8.21 is pretty old

On 06/04/2015 09:41 AM, Michael Felt wrote:

  GEN  src/coreutils.h
/bin/sh: 0403-057 Syntax error at line 1 : `;' is not expected.


Ah, thanks, that's a bug in the build procedure, which I have fixed by 
pushing the attached patch.  Please give it a try.
From f6dc56f4754838069037a2e03553e8badc065c05 Mon Sep 17 00:00:00 2001
From: Paul Eggert egg...@cs.ucla.edu
Date: Thu, 4 Jun 2015 12:15:35 -0700
Subject: [PATCH] build: port single_binary_prog to POSIX shell

Problem reported privately by Michael Felt.
* Makefile.am (install-exec-hook):
* src/local.mk (src/coreutils_symlinks, src/coreutils_shebangs)
(clean-local):
Port to POSIX shell, which doesn't allow 'for i in ; do ...'.
---
 Makefile.am  | 3 ++-
 src/local.mk | 9 ++---
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/Makefile.am b/Makefile.am
index 7d7e381..f5543dd 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -195,7 +195,8 @@ check-git-hook-script-sync:
 # the selected tools when installing.
 install-exec-hook:
 	$(AM_V_at)ctrans=$$(printf coreutils | sed -e $(transform));	\
-	for p in $(single_binary_progs); do\
+	for p in x $(single_binary_progs); do\
+	  test $$p = x  continue;	\
 	  ptrans=$$(printf '%s' $$p | sed -e $(transform));		\
 	  rm -f $(DESTDIR)$(bindir)/$$ptrans$(EXEEXT) || exit $$?;	\
 	  if test x$(single_binary_install_type) = xshebangs; then	\
diff --git a/src/local.mk b/src/local.mk
index 5a3b1b3..eaeed08 100644
--- a/src/local.mk
+++ b/src/local.mk
@@ -427,7 +427,8 @@ endif SINGLE_BINARY
 CLEANFILES += src/coreutils_symlinks
 src/coreutils_symlinks: Makefile
 	$(AM_V_GEN)touch $@
-	$(AM_V_at)for i in $(single_binary_progs); do \
+	$(AM_V_at)for i in x $(single_binary_progs); do \
+		test $$i = x  continue; \
 		rm -f src/$$i$(EXEEXT) || exit $$?; \
 		$(LN_S) -s coreutils$(EXEEXT) src/$$i$(EXEEXT) || exit $$?; \
 	done
@@ -435,7 +436,8 @@ src/coreutils_symlinks: Makefile
 CLEANFILES += src/coreutils_shebangs
 src/coreutils_shebangs: Makefile
 	$(AM_V_GEN)touch $@
-	$(AM_V_at)for i in $(single_binary_progs); do \
+	$(AM_V_at)for i in x $(single_binary_progs); do \
+		test $$i = x  continue; \
 		rm -f src/$$i$(EXEEXT) || exit $$?; \
 		printf '#!%s --coreutils-prog-shebang=%s\n' \
 			$(abs_top_builddir)/src/coreutils$(EXEEXT) $$i \
@@ -444,7 +446,8 @@ src/coreutils_shebangs: Makefile
 	done
 
 clean-local:
-	$(AM_V_at)for i in $(single_binary_progs); do \
+	$(AM_V_at)for i in x $(single_binary_progs); do \
+		test $$i = x  continue; \
 		rm -f src/$$i$(EXEEXT) || exit $$?; \
 	done
 
-- 
2.1.0

bug#20733: coreutils build problem

[Eric Blake]

[adding autoconf]

On 06/04/2015 01:17 PM, Paul Eggert wrote:
 
 On 06/04/2015 09:41 AM, Michael Felt wrote:
   GEN  src/coreutils.h
 /bin/sh: 0403-057 Syntax error at line 1 : `;' is not expected.
 

 Port to POSIX shell, which doesn't allow 'for i in ; do ...'.

Actually, POSIX _does_ allow for missing words between 'in' and the
terminator (; or newline) before 'do' (whether by a word that expands to
nothing, or by omission of words), requiring that the body of the for
statement is skipped in that case:

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_04

But it is also true that older shells did not always follow this rule,
so you are indeed better off always supplying at least one word that
won't be expanded into nothingness.

Hmmm, I thought that autoconf would document it as a portability
pitfall, but I don't see it under 'for' in this link:

https://www.gnu.org/software/autoconf/manual/autoconf.html#Limitations-of-Builtins

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

bug#20733: coreutils build problem

[Eric Blake]

On 06/04/2015 01:35 PM, Andreas Schwab wrote:
 Paul Eggert egg...@cs.ucla.edu writes:
 
 --- a/Makefile.am
 +++ b/Makefile.am
 @@ -195,7 +195,8 @@ check-git-hook-script-sync:
  # the selected tools when installing.
  install-exec-hook:
  $(AM_V_at)ctrans=$$(printf coreutils | sed -e $(transform));  \
 -for p in $(single_binary_progs); do \
 +for p in x $(single_binary_progs); do   \
 +  test $$p = x  continue; \
 
 A better way to solve that is to use a shell variable:
 
 progs='$(single_binary_progs)'; \
 for p in $$progs; do

You can also use something that expands into a no-op shell word:

for p in `` $(single_binary_progs); do

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature