Re: embedded newlines in shell function variable expansion

[Dmitry Goncharov via Bug reports and discussion for GNU make]

On Fri, Oct 9, 2020 at 11:11 AM Byrnes, Robert  wrote:
>
> If I use this Makefile ...
>
> 
> bash$ cat Makefile
> FOO := $(shell echo $(ENTRIES) ; )
> BAR := $(shell echo $(ENTRIES)   )
>
> all:
> @echo FOO = $(FOO)
> @echo BAR = $(BAR)
>
> .PHONY: all
> 
>
> ... and set ENTRIES with embedded newlines, then this happens:
>
> 
> bash$ make 'ENTRIES=
> blartz
> blurfl
> '
> FOO = blartzblurfl
> BAR = blartz blurfl
> 


Good morning.

i opened
https://savannah.gnu.org/bugs/index.php?59247
and attached a patch.
Thanks for your report and the example.

regards, Dmitry

RE: embedded newlines in shell function variable expansion

[Byrnes, Robert]

> Hm, interesting.  I guess it's possible we are stripping out newlines before 
> invoking the shell.  I don't remember that being the case but it could be.
>
> I'm not exactly sure what your objective is ...

We use a make variable that can be set to a whitespace-separated list of 
pathnames.  These are passed to functions like $(notdir ...), and that broke 
when a script happened to use newlines instead of spaces as separators, because 
all of the pathnames were glued together into a single word when the newlines 
were stripped.  We don't actually care about preserving newlines, and would be 
happy if they turned into spaces.  But we do care about preserving whitespace 
that separates words.

> ... but note that as written this probably won't do what you want, because if 
> you run:
>
> /bin/sh -c 'echo foo
> bar'
>
> then it will echo "foo" and try to run the program "bar".

I guess that particular case could be protected by quotes as $(shell 
"$(ENTRIES)").

--
Bob

Re: embedded newlines in shell function variable expansion

[Paul Smith]

On Fri, 2020-10-09 at 18:00 +, Byrnes, Robert wrote:
> [pid 144497] execve("/bin/sh", ["/bin/sh", "-c", "/bin/echo blartzblurfl ; 
> "], 0x7ffd98ee6bc0 /* 72 vars */) = 0

Hm, interesting.  I guess it's possible we are stripping out newlines
before invoking the shell.  I don't remember that being the case but it
could be.

I'm not exactly sure what your objective is but note that as written
this probably won't do what you want, because if you run:

/bin/sh -c 'echo foo
bar'

then it will echo "foo" and try to run the program "bar".

> For reference, here's the modified Makefile ...

Thx but this is not needed to be included with all the messages...
that's why we have an email archive :).

RE: embedded newlines in shell function variable expansion

[Byrnes, Robert]

> My suspicion is that it's a difference between your /bin/echo command and 
> your shell's builtin echo command.
>
> In the first case (with the ";") because there's a special character in the 
> command GNU make will run a shell like this:
>
>/bin/sh -c 'echo ... ; '
>
> and the shell's built-in echo will be used (if it has one, which most shells 
> do).
>
> In the second case (without the ";"), there's no special character and so GNU 
> make runs the fast path which invokes the echo command directly, not using a 
> shell, and so it will use your PATH to find an "echo" program and run that.
>
> If you change both commands to use /bin/echo explicitly, do they work the 
> same way?

The behavior doesn't change if I use /bin/echo for both commands.

strace shows the subprocesses exactly as you describe, but the newlines have 
already been stripped (by make, I presume) before the shell is invoked:


execve("/usr/bin/make", ["make", "ENTRIES=\nblartz\nblurfl\n"], 0x7ffdac2aff40 
/* 72 vars */) = 0

[pid 144497] execve("/bin/sh", ["/bin/sh", "-c", "/bin/echo blartzblurfl ; "], 
0x7ffd98ee6bc0 /* 72 vars */) = 0
[pid 144497] execve("/bin/echo", ["/bin/echo", "blartzblurfl"], 0x1631fb0 /* 71 
vars */) = 0

[pid 144498] execve("/bin/echo", ["/bin/echo", "blartz\nblurfl\n"], 
0x7ffd98ee6bc0 /* 72 vars */) = 0


For reference, here's the modified Makefile ...


FOO := $(shell /bin/echo $(ENTRIES) ; )
BAR := $(shell /bin/echo $(ENTRIES)   )

all:
@echo FOO = $(FOO)
@echo BAR = $(BAR)

.PHONY: all


And this is how to reproduce the behavior ...


bash$ make 'ENTRIES=
blartz
blurfl
'
FOO = blartzblurfl
BAR = blartz blurfl


Thanks,

--
Bob

Re: embedded newlines in shell function variable expansion

[Paul Smith]

On Fri, 2020-10-09 at 15:03 +, Byrnes, Robert wrote:
> Why is the embedded whitespace removed in the first (FOO) case?  It
> seems related to the semicolon shell metacharacter ...

My suspicion is that it's a difference between your /bin/echo command
and your shell's builtin echo command.

In the first case (with the ";") because there's a special character in
the command GNU make will run a shell like this:

   /bin/sh -c 'echo ... ; '

and the shell's built-in echo will be used (if it has one, which most
shells do).

In the second case (without the ";"), there's no special character and
so GNU make runs the fast path which invokes the echo command directly,
not using a shell, and so it will use your PATH to find an "echo"
program and run that.

If you change both commands to use /bin/echo explicitly, do they work
the same way?

embedded newlines in shell function variable expansion

[Byrnes, Robert]

If I use this Makefile ...


bash$ cat Makefile
FOO := $(shell echo $(ENTRIES) ; )
BAR := $(shell echo $(ENTRIES)   )

all:
@echo FOO = $(FOO)
@echo BAR = $(BAR)

.PHONY: all


... and set ENTRIES with embedded newlines, then this happens:


bash$ make 'ENTRIES=
blartz
blurfl
'
FOO = blartzblurfl
BAR = blartz blurfl


Why is the embedded whitespace removed in the first (FOO) case?  It seems 
related to the semicolon shell metacharacter ...


bash$ make --version
GNU Make 4.3
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.


Thanks,

--
Bob