[bug #59154] Multiline environment variables handled poorly

[Jörg Schilling]

Follow-up Comment #16, bug #59154 (project make):

Hi Dan,

using BSD make as an example in general is not a good idea since BSD make
implements plenty of non-POSIX compliant "features".

Since this command however works fine with SunPro Make and smake and produces
the same output as bmake, it must be seen as a gmake bug that prevents gmake
from being used as a make alternative.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[anonymous]

Follow-up Comment #15, bug #59154 (project make):

Also: I have great respect for the difficulty of getting stuff like this
right.  (I recently stared down the maw of commandline issues in response
files in meson... https://github.com/mesonbuild/meson/pull/7245 ) so while I
may have sounded glib, I didn't mean to imply a nonbreaking fix would be easy.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[anonymous]

Follow-up Comment #14, bug #59154 (project make):

[comment #10 comment #10:]
> GNU makefiles will allow the following makefile:
> 
> 
> define SOMECOMMAND
> cd foo && echo one
> cd foo && echo two
> endef
> 
> all: ; $(SOMECOMMAND)
> 
> 
> to print both "one" and "two" when there is a subdirectory "foo" which is
empty.
> 
> If the variable is passed to a single shell then it will print an error at
the second "cd".

Thanks for the example!  I looked briefly in tests/, but didn't immediately
see a regression test that verifies this behavior.  I imagine there is one.  
Anyway, so the task for the intrepid complainer would be to
support the BSD usage while not breaking the usage described
above.  If that's impossible, well, that's that.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[Paul D. Smith]

Follow-up Comment #13, bug #59154 (project make):

Yes, I said:

> when there is a subdirectory "foo" which is empty.

Your subdirectory "foo" is not empty, it contains a sub-subdirectory "foo"
:).


Basically people use define/endef to create recipes which can span multiple
lines, and they expect that to work identically to explicitly writing multiple
lines in the recipe itself, including resetting the environment on each new
line.

I'm not saying that I think this is the best way it could be done, but that's
how it works and how it has has always worked.  Maybe if a different choice
had been made when define was invented we would have less need for .ONESHELL. 
But maybe other things would be harder.

In any event, for sure it will break a lot of things to change it now so it's
kind of a non-starter IMO.  I don't see any straightforward way to do
something different: even treating environment variables specially is not a
solution due to recursive makefiles and the export command.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[Martin Dorey]

Follow-up Comment #12, bug #59154 (project make):

[comment #10 comment #10:]
> If the variable is passed to a single shell then it will print an error at
the second "cd".

Eh?


martind@sirius:~/tmp/make-59154$ bash -c 'cd foo && echo one
cd foo && echo two'
one
two
martind@sirius:~/tmp/make-59154$ 


Oh, you meant if there wasn't a foo/foo/.  I'm liking that easy way to pass a
multi-line script to a single shell without having .ONESHELL applied to
everything.  Devil's advocacy aside, perhaps
https://www.gnu.org/software/make/manual/make.html#One-Shell would be the best
place to explain that variable expansion takes place before the splitting into
lines.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[Kyle Rose]

Follow-up Comment #11, bug #59154 (project make):

[comment #9 comment #9:]
> I don't think this would require Make to parse double quotes.
> 
> My guess is it would mean passing along expanded variables to the shell
without parsing the variable contents for newlines.
> 
> If I have time, maybe I'll have a look at whether my guess is right.

This absolutely will break tons of makefiles, including my company's 20 year
old build automation suite and the Linux kernel makefile, for the reason Paul
stated. I recommend resolving WONTFIX.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[anonymous]

Follow-up Comment #9, bug #59154 (project make):

I don't think this would require Make to parse double quotes.

My guess is it would mean passing along expanded variables to the shell
without parsing the variable contents for newlines.

If I have time, maybe I'll have a look at whether my guess is right.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[Kyle Rose]

Follow-up Comment #8, bug #59154 (project make):

[comment #7 comment #7:]
> If the string isn't quoted, the first newline would naturally signal a new
command to the shell, wouldn't it?
> 
> Got an example of a real-world Makefile that would break?

Nope. But that doesn't mean there isn't one.

There are already existing ways to do what you want. I, for instance, use
structures like:

echo -e $(call quote-sh,$(subst $(NL),$(backslash)n,$(mystring)))

to do this (in part because "$(mystring)" doesn't work as expected when
$(mystring) contains a double quote).

Finally, does make even parse double quotes in the recipes currently?
Doubtful, and I'm not sure I'd want to add that functionality.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[Paul D. Smith]

Follow-up Comment #10, bug #59154 (project make):

I already explained below why it works the way it does... maybe my mic is not
on?  Am I on mute?  I do that all the time on my video calls.

BSD make doesn't support multiline variable definitions the way GNU make does,
so it makes sense that they do something different when they expand variables
containing newlines.

GNU makefiles will allow the following makefile:


define SOMECOMMAND
cd foo && echo one
cd foo && echo two
endef

all: ; $(SOMECOMMAND)


to print both "one" and "two" when there is a subdirectory "foo" which is
empty.

If the variable is passed to a single shell then it will print an error at the
second "cd".

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[anonymous]

Follow-up Comment #4, bug #59154 (project make):

"Newlines in variable expansions are treated as real newlines in the rule."

Well, yes, of course.  What I didn't expect is for the command to be truncated
at the first newline.  The shell script

BLAH="hi
there"
export BLAH

echo "$BLAH"

properly echoes out both lines of BLAH... yet a Makefile with the same line it
it fails.  But only in Gnu make; in BSD make, it works.  I fail to understand
how this could be by design.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[anonymous]

Follow-up Comment #7, bug #59154 (project make):

If the string isn't quoted, the first newline would naturally signal a new
command to the shell, wouldn't it?

Got an example of a real-world Makefile that would break?

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[Kyle Rose]

Follow-up Comment #6, bug #59154 (project make):

[comment #5 comment #5:]
> (To be precise, the line
>echo "${BLAH}"
> works in all the shells I've tested, and in BSD Make, but not GNU Make.  If
BSD Make can get this right, seems like gnu make should be able to, too.)

This could break a ton of software that relies on multiple lines in an
environment variable becoming multiple commands in a recipe. If you want this
behavior, it should be selectable on a target-by-target basis with a
.MULTILINECMD dependency or some other mechanism.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[anonymous]

Follow-up Comment #5, bug #59154 (project make):

(To be precise, the line
   echo "${BLAH}"
works in all the shells I've tested, and in BSD Make, but not GNU Make.  If
BSD Make can get this right, seems like gnu make should be able to, too.)

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[anonymous]

Follow-up Comment #3, bug #59154 (project make):

Since, IN THIS CASE, the value should really only be processed once, isn't the
solution to defer the expansion to the shell by using
echo PUB is "$${PUB}"

as the recipe?

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[Paul D. Smith]

Follow-up Comment #2, bug #59154 (project make):

This is done on purpose.  Newlines in variable expansions are treated as real
newlines in the rule.  If they weren't, then multiline definitions would not
work at all:


define MYRULE
echo hello
echo world
endef

all: ; $(MYRULE)


I suppose there may be some way to special-case things, I'm not sure.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[Martin Dorey]

Follow-up Comment #1, bug #59154 (project make):

I thought this might be related to
https://en.wikipedia.org/wiki/Shellshock_(software_bug) but make is exec()ing
the wrong command:


martind@sirius:~/tmp/make-59154$ export PUB="$(echo hello; echo world)"
martind@sirius:~/tmp/make-59154$ cat Makefile
foo:; echo PUB is "$(PUB)"
martind@sirius:~/tmp/make-59154$ strace -f make 2>&1 | grep execve
execve("/usr/bin/make", ["make"], [/* 209 vars */]) = 0
[pid 17978] execve("/bin/sh", ["/bin/sh", "-c", "echo PUB is \"hello"], [/*
212 vars */]) = 0
martind@sirius:~/tmp/make-59154$ 


As is suggested by the make output:


martind@sirius:~/tmp/make-59154$ make
echo PUB is "hello
/bin/sh: 1: Syntax error: Unterminated quoted string
Makefile:1: recipe for target 'foo' failed
make: *** [foo] Error 2
martind@sirius:~/tmp/make-59154$ 


make -p says PUB has an appropriate-looking definition:


define PUB
hello
world
endef


I got the same error from make-3.81, so not a recent regression, which makes
me more surprised that I haven't tripped on this.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #59154] Multiline environment variables handled poorly

[Dan Kegel]

URL:
  

 Summary: Multiline environment variables handled poorly
 Project: make
Submitted by: dankegel
Submitted on: Mon 21 Sep 2020 02:58:39 PM UTC
Severity: 3 - Normal
  Item Group: Bug
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: 4.2.1
Operating System: Any
   Fixed Release: None
   Triage Status: None

___

Details:

I accidentally used a multiline envrionment variable thusly:

$ export PUB="$(base64 < ~/.ssh/id_rsa.pub)"
$ cat Makefile
foo:
   echo PUB is "$(PUB)"

This worked fine on mac where base64 defaults to not wrapping
there.

But on Linux where base64 defaults to wrapping,
this explored multiline variable values.
With gnu make, it fails with
/bin/sh: 1: Syntax error: unterminated quoted string

With bsd make, it succeeds.

The shell is happy to handle multiline environment variables.
Given that, IMHO, Makefiles that contain properly quoted shell references to
an environment variable ought to continue working even if that environment
variable happens to contain a multiline value.  One other implementation gets
it right, so it seems reasonable to enhance gnu make to handle this case as
well.





___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/