[bug #62595] Add ability to read .env files into make

[Paul D. Smith]

Update of bug #62595 (project make):

 Summary: Add ability to read .env files into make and  => Add
ability to read .env files into make


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #62595] Add ability to read .env files into make and

[Chigozirim Chukwu]

Follow-up Comment #8, bug #62595 (project make):

> Of course you can also use the re-exec feature of make to write a rule to
generate the include file then you have the full power of a recipe.
I like the idea, especially because of the re-exec feature. However, my goal
was to avoid having to make this external to make. Also given that the rule
for recipe is not likely to change from project to project, I would like to
suggest an alternative (and possibly simpler) abstraction:

What if we have a special target rule, which has dependencies on any number of
".env" files:


.ENVFILES: .env production.env


This will have the job of parsing the env files and including them into make.

> I'm also not sure I agree that the right behaviour is to always export every
variable in the file. But these are details:...
I suggested the _always export_ option because that's usually the way I've
seen .env files being used.

For example, when working with docker-compose yaml files, docker-compose reads
a .env file and any other included yaml files can also see the variables. _See
the syntax rules for .env
_

The same is true for most people who use .env files with shell scripts. Some
will do something like this to automatically export and load the variables
into the current process:


set -a; source .env; set +a


Pipenv  which is used to manage a python
project dependencies will also usually load the .env file into the current
process, and every program that is started sees the environments.

In any case, I don't think lack of automatic export is a deal breaker. If the
current plan is to create an external program, or to use the special target,
then decision to export or not can be made later rather than now.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #62595] Add ability to read .env files into make and

[Paul D. Smith]

Follow-up Comment #7, bug #62595 (project make):

Thanks for the pointer to the spec, that's very helpful.  I agree that this
would be possible to implement.

I'm not interested in a flag like --envfile, that doesn't sound very useful. 
I would anticipate that most build systems would want to always include the
content of the file, and not require the user to add an option to the command
line.

I'm also not sure I agree that the right behavior is to always export every
variable in the file.  But these are details: the first thing is to decide
whether this needs to be added at all.

I was thinking about alternatives to adding new syntax, and the problem is
that there's not a very straightforward way in GNU make today to evaluate the
output of a program as a makefile.

We can run a program using $(shell ...), but the results are modified by
removing newlines and so this is not useful.

The only way to do it is to run $(shell ...) and have it write to a temporary
file, then include the temporary file.  Of course you can also use the re-exec
feature of make to write a rule to generate the include file then you have the
full power of a recipe.

Is that sufficient?  It would be something like:


include env.mk

env.mk: env.env
env2make < $< > $@


for some putative *env2make* conversion tool.

An alternative would be to invent some new type of function which would either
preserve newlines in the result, which could then be passed to the *eval*
function, or create an alternative to include which would run a program and
"include" the output, or something like that.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #62595] Add ability to read .env files into make and

[anonymous]

Follow-up Comment #6, bug #62595 (project make):

 > ⚠ External Email
 > 
 > Follow-up Comment #4, bug #62595 (project make):
 > 
 > No, no, that's not something one should expect from a .env file.
 > 
 > I said that _some_ dotenv parsers take it a step further to do all that
fancy
 > interpolation, but really .env files are just files containing a bunch of
 > key=value lines, no functions or shell parsing necessary. It's just a
 > configuration file. Variable interpolation is a nice-to-have feature of
most
 > dotenv parsers, not a requirement.



You're asking for a specialized parser, then.


 > So the goal is not to write a parser, but more of a translator that
converts
 > .env syntax to make syntax.
 > 

The write a translator of arbitrary shell syntax, you need a parser.

Rather than update Make for this special case, I think it would be
better to write a small shell wrapper for Make that reads each line of
the 'env' file, and either writes a post-processed file that Make can
read, or just export the variables into the current shell before
invoking make.

Here's a simple example to show you the concept.


#!/bin/bash

FILE=$(pwd)/example.env;

while read LINE ; do
IFS='=' read -r key value <<<${LINE};
eval ${LINE};
echo "input  : ${LINE}";
echo "key/value  : '${key}' / '${value}'";
echo "evaluated  : ${!key}";
echo "Make output: ${key}=${!key}";
echo "";
done <${FILE};

Escaping special characters (like '#') before exporting to make is not
implemented, but should be trivial to do before evaluating 'LINE' into
the environment.



___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #62595] Add ability to read .env files into make and

[Martin Dorey]

Follow-up Comment #5, bug #62595 (project make):

I'd never heard of these env file things, so appreciated the well-written
posts that those links point to.  Why would this feature be better in Make
rather than in a wrapper?  If you wanted, for some reason that I'm struggling
to guess, to keep "make" as the interface, then I think it'd be
straightforward to write a Makefile that would spot that an expected
environment variable wasn't available, then started a shell that sourced the
environment file, exported all its variables and reran make.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #62595] Add ability to read .env files into make and

[Chigozirim Chukwu]

Follow-up Comment #4, bug #62595 (project make):

No, no, that's not something one should expect from a .env file.

I said that _some_ dotenv parsers take it a step further to do all that fancy
interpolation, but really .env files are just files containing a bunch of
key=value lines, no functions or shell parsing necessary. It's just a
configuration file. Variable interpolation is a nice-to-have feature of most
dotenv parsers, not a requirement.



Unfortulately I don't think there is a definite spec I can point to, but I
found this one which explains the format
. And another one which
talks about the origin


To bring this back on topic, I believe "make" already has the capacity to do
what most dotenv parsers are doing. As you can see from the stackexchange page
 I
linked earlier, make can already read some simple .env files when included
using the "include" directive, but for others it just needs to be guided a
little. For example:


TEST_VAR='8X46Yj*aek3GQeiW7#bK!Eg@#6vjV'


should be translated into


export TEST_VAR := 8X46Yj*aek3GQeiW7\#bK!Eg@\#6vjV


And something like:


EMAIL=${USER}@example.org


is to be translated to


export EMAIL := $(USER)@example.org


So the goal is not to write a parser, but more of a translator that converts
.env syntax to make syntax.

This feature can be exposed either via a directive within a Makefile
(includeenv) or a flag (--envfile) to make.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #62595] Add ability to read .env files into make and

[Paul D. Smith]

Follow-up Comment #3, bug #62595 (project make):

Right, so that's just a shell script that doesn't happen to do anything except
set variables.

Your comment 'The format of a .env file is simply "key=value"' is not the
case, as you yourself point out when you say "the single quotes surrounding
the value are to be ignored".

That means that the value needs to be parsed/interpreted somehow.  And if it's
really a shell script then the value could contain any valid shell syntax: it
might be quoted with single quotes, or double quotes, and the contents could
be escaped with "\" and how that is handled depends on the type of quoting. 
The value could contain a simple variable that needs to be expanded, or a
variable of the form ${...}.  It could use fancy shell variable syntax like
${foo:-bar} or any of the other things.

It could even contain backticks or $() to run subcommands and retrieve the
output.

This is what I meant by, I don't really want to put an entire shell parser
into make.

If there's some limit to the actual syntax of these ".env files", where
there's some standard specification that reduces their scope from "any valid
shell variable assignment" to something more tractable, can you point to such
a spec?


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #62595] Add ability to read .env files into make and

[Chigozirim Chukwu]

Follow-up Comment #2, bug #62595 (project make):

Hi Paul

.env files are commonly used for specifying secrets for an application or a
build.

Please see this question to get an idea of how people are currently trying to
use .env files in their Makefile:
https://unix.stackexchange.com/questions/235223/makefile-include-env-file

The format of a .env file is simply "key=value". From the make.env project I
linked to, here is what a typical file may look like

APP_PASSWORD='8oyy!r#vNpRy2TT'
FOO_BAR='4dabfs#a%d73w$Z2TBN4!nYD4Y$TW'
TEST_VAR='8X46Yj*aek3GQeiW7#bK!Eg@#6vjV'
ANOTHER_VAR='"hello world'

---

In the above case, the single quotes surrounding the value are to be ignored,
but everything else is to be taken literally. Some .env parsers take it a step
further and also expand environment variables within the .env file, so values
that are surrounded by double quotes and have $ symbol inside are expanded and
the result is what is stored as part of the value. ex. user="$USER" expands to
user="username".

The solution I settled on for the make.env project is to convert each of those
lines to "export KEY:='$(value VALUE)'", then load them into make using the
--eval flag. This was the only way I found that reliably loads the value into
make without make attempting to interpret the value.

I hope that clears things up


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #62595] Add ability to read .env files into make and

[Paul D. Smith]

Follow-up Comment #1, bug #62595 (project make):

I don't know what an ".env" file is.

Do you mean, a shell script?  It's not possible to embed a shell parser into
make.  Or, I guess it would be possible but it's not something I think is a
good idea.

If you don't mean a shell script, is there some definition of exactly what
syntax an ".env" file supports and where such things are used?


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #62595] Add ability to read .env files into make and

[Chigozirim Chukwu]

URL:
  

 Summary: Add ability to read .env files into make and 
 Project: make
   Submitter: firstairbender
   Submitted: Wed 08 Jun 2022 03:43:10 AM CDT
Severity: 3 - Normal
  Item Group: Enhancement
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: 4.3
Operating System: None
   Fixed Release: None
   Triage Status: None


___

Follow-up Comments:


---
Date: Wed 08 Jun 2022 03:43:10 AM CDT By: Chigozirim Chukwu 
I would like the ability to allow make to read .env files. I kinda already got
started here with a python wrapper around make, called `make.env`:
https://github.com/smac89/make.env

I believe this is something many people will benefit from, and it can be
exposed via a flag of some sort.

I don't mind working on it, but I just need to know if it's something that has
been considered before and discarded, or if there are any concerns with giving
`make` the ability to read environment files







___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/