Re: bash 4.x filters out environmental variables containing a dot in the name
On Tuesday 30 June 2009 07:21:12 Stephane CHAZELAS wrote: 2009-06-29, 10:03(-04), Chet Ramey: The question is whether tolerant just means that the shell doesn't display a warning message about the assignment, as it does when you use an invalid variable name in an assignment statement, or exit with a variable assignment error, or dump core with a seg fault, as in many historical versions of sh. It may or may not also mean that the shell passes inherited invalid variable names to child processes. Historical versions of sh and ksh behave like bash-4.0. ksh93 doesn't but pdksh/mksh still does. the shells in busybox (ash/hush) as well as dash pass them along. -mike signature.asc Description: This is a digitally signed message part.
Re: bash 4.x filters out environmental variables containing a dot in the name
2009-06-29, 10:03(-04), Chet Ramey:
and it's a bug that bash-4 is filtering them.
Maybe, maybe not. That's open to interpretation. Here's how I see it.
not allowing them to be used in
the shell is fine (echo ${vmlinux.lds}), but removing them from the
environment and thus not allowing other applications to leverage them is not.
The shell is not like other applications that simply pass the environment
through as uninterpreted strings. It applies semantic interpretation to
the environment variables it inherits.
Many tools that use the environment apply semantic
interpretation to the environment variables they inherit, and
what variable is supported by a shell is shell dependant.
Shells scan the initial environment exactly once: at startup, when creating
variables. The environment passed to child processes is created from the
set of exported shell variables. This implies to me that the environment
passed to child processes consists only of valid shell variables.
For the rc or es shell, anything is valid except the empty string.
Certainly conforming applications should not create environment variables
with invalid names.
What about non-conforming applications? I agree that if they
should to use the shell API, they should conform to it, but
here, we're in case not covered by the API. That application
should be able to expect its environment to be transfered as is
through a shell to another non-conforming application.
POSIX explicitly states that applications shall tolerate the presence of
these
environment variables, and while that is open for interpretation (i.e.
you've
interpreted this to mean automatic removal), i dont see any reason for bash
to
be culling these.
Posix also says that variables are inherited from the environment. That
word has a very specific meaning, as was reiterated during the $@ and set -u
discussion. The same variables language is used when Posix talks about
creating the environment for shell execution environments.
Yes, but that's shell variables, it's not talking about the rest
of the environment.
The question is whether tolerant just means that the shell doesn't display
a warning message about the assignment, as it does when you use an invalid
variable name in an assignment statement, or exit with a variable assignment
error, or dump core with a seg fault, as in many historical versions of sh.
It may or may not also mean that the shell passes inherited invalid variable
names to child processes.
Historical versions of sh and ksh behave like bash-4.0.
ksh93 doesn't but pdksh/mksh still does.
It seems, though, that there might be enough use for me to try and make it
work. While I'm not wild about creating yet another class of variable,
there might be a way to do it simply.
I suppose the most obvious way would be to store away those
environment strings that bash doesn't know how to handle and
append it to each execve.
Now other points to consider could be what to do in cases like:
- there is both a=1 and a=2 in the environment passed to the
shell.
- there is both a=1 and a in the environment passed to the
shell. (and maybe also and =x).
I think it's probably worth asking the Austin group about it.
--
Stéphane
Re: bash 4.x filters out environmental variables containing a dot in the name
Stephane CHAZELAS wrote: Posix also says that variables are inherited from the environment. That word has a very specific meaning, as was reiterated during the $@ and set -u discussion. The same variables language is used when Posix talks about creating the environment for shell execution environments. Yes, but that's shell variables, it's not talking about the rest of the environment. Yes, well, that's the question, isn't it? Whether there is any rest of the environment. Is the environment passed to subshells made up of only shell variables with the export attribute? Since we agree that invalid shell assignments in the environment don't cause shell variables to be created, they must belong to some special class that is outside the rest of the shell to get them passed to subshells. Historical versions of sh and ksh behave like bash-4.0. ksh93 doesn't but pdksh/mksh still does. This behavior changed in ksh93s. I think it's probably worth asking the Austin group about it. Probably. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: bash 4.x filters out environmental variables containing a dot in the name
and it's a bug that bash-4 is filtering them.
Maybe, maybe not. That's open to interpretation. Here's how I see it.
not allowing them to be used in
the shell is fine (echo ${vmlinux.lds}), but removing them from the
environment and thus not allowing other applications to leverage them is not.
The shell is not like other applications that simply pass the environment
through as uninterpreted strings. It applies semantic interpretation to
the environment variables it inherits.
Shells scan the initial environment exactly once: at startup, when creating
variables. The environment passed to child processes is created from the
set of exported shell variables. This implies to me that the environment
passed to child processes consists only of valid shell variables.
Certainly conforming applications should not create environment variables
with invalid names.
POSIX explicitly states that applications shall tolerate the presence of
these
environment variables, and while that is open for interpretation (i.e. you've
interpreted this to mean automatic removal), i dont see any reason for bash
to
be culling these.
Posix also says that variables are inherited from the environment. That
word has a very specific meaning, as was reiterated during the $@ and set -u
discussion. The same variables language is used when Posix talks about
creating the environment for shell execution environments.
The question is whether tolerant just means that the shell doesn't display
a warning message about the assignment, as it does when you use an invalid
variable name in an assignment statement, or exit with a variable assignment
error, or dump core with a seg fault, as in many historical versions of sh.
It may or may not also mean that the shell passes inherited invalid variable
names to child processes.
Historical versions of sh and ksh behave like bash-4.0.
It seems, though, that there might be enough use for me to try and make it
work. While I'm not wild about creating yet another class of variable,
there might be a way to do it simply.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
Chet Ramey, ITS, CWRUc...@case.eduhttp://tiswww.tis.case.edu/~chet/
Re: bash 4.x filters out environmental variables containing a dot in the name
Thanks for heads up, Christian. Filtering out . will break our company's
accounting software, and presumably many other applications.
Checking for correct charset is sensible thing to do when setting or accessing
a variable. But, those variables that Bash did not change or set, should be
passed on to the next program untouched.
--William
--- On Fri, 6/26/09, Christian Krause c...@plauener.de wrote:
From: Christian Krause c...@plauener.de
Subject: Re: bash 4.x filters out environmental variables containing a dot in
the name
To: chet.ra...@case.edu
Cc: bug-bash@gnu.org
Received: Friday, June 26, 2009, 7:58 AM
Hi Chet,
Thanks for the answers. The problem is now, that this
behavior of the
bash creates some real problems outside, probably with a
larger impact.
Before asking the kernel developers to change parts of
linux kernel's
build system, I'd like to be sure whether bash-4.x's
behavior is correct
or not. Please see my comments below:
Chet Ramey wrote:
Mike Frysinger wrote:
On Thursday 25 June 2009 19:17:38 Chet Ramey
wrote:
Christian Krause wrote:
Bash Version: 4.0
Patch Level: 16
Release Status: release
Description:
During the compilation of the linux kernel
(configured to user mode
linux) I've discovered the following
problem of bash 4.0 (which is now
delivered by Fedora 11):
If an environmental variable contains a
. in its name, the new bash
4.0 filters out these variables. They will
not be visible via set nor
will they be inherited to executed
sub-shells or processes.
Such strings are invalid shell or environment
variable names. It was a
bug in bash-3.2 that it created invalid
variables from the initial
environment.
and it's a bug that bash-4 is filtering
them. not allowing them to be used in
the shell is fine (echo ${vmlinux.lds}), but
removing them from the
environment and thus not allowing other
applications to leverage them is not.
It's not a bug. Posix explicitly restricts
environment variable names
to consist of uppercase letters, lowercase letters,
digits, and
As far as I interpret the standard (IEEE Std 1003.1, 2004
Edition), the
general definition for environment variables is something
like this:
- names must not contain =
- if it should be portable, the names should only contain
characters
from the portable character set (which includes .)
Sure, there is a restriction that variables used by the
shell (and the
utilities described in the standard) should only contain
the characters
you described.
However, since not all programs belong to this set, I don't
see an
explicit statement which denies the usage of e.g. . in
environmental
variables in general.
underscores. There is no provision for variables
with invalid names
that don't exactly exist and are just passed down to
applications in
their environment. The environment is
constructed from variables with
the export attribute set (another thing Posix
explicitly states); things
that aren't valid variables don't get in there.
Hm, I'm not sure I can agree to this. The problem is, that
for other
programs names of environmental variables containing a .
are not
invalid (although they are invalid for the bash).
As far as I understand the behavior of the bash,
environmental variables
which were put in the process environmental during exec'ing
a shell are
exported to subsequent processes of this shell without
explicitly
exporting them.
So, if the bash passes variables containing in the bash's
process
environment to sub-processes anyway (without any explicit
export), I
would argue it would be more natural that the bash should
not filter
them. If any other process execs itself, the new process
image will have
the same environmental variables set (if not execle is
used)...
Given all of these facts I still tend to say that the bash
shouldn't
filter them...
Best regards,
Christian
__
Yahoo! Canada Toolbar: Search from anywhere on the web, and bookmark your
favourite sites. Download it now
http://ca.toolbar.yahoo.com.
Re: bash 4.x filters out environmental variables containing a dot in the name
2009-06-26, 13:58(+02), Christian Krause: Hi Chet, Thanks for the answers. The problem is now, that this behavior of the bash creates some real problems outside, probably with a larger impact. Before asking the kernel developers to change parts of linux kernel's build system, I'd like to be sure whether bash-4.x's behavior is correct or not. Please see my comments below: [...] POSIX doesn't prevent bash to do so I suppose, and bash is not the only shell to do so, pdksh based shells (which many systems sh are based on) also do so. bash3 and ksh93 also strip some environment variables (and override others like IFS) bash3 strips the environment strings that have no = in them, and the environment variable with an empty name. ksh93 also strips the environment strings without = except the empty string. Now, I can't think of any good reason why shells would do things like that (removes things from the environment which they have no business with) and it seems wrong to me as well, but one shouldn't rely on shells preserving environment variables that can't be mapped to shell variables as in practice that's no the case for a few shells, and it seems it's unspecified by POSIX. -- Stéphane
Re: bash 4.x filters out environmental variables containing a dot in the name
Christian Krause wrote: Given all of these facts I still tend to say that the bash shouldn't filter them... There's always the following argument: Other characters may be permitted by an implementation; applications shall tolerate the presence of such names. I agree with Christian here. As far as technically possible, Bash should respect every environment variable name. J.
Re: bash 4.x filters out environmental variables containing a dot in the name
Chet Ramey chet.ra...@case.edu writes: It's not a bug. Posix explicitly restricts environment variable names to consist of uppercase letters, lowercase letters, digits, and underscores. POSIX only talks about their use in POSIX utilities. It does not say anything about non-POSIX utilities and extensions (and using other characters _is_ a proper extension). There is no provision for variables with invalid names that don't exactly exist and are just passed down to applications in their environment. POSIX explicitly says that other characters may be permitted, and Unix does permit them, just like Unix permits any character except / and NUL in filenames. Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 And now for something completely different.
Re: bash 4.x filters out environmental variables containing a dot in the name
Hi Chet,
Thanks for the answers. The problem is now, that this behavior of the
bash creates some real problems outside, probably with a larger impact.
Before asking the kernel developers to change parts of linux kernel's
build system, I'd like to be sure whether bash-4.x's behavior is correct
or not. Please see my comments below:
Chet Ramey wrote:
Mike Frysinger wrote:
On Thursday 25 June 2009 19:17:38 Chet Ramey wrote:
Christian Krause wrote:
Bash Version: 4.0
Patch Level: 16
Release Status: release
Description:
During the compilation of the linux kernel (configured to user mode
linux) I've discovered the following problem of bash 4.0 (which is now
delivered by Fedora 11):
If an environmental variable contains a . in its name, the new bash
4.0 filters out these variables. They will not be visible via set nor
will they be inherited to executed sub-shells or processes.
Such strings are invalid shell or environment variable names. It was a
bug in bash-3.2 that it created invalid variables from the initial
environment.
and it's a bug that bash-4 is filtering them. not allowing them to be used
in
the shell is fine (echo ${vmlinux.lds}), but removing them from the
environment and thus not allowing other applications to leverage them is
not.
It's not a bug. Posix explicitly restricts environment variable names
to consist of uppercase letters, lowercase letters, digits, and
As far as I interpret the standard (IEEE Std 1003.1, 2004 Edition), the
general definition for environment variables is something like this:
- names must not contain =
- if it should be portable, the names should only contain characters
from the portable character set (which includes .)
Sure, there is a restriction that variables used by the shell (and the
utilities described in the standard) should only contain the characters
you described.
However, since not all programs belong to this set, I don't see an
explicit statement which denies the usage of e.g. . in environmental
variables in general.
underscores. There is no provision for variables with invalid names
that don't exactly exist and are just passed down to applications in
their environment. The environment is constructed from variables with
the export attribute set (another thing Posix explicitly states); things
that aren't valid variables don't get in there.
Hm, I'm not sure I can agree to this. The problem is, that for other
programs names of environmental variables containing a . are not
invalid (although they are invalid for the bash).
As far as I understand the behavior of the bash, environmental variables
which were put in the process environmental during exec'ing a shell are
exported to subsequent processes of this shell without explicitly
exporting them.
So, if the bash passes variables containing in the bash's process
environment to sub-processes anyway (without any explicit export), I
would argue it would be more natural that the bash should not filter
them. If any other process execs itself, the new process image will have
the same environmental variables set (if not execle is used)...
Given all of these facts I still tend to say that the bash shouldn't
filter them...
Best regards,
Christian
Re: bash 4.x filters out environmental variables containing a dot in the name
Christian Krause wrote: Configuration Information [Automatically generated, do not change]: Machine: i386 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i386' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i386-redhat-linux-gnu' -DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i586 -mtune=generic -fasynchronous-unwind-tables uname output: Linux noname 2.6.29.4-167.fc11.i686.PAE #1 SMP Wed May 27 17:28:22 EDT 2009 i686 i686 i386 GNU/Linux Machine Type: i386-redhat-linux-gnu Bash Version: 4.0 Patch Level: 16 Release Status: release Description: During the compilation of the linux kernel (configured to user mode linux) I've discovered the following problem of bash 4.0 (which is now delivered by Fedora 11): If an environmental variable contains a . in its name, the new bash 4.0 filters out these variables. They will not be visible via set nor will they be inherited to executed sub-shells or processes. Such strings are invalid shell or environment variable names. It was a bug in bash-3.2 that it created invalid variables from the initial environment. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: bash 4.x filters out environmental variables containing a dot in the name
On Thursday 25 June 2009 19:17:38 Chet Ramey wrote:
Christian Krause wrote:
Bash Version: 4.0
Patch Level: 16
Release Status: release
Description:
During the compilation of the linux kernel (configured to user mode
linux) I've discovered the following problem of bash 4.0 (which is now
delivered by Fedora 11):
If an environmental variable contains a . in its name, the new bash
4.0 filters out these variables. They will not be visible via set nor
will they be inherited to executed sub-shells or processes.
Such strings are invalid shell or environment variable names. It was a
bug in bash-3.2 that it created invalid variables from the initial
environment.
and it's a bug that bash-4 is filtering them. not allowing them to be used in
the shell is fine (echo ${vmlinux.lds}), but removing them from the
environment and thus not allowing other applications to leverage them is not.
POSIX explicitly states that applications shall tolerate the presence of these
environment variables, and while that is open for interpretation (i.e. you've
interpreted this to mean automatic removal), i dont see any reason for bash to
be culling these.
-mike
signature.asc
Description: This is a digitally signed message part.
Re: bash 4.x filters out environmental variables containing a dot in the name
Mike Frysinger wrote:
On Thursday 25 June 2009 19:17:38 Chet Ramey wrote:
Christian Krause wrote:
Bash Version: 4.0
Patch Level: 16
Release Status: release
Description:
During the compilation of the linux kernel (configured to user mode
linux) I've discovered the following problem of bash 4.0 (which is now
delivered by Fedora 11):
If an environmental variable contains a . in its name, the new bash
4.0 filters out these variables. They will not be visible via set nor
will they be inherited to executed sub-shells or processes.
Such strings are invalid shell or environment variable names. It was a
bug in bash-3.2 that it created invalid variables from the initial
environment.
and it's a bug that bash-4 is filtering them. not allowing them to be used
in
the shell is fine (echo ${vmlinux.lds}), but removing them from the
environment and thus not allowing other applications to leverage them is not.
It's not a bug. Posix explicitly restricts environment variable names
to consist of uppercase letters, lowercase letters, digits, and
underscores. There is no provision for variables with invalid names
that don't exactly exist and are just passed down to applications in
their environment. The environment is constructed from variables with
the export attribute set (another thing Posix explicitly states); things
that aren't valid variables don't get in there.
POSIX explicitly states that applications shall tolerate the presence of
these
environment variables, and while that is open for interpretation (i.e. you've
interpreted this to mean automatic removal), i dont see any reason for bash
to
be culling these.
That doesn't mean what you think it means. It means that applications
won't abort because of invalid names in the environment, like most
historical versions of sh.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
