[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
The following issue has a resolution that has been APPLIED. == https://austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: Applied Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text:https://austingroupbugs.net/view.php?id=1561#c5795 Resolution: Accepted As Marked Fixed in Version: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-11-30 16:35 UTC == Summary:clarify what kind of data shell variables need to be able to hold == Relationships ID Summary -- related to 0001560 clarify wording of command substitution related to 0001562 printf utility: clarify what is (byte) ... related to 0001564 clariy on what (character/byte) strings... == Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 2022-02-02 18:44 kreNote Added: 0005654 2022-02-06 11:18 mirabilos Note Added: 0005662 2022-02-06 18:18 chet_ramey Note Added: 0005665 2022-02-06 23:17 kreNote Added: 0005666 2022-02-08 15:14 calestyo Note Added: 0005668 2022-02-09 01:58 kreNote Added: 0005669 2022-04-07 16:29 geoffclare Relationship added related to 0001560 2022-04-07 16:30 geoffclare Relationship added related to 0001562 2022-04-07 16:30 geoffclare Relationship added related to 0001564 2022-04-11 13:52 geoffclare Note Added: 0005795 2022-04-15 23:38 calestyo Note Added: 0005807 2022-04-16 00:22 calestyo Note Added: 0005808 2022-10-31 16:25 geoffclare Final Accepted Text => https://austingroupbugs.net/view.php?id=1561#c5795 2022-10-31 16:25 geoffclare Status New => Resolved 2022-10-31 16:25 geoffclare Resolution Open => Accepted As Marked 2022-10-31 16:25 geoffclare Tag Attached: tc3-2008 2022-11-30 16:35 geoffclare Status Resolved => Applied ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
The following issue has been RESOLVED. == https://austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: Resolved Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text:https://austingroupbugs.net/view.php?id=1561#c5795 Resolution: Accepted As Marked Fixed in Version: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-10-31 16:25 UTC == Summary:clarify what kind of data shell variables need to be able to hold == Relationships ID Summary -- related to 0001560 clarify wording of command substitution related to 0001562 printf utility: clarify what is (byte) ... related to 0001564 clariy on what (character/byte) strings... == Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 2022-02-02 18:44 kreNote Added: 0005654 2022-02-06 11:18 mirabilos Note Added: 0005662 2022-02-06 18:18 chet_ramey Note Added: 0005665 2022-02-06 23:17 kreNote Added: 0005666 2022-02-08 15:14 calestyo Note Added: 0005668 2022-02-09 01:58 kreNote Added: 0005669 2022-04-07 16:29 geoffclare Relationship added related to 0001560 2022-04-07 16:30 geoffclare Relationship added related to 0001562 2022-04-07 16:30 geoffclare Relationship added related to 0001564 2022-04-11 13:52 geoffclare Note Added: 0005795 2022-04-15 23:38 calestyo Note Added: 0005807 2022-04-16 00:22 calestyo Note Added: 0005808 2022-10-31 16:25 geoffclare Final Accepted Text => https://austingroupbugs.net/view.php?id=1561#c5795 2022-10-31 16:25 geoffclare Status New => Resolved 2022-10-31 16:25 geoffclare Resolution Open => Accepted As Marked ==
Re: [Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
On Mon, 2022-04-25 at 12:02 +0100, Geoff Clare via austin-group-l at
The Open Group wrote:
> > > > May I suggest to rather directly write something like:
> > > > "The following four varieties of parameter expansion provide
> > > > for
> > > > substring
> > > > processing, with each of them requiring the value to be a
> > > > character
> > > > string."
> > > >
> > > > Especially the "substring" makes it IMO a tiny bit vague...
> > > > like
> > > > in:
> > > > foo="${binaryString}."
> > > > cmdSubstWithTrailingNL=${foo%.}
> > > >
> > > > One could claim: "well,... the '.' is a substring of $foo...
> > > > and
> > > > only that
> > > > is processed as character and matched... so fine"
...
>
> I don't see a problem with this use
> of "substring"
Well...
> but in any case there is no point working on minutiae
> of the wording until the bigger question has been settled.
Okay.
> > Well but then the wording of your sentence seems wrong or at least
> > confusing to me:
> >
> > > However, the shell may initialize parameters that do not have
> > > valid
> > > names from such environment variables.
> >
> > "initialize parameters that do not have valid names"
> >
> > => how could it initialise a parameter that does not have a valid
> > name? That's forbidden?!
> > 2.5 Parameters and Variables
> > "A parameter can be denoted by a name, a number, or one of the
> > special characters listed in Special Parameters."
> >
> > So either it's a (valid) name (and would be e.g.
> > BASH_INVALIDLY_NAME_ENV_VARS, and I guess it doesn't matter here
> > whether one would consider that a variable or parameter), or it's a
> > digit (and thus a positional param... and thus this cannot be used
> > for
> > that purpose) or a special char, which would require such shells to
> > use
> > such special char for that purpose.
>
> No. You are reading too much into that statement from 2.5. It says
> a parameter *can* be denoted by a name, a number, or a special
> character. It does not say that a parameter *can't* be denoted by
> other things as well.
Phew... well it may be intended like so, but that reading also seems
pretty vague to me, not to talk about the consequences for portability
if someone would actually do so.
Anyway... probably not worth arguing about this.
> > For the latter... does it really make sense to encourage them doing
> > this?
>
> The special characters listed in the referenced section all have
> strictly defined purposes. They can't be used for something else.
I rather meant (not) "encouraging" a shell to use any *other* special
character for a new special parameter (e.g. $^ or so).
> > Wouldn't it rather make sense to just allow them something like
> > BASH_INVALIDLY_NAME_ENV_VARS?
>
> I think my wording allows that. The array itself is an extension and
> thus not affected by what the standard says here. However, each
> element of the array would be considered to be a parameter (denoted
> by "name[index]"), and thus allowed to be initialised.
Well I guess that's anyway just nitpicking and in the end everyone
probably knows that this specific use of such invalid names is allowed.
Not sure how it's considered formally, but when you take e.g. bash and
you'd have an (indexed!) array like:
$ declare -a inv_from_env
$ inv_from_env=(value1 345 value1 .)
...where pars of element would be value/invalid_name...
$ echo $inv_from_env
value1
... then one would have initialised a variable from an invalid env var.
But obviously, that example is quite constructed...
...so I guess I can live with the current wording, thus let's defer
this until Harald has had a chance to do his work.
Cheers,
Chris.
Re: [Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
Christoph Anton Mitterer wrote, on 25 Apr 2022:
>
> On Fri, 2022-04-22 at 15:36 +0100, Geoff Clare via austin-group-l at
> The Open Group wrote:
> > > However some shells may e.g. wish to provide such invalidly named
> > > env vars
> > > in a special shell var (like an associative array)... so that
> > > should
> > > perhaps be allowed?
> >
> > Yes, that's allowed (see below), but to be clear - such things are
> > not "special shell variables". They cannot be shell variables of any
> > kind because shell variables always have valid names.
>
> With "special shell variable" I meant something like the variable
> (associative array) BASH_INVALIDLY_NAME_ENV_VARS (not that it would
> really exist in current bash versions).
> So that would still have a valid name itself, but contain those env
> vars, with invalid names.
See below.
> > > > ... for character substring processing.
> > >
> > > May I suggest to rather directly write something like:
> > > "The following four varieties of parameter expansion provide for
> > > substring
> > > processing, with each of them requiring the value to be a character
> > > string."
> > >
> > > Especially the "substring" makes it IMO a tiny bit vague... like
> > > in:
> > > foo="${binaryString}."
> > > cmdSubstWithTrailingNL=${foo%.}
> > >
> > > One could claim: "well,... the '.' is a substring of $foo... and
> > > only that
> > > is processed as character and matched... so fine"
> > >
> > > Further, this basically assume the current proposal of
> > > https://www.austingroupbugs.net/view.php?id=1562 ... i.e. that
> > > pattern
> > > matching notation would always be on character strings.
> > > However, on the mailing list, Harald van Dijk offered to look more
> > > deeply
> > > into that matter, ... So may I kindly request that you defer voting
> > > on this
> > > particular part of the above changes, until Harald had a chance to
> > > do so?
> >
> > Agreed.
>
> If your "agreed" goes also for the first part of my reply... (and not
> just the deferring) then it would perhaps make sense to change still
> change the proposal now (accordingly) and just don't vote on it.
>
> Otherwise I guess the first suggestion (about the vagueness of
> "substring") would be easily forgotten.
I was agreeing to the deferral. I don't see a problem with this use
of "substring", but in any case there is no point working on minutiae
of the wording until the bigger question has been settled.
> > > > However, the shell may initialize parameters that do not have
> > > > valid names
> > > from such environment variables.
> > >
> > > What's that intended for? To allow positional/special parameters to
> > > be
> > > initialised from the environment variable?
> >
> > This is what you wanted above when you said "some shells may e.g.
> > wish
> > to provide such invalidly named env vars in a special shell var (like
> > an associative array)". As I pointed out there, such things are not
> > any kind of shell variable, which is why I called them parameters
> > here.
>
> Well but then the wording of your sentence seems wrong or at least
> confusing to me:
>
> > However, the shell may initialize parameters that do not have valid
> > names from such environment variables.
>
> "initialize parameters that do not have valid names"
>
> => how could it initialise a parameter that does not have a valid
>name? That's forbidden?!
>2.5 Parameters and Variables
>"A parameter can be denoted by a name, a number, or one of the
>special characters listed in Special Parameters."
>
> So either it's a (valid) name (and would be e.g.
> BASH_INVALIDLY_NAME_ENV_VARS, and I guess it doesn't matter here
> whether one would consider that a variable or parameter), or it's a
> digit (and thus a positional param... and thus this cannot be used for
> that purpose) or a special char, which would require such shells to use
> such special char for that purpose.
No. You are reading too much into that statement from 2.5. It says
a parameter *can* be denoted by a name, a number, or a special
character. It does not say that a parameter *can't* be denoted by
other things as well.
> For the latter... does it really make sense to encourage them doing
> this?
The special characters listed in the referenced section all have
strictly defined purposes. They can't be used for something else.
> Wouldn't it rather make sense to just allow them something like
> BASH_INVALIDLY_NAME_ENV_VARS?
I think my wording allows that. The array itself is an extension and
thus not affected by what the standard says here. However, each
element of the array would be considered to be a parameter (denoted
by "name[index]"), and thus allowed to be initialised.
--
Geoff Clare
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
On Fri, 2022-04-22 at 15:36 +0100, Geoff Clare via austin-group-l at
The Open Group wrote:
> > Cause if it's that a shell must not "take over" e.g.
> > '+=whatAWeirdVariable', then this was (at least indirectly) already
> > clear
> > before, by variables being parameters and these, as per 2.5, having
> > names
> > (in the sense of 3.235 Name).
> >
> > If your intention is however to forbid that environment variables
> > with
> > invalid names must not be "taken over" by transcribing their
> > invalid name
> > into something valid (e.g. replacing any invalid chars with
> > '__ > code point>__' - which would of course be prone to collisions) than
> > I think
> > this wording is too unclear/indirect
>
> I disagree. If a shell variable with a valid (and therefore
> different)
> name is initialised from an environment variable with an invalid
> name,
> that clearly violates the requirement that shell variables are
> initialised only from environment variables that have valid names.
Uhm... okay,.. yes,... I see you're right and that solves point (4) and
also the question whether such transliterations would be allowed quite
nicely.
> > However some shells may e.g. wish to provide such invalidly named
> > env vars
> > in a special shell var (like an associative array)... so that
> > should
> > perhaps be allowed?
>
> Yes, that's allowed (see below), but to be clear - such things are
> not "special shell variables". They cannot be shell variables of any
> kind because shell variables always have valid names.
With "special shell variable" I meant something like the variable
(associative array) BASH_INVALIDLY_NAME_ENV_VARS (not that it would
really exist in current bash versions).
So that would still have a valid name itself, but contain those env
vars, with invalid names.
>
> > > ... for character substring processing.
> >
> > May I suggest to rather directly write something like:
> > "The following four varieties of parameter expansion provide for
> > substring
> > processing, with each of them requiring the value to be a character
> > string."
> >
> > Especially the "substring" makes it IMO a tiny bit vague... like
> > in:
> > foo="${binaryString}."
> > cmdSubstWithTrailingNL=${foo%.}
> >
> > One could claim: "well,... the '.' is a substring of $foo... and
> > only that
> > is processed as character and matched... so fine"
> >
> > Further, this basically assume the current proposal of
> > https://www.austingroupbugs.net/view.php?id=1562 ... i.e. that
> > pattern
> > matching notation would always be on character strings.
> > However, on the mailing list, Harald van Dijk offered to look more
> > deeply
> > into that matter, ... So may I kindly request that you defer voting
> > on this
> > particular part of the above changes, until Harald had a chance to
> > do so?
>
> Agreed.
If your "agreed" goes also for the first part of my reply... (and not
just the deferring) then it would perhaps make sense to change still
change the proposal now (accordingly) and just don't vote on it.
Otherwise I guess the first suggestion (about the vagueness of
"substring") would be easily forgotten.
> > > However, the shell may initialize parameters that do not have
> > > valid names
> > from such environment variables.
> >
> > What's that intended for? To allow positional/special parameters to
> > be
> > initialised from the environment variable?
>
> This is what you wanted above when you said "some shells may e.g.
> wish
> to provide such invalidly named env vars in a special shell var (like
> an associative array)". As I pointed out there, such things are not
> any kind of shell variable, which is why I called them parameters
> here.
Well but then the wording of your sentence seems wrong or at least
confusing to me:
> However, the shell may initialize parameters that do not have valid
> names from such environment variables.
"initialize parameters that do not have valid names"
=> how could it initialise a parameter that does not have a valid
name? That's forbidden?!
2.5 Parameters and Variables
"A parameter can be denoted by a name, a number, or one of the
special characters listed in Special Parameters."
So either it's a (valid) name (and would be e.g.
BASH_INVALIDLY_NAME_ENV_VARS, and I guess it doesn't matter here
whether one would consider that a variable or parameter), or it's a
digit (and thus a positional param... and thus this cannot be used for
that purpose) or a special char, which would require such shells to use
such special char for that purpose.
For the latter... does it really make sense to encourage them doing
this?
Wouldn't it rather make sense to just allow them something like
BASH_INVALIDLY_NAME_ENV_VARS?
> > - My original point (3) from the "Desired Actions" (the definition
> > of
> > "Stream" using "characters" although it can be bytes)... shall I
> > open
> > another ticket for this to be dealt with?
>
> Already fixed by bug 1371.
Ah thanks, I guess I had
Re: [Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
> --
> (0005807) calestyo (reporter) - 2022-04-15 23:38
> https://www.austingroupbugs.net/view.php?id=1561#c5807
> --
> Re: https://www.austingroupbugs.net/view.php?id=1561#c5795
>
> Some of the experts which all have undoubtedly much more knowledge than me
> should probably rather review that... nevertheless...
>
>
> > Since field splitting is performed on the results of
> > (unquoted) parameter expansions, it is also affected
> > by this issue, but the fix is included in my suggested
> > changes for bug 0001560 so is not repeated here.
>
> I assume with fix you mean the introduction of "bytes"/"byte sequence"
> there?
Yes.
> > Parameters can contain arbitrary byte sequences
>
> I assume that this wording implies in the language of the standard, that
> conforming shells *MUST* support that, right?
Yes. XBD 1.5 says "The feature or behavior is mandatory for an
implementation that conforms to POSIX.1-202x. An application can rely
on the existence of the feature or behavior."
> > Shell variables shall be initialized only from
> > environment variables that have valid names.
>
> What exactly is the intention of this?
To address point 4 in the desired action.
> Cause if it's that a shell must not "take over" e.g.
> '+=whatAWeirdVariable', then this was (at least indirectly) already clear
> before, by variables being parameters and these, as per 2.5, having names
> (in the sense of 3.235 Name).
>
> If your intention is however to forbid that environment variables with
> invalid names must not be "taken over" by transcribing their invalid name
> into something valid (e.g. replacing any invalid chars with '__ code point>__' - which would of course be prone to collisions) than I think
> this wording is too unclear/indirect
I disagree. If a shell variable with a valid (and therefore different)
name is initialised from an environment variable with an invalid name,
that clearly violates the requirement that shell variables are
initialised only from environment variables that have valid names.
> However some shells may e.g. wish to provide such invalidly named env vars
> in a special shell var (like an associative array)... so that should
> perhaps be allowed?
Yes, that's allowed (see below), but to be clear - such things are
not "special shell variables". They cannot be shell variables of any
kind because shell variables always have valid names.
> > ... for character substring processing.
>
> May I suggest to rather directly write something like:
> "The following four varieties of parameter expansion provide for substring
> processing, with each of them requiring the value to be a character
> string."
>
> Especially the "substring" makes it IMO a tiny bit vague... like in:
> foo="${binaryString}."
> cmdSubstWithTrailingNL=${foo%.}
>
> One could claim: "well,... the '.' is a substring of $foo... and only that
> is processed as character and matched... so fine"
>
> Further, this basically assume the current proposal of
> https://www.austingroupbugs.net/view.php?id=1562 ... i.e. that pattern
> matching notation would always be on character strings.
> However, on the mailing list, Harald van Dijk offered to look more deeply
> into that matter, ... So may I kindly request that you defer voting on this
> particular part of the above changes, until Harald had a chance to do so?
Agreed.
> > However, the shell may initialize parameters that do not have valid names
> from such environment variables.
>
> What's that intended for? To allow positional/special parameters to be
> initialised from the environment variable?
This is what you wanted above when you said "some shells may e.g. wish
to provide such invalidly named env vars in a special shell var (like
an associative array)". As I pointed out there, such things are not
any kind of shell variable, which is why I called them parameters here.
> --
> (0005808) calestyo (reporter) - 2022-04-16 00:22
> https://www.austingroupbugs.net/view.php?id=1561#c5808
> --
>
> As far as I can see all have been dealt with, except for the following:
>
>
> - My original point (3) from the "Desired Actions" (the definition of
> "Stream" using "characters" although it can be bytes)... shall I open
> another ticket for this to be dealt with?
Already fixed by bug 1371.
> - My original point (2) from the "Desired Actions" (would a shell be
> allowed to transform the value or to skip env vars which are not valid
> characters), is kinda still open.
> I mean it's specified now, that any byte values (except NUL) need to be
> supported, but not ruled out whether shells might still do any fancy
> transformations (e.g. mapping any such bytes that do not form characters
> into special Unicode
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-04-16 00:22 UTC == Summary:clarify what kind of data shell variables need to be able to hold == Relationships ID Summary -- related to 0001560 clarify wording of command substitution related to 0001562 printf utility: clarify what is (byte) ... related to 0001564 clariy on what (character/byte) strings... == -- (0005808) calestyo (reporter) - 2022-04-16 00:22 https://www.austingroupbugs.net/view.php?id=1561#c5808 -- I tried to go through all the possible points to deal with, that have come up so far in this ticket. I'd say that no adaptions (like re-iterations) are needed in these: - 3.239 Parameter, page 62 - 3.393 Variable, page 85 - 2.6.3 Command Substitution, page 2323 (unlike assumed in the description of this issue, I not longer think anything needs to be done there, which wouldn't have already been dealt with in other tickets). As far as I can see all have been dealt with, except for the following: - My original point (3) from the "Desired Actions" (the definition of "Stream" using "characters" although it can be bytes)... shall I open another ticket for this to be dealt with? - My original point (2) from the "Desired Actions" (would a shell be allowed to transform the value or to skip env vars which are not valid characters), is kinda still open. I mean it's specified now, that any byte values (except NUL) need to be supported, but not ruled out whether shells might still do any fancy transformations (e.g. mapping any such bytes that do not form characters into special Unicode regions). Should something be done about that? Like excluding it or declaring it explicitly unspecified - or should it simply be left out? - https://www.austingroupbugs.net/view.php?id=1561#c5669 That solves point (4) (in the sense that it's explicitly unspecified)... perhaps with the following to be considered by some expert: I've noticed, that the line KRE was quoting (page 2335, lines 75446-75449 and their counterparts on page 2336, lines 75469-75472) were all "only" about "Non-built-in Utility Execution". Does that open any holes with respect to regular built-in utilities? Utilities are not only the 3.369 Standard Utilities (none of which would use any such strangely named environment variables, I guess)... so a shell could, AFAIU it, make *any* program a built-in utility, right? Such program may however either expect that such ill-named variables are present - or the opposite - not present. Does anyone think that the same (i.e. that it's unspecified) should be included for regular built-ins, too? - With respect to https://www.austingroupbugs.net/view.php?id=1561#c5668 Should page 2351, line 76081-76082 include a note with respect to what's been said in page 2335, lines 75446-75449 and their counterparts on page 2336, lines 75469-75472 namely that in addition to the variables with export attribute, also such with invalid names *might* be passed on (respectively that it's unspecified whether or not)? Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue.
==
https://www.austingroupbugs.net/view.php?id=1561
==
Reported By:calestyo
Assigned To:
==
Project:Issue 8 drafts
Issue ID: 1561
Category: Shell and Utilities
Type: Enhancement Request
Severity: Editorial
Priority: normal
Status: New
Name: Christoph Anton Mitterer
Organization:
User Reference:
Section:various
Page Number:N/A
Line Number:N/A
Final Accepted Text:
==
Date Submitted: 2022-02-01 00:10 UTC
Last Modified: 2022-04-15 23:38 UTC
==
Summary:clarify what kind of data shell variables need to be
able to hold
==
Relationships ID Summary
--
related to 0001560 clarify wording of command substitution
related to 0001562 printf utility: clarify what is (byte) ...
related to 0001564 clariy on what (character/byte) strings...
==
--
(0005807) calestyo (reporter) - 2022-04-15 23:38
https://www.austingroupbugs.net/view.php?id=1561#c5807
--
Re: https://www.austingroupbugs.net/view.php?id=1561#c5795
Some of the experts which all have undoubtedly much more knowledge than me
should probably rather review that... nevertheless...
> Since field splitting is performed on the results of
> (unquoted) parameter expansions, it is also affected
> by this issue, but the fix is included in my suggested
> changes for bug 0001560 so is not repeated here.
I assume with fix you mean the introduction of "bytes"/"byte sequence"
there?
> Parameters can contain arbitrary byte sequences
I assume that this wording implies in the language of the standard, that
conforming shells *MUST* support that, right?
> Shell variables shall be initialized only from
> environment variables that have valid names.
What exactly is the intention of this?
Cause if it's that a shell must not "take over" e.g.
'+=whatAWeirdVariable', then this was (at least indirectly) already clear
before, by variables being parameters and these, as per 2.5, having names
(in the sense of 3.235 Name).
If your intention is however to forbid that environment variables with
invalid names must not be "taken over" by transcribing their invalid name
into something valid (e.g. replacing any invalid chars with '' - which would of course be prone to collisions) than I think
this wording is too unclear/indirect and it should rather be something
like:
"Environment variable that don't have valid names must not be made shell
variables, not even by transcribing the invalid names or similar means."
However some shells may e.g. wish to provide such invalidly named env vars
in a special shell var (like an associative array)... so that should
perhaps be allowed?
At least, I personally think, that the current sentence is a bit vague what
it exactly means.
> The shell shall process their values as characters
> only when performing operations that are described
> in this standard in terms of characters.
That one is quite nice!
> ... for character substring processing.
May I suggest to rather directly write something like:
"The following four varieties of parameter expansion provide for substring
processing, with each of them requiring the value to be a character
string."
Especially the "substring" makes it IMO a tiny bit vague... like in:
foo="${binaryString}."
cmdSubstWithTrailingNL=${foo%.}
One could claim: "well,... the '.' is a substring of $foo... and only that
is processed as character and matched... so fine"
Further, this basically assume the current proposal of
https://www.austingroupbugs.net/view.php?id=1562 ... i.e. that pattern
matching notation would always be on character strings.
However, on the mailing list, Harald van Dijk offered to look more deeply
into that matter, ... So may I kindly request that you defer voting on this
particular part of the above changes, until Harald had a chance to do so?
> However, the shell may initialize parameters that do not have valid names
from such environment variables.
What's that intended for? To allow positional/special parameters to
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-04-11 13:52 UTC == Summary:clarify what kind of data shell variables need to be able to hold == Relationships ID Summary -- related to 0001560 clarify wording of command substitution related to 0001562 printf utility: clarify what is (byte) ... related to 0001564 clariy on what (character/byte) strings... == -- (0005795) geoffclare (manager) - 2022-04-11 13:52 https://austingroupbugs.net/view.php?id=1561#c5795 -- Since field splitting is performed on the results of (unquoted) parameter expansions, it is also affected by this issue, but the fix is included in my suggested changes for bug https://austingroupbugs.net/view.php?id=1560 so is not repeated here. Suggested changes... On page 155 line 5331 section 8.1 Environment Variable Definition, change:The value of an environment variable is a string of characters.to:The value of an environment variable is an arbitrary sequence of bytes, except for the null byte. On page 155 line 5335 section 8.1 Environment Variable Definition, change:names shall not contain the character '='.to:names shall not contain any bytes that have the encoded value of the character '='. On page 155 line 5336 section 8.1 Environment Variable Definition, change:shall be composed of characters from the portable character setto:shall be composed of bytes that have the encoded value of characters from the portable character set On page 155 line 5342 section 8.1 Environment Variable Definition, change:Other characters may be permitted by an implementationto:Other characters, and byte sequences that do not form valid characters, may be permitted by an implementation On page 2314 line 74531 section 2.5 Parameters and Variables, add a new paragraph:Parameters can contain arbitrary byte sequences, except for the null byte. The shell shall process their values as characters only when performing operations that are described in this standard in terms of characters. On page 2316 line 74612 section 2.5.3 Shell Variables, insert:Shell variables shall be initialized only from environment variables that have valid names.before:If a variable is initialized from the environment, ... On page 2321 line 74857 section 2.6.2 Parameter Expansion, change:... for substring processing.to:... for character substring processing. After page 3626 line 125374 section C.2.5.3 Shell Variables, add a new paragraph:Since shell variables are parameters denoted by a name, the shell cannot initialize shell variables from environment variables that do not have a valid name. However, the shell may initialize parameters that do not have valid names from such environment variables. Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
The following issue has been set as RELATED TO issue 0001564. == https://austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-04-07 16:30 UTC == Summary:clarify what kind of data shell variables need to be able to hold == Relationships ID Summary -- related to 0001560 clarify wording of command substitution related to 0001562 printf utility: clarify what is (byte) ... related to 0001564 clariy on what (character/byte) strings... == Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 2022-02-02 18:44 kreNote Added: 0005654 2022-02-06 11:18 mirabilos Note Added: 0005662 2022-02-06 18:18 chet_ramey Note Added: 0005665 2022-02-06 23:17 kreNote Added: 0005666 2022-02-08 15:14 calestyo Note Added: 0005668 2022-02-09 01:58 kreNote Added: 0005669 2022-04-07 16:29 geoffclare Relationship added related to 0001560 2022-04-07 16:30 geoffclare Relationship added related to 0001562 2022-04-07 16:30 geoffclare Relationship added related to 0001564 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
The following issue has been set as RELATED TO issue 0001562. == https://austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-04-07 16:30 UTC == Summary:clarify what kind of data shell variables need to be able to hold == Relationships ID Summary -- related to 0001560 clarify wording of command substitution related to 0001562 printf utility: clarify what is (byte) ... == Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 2022-02-02 18:44 kreNote Added: 0005654 2022-02-06 11:18 mirabilos Note Added: 0005662 2022-02-06 18:18 chet_ramey Note Added: 0005665 2022-02-06 23:17 kreNote Added: 0005666 2022-02-08 15:14 calestyo Note Added: 0005668 2022-02-09 01:58 kreNote Added: 0005669 2022-04-07 16:29 geoffclare Relationship added related to 0001560 2022-04-07 16:30 geoffclare Relationship added related to 0001562 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
The following issue has been set as RELATED TO issue 0001560. == https://austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-04-07 16:29 UTC == Summary:clarify what kind of data shell variables need to be able to hold == Relationships ID Summary -- related to 0001560 clarify wording of command substitution == Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 2022-02-02 18:44 kreNote Added: 0005654 2022-02-06 11:18 mirabilos Note Added: 0005662 2022-02-06 18:18 chet_ramey Note Added: 0005665 2022-02-06 23:17 kreNote Added: 0005666 2022-02-08 15:14 calestyo Note Added: 0005668 2022-02-09 01:58 kreNote Added: 0005669 2022-04-07 16:29 geoffclare Relationship added related to 0001560 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-09 01:58 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005669) kre (reporter) - 2022-02-09 01:58 https://www.austingroupbugs.net/view.php?id=1561#c5669 -- Re https://www.austingroupbugs.net/view.php?id=1561#c5668: but one could implicitly deduce No, that' snot how it works, If the standard requires something to be done, it will say so. If it requires somethingnot to be done, it will say so. If it says nothing on an issue, then nothing is required. If this causes an interoperability problem, then the standard is broken, and should be fixed. Here, nothing needs to be done (oe perhaps beyond what has already been done) as it is already unspecified what happens in this case (which means applications cannot depend upon either behaviour): >From 2.9.1.6 in draft 2.1: It is unspecified whether environment variables that were passed to the shell when it was invoked, but were not used to initialize shell variables (see Section 2.5.3) because they had invalid names, are included in the environment passed to execl( ) and (if execl( ) fails as described above) to the new shell. Don't get all uptight that it says "environment variables" - as far as the stanrdard is concerned, everything in the environment is an environment variable, as that's all the standard defines to go there. Anything with a valid name (which must be terminated by '=' to be valid) gets turned into a shell variable, and exported. Everything else has an invalid name, and can be ignored by the shell (or used for any other purpose the shell desires). Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 2022-02-02 18:44 kreNote Added: 0005654 2022-02-06 11:18 mirabilos Note Added: 0005662 2022-02-06 18:18 chet_ramey Note Added: 0005665 2022-02-06 23:17 kreNote Added: 0005666 2022-02-08 15:14 calestyo Note Added: 0005668 2022-02-09 01:58 kreNote Added: 0005669 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-08 15:14 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005668) calestyo (reporter) - 2022-02-08 15:14 https://www.austingroupbugs.net/view.php?id=1561#c5668 -- Not sure whether that's relevant, but 2.12. Shell Execution Environment says: "Variables with the export attribute, along with those explicitly exported for the duration of the command, shall be passed to the utility environment variables" Doesn't that kinda rule out, that the shell may pass on any variables in its own environment that haven't had a valid Name (in the sense of POSIX) to any executed programs? - "Variables with the export attribute" => those are all shell variables - "along with those explicitly exported for the duration of the command" => those are the variable assignments on the command. - it doesn't rule out explicitly that no others "shall be passed" on... but one could implicitly deduce that (as those environment variables with non-Names are already part of POSIX and not just some vendor extension... so why should POSIX not list them as to be exported or at least call it "unspecified" ... if it were so?) Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 2022-02-02 18:44 kreNote Added: 0005654 2022-02-06 11:18 mirabilos Note Added: 0005662 2022-02-06 18:18 chet_ramey Note Added: 0005665 2022-02-06 23:17 kreNote Added: 0005666 2022-02-08 15:14 calestyo Note Added: 0005668 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-06 23:17 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005666) kre (reporter) - 2022-02-06 23:17 https://www.austingroupbugs.net/view.php?id=1561#c5666 -- Re https://www.austingroupbugs.net/view.php?id=1561#c5662 https://www.austingroupbugs.net/view.php?id=1561#c5665 ash derived shells work the same way. Valid names in the incomoming environment (except a few like IFS) are imported into the shell's variable table, and exported. The outgoing environment is built from the exported varibles in the shell, and nothing else. We could keep the discarded trash from the imported environment. but don't, and I see no reason to ever do that. Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 2022-02-02 18:44 kreNote Added: 0005654 2022-02-06 11:18 mirabilos Note Added: 0005662 2022-02-06 18:18 chet_ramey Note Added: 0005665 2022-02-06 23:17 kreNote Added: 0005666 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-06 18:18 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005665) chet_ramey (reporter) - 2022-02-06 18:18 https://www.austingroupbugs.net/view.php?id=1561#c5665 -- Re: "fighting about it". We may have, I don't remember when. I don't understand the second paragraph, unless mksh uses setenv and its siblings to create the child environment. Surely you already reject the bad names when importing variables from the environment, so you know when they occur. You could save them in some table and add them to the child environment when you create it. Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 2022-02-02 18:44 kreNote Added: 0005654 2022-02-06 11:18 mirabilos Note Added: 0005662 2022-02-06 18:18 chet_ramey Note Added: 0005665 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-06 11:18 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005662) mirabilos (reporter) - 2022-02-06 11:18 https://www.austingroupbugs.net/view.php?id=1561#c5662 -- It absolutely needs to stay unspecified whether “bad names” are passed to child processes. I thought we had fought about this already? In mksh, the child environment is created from scratch, using exported shell parameters and nothing else. Requiring passthrough here would be insane, a weird hell of trying to deal with setenv/unsetenv/whatever… and explode _more_ in the face of duplicates than the current method. Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 2022-02-02 18:44 kreNote Added: 0005654 2022-02-06 11:18 mirabilos Note Added: 0005662 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-02 18:44 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005654) kre (reporter) - 2022-02-02 18:44 https://www.austingroupbugs.net/view.php?id=1561#c5654 -- Re https://www.austingroupbugs.net/view.php?id=1561#c5653 Thanks. Undefined seems a little harsh however, I'd have thought unspecified should be sufficient for this. Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 2022-02-02 18:44 kreNote Added: 0005654 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-02 16:39 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005653) calestyo (reporter) - 2022-02-02 16:39 https://www.austingroupbugs.net/view.php?id=1561#c5653 -- Re: https://www.austingroupbugs.net/view.php?id=1561#c5650 8.1 Environment Variable Definition says: "If more than one string in an environment of a process has the same name, the consequences are undefined." Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 2022-02-02 16:39 calestyo Note Added: 0005653 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-02 15:15 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005652) chet_ramey (reporter) - 2022-02-02 15:15 https://www.austingroupbugs.net/view.php?id=1561#c5652 -- Re: https://www.austingroupbugs.net/view.php?id=1561#c5650 Bash supports both old and newer behavior wrt environment strings. If an environment string doesn't contain `=' or starts with `=' (no name), bash skips it (way, way back in the day, such strings would cause some applications -- notably `sh' -- to seg fault). An environment string that contains an `=', doesn't follow the naming rules for an exported shell function, but has a name that is not a valid shell variable name, gets passed to child processes in their environment but otherwise ignored. Bash doesn't export or import array variables. There is code in there to do it, just commented out. It encodes the value as a compound variable assignment, similar to the output of `declare -p'. I've never released a version of bash with that code enabled. One of the concerns has been distinguishing it from a scalar variable with value `( some list of words )'. There aren't any shells that export arrays as arrays. ksh93 and mksh export element 0 as a scalar variable. yash exports the values as a colon-separated list, but doesn't import the variable as an array. bash and zsh don't allow it. Bash does export and import shell functions, encoding the names in a way that allows the shell to recognize them as functions. Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 2022-02-02 15:15 chet_ramey Note Added: 0005652 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-01 23:07 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005650) kre (reporter) - 2022-02-01 23:07 https://www.austingroupbugs.net/view.php?id=1561#c5650 -- Re 4) again, I agree, unspecified. But re https://www.austingroupbugs.net/view.php?id=1561#c5649 I thought that it was unspecified whether such things are passed through to children. Our shell creates vars from the environment (exported ones of course) when the names are valid, and ignores anything else it sees there. When running a child process all exported sh variables (and nothing else) is passed to the child as its environment. I suspect that the question about arrays was assuming that the environment might contain something like: ARRAY[4]=7 if some shel decided to permit exporting individual elements of an array, Lastly, and I'm not sure if this shold be considered as part of 2) or added as a new 5) - but is it anywhere specified what the shell is supposed to do if it receives an environment which contains X=1 X=2 X=3 ??Shells will never create such a thing, and I don't believe the env command can do it either, but purpose written C code can easily set that up. rather than the whole array (in the case that a whole array is exported, how does bash, when importing it, distinguish that from a scalar var whose contents just happen to be identical to whatever serialized format the array contents are exported like ?) And POSIX only requires POSIX names to work as shell function names, though that is a truly stupid limitation (it permits implementations to extend it, and most do I think). We (NetBSD sh) don't allow exported functions, but we do allow for every shell to read a startup file (not just interactive shells) so functions that one mighth want to export can be written there, and then are available to all who follow (a script can make its own file and arrange for its children to read that one, if it wants to). Lastly, when reading the standard, I believe that we must sometimes use care in assuming that "characters" means locale specific chars, rather than bytes. Much of the text is quite old, and has its origins in even older material, and that far ago, for most concerned people, characters and bytes were the same thing (which is why what we might now call an int8_t in C started out (and still is though sometimes as a uint8_t) also called "char"). Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 2022-02-01 23:07 kreNote Added: 0005650 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-01 20:52 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005649) chet_ramey (reporter) - 2022-02-01 20:52 https://www.austingroupbugs.net/view.php?id=1561#c5649 -- Re: 4) This has come up before and the current standard contains text to the effect that "applications shall tolerate the presence of such names." The consensus seems to be to pass them through to child processes in their environment but not attempt to create shell variables from them. It seems to me that arrays don't affect this: they're normal shell variables and have normal shell variable name restrictions. Shell functions do, since they can have nearly arbitrary names, the bash function name encoding aside. Bash tolerates such names by -- you guessed it -- creating shell functions from them. Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 2022-02-01 20:52 chet_ramey Note Added: 0005649 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-01 19:44 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005647) calestyo (reporter) - 2022-02-01 19:44 https://www.austingroupbugs.net/view.php?id=1561#c5647 -- I'd agree with that. Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 2022-02-01 19:44 calestyo Note Added: 0005647 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
A NOTE has been added to this issue. == https://www.austingroupbugs.net/view.php?id=1561 == Reported By:calestyo Assigned To: == Project:Issue 8 drafts Issue ID: 1561 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Christoph Anton Mitterer Organization: User Reference: Section:various Page Number:N/A Line Number:N/A Final Accepted Text: == Date Submitted: 2022-02-01 00:10 UTC Last Modified: 2022-02-01 19:33 UTC == Summary:clarify what kind of data shell variables need to be able to hold == -- (0005645) mirabilos (reporter) - 2022-02-01 19:33 https://www.austingroupbugs.net/view.php?id=1561#c5645 -- For 4) I suggest unspecified; many shells allow arrays to be imported from the environment, and GNU bash even has imported functions… I’d be very much against a requirement to transform them somehow (especially as that would open the door to more attacks), so (shell-locally) extending the permitted values and ignoring the rest is a sane way to go. Issue History Date ModifiedUsername FieldChange == 2022-02-01 00:10 calestyo New Issue 2022-02-01 00:10 calestyo Name => Christoph Anton Mitterer 2022-02-01 00:10 calestyo Section => various 2022-02-01 00:10 calestyo Page Number => N/A 2022-02-01 00:10 calestyo Line Number => N/A 2022-02-01 19:33 mirabilos Note Added: 0005645 ==
[Issue 8 drafts 0001561]: clarify what kind of data shell variables need to be able to hold
The following issue has been SUBMITTED.
==
https://www.austingroupbugs.net/view.php?id=1561
==
Reported By:calestyo
Assigned To:
==
Project:Issue 8 drafts
Issue ID: 1561
Category: Shell and Utilities
Type: Enhancement Request
Severity: Editorial
Priority: normal
Status: New
Name: Christoph Anton Mitterer
Organization:
User Reference:
Section:various
Page Number:N/A
Line Number:N/A
Final Accepted Text:
==
Date Submitted: 2022-02-01 00:10 UTC
Last Modified: 2022-02-01 00:10 UTC
==
Summary:clarify what kind of data shell variables need to be
able to hold
Description:
In:
https://collaboration.opengroup.org/austin/plato/protected/mailarch.php?soph=N=show=austin-group-l=33722=100=0=
I've raised the question, on which data shell variables are required to be
able to hold.
In various replies following it became clear that there is some ambiguity
with respect to that question:
In:
https://collaboration.opengroup.org/austin/plato/protected/mailarch.php?soph=N=show=austin-group-l=33723=100=0=
Geoff Clare brought up that:
»but POSIX clearly requires that a variable can be
assigned any value obtained from a command substitution that does not
include a NUL byte, and specifies utilities that can be used to
generate arbitrary byte values, therefore a variable can contain any
sequence of bytes that does not include a NUL byte.«
Which AFAIU means that shell variables are expected to hold any bytes
except NUL, and only the use of these shell variables in certain other
constructs (e.g. ${#var}) interprets them as characters according to the
current locale.
It was brought up, that e.g. yash discards any bytes from shell variables
that don't make up a valid encoding:
https://collaboration.opengroup.org/austin/plato/protected/mailarch.php?soph=N=show=austin-group-l=33724=100=0=
In:
https://collaboration.opengroup.org/austin/plato/protected/mailarch.php?soph=N=show=austin-group-l=33725=100=0=
Chet Ramey brought up, that shell variables are initialised from
environment variables, which themselves may contain anything except NUL as
value, as long as anything before the "=" is a valid Name (in the sense of
POSIX).
And in the later:
https://collaboration.opengroup.org/austin/plato/protected/mailarch.php?soph=N=show=austin-group-l=33731=100=0=
that:
»applications can obviously put whatever they want into the value of an
environment variable in envp and call execve.«
In:
https://collaboration.opengroup.org/austin/plato/protected/mailarch.php?soph=N=show=austin-group-l=33730=100=0=
Harald van Dijk countered, that:
»That is not what POSIX says. It says "The value of an environment
variable is a string of characters" (8.1 Environment Variable Definition),
and "character" is defined as "a sequence of one or more bytes representing
a single graphic symbol or control code" (3 Definitions), with a note that
says it corresponds to what C calls a multi-byte character. Environment
variables are not specified to allow arbitrary bytes.«
There was some further discussion on whether the definition of command
substitutions implies whether or not any bytes other than NUL need to be
able to be stored in shell variables.
One argument brought up was, that there the wording " character"
is used - another, that this would clearly refer *only* to the
itself which is per definition the same (byte) in every locale.
(for that particular part see also the proposed clarifications in
https://www.austingroupbugs.net/view.php?id=1560 ).
In:
https://collaboration.opengroup.org/austin/plato/protected/mailarch.php?soph=N=show=austin-group-l=33736=100=0=
I brought up that in addition to what Harald pointed out earlier, in 8.1
Environment Variables it says:
»These strings have the form name=value; names shall not contain the
character '='. For values to be portable across systems conforming to
POSIX.1-2017, the value shall be composed of characters from the
portable character set (except NUL and as indicated below).«
but a bit further down it says the contradicting:
»The values that the environment variables may be assigned are not
restricted except that they are considered to end with a null byte and
the total space used to store the environment and the arguments to the
process is limited to {ARG_MAX} bytes.«
And in:
