Re: Arithmetic expansion with increments and output redirection
> > "Utilities other than the special built-ins (see Special Built-In > Utilities) shall be invoked in a separate environment that consists of the > following...[includes redirections specified to the utility]...The > environment of the shell process shall not be changed by the utility" > > > http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12 > The utility here in question is the external command (/bin/true in my example), which doesn't do the expansion itself. In this case, it's not the utility affecting the environment, it's the expansion performed by the (sub)shell itself before the exec*() that has an effect on the environment. They're different from word expansions. > The bash manual calls it a word, which is one of the sources of my confusion. "The general format for redirecting output is: [n]>word" https://linux.die.net/man/1/bash
Re: Arithmetic expansion with increments and output redirection
On 4/24/19 10:20 AM, Ian Neal wrote: > "Utilities other than the special built-ins (see Special Built-In > Utilities) shall be invoked in a separate environment that consists of the > following...[includes redirections specified to the utility]...The > environment of the shell process shall not be changed by the utility" > > > http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12 > > The utility here in question is the external command (/bin/true in my > example), which doesn't do the expansion itself. In this case, it's not the > utility affecting the environment, it's the expansion performed by the > (sub)shell itself before the exec*() that has an effect on the environment. I think the POSIX language is crafted to allow both behaviors, and bash has always interpreted it to mean that the redirections are processed, including any word expansions specified, in the child process forked to execute the command. > > > They're different from word expansions. > > The bash manual calls it a word, which is one of the sources of my confusion. > "The general format for redirecting output is: > > [n]>word" Imprecise language. Let's see if I can do better. A word has a specific definition, which is given earlier in the man page. It determines which characters make up the token following the redirection operator. That word undergoes a set of expansions, which are also specified. These are a subset of the full set of expansions a word that is not part of a redirection undergoes (for example, non-interactive shells don't perform globbing on the word in a redirection, and word splitting is never performed). These are performed separately, and not at the same time as the expansions on a word that is an argument to a simple command. In fact, there was a debate over whether or not the effects of the other word expansions are visible to the expansions performed when processing redirections. That's what I mean when I say they're different from word expansions. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Arithmetic expansion with increments and output redirection
On Wed, Apr 24, 2019 at 09:58:42AM -0400, Chet Ramey wrote:
> The SVR4.2 Bourne shell, as another data point, behaves like bash. Since
> it doesn't have arithmetic expansion, you have to use something it does
> handle internally, like ${n:=2}. I don't know what ksh88, the other POSIX
> historical reference implementation, does.
# uname -a
HP-UX vandev B.10.20 A 9000/778 2000153729 two-user license
# exec ksh
# set -o vi
# n=0; : $((n=n+1)); echo "$n"
1
# n=0; /bin/true > $((n=n+1)); echo "$n"
0
Looks like it changed between ksh88 and ksh93.
Re: Arithmetic expansion with increments and output redirection
On 4/24/19 9:49 AM, Ilkka Virta wrote:
> On 24.4. 16:37, Chet Ramey wrote:
>> "Utilities other than the special built-ins (see Special Built-In
>> Utilities) shall be invoked in a separate environment that consists of the
>> following...[includes redirections specified to the utility]...
>
> It does say
>
> "Open files inherited on invocation of the shell, open files controlled by
> the exec special built-in plus any modifications, and additions specified
> by any redirections to the utility"
>
> which could also be read to apply only the open files themselves, not the
> byproducts of finding out their names.
I think the POSIX language is crafted to allow both behaviors, and bash
has always interpreted it to process redirections, including the expansions
performed on the word, in the subshell forked to execute the command.
> Anyway, as little as it's worth, Zsh seems to do it the same way Bash does,
> all others leave the changed value visible.
The SVR4.2 Bourne shell, as another data point, behaves like bash. Since
it doesn't have arithmetic expansion, you have to use something it does
handle internally, like ${n:=2}. I don't know what ksh88, the other POSIX
historical reference implementation, does.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Arithmetic expansion with increments and output redirection
On 24.4. 16:37, Chet Ramey wrote: "Utilities other than the special built-ins (see Special Built-In Utilities) shall be invoked in a separate environment that consists of the following...[includes redirections specified to the utility]... It does say "Open files inherited on invocation of the shell, open files controlled by the exec special built-in plus any modifications, and additions specified by any redirections to the utility" which could also be read to apply only the open files themselves, not the byproducts of finding out their names. The environment of the shell process shall not be changed by the utility" It's not the utility that changes the environment when processing the expansion, but the shell itself, isn't it? - - Anyway, as little as it's worth, Zsh seems to do it the same way Bash does, all others leave the changed value visible. $ for shell in 'busybox sh' dash yash ksh93 mksh bash zsh; do $shell -c 'i=1; /bin/echo foo > $(( i += 1 )); printf "%-15s %s\n" "$1:" "$i";' sh "$shell"; done busybox sh: 2 dash: 2 yash: 2 ksh93: 2 mksh: 2 bash: 1 zsh:1 I also did find the Bash/Zsh behaviour a bit surprising. But I'm not sure it matters other than here and with stuff like $BASHPID? It's easy to work around here by splitting the increment/decrement to a separate line: /bin/echo foo > "$i" : "$(( i += 1 ))" Some find that easier to read, too: the increment isn't "hidden" within the other stuff on the command line. -- Ilkka Virta / itvi...@iki.fi
Re: Arithmetic expansion with increments and output redirection
On 4/24/19 9:07 AM, Andreas Schwab wrote: >> The /bin/true forces the use of an external command, which means a subshell >> is forked. The redirection occurs in the forked subshell. > > But the expansion isn't required to be performed in the subshell. I > don't see POSIX having any wording to require one or the other. I think the POSIX wording is carefully crafted to allow all behaviors. > Doing > the expansion before the fork looks reasonable, even if only to make it > consistent with builtins. Is that consistency more valuable than avoiding side effects? -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Arithmetic expansion with increments and output redirection
On Wed, Apr 24, 2019, 07:12 Chet Ramey wrote: > On 4/24/19 8:47 AM, Ian Neal wrote: > > > At what point is a subshell being invoked? There's no pipeline, command > > substitution, coprocess, background process, or explicit () subshell > here, > > which are the only cases a subshell should be created. Otherwise, the > > entire operation should be evaluated by the main shell. Arithmetic > > expansion is not in that list. > > Why would you think that /bin/true would be "evaluated by the main shell?" > It's not a shell compound command or a builtin, and non-builtin commands > are run in child processes. > Calling what should be a simple fork(); exec(); a "subshell" is a little disingenuous unless it has its own parameter expansion options, which isn't true -- if the $((n++)) was on the left side of the file redirection (which is to say a parameter), it would be expanded by the shell before the fork(), so why is this true with the redirection itself? What makes that case so special? I posit that it shouldn't be. >
Re: Arithmetic expansion with increments and output redirection
On Apr 24 2019, Greg Wooledge wrote: > On Wed, Apr 24, 2019 at 06:47:41AM -0600, Ian Neal wrote: >> At what point is a subshell being invoked? There's no pipeline, command >> substitution, coprocess, background process, or explicit () subshell here, >> which are the only cases a subshell should be created. Otherwise, the >> entire operation should be evaluated by the main shell. Arithmetic >> expansion is not in that list. > > The context got snipped but IIRC it was something like > > /bin/true > $((n++)) > > The /bin/true forces the use of an external command, which means a subshell > is forked. The redirection occurs in the forked subshell. But the expansion isn't required to be performed in the subshell. I don't see POSIX having any wording to require one or the other. Doing the expansion before the fork looks reasonable, even if only to make it consistent with builtins. Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Re: Arithmetic expansion with increments and output redirection
On 4/23/19 5:49 PM, Ian Neal wrote: > Bash Version: 4.2 > Patch Level: 46 > Release Status: release > > Description: > When using arithmetic expansion with variable pre- and > post-increments/decrements in the output redirection file path, > specifically on external executables (not builtins or functions), the state > of the variable being incremented/decremented is not persisted in the > environment. The redirection is expanded in the environment of the command. In the second case, that means the child process forked to run /bin/true. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Arithmetic expansion with increments and output redirection
On 4/24/19 8:47 AM, Ian Neal wrote: > At what point is a subshell being invoked? There's no pipeline, command > substitution, coprocess, background process, or explicit () subshell here, > which are the only cases a subshell should be created. Otherwise, the > entire operation should be evaluated by the main shell. Arithmetic > expansion is not in that list. Why would you think that /bin/true would be "evaluated by the main shell?" It's not a shell compound command or a builtin, and non-builtin commands are run in child processes. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Arithmetic expansion with increments and output redirection
On Wed, Apr 24, 2019 at 06:47:41AM -0600, Ian Neal wrote: > At what point is a subshell being invoked? There's no pipeline, command > substitution, coprocess, background process, or explicit () subshell here, > which are the only cases a subshell should be created. Otherwise, the > entire operation should be evaluated by the main shell. Arithmetic > expansion is not in that list. The context got snipped but IIRC it was something like /bin/true > $((n++)) The /bin/true forces the use of an external command, which means a subshell is forked. The redirection occurs in the forked subshell.
Re: Arithmetic expansion with increments and output redirection
> Date:Tue, 23 Apr 2019 15:49:18 -0600 > From:Ian Neal > Message-ID: 1_z7uy+7rv...@mail.gmail.com> > > | When using arithmetic expansion with variable pre- and > | post-increments/decrements in the output redirection file path, > | specifically on external executables (not builtins or functions), the > state > | of the variable being incremented/decremented is not persisted in the > | environment. > > That is as it should be, redirects are eveluated in the sub-shell > context. If anything the bug you showed is that redirects when the > shell does not fork are being evaluated in the context of the shell > (but I think that might be an unspecified case). > > In general it is best not to even consider using any evaluation with > side effects in any redirection, and unless you really consider what > you're doing, not in var-assigns either (there, if you're not expecting > the side effects to be visible in the same, or any other, assignemnt > in the same command you should be OK). > > kre > At what point is a subshell being invoked? There's no pipeline, command substitution, coprocess, background process, or explicit () subshell here, which are the only cases a subshell should be created. Otherwise, the entire operation should be evaluated by the main shell. Arithmetic expansion is not in that list. >
Re: Arithmetic expansion with increments and output redirection
Date:Tue, 23 Apr 2019 15:49:18 -0600 From:Ian Neal Message-ID: | When using arithmetic expansion with variable pre- and | post-increments/decrements in the output redirection file path, | specifically on external executables (not builtins or functions), the state | of the variable being incremented/decremented is not persisted in the | environment. That is as it should be, redirects are eveluated in the sub-shell context. If anything the bug you showed is that redirects when the shell does not fork are being evaluated in the context of the shell (but I think that might be an unspecified case). In general it is best not to even consider using any evaluation with side effects in any redirection, and unless you really consider what you're doing, not in var-assigns either (there, if you're not expecting the side effects to be visible in the same, or any other, assignemnt in the same command you should be OK). kre
Arithmetic expansion with increments and output redirection
Configuration Information [Automatically generated, do not change]: Machine: x86_64 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-redhat-linux-gnu' -DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic uname output: Linux s1028505 3.10.0-957.10.1.el7.x86_64 #1 SMP Thu Feb 7 07:12:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Machine Type: x86_64-redhat-linux-gnu Bash Version: 4.2 Patch Level: 46 Release Status: release Description: When using arithmetic expansion with variable pre- and post-increments/decrements in the output redirection file path, specifically on external executables (not builtins or functions), the state of the variable being incremented/decremented is not persisted in the environment. Repeat-By: # n=1; true > $((--n)); echo $n; ls outputs: 0 0 but # n=1; /bin/true > $((--n)); echo $n; ls outputs: 1 0
