2025年8月28日(木) 17:20 Kevin Pulo <kevin.p...@mongodb.com>:
> On Wed, 27 Aug 2025 at 19:53, Koichi Murase <myoga.mur...@gmail.com> wrote:
> > We constantly receive reports from users saying "the frameworks should
> > work flawlessly with this set of options because it is the built-in
> > feature of Bash", etc. in bash-completion, oh-my-bash, ble.sh, etc. It
> > won't stop until we enclose all command substitutions within « local
> > saved=$(shopt -p cmdsubst_trailing_nls); shopt -s
> > cmdsubst_trailing_nls ... eval -- "$saved" ».
>
> Similar arguments can be made about any behavior-modifying option, but they
> continue to be added to bash (eg. patsub_replacement, localvar_inherit). [...]
Yes, you are right, and in fact, I raised discussions for
`patsub_replacement' and `localvar_{unset,inherit}' in the past. In
general, I think the bar to add new shell options should be high when
the options may change the global behavior of the shell.
Coincidentally, both the options you mentioned, `patsub_replacement'
and `localvar_inherit', are the ones that made me decide to
participate in the discussion about the compatibility more actively.
When `patsub_replacement' was added by Chet and when discussions that
introduced `localvar_inherit' happened, I considered their impact and
raised/participated in discussions on the mailing lists. I'm just
repeating the same for the suggested `cmdsubst_trailing_nls' here.
Nevertheless, I think the addition of the trailing-newline-preserving
feature through shell options has a larger impact on existing codes
than `patsub_replacement' and `localvar_inherit'.
> [...] This
> is partly why I thought it was not unreasonable to ask if these shopts can be
> considered.
Yes, it is not unreasonable to ask for the possibility of new shell
options, but at the same time, it is not unreasonable to show an
opposing opinion.
1) First, `cmdsubst_trailing_nls' affects all the command
substitutions where the output ends with a newline (which is the
normal behavior in the CLI tools), while `patsub_replacement' only
affects the cases where ${var/xxx/yyy} is used and yyy may contain
`&', and `localvar_inherit' only affects the case where the user
referenced uninitialzied variables. When the patsub_replacement was
added, I made a survey to check how many lines in major frameworks and
Bash programs would be affected by the change of patsub_replacement
(see an attachment [1] to this mailing list in 2022-01). About 60
lines were affected by the addition of patsub_replacement, which isn't
a small number, but I'm sure that it's much less than the number of
command substitutions in those projects. Have you gauged the impact of
adding "cmdsubst_trailing_nls" on the existing frameworks in a similar
way?
[1] https://lists.gnu.org/r/bug-bash/2022-01/txttsBegI6e9u.txt
2) Next, for `patsub_replacement' and `localvar_inherit', we don't
need a complicated workaround like « local saved=$(shopt -p
cmdsubst_trailing_nls); shopt -s cmdsubst_trailing_nls ... eval --
"$saved" » to make the code work with any sides of the options.
For `patsub_replacement', one can simply quote the replacement "yyy"
as « a=${rep/xxx/"yyy"}» so that it works in the expected way
regardless of the shell option `patsub_replacement'. This was what I
requested in Ref. [2] soon after the initial implementation of
`patsub_replacement' appeared in the devel branch. (I also requested
to make `patsub_replacement' an opt-in feature, but it was rejected.
Later, this triggered another discussion [3] after 5.2 was released,
but it was too late. We should definitely discuss new shell behaviors
sufficiently in advance. After thar, I decided to vocally raise
discussions when I have any concerns). When you also care about 4.2 or
`compat42', you need to apply the replacement in the variable
assignment, but it's still not so complicated. The only nontrivial
case is the array expansion with `pathsub_replacement' [4], but the
chances are limited.
[2] https://lists.nongnu.org/archive/html/bug-bash/2021-10/msg00050.html
[3] https://lists.nongnu.org/archive/html/bug-bash/2022-11/threads.html#00001
[4]
https://github.com/scop/bash-completion/blob/c55ee7f6fb75300786cb522261f68eb80366c41f/doc/styleguide.md#patsub_replacement-for-array-elements
For `localvar_inherit', one can write « local v= » to make sure that
the variable is empty, or if you want to make the variable the `unset'
state, one can write « local v; unset -v v ». There is a minor
technique to receive optional parameters by « local v » through
temporary environments without being affected by existing variables,
and it would be broken by `localvar_inherit', but I think it is a very
minor technique only known on this list. I think the following
discussions partly reflect my position on compatibility change related
to the dynamic scoping of the local variables.
[5] https://lists.nongnu.org/archive/html/bug-bash/2018-03/msg00020.html
[6] https://lists.nongnu.org/archive/html/help-bash/2023-03/msg00151.html
After all, it would be a matter of extent. A slight compatibility
breakage that doesn't have a significant impact on real existing
scripts might be added, while a larger compatibility change that
breaks a wide range of existing codebases should not be introduced.
The question here is whether it's worth introducing the compatibility
issue to all existing command substitutions (instead of introducing a
new syntax that doesn't break any existing codes or working with a
simple shell function).
> It would only be necessary to preserve cmdsubst_trailing_nls around command
> substitutions where trailing newline behavior actually matters, eg. any
> unquoted $() is unlikely to be affected, because the subsequent word splitting
> will give the same results with or without trailing newlines.
OK, unquoted $()'s (that undergo word splitting) are unaffected,
assuming the normal IFS. However, the quoted command substitutions (or
command substitutions in variable assignments) are more common than
the unquoted $() and may still be affected by the new option.
> But this also means there could be another approach for frameworks - to ensure
> that the commands run inside substitutions emit no trailing newlines. Then
> the
> result will be the same regardless of cmdsubst_strip_newlines.
If it is acceptable to change the behavior at the side of the command
inside $(), isn't it easier to modify the commands that want to
preserve the trailing newlines so that they output an extra `.',
allowing to simply call it as « foo=$(cmd); x=${x%.} »? Since the CLI
tools are basically designed to be line-oriented for the historical
reason, the majority of the command substitutions expects the
traditional command substitution, and the cases of preserving the
newlines are minor compared to the historical one. Then, the behavior
of the minor cases should be modified, instead of trying to change the
majority, which follows the traditions.
> This could be done by piping into an ugly sed/awk program, [...]
> Also note that, AFAIK, the reverse is not possible. That is, if bash doesn't
> have a trailing newline preserving command substitution, it is not possible to
> write a preserve_trailing_nls function that makes $(foo |
> preserve_trailing_nls) work as intended.
You can define e.g. « preserve_trailing_nls() { REPLY=$(cat;echo .)
REPLY=${REPLY%.}; } » and turn on `lastpipe`, so ${| foo |
preserve_trailing_nls; } works as you expect. If you can assume NUL
doesn't appear in the stream, you can even do ${| foo | read -rd ''; }
without introducing any functions. If you want to preserve the exit
status, you can combine these with a process substitution `< <(foo)'.
> I'm not insisting, I'm asking, and trying to explain that this way does have
> some advantages, as well as some disadvantages. Just the same as how new
> syntax has both advantages and disadvantages. It's ok if the conclusion is
> not
> to have a shopt, but the pros and cons of each approach should be properly
> understood and weighed.
I totally agree. We are discussing the advantages and disadvantages of
different ways to implement the feature: a shell option vs a new
syntax vs a shell function.
> But I am aware that just because it was easy to do, does not necessarily mean
> that it is the best approach,
Yes, that's my point.
> and I have said as much from the start (and it's
> why I am _asking_ for this to be considered for inclusion).
Yes, so I replied to exactly that part "(1) the choice of using shopts
to modify behavior" as I quoted at the top of my first reply.
> I don't know what those main burdens are, but aiming for a light touch
> is usually better than suggesting large and invasive changes.
If the resulting behavior is the same, yes.
> > The cost is negligible as long as it is properly implemented.
>
> Here is a hypothetical example of a burden that could arise from doing this as
> new syntax.
>
> [...] ${|foo;} can only be used for one or the other of
> the trailing newline preserving version of ${ foo;}, or the $REPLY-style
> no-fork command substitution, but not both.
There are still other delimiters `;' `;;' `&&', `|&', `||', etc. It is
even possible to use an arbitrary name (such as `preserve_nls') plus
space because ${preserve_pls ...} is currently a syntax error. In the
first place, t doesn't need to be in the form ${<something> xxx; }.
Then, I don't think there would be so many variations of command
substitutions that deserve a new feature at the language level.
> > What you propose is actually the opposite. With the shell options to
> > change existing behaviors, existing codes expecting the traditional
> > behavior may produce unexpected results *without causing an explicit
> > error*, when they are combined with a code that changes the option
> > poorly.
>
> As above, the same is true when other behavior-modifying shell options are
> introduced.
Yes, so I'm against the suggested option in the same way as I was
against the other new behavior-modifying shell options.
> Sorry, I wasn't clear with this. What I mean is, suppose you have some
> scripts
> [...] one day you discover that the
> trailing newline stripping is causing incorrect behavior. You realize that
> you
> hadn't considered this case,
> [...] with user-selectable shell options to control the trailing newline
> handling, instead of updating the syntax at many locations, you can instead
> just set the shopts once at the start of each script.
I don't think it is a good idea to add an option to turn someone's
misunderstanding into a real language feature. If we start to
implement every kind of misunderstanding as real shell features, the
language becomes unmanageable. I cannot agree with this specific
reasoning.
> Please note that I am not saying "therefore the shopt approach should be
> used".
> I am just saying that this is an advantage of this approach, in this
> circumstance, which should be weighed against the alternatives.
OK.
> > > For example, while it's possible for me to write a shell function that
> > > wraps builtins/commands and "adjusts" their behavior, the inability to use
> > > shell functions (or anything else) to modify the behavior of shell syntax
> > > constructs like $(), is precisely why I looked into adding these shopts in
> > > the first place.
> >
> > This is tautological. The above seems to attempt to explain the reason
> > why the option to change the existing behavior of shell constructs
> > like $() is introduced instead of using a shell function. It says the
> > reason is that the existing constructs like $() cannot be modified by
> > using shell functions. More than half of the arguments seem to be
> > nonsensical. Do you use LLM?
>
> I don't think you've understood what I was trying to say.
>
> Suppose that in my scripts, I want read to always use -r, unless I override
> that by specifying +r. I can achieve this by defining something like:
> [...]
So you simply stated the triavil fact that the behavior $() cannot be
modified by a shell function. I'm not sure how it is relevant to the
present discussion. We were talking about the way to implement the
suggested feature: "changing the behavior of $() through an option" vs
"introducing a new syntax" vs "using a shell function like
`grab_output'".
> Suppose bash had a feature where, if a shell function called _comsub is
> defined, then that function gets called whenever `` or $() is encountered.
It's fine if you are talking about a hypothetical feature just for
fun, but I'm not sure if it is related to the current discussion. If
you seriously suggest this as a new feature, I have to repeat the same
discussion of the impact on the existing scripts using the traditional
command substitutions $(). This should introduce a new syntax (such as
« ${func_name cmd;} » or « ${func_name(...)} »). Actually, it's much
worse than the suggested shell option. I don't think we need to seek
an even worse approach for preserving newlines than the shell-option
approach.
> This would still break any external code that assumes traditional behavior,
> but
> if that's a concern I could add some sophistication to make it only apply to
> my
> own scripts:
Not all users can do this properly. This becomes a new unnecessary pitfall.
> What I am saying is that, because no such feature exists, I (as a user) have
> no
> way to control what $() does or doesn't do. And that is why I tried adding
> the
> shopt to give users some small control over what $() does.
My claim is that for the present purpose of providing a feature to
allow preserving the trailing newlines, it's not necessary to prepare
a way that may allow changing the global behavior of the command
substitutions.
> [...] (or even $(foo |
> bar | special) or $(special | bar | foo)), I would be happy to hear it. But
> my
> understanding is that this is currently impossible in bash.
I answered this above.
--
Honestly, I still think a "not-ideal but realistic" approach is to use
a shell function like `grab_output', though I agree that the
appearance isn't nice, and I also point out that we need an extra
quoting for the command to be run (just like the argument of the
`eval' builtin). However, these are just a matter of appearance, and I
feel it wouldn't deserve introducing complication in the existing
codebase or new places for pitfalls. If the feature is really desired
by the community as an essential feature, it should be implemented as
a new syntax.
--
Koichi
