On Wed, 27 Aug 2025 at 19:53, Koichi Murase <myoga.mur...@gmail.com> wrote:
> We constantly receive reports from users saying "the frameworks should
> work flawlessly with this set of options because it is the built-in
> feature of Bash", etc. in bash-completion, oh-my-bash, ble.sh, etc. It
> won't stop until we enclose all command substitutions within « local
> saved=$(shopt -p cmdsubst_trailing_nls); shopt -s
> cmdsubst_trailing_nls ... eval -- "$saved" ».
Similar arguments can be made about any behavior-modifying option, but they
continue to be added to bash (eg. patsub_replacement, localvar_inherit). This
is partly why I thought it was not unreasonable to ask if these shopts can be
considered.
It would only be necessary to preserve cmdsubst_trailing_nls around command
substitutions where trailing newline behavior actually matters, eg. any
unquoted $() is unlikely to be affected, because the subsequent word splitting
will give the same results with or without trailing newlines.
But this also means there could be another approach for frameworks - to ensure
that the commands run inside substitutions emit no trailing newlines. Then the
result will be the same regardless of cmdsubst_strip_newlines.
This could be done by piping into an ugly sed/awk program, but even better
would to have a shell function that uses a subshell:
function strip_trailing_nls {
( shopt -u cmdsubst_trailing_nls 2>/dev/null; echo -n "$(cat)"; )
}
Changing each $(foo) to $(foo | strip_trailing_nls) is not great, but it's a
lot better than changing each $(foo) to « local saved=$(shopt -p
cmdsubst_trailing_nls); shopt -s cmdsubst_trailing_nls; ... $(foo) ...; eval --
"$saved" ».
Also note that, AFAIK, the reverse is not possible. That is, if bash doesn't
have a trailing newline preserving command substitution, it is not possible to
write a preserve_trailing_nls function that makes $(foo |
preserve_trailing_nls) work as intended.
> Even if a consensus that "the frameworks don't need to handle
> non-historical options" could be formed in the future, such a shell
> option to change the traditional behavior still remains to be a
> pitfall. Why do you stick with changing the historical behavior?
I'm not insisting, I'm asking, and trying to explain that this way does have
some advantages, as well as some disadvantages. Just the same as how new
syntax has both advantages and disadvantages. It's ok if the conclusion is not
to have a shopt, but the pros and cons of each approach should be properly
understood and weighed.
> > - It's harder to implement, and I would expect a higher maintenance burden.
>
> It's much easier than implementing an entire shell from scratch.
I don't understand the comparison to an entire shell. What I meant was that
guarding some existing functionality behind some user-selectable flags was
quite easy to do, with minimal changes, particularly as someone not familiar
with the bash codebase - whereas adding new syntax would almost certainly
involve more significant and/or invasive changes across the codebase.
But I am aware that just because it was easy to do, does not necessarily mean
that it is the best approach, and I have said as much from the start (and it's
why I am _asking_ for this to be considered for inclusion). My goal is not to
add burdens for the maintainers, but rather to help others who might be
affected by the same limitations, while minimizing any additional maintenance
burden. I don't know what those main burdens are, but aiming for a light touch
is usually better than suggesting large and invasive changes.
> The cost is negligible as long as it is properly implemented.
Here is a hypothetical example of a burden that could arise from doing this as
new syntax.
Suppose that this new syntax for trailing newline preserving command
substitutions had been added prior to bash 5.3, when there was only `` and $()
to consider. Further suppose that the syntax was to start the command
substitution with an otherwise invalid character, such as &, so that `&foo` and
$(&foo) are the trailing newline preserving versions of `foo` and $(foo).
Now, when it comes to adding support for the ksh-style no-fork command
substitution ${ foo;}, it's necessary to also add a ${&foo;} version, for the
trailing newline preserving no-fork command substitution. And, if the initial
syntax had chosen | instead of &, then this situation would be even more
complex to navigate, since ${|foo;} can only be used for one or the other of
the trailing newline preserving version of ${ foo;}, or the $REPLY-style
no-fork command substitution, but not both.
By contrast, if the trailing-newline-preservation had been implemented via a
shopt, then this problem would not exist. Nothing special was needed in the
cmdsubst_strip_newlines patch to handle the ${ foo;} or $(<foo) cases - they
both received the selectable behavior automatically alongside $(foo) (because
they all use read_comsub()). If later another new type of command substitution
were to be added, then I would expect it to also automatically get trailing
newline selection via cmdsubst_strip_newlines. This isn't necessarily the case
with syntax-based approaches.
> What you propose is actually the opposite. With the shell options to
> change existing behaviors, existing codes expecting the traditional
> behavior may produce unexpected results *without causing an explicit
> error*, when they are combined with a code that changes the option
> poorly.
As above, the same is true when other behavior-modifying shell options are
introduced.
> > - Modifying existing code which uses traditional trailing newline behavior,
> > to instead use the alternate trailing newline behavior, requires
> > switching to the new syntax throughout the scripts, rather than choosing
> > the desired behavior once at the start of the script.
>
> This doesn't seem to make sense. When you want to switch to use the
> new behavior, even if you decided to change the existing behavior of
> command substitutions instead of adding a new syntax, you still need
> to rewrite the entire scripts to remove the extra handling of $? and
> ${result%.} in the existing scripts, or you need to scan through the
> entire shell scripts to see whether the behavioral change do not
> affect the behavior of the existing codes.
Sorry, I wasn't clear with this. What I mean is, suppose you have some scripts
which use the normal historical command substitution (and here strings) in many
places. They are just regular large scripts, which don't use libraries or
frameworks, and they seem to work fine - until one day you discover that the
trailing newline stripping is causing incorrect behavior. You realize that you
hadn't considered this case, and that preventing trailing newline stripping
(and newline appending for here strings) would fix the problem.
If bash has alternate command substitution syntax for these, then you need to
go through the scripts and update every command substitution and here string to
use the alternate syntax. (This is similar to the current situation, where you
have to add the extra handling for the trailing ., $?, and ${result%.}.)
By contrast, with user-selectable shell options to control the trailing newline
handling, instead of updating the syntax at many locations, you can instead
just set the shopts once at the start of each script.
Please note that I am not saying "therefore the shopt approach should be used".
I am just saying that this is an advantage of this approach, in this
circumstance, which should be weighed against the alternatives.
> > Unfortunately "if one accepts a different way" is exactly saying "if one
> > accepts non-idiomatic code"
>
> Do you mean that a new syntax is also non-idiomatic? Do you think the
> only idiomatic code is to change the existing behavior of the command
> substitutions with exactly the same syntax? If so, I cannot agree.
No, I agree with you that after the introduction of a new syntax, it would then
become the idiomatic of doing whatever it does. If bash had a syntax for doing
trailing newline preserving command substitution, then I would be happy to use
it. What I mean is, in the absence of such a natural built-in syntax for
non-stripping command substitution, the only "different ways" currently
available either do not use command substitution, or use it in a convoluted
way.
> > For example, while it's possible for me to write a shell function that
> > wraps builtins/commands and "adjusts" their behavior, the inability to use
> > shell functions (or anything else) to modify the behavior of shell syntax
> > constructs like $(), is precisely why I looked into adding these shopts in
> > the first place.
>
> This is tautological. The above seems to attempt to explain the reason
> why the option to change the existing behavior of shell constructs
> like $() is introduced instead of using a shell function. It says the
> reason is that the existing constructs like $() cannot be modified by
> using shell functions. More than half of the arguments seem to be
> nonsensical. Do you use LLM?
I don't think you've understood what I was trying to say.
Suppose that in my scripts, I want read to always use -r, unless I override
that by specifying +r. I can achieve this by defining something like:
function read {
if [[ "${BASH_SOURCE[1]}" == /my/scripts/* ]]; then
# FIXME: handle combined args properly, eg. read -ers
local include_r=y
local -a args
while [ $# -gt 0 ]; do
case "$1" in
-r) include_r=y ;;
+r) include_r=n ;;
*) args+=("$1") ;;
esac
shift
done
if [ $include_r = y ]; then
args=(-r "${args[@]}")
fi
set -- "${args[@]}"
fi
command read "$@"
}
This is what I meant by "it's possible for me to write a shell function that
wraps builtins/commands and "adjusts" their behavior". By defining this
function, I have specified what bash should do when it sees a "read" command.
By contrast, it is not possible for me to specify what bash should do when it
sees $().
Suppose bash had a feature where, if a shell function called _comsub is
defined, then that function gets called whenever `` or $() is encountered. The
function would receive a single argument which is the string contained inside
the command substitution, and the result would be whatever is in $REPLY after
the function ends. This feature would allow me to get the command substitution
behavior I want by doing something like:
function _comsub {
read -rd '' < <(eval -- "$1") || [[ $REPLY ]]
}
This would still break any external code that assumes traditional behavior, but
if that's a concern I could add some sophistication to make it only apply to my
own scripts:
function _comsub_define {
function _comsub {
if [[ "${BASH_SOURCE[1]}" == /my/scripts/* ]]; then
read -rd '' < <(eval -- "$1") || [[ $REPLY ]]
else
unset -f _comsub
REPLY=$(eval -- "$1")
local rc=$?
_comsub_define
return $rc
fi
}
}
_comsub_define
What I am saying is that, because no such feature exists, I (as a user) have no
way to control what $() does or doesn't do. And that is why I tried adding the
shopt to give users some small control over what $() does.
If you are aware of any way that I can adjust the behavior of command
substitution to not strip trailing newlines, while still having the command
substitution look like a normal $(foo) command substitution (or even $(foo |
bar | special) or $(special | bar | foo)), I would be happy to hear it. But my
understanding is that this is currently impossible in bash.
Kev
