Re: Unclosed quotes on heredoc mode

2021-11-27 Thread Chet Ramey

On 11/24/21 10:40 AM, Robert Elz wrote:

 Date:Tue, 23 Nov 2021 11:09:51 -0500
 From:Chet Ramey 
 Message-ID:  <3a5f6f3a-aa73-d8ac-46f4-46467d5b3...@case.edu>

   | > I'll run our tests against the newest (released) bash
   |
   | OK. However, since, as I said, the devel branch has a completely different
   | implementation, this is not particularly useful.

OK, then I won't bother ... running the tests is easy (about 5 mins
elapsed time, after about 1 second setup) but analysing the results to
separate out the real bugs from the places where bash is just different
from our shell, and neither is right or wrong (our tests are testing to
make sure we don't change things by accident) takes longer.


OK, if you do end up building the devel branch, I'd be interested in these
results.



   | It's the build version: how many times have you built in this build tree?

Once, of course ... why would I ever build it again?


Patches exist. There are vendors who take the original release, apply their
own special-sauce patches, then apply the patches I release as they come
out, as part of their own distribution release process.



   | Whatever. You do you. Don't be surprised if many of my answers turn out to
   | be "that's already fixed in the devel branch."

First, thanks to those (several) people who indicated how I could
fetch the devel code, I might look at that sometime, but in general I
prefer to wait for the released versions (and then for those to get
included in NetBSD's pkgsrc, which usually happens quite quickly).


Usually, that's ok. In this instance, where we're discussing a feature
whose implementation is substantially different between the released and
development versions, it's more relevant.



   | Refer to my previous message about the reading-full-lines strategy.

I have no problem with reading full lines, but whenever a "full line"
includes a newline token, any pending here docs should be read. 


So the ultimate question is whether or not the act of reading a command
substitution should reset this requirement. That's where we disagree.
The grammar is, at that point, reading a different command.



   | The devel branch produces
   |
   | TRACE: pid 78934: parse_comsub: need_here_doc = 1 after yyparse()?
   | cat: abc: No such file or directory
   | cat: def: No such file or directory

That looks much better.


Like I said, it's a conscious choice that is still fluid.


   | We talked about this. The command substitution starts a new parsing context
   | to implement the "any valid shell script" part of the standard.

Then we get to whether heredoc data is part of a valid shell script
in that sense - when there is yet to be a newline token to introduce it.


What does this mean? In all cases, the here-documents are not read until
after a newline token. That's not the issue.



This is where we started this, the question of which newline is the one
after which heredoc data starts.   It isn't at all as clear as you make
it appear to be.


Apparently not.



   | The netbsd shell appears to be the outlier here. The parser reads the
   | command substitution so it can parse the entire and-or list before trying
   | to gather any here-documents.

You cannot possibly really mean that I hope.   That is, in

cmd1 <

There is no command substitution in this example.


Once again, heredoc gathering has nothing at all to do with the grammar,
and so obviously not the parser either, beyond it informing the lexer which
heredocs are pending.


So, again, the question is whether or not input data that is logically
part of the command substitution (it appears between the opening and
closing parentheses) should affect the `outer' command. That's the
question. We have different answers.





   | The fundamental point of disagreement is what to do if the lexer (after,
   | presumably, calling the parser recursively) finds that it still has here-
   | documents to read after reading the end of the command substitution.


It was. Now we have moved away from that and added the `should text in the
command substitution satisfy here-documents outside it?'

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Greek messages output to iOS ssh clients

2021-11-25 Thread Chet Ramey

On 11/24/21 11:55 AM, jer...@jrandall.duckdns.org wrote:


Bash Version: 5.1
Patch Level: 8
Release Status: release

Description:
Upon ssh to bash 5.1.8 from a mobile device running iOS 15.1, error and 
status messages are echoed by bash to the screen in the Greek alphabet and 
language.


The iOS client is sending the locale information to the macOS system
through the environment. You can see what the locale is on macOS by running
`locale'. You'll have to check through the settings on your ssh client to
see how to get it to send your desired locale.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Unclosed quotes on heredoc mode

2021-11-23 Thread Chet Ramey
s
returning to it, the grammar just increments the "number of heredocs needed"
counter, supplies the end words for each, and the lexer takes care of the rest.


The fundamental point of disagreement is what to do if the lexer (after,
presumably, calling the parser recursively) finds that it still has here-
documents to read after reading the end of the command substitution.

What happens in the command substitution stays in the command substitution.
If you subscribe to that, you need to specify your here-documents inside
the command substitution.



And then there is of course the combination of the two of those examples:

cat <

Which has the same fundamental disagreement.



I'll stop it there, probably what follows is ')' on the same
line, but whatever happens next (assumed syntactically corrrect),
if your requirement is that END precedes EOF in what follows you're
clearly wrong, as POSIX is quite clear that the order in which the
heredocs are to be read is left to right across the line (regardless
of which commands they're attached to), so the EOF ending one *must*
appear first, and the END ending one second.   And the two of them
follow one newline token.


The logical conclusion of this line of thinking is that a `done' in a
command substitution can terminate a `for' loop that starts outside it.
Either you reset the parsing state, including the "hey I need a here-
document now," or you don't (or you do some partial half-assed job of it
that doesn't help anyone).

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Crash when using ulimit -n

2021-11-22 Thread Chet Ramey
On 11/21/21 8:34 PM, balasr wrote:

> Bash Version: 5.1
> Patch Level: 8
> Release Status: release
> 
> Description:
>     Calling ulimit -n 1024 results in the message "Warning: Program 
> '/bin/bash' crashed." and bash locks up.
> 
> Repeat-By:
>     1. Open bash
>     2. Enter `ulimit -n 1024'

Sorry, I can't reproduce this.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
     ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Strange complete -C output

2021-11-21 Thread Chet Ramey

On 11/20/21 10:40 PM, Emily Seville wrote:

Bash version: GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)
OS version: Linux Mint 20.2 Cinnamon

Hello! Why when I create *complete -C 'echo -e "aa\nab\n"' : *completion 
and try use it I obtain additional garbage in output instead of only *ab 
ab* hints?


"When the function or command is invoked, the
first argument ($1) is the name of the command whose arguments are  be-
ing  completed,  the  second argument ($2) is the word being completed,
and the third argument ($3) is the word preceding the word  being  com-
pleted on the current command line. "

So your echo command is echoing the arguments it receives when invoked by
the completion system.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Unclosed quotes on heredoc mode

2021-11-20 Thread Chet Ramey

On 11/20/21 12:35 PM, Robert Elz wrote:

 Date:Sat, 20 Nov 2021 11:33:37 -0500
 From:Chet Ramey 
 Message-ID:  <4addb789-50b6-12a5-7b8a-8a082abaa...@case.edu>

   | I'm skeptical, but willing to be convinced. Bourne's shell allowed EOF to
   | terminate all sorts of things (quoted strings, command substitutions, here
   | documents) -- enough to make it purposeful.

More likely economical.   Making things fit in that sh was a real challenge.


Right. Purposeful.


That's a good starting point, provided you're willing to actually implement
that.  That's what I'd like. 


How about this. You show me examples where bash (devel bash) does what you
think is the wrong thing, and we agree it's a bug, I'll fix it.


 But for this you need to understand that
the shell has to parse and understand command substitutions, as they're read,
in order to correctly find the end,


The devel bash already does this. We've talked about it before. You need to
use bison, not byacc, and a new enough version of bison, but it works fine.


and a newline token in the middle of
a command substitution counts for a here doc operator that occurred before
it, 


What does `counts' mean? You're not really reading the lines as shell
words, so a command substitution isn't really a command substitution while
you're reading the body of a here-document. You mean something like this?

cat << EOF
echo $(echo this EOF is
not the end of
the command substitution
EOF
but it is the end of the
here-document
)



and a here doc operator in a command substitution might not encounter
a newline until after the cmdsub text has ended - the next following newline
token provides there here doc text.


I can't imagine a useful example of this that isn't an error.

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Unclosed quotes on heredoc mode

2021-11-20 Thread Chet Ramey

On 11/19/21 9:18 AM, Robert Elz wrote:


illusory compat issues.  I have no idea what inspired this initially, but
my guess would be a code bug no-one noticed.


I'm skeptical, but willing to be convinced. Bourne's shell allowed EOF to
terminate all sorts of things (quoted strings, command substitutions, here
documents) -- enough to make it purposeful.



So just how many complaints do you get about the warning message?
"ksh doesn't complain wbout this, why does bash?"


It's usually people who have misplaced or mistyped the ending delimiter.
It took only a few seconds to find this:

https://unix.stackexchange.com/questions/657488/warning-here-document-at-line-2-delimited-by-end-of-file-wanted-eof

I don't have time right now to look for other reports that might have
tested it against other shells.



   | Which instance of `ola"'? The first or the second?

The first.

   | This cannot be a serious question unless you mean the second.

It is a very serious question, but not as to what should hppen
but how the standard needs to describe it.


That's why I suggested what I did.

Some variant of the existing

"When an io_here token has been recognized by the grammar (see Shell
Grammar), one or more of the subsequent lines immediately following the
next NEWLINE token form the body of one or more here-documents and shall be
parsed according to the rules of Here-Document."

could probably work as a basis. That implies that the shell goes off and
reads lines before parsing the rest of the current line as a list.



   | The delimiter is a `word', and we both know what a shell word is.

yes, but that's irrelevant, it is merely a coincidence here that
the newline in question occurs in the delimiter.
Another example
cat < file ; echo "abc
def
EOF
ghi" \
EOF
EOF
What is the here doc, and what does echo say.


That's a good example. The here-doc is empty (the delimiter is the third
EOF) and the echo prints the rest of the text, with the backslash-newline
disappearing.

I'd say that this is somewhat deceptive, and is a decent illustration of my
point. The shell -- bash, at least -- always reads complete lines from the
input before parsing any here documents, so it's going to keep reading
through the second EOF to read the `complete' first line, due to the quoted
string and the quoted newline. The `current' token is going to be the
newline that follows the second EOF even before it starts figuring out that
it has a here-document and goes off to collect the body.

So, the shell reads the here-document body and creates the here document
after it reads an unquoted newline token -- the first newline token after
finding the here-document delimiter.



The first newline after the << is the one after abc.
Do remember that here doc data collection is entirely a
lexical issue, that's why tgey dot appear anywhere in
the sh grammar.


Oh, I do.



   | The newline after the delimiter is both, but sure, newline token would
   | probably work better.

The example above shows the issue better.  That includes the \newline
which can only be a \ newline because the 2nd char there is a newline,
and that has to be seen at the lexical level.


Yes. Here-documents are one of those features that requires mutual feedback
between the parser and lexer.



   | So it doesn't read `lines' in the POSIX sense? Huh. Who knew?

For this.  No.   An extension.  One that comes for feee.


I like the Freudian slip there.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
     ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-11-19 Thread Chet Ramey

On 10/12/21 3:38 PM, Julien Moutinho wrote:

Le mar. 05 oct. 2021 16h12 -0400, Chet Ramey a écrit :

On 10/5/21 1:50 PM, Dominique Martinet wrote:

If I change malloc_usable_size to return p->mh_nbytes instead of
maxbytes, then the crash disappears.[2]


That's the right fix.


Chet, when you'll have time, would you mind publishing such fix at:
https://ftp.gnu.org/gnu/bash/bash-5.1-patches/
Because then I could try to get it merged into Nixpkgs before
the next biannual release of NixOS next November.


This fix is in the most recent set of patches I released this week
(it's patch 9).

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: unbalanced parenthesis not recognized

2021-11-19 Thread Chet Ramey

On 11/19/21 2:02 AM, Harald Dunkel wrote:



"Some scenarios" is the point here. The parenthesis have to balance as
soon as it comes to shell parameter expansion, which is (or should have
been) the case here. 


OK. Let's look at the original example:

: ${SSLDIR}:="${JM_WORK}/ssl"}

Where do you think parameter expansion applies here, as far as the final
`}' in the string, and why should it "have been the case?" There are no
incomplete parameter expansions in this command; why should the shell
assume that a close brace appearing by itself should be matched somewhere?


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Unclosed quotes on heredoc mode

2021-11-18 Thread Chet Ramey
On 11/17/21 7:01 PM, Robert Elz wrote:
> Date:Wed, 17 Nov 2021 15:47:37 -0500
> From:    Chet Ramey 
> Message-ID:  <420281e7-f3c4-8054-d390-9378080c2...@case.edu>
> 
>   | Every modern shell uses `$PATH' as the here-document delimiter
> 
> Depends what you call modern shells - some ash derived shells (at least)
> don't, because they parse the $PATH into an internal form (in all words
> where that makes sense, before knowing what the word is to be used for)
> and then cannot match that properly.   While that isn't actually expanding
> the word, it still makes things fail badly.

Yeah, that's a bug. But it's probably baked in.

> But:
> 
> [D] sh-current $ cat foo <<$PATH
> sh: 80: Syntax error: Illegal eof marker for << redirection
> 
> at least we error out when the user tries, not just fail to ever
> find the end of the here doc.

OK, that's clearly a bug. Is this specific to the literal string `$PATH',
or are there more things that trigger it? The error message uses some
sloppy language, but that's neither here nor there -- this is a perfectly
valid script:

cat <<$PATH
hello
$PATH

that should echo `hello'.


>   | > First, the EOF should not work, that's a bash bug (IMO) - that should
>   | > generate an error, not just a warning.
>   |
>   | It's not. The historical shells used for the basis of the POSIX standard
> 
> I didn't say it was a standards violation, I said it was a bug.
> That the same bug exists in some other ancient shells isn't a justification.

"Some other ancient shells?" Like dash, or (the current and actively-
developed) ksh93, or the FreeBSD sh, or zsh? The ones I listed in the part
of my message you chopped? You're certainly free to consider it a bug, and
not to consider the compatibility concerns that inspired its inclusion in
bash in the first place, but let's not pretend that this is something that
died out a long time ago.

> 
> Further, no-one (not anyone I
> have ever seen) deliberately relies upon the here doc ending at EOF, not
> even if a here doc is in a -c command string or similar).

You never really know, do you?

>   | > OK, here we have another of the oddities of shell syntax.   The spec
>   | > says that a here document starts at the next newline after the << 
> operator,
>   | > but that's not what it really means. 
>   |
>   | I think the intent there is that the here document starts at the next
>   | newline after the delimiter.
> 
> You mean at the newline after the ola" in the example given?   Really?

Which instance of `ola"'? The first or the second? This cannot be a serious
question unless you mean the second. The delimiter is a `word', and we both
know what a shell word is. In other messages from both of us, we agree that
the delimiter is "ola\nI,\nola\nola". The here document body starts at the
next newline following that delimiter. If you want to reject it because the
delimiter contains a newline, that's fine, but let's also not pretend we
don't know what the delimiter is.

> Surely it must mean newline token, not newline character, mustn't it?

The newline after the delimiter is both, but sure, newline token would
probably work better.

> (Even then, there are more, messier, issues, which I know you're aware of;
> if we could make it as simple as "after the lexically next newline token"
> it would make everything much simpler - that's what it should be.)
> 
>   | > Being able to do that (include embedded newline characters
>   | > do in some other shells).
>   |
>   | I couldn't fine one where it does.
> 
> They work in (at least) the NetBSD shell, FreeBSD too I expect, since the
> two use essentially the same mechanism for recognising the end of the
> here doc -- (effectively) after a newline, read chars (from a buffer) one
> at a time, comparing them with the end delimiter, until either there is a
> match failure, or until the end of the end delimiter (after which one more
> char from the buffer is compared to \n).   (Add tab stripping as required).

So it doesn't read `lines' in the POSIX sense? Huh. Who knew?

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Bash-5.1 Official Patch 12

2021-11-17 Thread Chet Ramey
rap.c 2020-11-28 12:04:07.0 -0500
--- trap.c  2020-11-28 10:22:10.0 -0500
***
*** 482,485 
--- 482,511 
  }
  
+   /* This means we're in a subshell, but have not yet reset the handler for
+  trapped signals. We're not supposed to execute the trap in this 
situation;
+  we should restore the original signal and resend the signal to ourselves
+  to preserve the Posix "signal traps that are not being ignored shall be
+  set to the default action" semantics. */
+   if ((subshell_environment & SUBSHELL_IGNTRAP) && trap_list[sig] != (char 
*)IGNORE_SIG)
+ {
+   sigset_t mask;
+ 
+   /* Paranoia */
+   if (original_signals[sig] == IMPOSSIBLE_TRAP_HANDLER)
+   original_signals[sig] = SIG_DFL;
+ 
+   restore_signal (sig);
+ 
+   /* Make sure we let the signal we just caught through */
+   sigemptyset ();
+   sigprocmask (SIG_SETMASK, (sigset_t *)NULL, );
+   sigdelset (, sig);
+   sigprocmask (SIG_SETMASK, , (sigset_t *)NULL);
+ 
+   kill (getpid (), sig);
+ 
+   SIGRETURN (0);
+ }
+ 
if ((sig >= NSIG) ||
(trap_list[sig] == (char *)DEFAULT_SIG) ||

*** ../bash-5.1/patchlevel.h2020-06-22 14:51:03.0 -0400
--- patchlevel.h2020-10-01 11:01:28.0 -0400
***
*** 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 11
  
  #endif /* _PATCHLEVEL_H_ */
--- 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 12
  
  #endif /* _PATCHLEVEL_H_ */

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
     ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Bash-5.1 Official Patch 11

2021-11-17 Thread Chet Ramey
 BASH PATCH REPORT
 =

Bash-Release:   5.1
Patch-ID:   bash51-011

Bug-Reported-by:Alex fxmbsw7 Ratchev 
Bug-Reference-ID:   

Bug-Reference-URL:  
https://lists.gnu.org/archive/html/bug-bash/2020-11/msg00064.html

Bug-Description:

When reading a compound assignment, and running it through the parser to
split it into words, we need to save and restore any alias we're currently
expanding.

Patch (apply with `patch -p0'):

*** ../bash-5.1-patched/parse.y 2020-11-28 12:10:06.0 -0500
--- parse.y 2021-10-13 11:04:27.0 -0400
***
*** 6494,6501 
  
push_stream (1);
- #if 0 /* TAG: bash-5.2 Alex fxmbsw7 Ratchev  11/17/2020 */
if (ea = expanding_alias ())
  parser_save_alias ();
- #endif
last_read_token = WORD; /* WORD to allow reserved words here */
current_command_line_count = 0;
--- 6494,6499 
***
*** 6532,6539 
pop_stream ();
  
- #if 0 /* TAG: bash-5.2 */
if (ea)
  parser_restore_alias ();
- #endif
  
  #if defined (HISTORY)
--- 6530,6535 
*** ../bash-5.1-patched/y.tab.c 2020-11-28 12:17:19.0 -0500
--- y.tab.c 2021-11-17 10:47:35.0 -0500
***
*** 8788,8795 
  
push_stream (1);
- #if 0 /* TAG: bash-5.2 Alex fxmbsw7 Ratchev  11/17/2020 */
if (ea = expanding_alias ())
  parser_save_alias ();
- #endif
last_read_token = WORD; /* WORD to allow reserved words here */
current_command_line_count = 0;
--- 8777,8782 
***
*** 8826,8833 
pop_stream ();
  
- #if 0 /* TAG: bash-5.2 */
if (ea)
  parser_restore_alias ();
- #endif
  
  #if defined (HISTORY)
--- 8813,8818 
*** ../bash-5.1/patchlevel.h2020-06-22 14:51:03.0 -0400
--- patchlevel.h2020-10-01 11:01:28.0 -0400
***
*** 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 10
  
  #endif /* _PATCHLEVEL_H_ */
--- 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 11
  
  #endif /* _PATCHLEVEL_H_ */

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Bash-5.1 Official Patch 9

2021-11-17 Thread Chet Ramey
 BASH PATCH REPORT
 =

Bash-Release:   5.1
Patch-ID:   bash51-009

Bug-Reported-by:Julien Moutinho 
Bug-Reference-ID:   <20211004035906.5kiobuzkpeckm...@sourcephile.fr>
Bug-Reference-URL:  
https://lists.gnu.org/archive/html/bug-bash/2021-10/msg00022.html

Bug-Description:

The bash malloc implementation of malloc_usable_size() does not follow the
specification. This can cause library functions that use it to overwrite
memory bounds checking.

Patch (apply with `patch -p0'):

*** ../bash-5.1-patched/lib/malloc/malloc.c 2020-07-08 10:19:30.0 
-0400
--- lib/malloc/malloc.c 2021-10-05 16:10:55.0 -0400
***
*** 1287,1297 
  }
  
!   /* XXX - should we return 0 if ISFREE? */
!   maxbytes = binsize(p->mh_index);
! 
!   /* So the usable size is the maximum number of bytes in the bin less the
!  malloc overhead */
!   maxbytes -= MOVERHEAD + MSLOP;
!   return (maxbytes);
  }
  
--- 1358,1367 
  }
  
!   /* return 0 if ISFREE */
!   if (p->mh_alloc == ISFREE)
! return 0;
!   
!   /* Since we use bounds checking, the usable size is the last requested 
size. */
!   return (p->mh_nbytes);
  }
  
*** ../bash-5.1/patchlevel.h2020-06-22 14:51:03.0 -0400
--- patchlevel.h2020-10-01 11:01:28.0 -0400
***
*** 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 8
  
  #endif /* _PATCHLEVEL_H_ */
--- 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 9
  
  #endif /* _PATCHLEVEL_H_ */

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Bash-5.1 Official Patch 10

2021-11-17 Thread Chet Ramey
 BASH PATCH REPORT
 =

Bash-Release:   5.1
Patch-ID:   bash51-010

Bug-Reported-by:Jonas Alfredsson 
Bug-Reference-ID:   

Bug-Reference-URL:  
https://lists.gnu.org/archive/html/bug-bash/2021-05/msg00059.html

Bug-Description:

If `wait -n' is interrupted by a trapped signal other than SIGINT, it does
not completely clean up state, and that can prevent subsequent calls to
`wait -n' from working correctly.

Patch (apply with `patch -p0'):

*** ../bash-5.1-patched/builtins/wait.def   2020-12-16 17:13:12.0 
-0500
--- builtins/wait.def   2021-11-17 10:25:15.0 -0500
***
*** 112,116 
   WORD_LIST *list;
  {
!   int status, code, opt, nflag, wflags;
char *vname;
SHELL_VAR *pidvar;
--- 112,117 
   WORD_LIST *list;
  {
!   int status, code, opt, nflag;
!   volatile int wflags;
char *vname;
SHELL_VAR *pidvar;
***
*** 181,184 
--- 188,193 
status = 128 + wait_signal_received;
wait_sigint_cleanup ();
+   if (wflags & JWAIT_WAITING)
+   unset_waitlist ();
WAIT_RETURN (status);
  }

*** ../bash-5.1/patchlevel.h2020-06-22 14:51:03.0 -0400
--- patchlevel.h2020-10-01 11:01:28.0 -0400
***
*** 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 9
  
  #endif /* _PATCHLEVEL_H_ */
--- 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 10
  
  #endif /* _PATCHLEVEL_H_ */

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Unclosed quotes on heredoc mode

2021-11-17 Thread Chet Ramey
On 11/17/21 3:02 PM, Robert Elz wrote:

>   | bash-5.1$ cat << $PATH
> 
> 
>   | it should have terminated with the upper delimiter!
> 
> What do you consider the "upper delimiter" ?
> 
> This is one of the weirder aspects of shell syntax, and perhaps one
> of bash's oddities.

It's not. Every modern shell uses `$PATH' as the here-document delimiter
and checks for the delimiter before any part of the process that expands
the lines in the here-document body.

> First, the EOF should not work, that's a bash bug (IMO) - that should
> generate an error, not just a warning.

It's not. The historical shells used for the basis of the POSIX standard
(ksh88 and the SVR4 sh) silently allow EOF to terminate a here-document;
ksh93 preserves this behavior. Some of the common shells allow this as
well (e.g., dash, zsh and the version of the FreeBSD from a couple of years
ago when I last built it), some do not (e.g., mksh and the netbsd sh). Bash
at least warns you about it.

> 
>  Example:
>   |
>   | bash-5.1$
>   | bash-5.1$ cat << ola"
> 
> OK, here we have another of the oddities of shell syntax.   The spec
> says that a here document starts at the next newline after the << operator,
> but that's not what it really means. 

I think the intent there is that the here document starts at the next
newline after the delimiter.


> Being able to do that (include embedded newline characters
> in a "line") isn't required by the shell specification, and (it has been
> a while since I checked) I do not believe that those work in bash (they
> do in some other shells).

I couldn't fine one where it does.


> Since bash doesn't allow end delimiter words that contain newlines to
> work, it should probably generate an error when you try to use one, that
> would have made things clear.

See above.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Unclosed quotes on heredoc mode

2021-11-17 Thread Chet Ramey
On 11/17/21 10:33 AM, Robert Elz wrote:

> There are several (IMO)
> bugs in the way bash processes here documents, 

Such as?


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Unclosed quotes on heredoc mode

2021-11-17 Thread Chet Ramey
On 11/17/21 1:45 PM, João Almeida Santos wrote:
> No, it’s on the email...Anyway, here’s the text!
> 
> bash-5.1$ echo $PATH
> /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/usr/local/bin/:/usr/local/bin/
> 
> bash-5.1$ cat << $PATH
>> /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/usr/local/bin/:/usr/local/bin/
>> it should have terminated with the upper delimiter! but, bash does not seem 
>> to expand PATH.
>> $PATH
> /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/usr/local/bin/:/usr/local/bin/
> it should have terminated with the upper delimiter! but, bash does not seem 
> to expand PATH.

The here document delimiter does not undergo any expansions except quote
removal, so the delimiter is the literal string `$PATH'. The lines of the
here-document undergo a different set of expansions, which happen after the
check for the delimiter is performed, which means that you need to have a
line that consists solely of `$PATH' to terminate the here-document (as you
discovered). I cannot see how you're going to be able to do anything useful
with this construct; it just seems too clever by (more than) half.

This is all in the bash documentation.


> ok, but now addressing the actual question. If I use unclosed quotes on 
> heredoc, I can't use 
> the given delimiter to end the heredoc, I end up having to use an EOF. 
> Example:
> 
> bash-5.1$
> bash-5.1$ cat « ola"
>> I,
>> ola""
>> ola"
>> ola

The delimiter is not what you think it is. The delimiter for a here-
document is a shell word (which can include quoted substrings), and after
it undergoes the appropriate quote removal, your delimiter is
"ola\nI,\nola\nola" (using C string notation).

Now, you're never going to be able to match this; it contains a newline.
When the shell constructs the here-document body, it reads individual lines
from the input source and, after removing the trailing newline, tries to
match them against the delimiter (and backslash doesn't work to quote the
newline). This will obviously never match a delimiter containing a newline.

Some shells (e.g., yash) choose to make this a syntax error. Bash does not.

> In the above example, I don't unterstand how to provide the wanted delimiter!

You simply cannot, not the way you specify it. If you really want to have
the double quote as part of the here-document delimiter, write it as

cat << ola\"

I can't imagine this being useful, either.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: bash conditional expressions

2021-11-17 Thread Chet Ramey
On 11/17/21 5:16 AM, Michael J. Baars wrote:

>> Why do you think `touch -am', which sets the atime and mtime to the same
>> value, should make -N true?
> 
> When -N stands for NEW

It doesn't, though. It could just as easily be a mnemonic for "new activity
in the file." You're using it to mean `new' because it's convenient for
your application, or the mental model you have for it, but that's just an
invention.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: I've found a vulnerability in bash

2021-11-17 Thread Chet Ramey
On 11/17/21 4:16 AM, Marshall Whittaker wrote:

> This shouldn't happen beacuse you can drop a file and then redirect
> other code for example calling a script if you only have access to drop
> a file.  Say a cronjob was running every hour, and it did rm * on some
> folder, by expansion, you could expand it to -riv or whatever you
> wanted and redirect program flow from there.

That's just bad scripting.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: bash conditional expressions

2021-11-15 Thread Chet Ramey

On 11/12/21 12:16 PM, Lawrence Velázquez wrote:


As I understand it, -N stands for NEW and therefore should return a true
when either a 'touch -a test' or a 'touch -am test' is given.


FWIW, there's some disagreement on this.


Not very much.


 % cat foo_test
 test -N foo
 echo "$?"
 % touch foo
 % /bin/bash -c 'echo "$BASH_VERSION"; . ./foo_test'
 3.2.57(1)-release
 0


Yes, this is the substance of the bug report. That version of the bash
builtin `test', which is around 15 years old, did it wrong. It was
finally fixed in bash-5.1.



 % /opt/local/bin/bash -c 'echo "$BASH_VERSION"; . ./foo_test'
 5.1.8(1)-release
 1
 % ksh -c 'echo "${.sh.version}"; . ./foo_test'
 Version AJM 93u+ 2012-08-01
 1
 % yash -c 'echo "$YASH_VERSION"; . ./foo_test'
 2.51
 1
 % zsh -c 'echo "$ZSH_VERSION"; . ./foo_test'
 5.8
 0


So zsh is the outlier.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
     ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: bash conditional expressions

2021-11-15 Thread Chet Ramey

On 11/12/21 4:36 AM, Mischa Baars wrote:


Could you please restore the Fedora 32 behaviour? Someone must have read
the bash manual a little too precise, because now the statement only
returns true when a 'touch -a test' is given and not when a 'touch -am
test' is given.

As I understand it, -N stands for NEW and therefore should return a true
when either a 'touch -a test' or a 'touch -am test' is given.


Why do you think `touch -am', which sets the atime and mtime to the same
value, should make -N true?

If test -N is a strict test that mtime > atime, it is working correctly
and you have managed to defeat it by setting atime == mtimne.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Document -x and -vx give the same results

2021-11-15 Thread Chet Ramey

On 11/14/21 11:40 AM, 積丹尼 Dan Jacobson wrote:

Man page says:
-vPrint shell input lines as they are read.
-xPrint commands and their arguments as they are executed.
Perhaps mention that -x and -vx give the same results, often or always.


They don't.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: coproc does not work with a lower case variable name

2021-11-12 Thread Chet Ramey

On 11/11/21 11:09 PM, g...@as.lan wrote:


Bash Version: 5.1
Patch Level: 8
Release Status: release

Description:
coproc gives a syntax error if I try to use it with a lower case 
variable name and a compound command:
bash: syntax error near unexpected token `}'
bash: syntax error near unexpected token `('


Repeat-By:
coproc bc { bc; }
coproc bc (bc)


I can't reproduce this. What does `type bc' produce?


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Surprising interaction between edit-and-execute-command, the cd builtin, and the prompt

2021-11-06 Thread Chet Ramey

On 11/6/21 9:57 AM, Piotr P. Stefaniak wrote:

Hi,

Using bash 5.0.17 edit-and-execute-command I type in something like
cd /tmp
save and exit to execute. $PWD is properly updated but the prompt
doesn't reflect the change (not until the next time it is printed).


The answer is the same as to this question from back in March:

https://lists.gnu.org/archive/html/bug-bash/2021-03/msg00145.html

`edit-and-execute-command' is a bindable readline command. The prompt
passed to readline persists throughout the entire duration of a single
call to readline(). When you run edit-and-execute-command via a readline
key binding, you're still within the same call to readline(), and readline
takes care of redisplaying the prompt and continuing to read input when
that command completes. There is no point where readline() returns a line
to its caller, so there's no point where the prompt is re-evaluated.

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: BASH_COMMAND does not expand correctly in subshells inside traps.

2021-11-05 Thread Chet Ramey

On 11/4/21 5:49 PM, Emanuele Torre wrote:


Bash Version: 5.1
Patch Level: 8
Release Status: release

Description:
   BASH_COMMAND does not expand to the expected value when used in a
   subshell inside a trap.


This is a variant of

https://lists.gnu.org/archive/html/help-bash/2021-10/msg00269.html

In this case, as explained in the above message, the subshell `forgets'
that it's executing as part of the DEBUG trap. Since we're not in the
DEBUG trap, the value of BASH_COMMAND doesn't get updated. As a
consequence, since the expansion of the command happens in the subshell,
the original value (unexpanded in the parent shell) remains.

I think I might have to rethink this strategy. What do you think?

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Newlines in ERR trap affect caller 0 line number

2021-11-04 Thread Chet Ramey
On 11/4/21 11:55 AM, Alex fxmbsw7 Ratchev wrote:
> thats around the same as aliases, inline text replacements , yes ?
> informative question

No. It's a different processing stage.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Newlines in ERR trap affect caller 0 line number

2021-11-04 Thread Chet Ramey
On 10/31/21 6:06 PM, Quinn Grier wrote:

> Bash Version: 5.1
> Patch Level: 8
> Release Status: release
> 
> Description:
> When an ERR trap includes newlines, the line number returned by
> "caller 0" is affected.

Thanks for the report. I'll take a look. Right now, the ERR trap (really
all traps) is treated as if the text of the trap appears inline in the
source right after the command that triggered it, so parsing the text of
the trap command increments the line number starting at the line number
of the triggering command.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: bash support for XDG Base Directory spec (~/.config/bash/)

2021-11-03 Thread Chet Ramey
On 11/2/21 9:53 AM, Siteshwar Vashisht wrote:
> On Fri, May 7, 2021 at 11:48 AM Allison Karlitskaya <
> allison.karlitsk...@redhat.com> wrote:
> 
>> hello,
>>
>> Please consider these two patches for inclusion in bash to support
>> storing shell initialisation files (profile, bashrc) in a subdirectory
>> of ~/.config/ as most programs do these days.
>>
>> I'm happy to make changes to address any feedback.
>>
>> Even if you'd prefer not to apply the second patch, applying the first
>> patch is a nice cleanup and would make it easier for distributions
>> such as Fedora to apply the second patch for themselves.
>>
>> Thank you very much for your consideration,
>>
> 
> This issue is also discussed in Red Hat bug[1]. I will leave few notes
> about it here.

[...]

> Proposed patch doesn't fully implement this specification:

OK. So we have an incomplete patch for a novel piece of functionality (in
the sense that no other shell implements it). That's going to impose an
implementation and maintenance cost that is better spent elsewhere -- at
least to this point.

The OP assumes that the value of this functionality is self-evident. I'm
a little more skeptical. How does it help make things easier for users?
How does it aid portability? Is it demonstrably better than the current
scheme, which has been in place for many years and is settled and well-
understood?

I'm not seeing a large number of reports about this (before this, the
last report was in 2016).

If someone wants to keep their config files in ~/.config/bash, what is
wrong with using symlinks from ~/.bash_profile and ~/.bashrc to those
files?

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Missing Backspace command in the readline

2021-11-02 Thread Chet Ramey

On 11/2/21 1:26 PM, Budi wrote:

What is the Backspace code in the readline, i.e. .inputrc file (as the
source come with explanation Ctrl-Backspace ( \b ) but not Backspace)
?


Backspace is Control-H (^H, C-h). The default binding in emacs mode is
backward-delete-char:

$ bind -p | grep backward-delete-char
"\C-h": backward-delete-char

That binding usually comes in from the stty `erase' special character,
but it's the default in the source. The usual result is that both DEL
and BS are bound to backward-delete-char.

This is not a bash bug. The question is more appropriate for help-bash.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Assignment with colons *should* be tilde expanded in POSIX mode

2021-11-02 Thread Chet Ramey

On 11/1/21 6:37 PM, Anders Kaseorg wrote:

As you know, POSIX requires tilde expansion following an an unquoted colon in 
an assignment [1]. A bug was reported [2] against bash 5.1-alpha that the 
tildes in

 $ echo foo=~:~
 foo=~:~

should not be expanded in POSIX mode, because this is not an assignment. That 
was fixed in 5.1-beta. However, that fix also seems to have broken the actual 
assignment

 $ foo=~:~
 $ echo "$foo"
 /home/anders:~

where both tildes should be expanded in POSIX mode, because this is an 
assignment.


Thanks for the report. I'll take a look and fix whatever is wrong.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: hash not restored after running command -p

2021-11-01 Thread Chet Ramey
On 11/1/21 11:26 AM, Chet Ramey wrote:
> On 10/31/21 12:06 PM, Roger Morris wrote:
>> Thanks for the reply.  Though POSIX may allow this, still the last
>> line of the following example is rather unexpected behavior
>>
>> $
>> $ echo echo MY LOCAL tmp/date SCRIPT > tmp/date
>> $ chmod +x tmp/date
>> $
>> $ PATH=.:/bin
>> $ date
>> Sun 31 Oct 2021 11:59:07 AM EDT
>> $ hash -l
>> builtin hash -p /bin/date date
>> $ cd tmp ; date ; cd ..
>> MY LOCAL tmp/date SCRIPT
> 
> This seems weird and non-intuitive (why not use the hashed value?), but
> it's actually a special case that's handled explicitly in the code. If the
> PATH search that results in a value being put into the hash table finds `.'
> in the PATH before finding the full path, the hash table remembers that
> fact and -- as a special case -- checks whether `./name' is an executable
> file before returning the hashed value on subsequent lookups. It's strange,
> but everyone seems to do it.

Before we all pile on here, yes, I understand why we all do it. I still
think it's non-intuitive, but you have to behave as if the hash table
weren't there at all.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: hash not restored after running command -p

2021-11-01 Thread Chet Ramey
On 10/31/21 12:06 PM, Roger Morris wrote:
> Thanks for the reply.  Though POSIX may allow this, still the last
> line of the following example is rather unexpected behavior
> 
> $
> $ echo echo MY LOCAL tmp/date SCRIPT > tmp/date
> $ chmod +x tmp/date
> $
> $ PATH=.:/bin
> $ date
> Sun 31 Oct 2021 11:59:07 AM EDT
> $ hash -l
> builtin hash -p /bin/date date
> $ cd tmp ; date ; cd ..
> MY LOCAL tmp/date SCRIPT

This seems weird and non-intuitive (why not use the hashed value?), but
it's actually a special case that's handled explicitly in the code. If the
PATH search that results in a value being put into the hash table finds `.'
in the PATH before finding the full path, the hash table remembers that
fact and -- as a special case -- checks whether `./name' is an executable
file before returning the hashed value on subsequent lookups. It's strange,
but everyone seems to do it.

> $
> $ PATH=.:/bin
> $ command -p date
> Sun 31 Oct 2021 11:59:50 AM EDT
> $ hash -l
> builtin hash -p /bin/date date
> $ cd tmp ; date ; cd ..
> Sun 31 Oct 2021 12:00:03 PM EDT

Once you know the above, this makes sense: `command -p' uses a magic value
for PATH, and since that value doesn't include `.' the search doesn't find
`.'. This means the full path gets put into the hash table without the flag
that says `look at ./date first', and the hashed value gets used.

The behavior would, of course, differ if `command -p' didn't do anything
with the hash table.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: hash not restored after running command -p

2021-11-01 Thread Chet Ramey
On 11/1/21 4:42 AM, Robert Elz wrote:
> I agree, this looks to be broken in bash - "command -p cmd" is (logically)
> 
>   oldpath=$PATH
>   PATH=/standard:/system:/path
>   cmd
>   PATH=$oldpath
> 
> and should act (as if) that way.

Clearly that's not the case. None of the side effects from assigning to
PATH occur.

What the OP wants here is for `command -p' not to

1. use the hash table

2. modify the hash table

There's nothing in the POSIX description of `command' or `hash' that
implies this. Both end up referencing

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01

which says, in part,

"Once a utility has been searched for and found (either as a result of this
specific search or as part of an unspecified shell start-up activity), an
implementation may remember its location and need not search for the
utility again unless the PATH variable has been the subject of an
assignment."

which seems at best inconclusive. I can see not using the hash table
(otherwise why use `command -p' at all?), but I don't see any support
for 2.

> 
> which is how it should be - the hash table is intended to speed PATH
> searches for commonly used commands, nothing should be found there
> which wouldn't be found from a PATH search - 

This isn't true in bash, either: `hash -p' exists.


> This is also somewhat inconsistent in bash, if one does, with the
> same setup as above, but without having done the "command -p ls",
> then:
>   PATH=/bin ls
> 
> the ls command works, "ls" is found by the PATH search, but is not added
> to the hash table (which is correct).   A subsequent "ls" correctly
> says:
>   bash: ls: command not found
> 
> If it is to be considered correct to add the entry in the command -p
> case, why would it not be correct in this case as well? 

Bash actually used to implement `command -p' that way
(`command -p name stuff' was implemented as the equivalent of
`PATH=magicpath name stuff'), but it's not what you want: if you do it like
that, `name' ends up inheriting the magic value of PATH and I got bug
reports about that.

So `command -p' has to be magic: it uses a magic internal version of $PATH
for lookups, which may come from a call to getconf(3) but doesn't have to,
but otherwise behaves exactly as if PATH were set to that value for the
purpose of POSIX's command search and execution.

You can argue for a change in behavior, which I suppose is what's happening
here, but there's little support for it in the standard.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: hash not restored after running command -p

2021-10-30 Thread Chet Ramey

On 10/29/21 6:06 PM, Roger Morris wrote:


Bash Version: 5.0
Patch Level: 17
Release Status: release

Description:
I believe there's a bug in bash 4.4 (and still in 5.1) that wasn't
there in 4.3.30

When 'command -p' runs, it no longer seems to restore the hash to its
previous value.


This came in in mid-2015 as part of a set of fixes to `command -p'. I think
the bash-4.4 and later behavior is correct: just because you're telling
command to use a standard path it doesn't mean it should not act like it
would in any other circumstance. There's nothing in POSIX that contradicts
that.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Why should `break' and `continue' in functions not break loops running outside of the function?

2021-10-30 Thread Chet Ramey

On 10/30/21 11:02 AM, Oğuz wrote:

On Sat, Oct 30, 2021 at 4:50 PM Greg Wooledge  wrote:

As Chet said, it's counterintuitive.  Most people don't expect function A
to be able to affect loops inside function B.


I do, and a subshell can prevent function A from affecting loops
inside function B. But that is not a real problem, you wouldn't call,
say `break 3', when you're only 2 loop levels deep in a function
unless you wanted to exit from the loop from within the function is
called after returning.


It's a violation of scope.


It's a violation of lexical scope, I'm asking why not implement
dynamic scope, what's wrong with it?


You might be interested in

https://www.austingroupbugs.net/view.php?id=842

There was a discussion on the austin-group mailing list back in 2016
accompanying this defect report.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: ${y@a} on an empty array y fails with 'y: unbound variable' if run with set -u

2021-10-26 Thread Chet Ramey
On 10/25/21 8:24 PM, Mark March wrote:
> If -u is on and you declare a simple or associative array 'y' and set it 
> empty, then ${y@a} will fail with 'unbound variable'.

It really is unset:

"An array variable is considered set if a subscript has been assigned  a
value."

> I was able to repro this under 5.1.0(1)-release and 5.1.8. 5.0.17(1)-release 
> does not seem to be affected.

Bash-5.1 fixed some bugs in this area.


> The code to reproduce (output lines begin with #>):
> 
> echo $BASH_VERSION
> #> 5.1.0(1)-release
> set -u
> declare -a y=()
> echo ${y@a}
> #> bash: y: unbound variable

Yes, because it's unset, but see below.

> declare -p y
> #> declare -a y=()
> echo ${y[@]}
> #>

The `@' and `*' subscripts are exempted from the nounset option to parallel
the behavior of `$@' and `$*' when there are no positional parameters (the
latter is specified by POSIX).

> set +u
> echo ${y@a}
> #> a
> 
> As you can see, turning off -u makes ${y@a} work correctly. 

If the `nonunset' option is not enabled, there is a special case for the
`a' transform so it will print the attributes of an unset array variable.

That was the result of a long discussion:

https://lists.gnu.org/archive/html/bug-bash/2020-02/msg00050.html


> I wonder if this is a side-effect of the fix that you described in item (n) 
> of the most recent change log:
> 
> n. Fixed a bug that caused ${foo@a} to treat foo as an unset variable if it 
> was an array without a value for subscript 0/"0" but had other set elements

This is not relevant, there are no set elements.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: current bash.git.devel segfaults on my code

2021-10-24 Thread Chet Ramey

On 10/24/21 4:00 AM, Alex fxmbsw7 Ratchev wrote:

and whats the cmd to fetch or change a commit state instead of newest


At some point, you have to be able to read through the things people
send you.

For instance:

https://thoughtbot.com/blog/git-bisect

gives you all the information you need.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Option for read to handle incomplete last line

2021-10-24 Thread Chet Ramey

On 10/24/21 12:22 PM, Martin Schulte wrote:


Before reading the source I would never have thought that read sets variables 
although it returns FAILURE.


Think of them as orthogonal conditions. `read' reads until a newline, or
EOF or other error condition (ignoring timeouts or reading N characters for
a minute), storing the characters it reads into the variable(s).

The variable(s) continue(s) to hold the characters read. `read' returns a
status dependent on the status of the last read(2). If it gets EOF (or
error), the return status is > 0, but the characters read are not lost.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: current bash.git.devel segfaults on my code

2021-10-23 Thread Chet Ramey

On 10/22/21 9:03 PM, Alex fxmbsw7 Ratchev wrote:

./bash --noprofile --norc

bash-5.1# ./bash ../xbl5/xbl

malloc: unknown:0: assertion botched
malloc: 0x3000211ab0: allocated: last allocated from glob.c:1150
realloc: underflow detected; mh_nbytes out of range
Aborting...Aborted
bash-5.1#


You have everything you need to automatically run `git bisect': a non-
interactive reproducer, a failing commit, and a successful commit.

You can get the commit IDs from

http://git.savannah.gnu.org/cgit/bash.git/log/?h=devel

(click on an individual commit message to view the commit ID).

Look at my previous message for a link to a quick description of how to
use `git bisect'.

Good luck! Let us know what you find.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: current bash.git.devel segfaults on my code

2021-10-22 Thread Chet Ramey

On 10/20/21 8:27 PM, Alex fxmbsw7 Ratchev wrote:

i see
makes sense

i was hoping for finished cmds to test / save debugging info , rather than 
for example this git docs


i can say it for anyone, ..imho 99 % to all docs are.. not for me
like 

how to check bash devel version and version history
to mark as good / bad
.. is one small question about that all


The bottom line is that even with your code, it will take the rest of your
environment to reproduce the issue. I couldn't reproduce it with what you
previously posted. I doubt you can reproduce it with `bash --noprofile 
--norc' as well.


The idea behind `git bisect' is to find a commit where it works, find a
commit where it doesn't (for some value of `it'), and do a binary search on
commits between those two to find which one starts breaking. `git bisect'
automates much of this process.

I found this short explainer after about ten seconds of searching:

https://thoughtbot.com/blog/git-bisect

Since the problem appears to be interactive startup, I'm not sure you'll
be able to automate this, unless something like `bash -i http://tiswww.cwru.edu/~chet/



Re: current bash.git.devel segfaults on my code

2021-10-20 Thread Chet Ramey

On 10/20/21 8:22 PM, Alex fxmbsw7 Ratchev wrote:

i see

no good

..

again such a stuckcase


If you're the only one who can reproduce it, you're going to have to take
the lead in debugging it.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: current bash.git.devel segfaults on my code

2021-10-20 Thread Chet Ramey

On 10/20/21 8:24 PM, Alex fxmbsw7 Ratchev wrote:

erm sorry what


The issue, if any, is obviously triggered by something in your environment.
You can run bash --noprofile --norc to verify that.

No one else has access to everything in your environment. I could not
reproduce the crash with the parts of your startup scripts you posted.

You are the only one who can reproduce this crash. There is little anyone
can do to help you at this point without you taking the lead to isolate
the crash to either a specific piece of shell code or a specific change
in bash.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: current bash.git.devel segfaults on my code

2021-10-20 Thread Chet Ramey

On 10/20/21 4:57 PM, Alex fxmbsw7 Ratchev wrote:
look, i do ./bash it segfaults, also when make valgrind, then i do valgrind 
./bash it shows what i pasted

the segfault didnt happen a month or two ago


The way you present this makes it impossible for anyone but you to debug it.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: current bash.git.devel segfaults on my code

2021-10-20 Thread Chet Ramey

On 10/19/21 7:47 PM, Alex fxmbsw7 Ratchev wrote:

any reply to this ?


There's nothing helpful there, and nothing that might point to any problem,
much less a solution.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: the alias bugger maybe again

2021-10-13 Thread Chet Ramey

On 10/11/21 3:08 PM, Alex fxmbsw7 Ratchev wrote:

the alias bug(s) i experience again in simple code


This is fixed in the devel branch, and the fix will be in the next release.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: current bash.git.devel segfaults on my code

2021-10-13 Thread Chet Ramey

On 10/12/21 9:15 PM, Alex fxmbsw7 Ratchev wrote:
could you give me maybe good urls about such to learn ( maybe not too long 
ones .. ) ?


You should, with a current bash devel, just be able to run

make clean
make valgrind

That disables some bash malloc wrapping functions that confuse valgrind
severely. Then just run

valgrind bash
or
valgrind --leak-check=full bash

and see what you get. (Assuming you have valgrind installed, of course, and
depending on what you want to find out).

If you want to try address sanitizer, make sure you have a compiler that
supports it, then run

make clean
make asan

I put the shorthand targets in for convenience. Once you build a bash with
address sanitizer support, you just have to run it and see what it reports.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: current bash.git.devel segfaults on my code

2021-10-12 Thread Chet Ramey
On 10/11/21 11:17 AM, Alex fxmbsw7 Ratchev wrote:
> as the topic says
> i fetched it earlier today, compiled, and tho it rather segfaults
> 
> due to i dunno to debug this, this mail consists of 3 parts, 1 gdb + bt
> ./bash (loading in bashrc a script which makesnit segfault)
> and once with gdb ./bash -vx
> third part is the code
> forth part is around where in the code it happens

I can't reproduce it. `mkPATH' isn't there, and there's probably a lot of
your environment that needs to be reproduced for it to work. It looks like
somewhere a free list block is getting clobbered, but I can't really tell.
Since I can't reproduce it, you'll have to see if you can get more detail
from something like valgrind or address sanitizer.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: devel: Questions about quoting in the new replacement ${var/pat/&}

2021-10-11 Thread Chet Ramey
On 10/5/21 4:41 AM, Koichi Murase wrote:
> I have questions on the new feature ${var/pat/&} in the devel branch.
> 
>> commit f188aa6a013e89d421e39354086eed513652b492 (upstream/devel)
>> Author: Chet Ramey 
>> Date:   Mon Oct 4 15:30:21 2021 -0400
>>
>> enable support for using `&' in the pattern substitution replacement 
>> string
>>
>> Any unquoted instances of & in STRING are replaced with the matching
>> portion of PATTERN.  Backslash is used to quote & in STRING; the
>> backslash is removed in order to permit a literal & in the
>> replacement string.  Users should take care if STRING is
>> double-quoted to avoid unwanted interactions between the backslash
>> and double-quoting.  Pattern substitution performs the check for &
>> after expanding STRING; shell programmers should quote backslashes
>> intended to escape the & and inhibit replacement so they survive any
>> quote removal performed by the expansion of STRING.
> 
> I would very much like this change introduced in the latest commit
> f188aa6a in devel as it would enable many more string manipulations
> with a simple construct, but I feel the current treatment of quoting
> has problems:
> 
> 1. There is no way to specify an arbitrary string in replacement in a
>   way that is compatible with both bash 5.1 and 5.2.

It's a change that assigns meaning to a character that was previously
valid, not an error. It's probably going to require a shell option.

> 
> 2. There is no way to insert a backslash before the matched part
>   (which I'd think would be one of the typical usages of &).

This is quite reasonable, and a minor change. If the replacement function
treats backslash specially by allowing it to quote `&', it should also
allow it to escape a backslash.

> 
> I below describe the details of each, followed by my suggestion or
> discussion on an alternative design.
> 
> --
> 1. How to specify an arbitrary string in replacement copatibly with
> both bash 5.1 and 5.2?
> 
> Currently any & in the replacement is replaced by the matched part
> regardless of whether & is quoted in the parameter-expansion context
> or not.  Even the result of the parameter expansions and other
> substitutions are subject to the special treatment of &, which makes
> it non-trivial to specify an arbitrary string to the replacement
> ${var/pat/rep}.

The documentation goes into this in some detail, including specifying the
expansions that REP undergoes.

>   $ str='X' pat='Y' rep='A'
>   $ echo ${str/$pat/}
>   X
> 
> where  is some string that represents the literal "$rep" (i.e.,
> 'A').  A naive quoting of "$rep" does not work:
> 
>   $ echo "1:${str/$pat/"$rep"}"
>   1:X

Wouldn't it be better to treat it in the standard way a double-quoted
parameter expansion would be treated? The double-quoted expansion is
already well-specified. People know how to get a backslash through
double quoting, even in a context, like this one, where quote removal
is performed.

> 
> I would have expected it to work because $pat will lose special
> meaning and be treated literally when it is quoted as "$pat". 
 For
> example, the glob patterns *?[ etc. and anchors # and % in $pat will
> lose its special meaning when it is quoted:
> 
>   $ v='A' p='?'; echo "${v/$p/B}"; echo "${v/"$p"/B}"
>   B
>   A
>   $ v='A' p='#'; echo "${v/$p/B}"; echo "${v/"$p"/B}"
>   BA
>   A
>   $ v='A' p='%'; echo "${v/$p/B}"; echo "${v/"$p"/B}"
>   AB
>   A
> 
> Of course, if $rep is not quoted, & in $rep is replaced by the matched
> part.
> 
>   $ echo "2:${str/$pat/$rep}"
>   2:X
> 
> * To properly specify an arbitrary string in the replacement, one
>   needs to replace all the characters.
> 
>   $ echo "${str/$pat/${rep//&/&}}"
> 
> * When the replacement is not stored in a variable, one needs to
>   create a variable for the replacement, i.e.,
> 
>   $ echo "${str/$pat/$(something)}"
> 
>   in Bash 5.1 needs to be converted to
> 
>   $ tmp=$(something)
>   $ echo "${str/$pat/${tmp//&/&}}"
> 
>   in Bash 5.2.
> 
> * Also, there is no way of writing it so that it works in both Bash
>   5.1 and 5.2.  To make it work, one needs to switch the code
>   depending on the bash version as:
> 
>   if ((BASH_VERSINFO[0]*1+BASH_VERSINFO[1]*100>=50200)); then
> echo "${str/$pat/${rep//&/&}}"
>   else
> echo "

Re: Misleading error when attempting to run foreign executable

2021-10-11 Thread Chet Ramey
On 10/11/21 7:09 AM, Robert Elz wrote:
> Date:Sun, 10 Oct 2021 21:09:53 -0400
> From:Eli Schwartz 
> Message-ID:  <4a362385-066d-0795-9a02-ff8bbb920...@archlinux.org>
> 
>   | So I wonder, if bash already in this exact case attempts to open() the
>   | file and read() it to look for a shebang, what's the harm in assuming
>   | (or checking) that it exists in this patch?
> 
> This is clearly an OS problem, not one in bash.

This is true. And yet:

> POSIX says of ENOENT as it applies to the exec*() set of functions:
> 
> [ENOENT]  A component of path or file does not name an existing file
>   or path or file is an empty string.

When execve returns this error for what is a clearly invalid reason, it's
not unreasonable that the shell help. It's a quality of implementation
issue at that point. If execve ever returns a different error for this
situation, the 5 lines of code will no longer be needed.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Incorrect LINENO with exported nested functions with loops

2021-10-10 Thread Chet Ramey

On 10/9/21 3:47 PM, Chet Ramey wrote:


The FOR token is
what causes the parser to increment word_top. The ARITH_FOR_EXPRS token
parses just one part of that production (the ((...;...;...)) part). It's
true that the function that parses the stuff between the double parens
doesn't use word_lineno[word_top], and maybe it should, but having
incremented word_top after seeing FOR, the action should decrement it after
parsing the complete statement.


Thanks for pointing me in the right direction. The issue was indeed that
parse_dparen wasn't using word_lineno[word_top] -- in fact, as you
suspected, it wasn't being set properly. Making sure it was set correctly
so the assumptions made elsewhere aren't violated (i.e., every use of FOR
causes word_top to be incremented) is the way to go. My fix is slightly
different from yours, but yours works as well.

Chet



--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Incorrect LINENO with exported nested functions with loops

2021-10-09 Thread Chet Ramey

On 10/7/21 5:06 PM, Tom Coleman wrote:
You are correct that FOR loops increment the word_top variable in the 
read_token_word function, but, ARITH_FOR_EXPRS loops do not. They are 
treated separately and do not increment the word_top variable from what I 
can see.


#define FOR 265
#define ARITH_FOR_EXPRS 286


The entire grammar production we're talking about here -- the one you're
modifying -- is

arith_for_command:  FOR ARITH_FOR_EXPRS list_terminator newline_list DO 
compound_list DONE


(for example; there are other rules in that production). The FOR token is
what causes the parser to increment word_top. The ARITH_FOR_EXPRS token
parses just one part of that production (the ((...;...;...)) part). It's
true that the function that parses the stuff between the double parens
doesn't use word_lineno[word_top], and maybe it should, but having
incremented word_top after seeing FOR, the action should decrement it after
parsing the complete statement.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: declare does not always set variable flags prior to assignment

2021-10-08 Thread Chet Ramey

On 10/4/21 10:17 AM, Léa Gris wrote:
Found out that the declare statement does not properly set all variable 
flags before assign values:



unset arr
declare -i -a arr=(1 2 3)
declare -p arr

declare -ai arr=([0]="1" [1]="2" [2]="3")


this is ok

declare +i -a arr=(hello world)
declare -p arr

declare -a arr=([0]="0" [1]="0")


this is not ok as arr assignment was handled as integers


It's an interesting order-of-operations question. You can make a case that
the shell should unset the attributes before expanding the words in the
compound assignment statement, since we special-case setting the attributes
for declaration commands. Let me look at that.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Misleading error when attempting to run foreign executable

2021-10-08 Thread Chet Ramey

On 10/4/21 10:06 AM, Ilkka Virta wrote:
On Mon, Oct 4, 2021 at 4:46 PM Chet Ramey <mailto:chet.ra...@case.edu>> wrote:


Bash reports the error it gets back from execve. In this case, it's
probably that the loader specified for the executable isn't present on your
system.


OTOH, for a script, Bash checks to see if the problem is with the 
interpreter and reports accordingly:


  $ ./foo.sh
bash: ./foo.sh: /bin/noexist: bad interpreter: No such file or directory

The shell does go on to stat() the file after getting ENOENT from execve(), 
so I suppose it could
add some clarifying note to the error message for the case of a binary file 
too.


About the only other thing it could say would be "cannot execute" or
"exists but cannot execute" in addition to "no such file or directory."
The shell doesn't know anything else. I'm not sure that would be a
significant improvement: you already know you can't execute the file and
you can easily check yourself whether or not the file exists.

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: EXIT trap is not executed after an exec failure in a non-interactive shell

2021-10-08 Thread Chet Ramey

On 10/1/21 2:16 PM, Mark March wrote:

Ok, thank you for clarifying. There is nothing in the documentation about this 
behavior as far as I can tell. I would suggest adding a line about traps 
getting reset after a failed exec to the paragraph on 'execfail'.


I think it will be a cleaner fix, and more intuitive, to make sure the
traps are preserved across a failed `exec'. I'll look at changing that
behavior.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: [PATCH] devel: fix segfault by unset 'assoc[${x[0]}]'

2021-10-08 Thread Chet Ramey

On 10/6/21 10:47 PM, Koichi Murase wrote:

I'd like the default behavior to be closer to what it is when
assoc_expand_once is enabled, as I said back in March. I think it's
going to be better for users in the long run.


Does that mean the behavior with `assoc_expand_once' being disabled
also modified in a way incompatible with older Bash versions?  I have
been thinking that `shopt -s assoc_expand_once' would be the default
in the future keeping the behavior of `shopt -u assoc_expand_once'.


My original vision was to modify behavior so that, at least for features
like arithmetic expansion and the `[[' command, it would not be needed at
all, and the behavior that users expect naturally would be the default.
Shell builtins are a different issue, since their arguments already undergo
the standard set of word expansions. I had hoped to be able to modify the
default behavior without too much incompatibility.


If the behavior of `shopt -u assoc_expand_once' would also be
modified, I would like to request another switch for the
backward-compatible behavior, in particular, a specific shopt switch
(but not a setting something like `BASH_COMPAT=51' which would involve
other behavior changes). Anyway, we need to maintain the code of the
backward-compatible behavior.


So far, I think I've kept things mostly compatible.



I see.  In order to make such architectural changes, I feel we first
need to determine how the behavior should be changed.  I guess such a
discussion would be again as long as the one in March.  Maybe this
would become just a repetition of the discussion in March, but to
summarize,

* I still feel that the cleanest solution is to introduce a special
   the syntax-level rule for `unset arr[...]' where the part `arr[...]'
   is treated as if the right-hand side of a variable assignment (just
   like in other assignment builtins such as `declare', `local',
   `export', etc.), i.e., pathname expansions and word splitting is not
   performed on the arguments of the form `name[xxx]'.


I'm not there yet with this one. I feel like the existing quoting can do
the job. The difference between this and a variable assignment is that
variable assignments already have defined behavior. This would be a new
change.


   This might be also useful to distinguish the all-element unset «
   unset a[@] » from the unset of the element of key='@' « unset a['@']
   ».


There are some changes along these lines in the devel branch already.



* I would like to request a backward-compatible mode where the extra
   expansions of array subscripts are performed the same as the older
   versions of Bash.  I would like to see a specific option for this
   mode rather than `BASH_COMPAT=51' which would involve other
   behavioral changes.


It's those backward-compatible behaviors that are the problem. Expanding
the subscript multiple times, no matter what the intention was initially,
is the source of most of the issues we're trying to address.



* I feel we need to care about the consistency with the extra
   expansions performed in other contexts:

   - printf -v 'a[$key]'
   - read 'a[$key]'
   - declare 'a[$key]=1'
   - vref='a[$key]'; echo "${!vref}"
   - declare -n nref='a[$key]'


I'm working through these. I think there's a decent framework for them in
the devel branch right now, but I haven't made any substantial changes in
behavior.


   - etc.


a better architectural solution.


This will not change the observable behavior, but if I would refactor
it, I'd make `valid_array_referecen' return the extracted subscript
and let `unbind_array_element' just receive the extracted subscript
rather than make `unbind_array_element' again extract the subscript.


Thanks, I'll look at this.

Chet



--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Incorrect LINENO with exported nested functions with loops

2021-10-07 Thread Chet Ramey
On 10/7/21 12:58 AM, Tom Coleman wrote:
> I did some experimenting and am confident on the fix. I don't know how to
> supply a patch suggestion officially, but have pasted it below. Four lines
> in parse.y need deleting. Arithmetic for loops should NOT be decrementing
> the 'word_top' variable, they do not make use of it.

I haven't looked closely at this yet, but I'm curious about your reasoning
for the final sentence. The FOR token causes word_top to be incremented by
read_token_word(); why should the production not decrement it after the
entire loop is parsed?

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: [PATCH] devel: fix segfault by unset 'assoc[${x[0]}]'

2021-10-06 Thread Chet Ramey
On 10/5/21 10:39 PM, Koichi Murase wrote:

>> You're right, there should be no `nesting' considered at all. By the time
>> unbind_array_element is called, since it's only called from unset_builtin,
>> the word and subscript should have already undergone all the expansion
>> they're going to. There should be no need to interpret ${} or $() in the
>> subscript: since associative arrays can have arbitrary subscripts, you
>> should not attempt to parse the subscript contents.
> 
> Yeah, I think the above paragraph describes the expected behavior when
> `assoc_expand_once' is turned on.
> 
> But in this patch, I actually aim to fix the behavior of the
> backward-compatible mode (with `assoc_expand_once' turned off).  In
> the patch, I suggested to remove `(var && assoc_p(var))' from the
> skipsubscript flag for the nesting consideration 

I understand. I'd like the default behavior to be closer to what it is
when assoc_expand_once is enabled, as I said back in March. I think it's
going to be better for users in the long run.


>> It can shortcut and say "ok, if it passes valid_array_reference, we should
>> just consider the entire argument as one word as long as the final
>> character is `]'." This is again where backwards compatibility and
>> assoc_expand_once matter.
>>
>> We can apply your change, but it is still incomplete
> 
> What is exactly the incompleteness that you focus on in this context?

Only that I'd like a more comprehensive behavioral change. Your fix is
fine for the limited scope it tackles (resolving the discrepancy between
valid_array_reference and unbind_array_element).


>> The real issue is that it's still going to expand the subscript for
>> backwards compatibility. I think that's going to have to be solved
>> at a higher level.
> 
> Yes, but I feel like this is another design issue which is irrelevant
> for the fix of the present small problem.

Sure, but that's why we're talking through the issue. Your fix is fine
for the problem it intends to solve, now I'd like to go beyond it and
figure out a better architectural solution.

Chet


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: [PATCH] 4.0..devel: fix a problem that unset 'a[`echo 0`]' causes "bad array subscript" error

2021-10-06 Thread Chet Ramey
On 10/5/21 10:48 PM, Koichi Murase wrote:
> [ I initially intended to submit this report after the previous report
> at https://lists.gnu.org/archive/html/bug-bash/2021-10/msg00051.html
> has been settled.  I decided to submit this patch now because a
> problem related to this report is mentioned in the above thread.  Note
> that this patch bases on the code after applying the patches provided
> at the previous report. ]>
> Bash Version:
>   All the versions from 4.0 to devel are affected.  The problem should
>   be reproduced independent of the machine type.
> 
> Description:
> 
>   « unset -v 'a[`echo 0`]' » fails because the first character « ` »
>   of the subscript « `echo 0` » is skipped when checking the ending of
>   the array subscript.

Thanks for the comprehensive report and fix.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
     ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-05 Thread Chet Ramey
On 10/5/21 1:50 PM, Dominique Martinet wrote:

> If I change malloc_usable_size to return p->mh_nbytes instead of
> maxbytes, then the crash disappears.[2]
> 
> I did not read the full bash malloc code but I suspect the buffer really
> could be grown, but we would need to fix p->mh_nbytes to maxbytes and
> also adjust the end block to pass sanity checks on free -- e.g. it
> should be considered as a lightweight inplace realloc.
> 
> I'm not sure we care enough to be honest and returning what is really
> usable feels like the simplest solution, what do you think?
> 

Thanks for your work tracking this down.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-05 Thread Chet Ramey
On 10/5/21 1:50 PM, Dominique Martinet wrote:

> Turns out I'm lucky enough on address consistency..
> 
> So,
>  - since we have a nice before/after with systemd, I took a moment to
> bisect it.
> It comes down to this commit[1] which is basically using
> malloc_usable_size() to use buffers beyond what it initially requested
> 
> [1] 
> https://github.com/systemd/systemd/commit/319a4f4bc46b230fc660321e99aaac1bc449deea
> 
> - through gdb/printf I see that the failing pointer has been allocated
> with (what comes down to) malloc(64), but malloc_usable_size returns
> 108, so systemd happily writes beyond the 64 bytes it originally
> requested -- which bash allocator treats as an overflow.

Sure. The bash malloc_usable_size basically returns the size of the chunk:
what you could realloc it to without having to go back for more memory
(that's how I interpreted `usable bytes', but that's clearly not how it's
intended). I suppose without the bounds checking that would have been fine.


> If I change malloc_usable_size to return p->mh_nbytes instead of
> maxbytes, then the crash disappears.[2]

That's the right fix.

> 
> I did not read the full bash malloc code but I suspect the buffer really
> could be grown, but we would need to fix p->mh_nbytes to maxbytes and
> also adjust the end block to pass sanity checks on free -- e.g. it
> should be considered as a lightweight inplace realloc.

If you want realloc, use realloc.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: [PATCH] devel: fix segfault by unset 'assoc[${x[0]}]'

2021-10-05 Thread Chet Ramey
On 10/5/21 7:05 AM, Koichi Murase wrote:

>   The segmentation fault is fixed by the above patch, but there
>   still remains the same error as bash 4.4.
> 
> bash-patch1: ${x[0: bad substitution
> 
>   This is caused by an inconsistency between `valid_array_reference
>   (name,flags)' (arrayfunc.c:1187) and `unbind_array_element
>   (var,sub,flags)' (arrayfunc.c:1033) in the extraction of
>   associative-array subscripts.  Note that `valid_array_reference' is
>   called from `unset_builtin' (builtins/set.def:834) to check if the
>   unset name has the form of an array element.  Also,
>   `unbind_array_element' is called from `unset_builtin' to perform the
>   actual unset.  In `valid_array_reference', the length of the
>   associative-array subscripts are determined as
> 
>   else if (isassoc)
> len = skipsubscript (t, 0, flags_NOEXPAND);  /* VA_NOEXPAND
> must be 1 */

The difference is that valid_array_reference can be called before any of
the subscript is expanded, in which case you need to parse things that
can be expanded, where unbind_array_element is called after all the
expansions are performed (but see below).

So let's see if we can talk through this.

> 
>   whereas in `unbind_array_element', the length is determined as
> 
>   if (var && assoc_p (var) && (flags_ONEWORD))
> len = strlen (sub) - 1;
>   else
> len = skipsubscript (sub, 0, (flags_NOEXPAND) || (var &&
> assoc_p(var)));  /* XXX */
> 
>   `skipsubscript' does not consider the nesting of ${} and $() when
>   bit 1 is set to the third argument.  In the former code, nesting is
>   not considered only when VA_NOEXPAND is specified.  However, in the
>   latter code, nesting is never considered for associative arrays
>   (even when VA_NOEXPAND is not specified).  I believe the former code
>   should be the expected one.

You're right, there should be no `nesting' considered at all. By the time
unbind_array_element is called, since it's only called from unset_builtin,
the word and subscript should have already undergone all the expansion
they're going to. There should be no need to interpret ${} or $() in the
subscript: since associative arrays can have arbitrary subscripts, you
should not attempt to parse the subscript contents. That was obviously one
of the problems with the code through bash-5.1, one of the original reasons
I introduced `assoc_expand_once', and an issue that's still looking for a
final resolution.

However, there is backwards compatibility to consider, which is why
assoc_expand_once isn't set by default and the code attempts to run the
subscript through word expansion.

In this example, the quoting prevents the shell from recognizing the word
as an array reference before the quote removal occurs and the word gets
passed to the unset builtin, so it can't set any hints for unset. unset
sees `a[${b[0]}]', which is ambiguous.

It can shortcut and say "ok, if it passes valid_array_reference, we should
just consider the entire argument as one word as long as the final
character is `]'." This is again where backwards compatibility and
assoc_expand_once matter.

We can apply your change, but it is still incomplete (plus it breaks things
that currently work, like

declare -A a
key='$(echo foo)'

a['$key']=2
declare -p a

unset -v "a['\$key']"
declare -p a

). The real issue is that it's still going to expand the subscript for
backwards compatibility. I think that's going to have to be solved at a
higher level.

Maybe the unset builtin code should just punt as described above --
depending on the shell compatibility level -- and turn on
VA_ONEWORD|VA_NOEXPAND if there is an unquoted `[', the last character is a
`]', and valid_array_reference() returns true.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-04 Thread Chet Ramey

On 10/4/21 9:44 PM, Dominique Martinet wrote:

Chet Ramey wrote on Mon, Oct 04, 2021 at 09:23:11PM -0400:

   - I could reproduce the same as Julien, with -DDISABLE_MALLOC_WRAPPERS
the crash still happens when bash is run directly but nothing complains
in valgrind.


I assume you mean using systemd. Has anyone tried running a bash linked to
the systemd library that provides the getpw functions, but not as a systemd
unit? You could then run it in a debugger if it crashes, for instance.


I'm running busybox sh in a unit (which starts properly), then
interactively test things from there.

Running in gdb does fail the same way as running normally, so I've also
been looking at that a bit, but nothing obvious poped up.
I'd like to trace back which allocation corresponds to the failing one,
and break from there next time.


Without the allocation tracing, which you don't get from allocations that
aren't directly from bash, it's tedious, but doable. It only works if the
memory layout is consistent enough that you get the same address failing
every time (that is, the address being passed to the failing free() is the
same).

If you do, you write it down, put a breakpoint in internal_malloc at the
point where it returns the memory to its caller, and just run, breaking
at that point, until malloc returns that address. Then run a backtrace and
see where you are.



If you have a nixos system, not necessarily on master, this should be
enough to reproduce:


I don't.


Hm, that won't necessary work with LTO though.
If they call reallocarray, which is in libc, LTO means reallocarray can
call libc's realloc without going through symbol interposition.
(That's a discussion that came up on fedora-devel mailing list when
talking about LTO, and breaking LD_PRELOAD no longer overloading calls
internal to a lib)


Yeah, that would break a whole bunch of things that have worked forever.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-04 Thread Chet Ramey

On 10/4/21 8:39 PM, Dominique Martinet wrote:


(not sure how that works? bash internal
malloc just passes to free pointers it doesn't know about?)


What does this mean? What pointers it doesn't know about? (If bash realloc
or free gets a pointer that wasn't allocated by the bash malloc, it
throws an error.)

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-04 Thread Chet Ramey

On 10/4/21 8:15 PM, Dominique Martinet wrote:


(I've been following this with Julien as I can reproduce the behaviour
on my nixos system -- you don't have to run the latest systemd, just
install the derivation and use its path in LD_LIBRARY_PATH instead of
the system's... That also probably could bring its own set of
incompatibility but so far I'm getting the same behaviour as him running
systemd properly)


I'd love to have a simpler reproducer here. It's just hard for me to see
how just changing systemd, and nothing else, produces an error. There might
be some obscure bug in the bash malloc, but the code we're talking about
here -- the code that detects overflow -- hasn't changed in years.


OTOH, valgrind *should* complain when using the system malloc (configure
--without-bash-malloc), and it does not, so for me that means there
really is some weird thing happening. Forgive me for trusting valgrind
analysis more than bash malloc debugging here...


The bash overflow detection is much less sophisticated than valgrind,
that's for sure, but it's so simple it's fairly easy to verify.



  - I could reproduce the same as Julien, with -DDISABLE_MALLOC_WRAPPERS
the crash still happens when bash is run directly but nothing complains
in valgrind.


I assume you mean using systemd. Has anyone tried running a bash linked to
the systemd library that provides the getpw functions, but not as a systemd
unit? You could then run it in a debugger if it crashes, for instance.


This could mean that systemd is overflowing bash malloc safeguards as
you pointed out (I just don't understand why it wouldn't overflow with
internal malloc), but it could also mean that the memory has been
allocated somewhere else (e.g. libc's malloc) and freed by bash malloc.


I have a tough time with that one. If the bash free/realloc get memory that
the bash malloc hasn't allocated, you're going to fail several sanity tests
before you get to the point of checking for overflow.



nss systemd has started using reallocarray() since v247 and that is not
tracked by bash, I would think that's a good candidate?


I can't see how? reallocarray() is not a memory allocation primitive. It's
going to call malloc/realloc to do its work (it's essentially just a call
to realloc(mem, nmemb * size)). Those will eventually call the bash
malloc/realloc/free.


I don't have time right now, but I think adding an implementation for
reallocarray (wrapper around realloc which does exist) would be the next
thing to do.


You could try it, but I'm skeptical that it will have any effect.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-04 Thread Chet Ramey

On 10/4/21 4:44 PM, Andreas Schwab wrote:

On Okt 04 2021, Chet Ramey wrote:



I suspect this is a buffer overflow introduced between systemd-247 and
systemd-249. It's not caught when building bash without the bash malloc
because the default libc malloc probably doesn't do the bounds checking
the bash malloc does, even without malloc debugging turned on.


If it's a buffer overflow, then valgrind should be able to catch it
(when bash is configured --without-bash-malloc).  valgrind's bounds
checking is much more advanced than what a checking malloc can do.


You'd think. This is the kind of overflow that will produce that error
message from the bash malloc:

int
main(int c, char **v)
{
char*buf;

buf = (char *)malloc (40);
if (buf == 0) {
fprintf(stderr, "malloc failed\n");
exit(1);
}
buf[40]='\254';

buf = realloc(buf, 218);
buf[218]='\214';

free(buf);
exit(0);
}




--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-04 Thread Chet Ramey

On 10/4/21 5:28 PM, Julien Moutinho wrote:

On Okt 04 2021, Chet Ramey wrote:

I suspect this is a buffer overflow introduced between systemd-247 and
systemd-249. It's not caught when building bash without the bash malloc
because the default libc malloc probably doesn't do the bounds checking
the bash malloc does, even without malloc debugging turned on.


Chet, thanks for you detailed analysis,
I've opened an issue to get some inputs from systemd's devs:
https://github.com/systemd/systemd/issues/20931

Le lun. 04 oct. 2021 22h44 +0200, Andreas Schwab a écrit :

If it's a buffer overflow, then valgrind should be able to catch it
(when bash is configured --without-bash-malloc).  valgrind's bounds
checking is much more advanced than what a checking malloc can do.


Andreas, just to confirm that so far I'm unable to get a crash or error
when using --without-bash-malloc, even in valgrind (but I'm a newbie at 
valgrind).


If you want to make sure that you're checking bash with valgrind, even in
the presence of bash optimizing execution and performing an `exec' of `id'
without forking, use a bash builtin like `true' instead of `id'.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-04 Thread Chet Ramey
-> systemd nss code). The only thing that's
changed is the systemd version.

> - No crash happens with bash5-without-bash-malloc on systemd-247 or 
> systemd-249.

Because the default malloc doesn't do this checking.

> - A slightly different crash, still involving _nss_systemd_getpwuid_r,
>   happens with bash4-with-bash-malloc on systemd-247.
>   It might have been fixed when a new definition of MALIGN_MASK
>   was introduced in bash-5.1:

This is the change from 8-byte to 16-byte alignment. One consequence of
this change is that the additional space in the block header is used to
catch underflow. Since there's no underflow message, we can assume the
sizes don't match because the overflow overwrote the size information
stored after the block.

>   
> https://git.savannah.gnu.org/cgit/bash.git/commit/lib/malloc/malloc.c?id=8868edaf2250e09c4e9a1c75ffe3274f28f38581
>   See also: https://www.mail-archive.com/bug-bash@gnu.org/msg24306.html

This was one of the things that prompted the change between bash-5.0 and
bash-5.1.

> - No crash happens with bash4-without-bash-malloc on systemd-247 or 
> systemd-249.

For the same reason as above.

> - bash crashes inside valgrind too,
>   but apparently something different is happening
>   because it crashes even without systemd being involved:

This is a red herring.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-04 Thread Chet Ramey
On 10/4/21 12:23 PM, Julien Moutinho wrote:
> Le lun. 04 oct. 2021 10h34 +0200, Andreas Schwab a écrit :
>> Here is a patch:
> Thanks Andreas, that particular crash disappears with this patch.
> However the crash after _nss_systemd_getpwuid_r() is still happening for me,
> and valgrind can still find a similar crash after source_builtin():

It's a problem with valgrind, described in another thread with this
subject. Build bash with -DDISABLE_MALLOC_WRAPPERS to work around it.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-04 Thread Chet Ramey
On 10/4/21 12:27 PM, Andreas Schwab wrote:
> On Okt 04 2021, Chet Ramey wrote:
> 
>> Nope. These are all functions internal to bash and the bash malloc, and
>> they are all defined.
> 
> The problem is that the xmalloc macro redirects directly to the internal
> malloc implementation, whereas the xfree function calls it indirectly
> through the free function.  The latter is seen by valgrind, the former
> isn't, so it didn't track it.  Thus the xmalloc, xrealloc, xfree macros
> need to be disabled if valgrind is used.

OK, now we're back to the invalidating some of valgrind's assumptions.
For this, you can define DISABLE_MALLOC_WRAPPERS while building bash,
which has been there since at least bash-2.05b. I haven't built it that
way in quite a while, but it shouldn't have any problems.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-04 Thread Chet Ramey
On 10/4/21 11:08 AM, Andreas Schwab wrote:
> On Okt 04 2021, Chet Ramey wrote:
> 
>> Nope, I don't buy that as the reason. xfree (name of function) and xfree(x)
>> (macro defined in xmalloc.h) are not the same thing.
> 
> That's exactly the problem.  You cannot pass the return value from
> sh_xmalloc to xfree, only sh_xfree.

That is absolutely not true with the bash malloc. It might be that valgrind
makes certain assumptions about these functions, but those are not valid
assumptions against the bash malloc implementation. You can freely mix
calls to xmalloc/sh_xmalloc/malloc and xfree/sh_xfree/free. They all end up
using the same set of internal functions and the same allocation pool.

I've explained this previously, most recently in

https://lists.gnu.org/archive/html/bug-bash/2017-04/msg00053.html

>> and everything works correctly.
> 
> Nope, it's undefined behaviour, as pointed out by valgrind.

Nope. These are all functions internal to bash and the bash malloc, and
they are all defined.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-04 Thread Chet Ramey
On 10/4/21 10:17 AM, Andreas Schwab wrote:
> On Okt 04 2021, Chet Ramey wrote:
> 
>> On 10/4/21 4:34 AM, Andreas Schwab wrote:
>>> On Okt 04 2021, Julien Moutinho wrote:
>>>
>>>> - bash crashes inside valgrind too,
>>>>   but apparently something different is happening
>>>>   because it crashes even without systemd being involved:
>>>>
>>>> $ nix build .#bash5-with-bash-malloc
>>>> $ valgrind result/bin/bash --norc -c true
>>>>> ==307088== Memcheck, a memory error detector
>>>>> ==307088== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
>>>>> ==307088== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright 
>>>>> info
>>>>> ==307088== Command: result/bin/bash --norc -c true
>>>>> ==307088== 
>>>>> ==307088== Invalid free() / delete / delete[] / realloc()
>>>>> ==307088==at 0x483F8E9: free (in 
>>>>> /nix/store/7s7hzqaf5imxmpjlxh2n6fs7ixml98ya-valgrind-3.16.1/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>> ==307088==by 0x47330F: xfree (xmalloc.c:150)
>>>>> ==307088==by 0x4644FA: unwind_frame_run_internal (unwind_prot.c:325)
>>>>> ==307088==by 0x4640B6: without_interrupts (unwind_prot.c:117)
>>>>> ==307088==by 0x464656: run_unwind_frame (unwind_prot.c:143)
>>>>> ==307088==by 0x479ACA: parse_and_execute (evalstring.c:523)
>>>>> ==307088==by 0x41C0A5: run_one_command (shell.c:1440)
>>>>> ==307088==by 0x41D6A1: main (shell.c:741)
>>>>> ==307088==  Address 0x404be10 is in the brk data segment 
>>>>> 0x4033000-0x4054fff
>>>
>>> Here is a patch:
>>
>> How does this fix the problem with valgrind? How does wrapping xfree in a
>> local function help?
> 
> Because xfree is a function-like macro, so taking the address does not
> work.

Nope, I don't buy that as the reason. xfree (name of function) and xfree(x)
(macro defined in xmalloc.h) are not the same thing. Unless you mean that
the existence of the macro confuses valgrind? Or something else?

Indeed, when you step through the code in a debugger, breaking at the spot
in parse_prologue that installs the unwind-protect, you see:

(gdb) b evalstring.c:257
Breakpoint 1 at 0x48d1b5: file
/usr/homes/chet/src/bash/src/builtins/evalstring.c, line 257.
(gdb) r -c 'echo a'
Starting program:
/mnt/src-build/build/linux/chet/bash/bash-current.rhe7/bash -c 'echo a'

Breakpoint 1, parse_prologue (string=0x7c6150 "echo a", flags=20,
tag=0x4f4c33 "parse_and_execute top")
at /usr/homes/chet/src/bash/src/builtins/evalstring.c:257
257 add_unwind_protect (xfree, orig_string);
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-324.el7_9.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64
(gdb) p orig_string
$1 = 0x7c6150 "echo a"
(gdb) s
add_unwind_protect (cleanup=0x486220 , arg=0x7c6150 "echo a")
at /usr/homes/chet/src/bash/src/unwind_prot.c:152
152   without_interrupts (add_unwind_protect_internal, (char *)cleanup, 
arg);

and everything works correctly.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

2021-10-04 Thread Chet Ramey
On 10/4/21 4:34 AM, Andreas Schwab wrote:
> On Okt 04 2021, Julien Moutinho wrote:
> 
>> - bash crashes inside valgrind too,
>>   but apparently something different is happening
>>   because it crashes even without systemd being involved:
>>
>> $ nix build .#bash5-with-bash-malloc
>> $ valgrind result/bin/bash --norc -c true
>>> ==307088== Memcheck, a memory error detector
>>> ==307088== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
>>> ==307088== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright 
>>> info
>>> ==307088== Command: result/bin/bash --norc -c true
>>> ==307088== 
>>> ==307088== Invalid free() / delete / delete[] / realloc()
>>> ==307088==at 0x483F8E9: free (in 
>>> /nix/store/7s7hzqaf5imxmpjlxh2n6fs7ixml98ya-valgrind-3.16.1/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>>> ==307088==by 0x47330F: xfree (xmalloc.c:150)
>>> ==307088==by 0x4644FA: unwind_frame_run_internal (unwind_prot.c:325)
>>> ==307088==by 0x4640B6: without_interrupts (unwind_prot.c:117)
>>> ==307088==by 0x464656: run_unwind_frame (unwind_prot.c:143)
>>> ==307088==by 0x479ACA: parse_and_execute (evalstring.c:523)
>>> ==307088==by 0x41C0A5: run_one_command (shell.c:1440)
>>> ==307088==by 0x41D6A1: main (shell.c:741)
>>> ==307088==  Address 0x404be10 is in the brk data segment 0x4033000-0x4054fff
> 
> Here is a patch:

How does this fix the problem with valgrind? How does wrapping xfree in a
local function help?

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Misleading error when attempting to run foreign executable

2021-10-04 Thread Chet Ramey
On 10/3/21 6:05 PM, Andrea Monaco wrote:
> 
> Hello,
> 
> 
> I'm using bash 5.0.3 on GNU/Linux. When I mount a GNU/Hurd filesystem and
> try to run an executable found there, I get for example:
> 
>   bash: ./ext2fs: No such file or directory
> 
> This seems a misleading error to me. The file is there and is
> executable, I guess it can't be run on my system but I think that
> another error message would be more appropriate.
> 
> Is this an issue with bash or maybe with the kernel?

Bash reports the error it gets back from execve. In this case, it's
probably that the loader specified for the executable isn't present on your
system.

Chet


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Request change to output of declare -f funcname

2021-10-03 Thread Chet Ramey

On 10/2/21 1:02 PM, Léa Gris wrote:

Le 02/10/2021 à 18:45, Greg Wooledge écrivait :

On Sat, Oct 02, 2021 at 06:06:32PM +0200, Léa Gris wrote:

Better illustrated how newlines are discarded:

$ sudo bash -c 'echo hello
echo world'

hello
world


$ sudo -i bash -c 'echo hello
echo world'

helloecho world


OK, that's news to me.  But that looks like a bug in sudo.  Asking bash
to change its behavior to work around a bug in some other program seems
like misdirected effort.


It is not wholly misdirected since Bash already add semicolon after 
statements where it is optional since these are followed by newlines.


Adding a semicolon after the last statement of a command group would not 
hurt in any circumstance.


But what would such a change help, other than to avoid the consequences of
what appears to be a bug in `sudo -i'?

Now while reworking the output of declare -f, there could be an option to 
produce the most compact output without newlines and without indentation; 
for the purpose of serializing function declarations, similarly as declare 
-p already serialize variables declaration in a compact one-line even for 
arrays.


Corollary declare -p could have an indented expanded output that would be 
useful for arrays with an element per line rather than space delimited 
elements.


If you'd like to see these changes, maybe do a sample implementation that
you (and others, if you'd like) can evaluate?

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: issue compiling bash.git with new sw on phone

2021-10-03 Thread Chet Ramey

On 10/2/21 7:40 AM, Alex fxmbsw7 Ratchev wrote:


malloc: make_cmd.c:656: assertion botched
malloc: 0x3000220af0: allocated: last allocated from make_cmd.c:656
realloc: underflow detected; mh_nbytes out of range
Aborting/configure: line 8975: 11907 Aborted CC="$CC"
GCC="$GCC" LDFLAGS="$LDFLAGS" LD="$LD" with_gnu_ld="$with_gnu_ld"
${CONFIG_SHELL-/bin/sh} "$ac_aux_dir/config.rpath" "$host" > conftest.sh
done


This is the code that reads here-documents and builds the complete document
in memory. I can't reproduce it using configure on RHEL or macOS, but if
you can find a simpler way to reproduce it, I'd be glad to look at that.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: EXIT trap is not executed after an exec failure in a non-interactive shell

2021-10-01 Thread Chet Ramey

On 9/30/21 7:24 PM, Mark March wrote:

If execfail is set, a failed exec does not cause a non-interactive shell to 
exit, but it seems to reset the EXIT trap:


Yes. When the shell runs `exec', it assumes the execed program will overlay
the shell process. To make that happen transparently, it has to undo things
it has done: it ends job control and restores the original process groups,
it restores the signal dispositions that it got from its parent, and it
clears other shell state like the EXIT trap.

If the exec fails, it tries to restore some things as well as it can, but
it doesn't try to restore any traps.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: receiving SIGHCHLD if job control is disabled - expected behavior?

2021-10-01 Thread Chet Ramey

On 10/1/21 4:24 AM, Vladimir Marek wrote:

Hello,

The code in question is

set +m
echo $BASH_VERSION
echo $SHELLOPTS
trap 'echo =SIGCHLD=' 18
sleep 1
echo done


Bash 5 output:
5.0.17(1)-release
braceexpand:hashall:interactive-comments
=SIGCHLD=
done


Bash 4 output:
4.4.19(1)-release
braceexpand:hashall:interactive-comments
done



I was trying to find a relevant note in changelog and I found this in CHANGES:

n. The SIGCHLD trap is run once for each exiting child process even if job
control is not enabled when the shell is in Posix mode.

and this in CWRU/changelog:

 - waitchld: run SIGCHLD trap for each child exited even if job control
   is not enabled when in Posix mode. Prompted by an austin-group-l
   discussion

I was trying to find the discussion and the closest thing I found was

https://www.mail-archive.com/austin-group-l@opengroup.org/msg00898.html


That's the one.

A SIGCHLD is generated when a child process dies, whether or not the shell
currently has job control active, since monitor mode is basically about
process groups. You can trap any signal, including SIGCHLD, so why not
make it as reliable as you can?

It's useful to be able to guarantee that you'll get a SIGCHLD trap for
each child exiting, to do exactly the sort of counting that was the subject
of that discussion, so I made that work whether or not monitor mode is
enabled.

The documentation, even though it talks about this feature in the JOB
CONTROL section, never made any distinction.


As a side note, another interesting bit I found is that if I replace the
'sleep 1' with 'read' I will not get SIGCHLD in bash 5.


Well, of course (?). `read' is a builtin, no child process is created, and
so the shell doesn't get a SIGCHLD.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: mapfile -t doesn't always remove the delimiter

2021-09-30 Thread Chet Ramey
On 9/30/21 10:50 AM, Greg Wooledge wrote:
> The mapfile builtin command doesn't correctly strip delimiters (when
> requested with the -t option) in all cases.

Thanks for the report. It's an easy fix.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Line counting is ideosyncratic with eval.

2021-09-24 Thread Chet Ramey

On 9/22/21 2:27 PM, dfsm...@us.ibm.com wrote:


Bash Version: 5.0
Patch Level: 3
Release Status: release

Description:
When running commands through eval, the script's line counter
is incremented, even though the eval line is fixed.


Thanks for the report. This is a tricky one (ksh93, for instance, reports
line 1001).

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: [PATCH] fix the termcap string for _rl_term_kP

2021-09-22 Thread Chet Ramey
On 9/22/21 7:59 AM, Koichi Murase wrote:
> Bash Version:
>   devel branch (441078402919f6f0dd677cad18d55c7a89d294fc),
>   5.1.8(2)-maint (x86_64-pc-linux-gnu)
> 
> Description:
> 
>   The key `page-up', which is supposed to be bound to the readline
>   bindable function `history-search-backward' after commit 65822e50,
>   does not actually work because of a typo in `lib/readline/terminal.c'.

Thanks for the report.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Extra timing statistics for failing function calls when errexit is enabled

2021-09-14 Thread Chet Ramey
On 9/13/21 4:32 PM, Sergej Alikov wrote:

> Bash Version: 5.1
> Patch Level: 8
> Release Status: release
> 
> Description:
>     When the "time" keyword is used in a pipeline with
>     function calls, and errexit is enabled - each failing
>     function call will emit the timing statistics before the
>     final statistics for the whole pipeline.

Thanks for the report. This will be fixed in the next version.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: POSIX defines struct timeval in

2021-09-08 Thread Chet Ramey
On 9/5/21 10:57 AM, Ori Sky Farrell wrote:
> I've discovered a gap in POSIX compliance in lib/sh/strftime.c on the devel 
> branch. POSIX mandates that the timeval struct be defined in , 
> although as can be seen on Line 67 of the file in question, the file is only 
> included here if the tm struct -- which POSIX mandates be defined in  
> -- is defined in .

It's interesting that your libc implementation claims POSIX conformity with
respect to this:

> The libc implementation I'm working with defines timeval in  and 
> tm in  as mandated by POSIX so here I encounter a compile failure on 
> Line 193.

and yet does not include the POSIX-required strftime().

I'll look at the  and  issues.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Interactive commands cant be backgrounded if run from bashrc

2021-09-02 Thread Chet Ramey

On 9/2/21 5:06 PM, C. Yang wrote:
Could you please explain why? I thought the reason for the behavior 
described in my original post was that bash does not complete 
initialization until .bashrc completes, which doesn't happen until the 
emacs process started from within it completes? 


OK. You said you enabled job control (set -m), started emacs, stopped it
and put it into the background. As soon as you background the process,
the shell goes ahead and reads and executes the next command, which, since
this is the last thing in .bashrc, results in EOF. Once the shell is
finished reading commands from .bashrc, it completes initialization and
continues by printing the interactive prompt.

The same thing effectively happens if you start emacs in the background
after enabling job control: since it will not have access to its
controlling terminal (it is in a different process group from the
terminal's process group) it will get a SIGTSTP when it tries to read from
the terminal and stop. Once the shell starts the process in the background,
it goes on immediately and doesn't wait.

If you try to start emacs in the foreground without enabling job control,
you will not be able to use job control signals to manipulate its state,
and the shell will have to wait for it to terminate before it can go on
and finish reading from .bashrc, as it would with any other foreground
process.

If you start emacs in the background without enabling job control, the
shell will complete reading and executing commands from .bashrc as
described above, and go on with normal interactive execution. Bash does
not make itself a session leader, or allocate a new controlling terminal,
so it and the backgrounded emacs will both have access to the controlling
terminal and will fight over input.

In the first two scenarios, the job you started by invoking emacs and
backgrounding it will eventually complete on its own, and the shell will
reap the terminated process, as part of the normal interactive shell
execution.

And it sounds like some 
things, like enabling job control, do not happen until that happens?


Unless you force job control using `set -m', as you say you did.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Interactive commands cant be backgrounded if run from bashrc

2021-09-02 Thread Chet Ramey
On 9/2/21 12:15 PM, C. Yang wrote:

> However, is it possible that there may be further unexpected consequences,
> since bash is still waiting to complete initialization this entire time?
> 
> For instance, if I stop and background emacs, then I find myself back to
> the bash
> shell. But technically, bash is still waiting for .bashrc to complete.

This is not correct.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Interactive commands cant be backgrounded if run from bashrc

2021-09-02 Thread Chet Ramey
On 9/1/21 2:10 PM, C. Yang wrote:

> Machine: x86_64
> OS: linux-gnu
> Compiler: gcc


> Description:
> 
>   Whenever I start my session, I'd like to automatically open emacs to a 
> specific file.
> 
>   So, I added the emacs command to the bottom of my ~/.bashrc file. This 
> opens emacs
> 
>   correctly when I start the session.
> 
>  
> 
>   Normally, when I start emacs, I can background the process with CTRL+Z, and 
> foreground
> 
>   with `fg` command. When emacs is started from .bashrc as above, pressing 
> CTRL+Z does
> 
>   not correctly background the process. Instead, the terminal session goes 
> blank and
> 
>   becomes unresponsive.

Bash doesn't initialize job control until after reading the startup files,
which are executed in a nominally non-interactive environment.

You can force that initialization by running `set -m'. It may work for your
purposes.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: help adding some features to 5.1

2021-09-01 Thread Chet Ramey
On 8/31/21 6:38 PM, Ananth Chellappa wrote:
> Hi Team,
>        Could I get some help locating portions of the code that would need
> to be tweaked to add these features?
> 
> If I had the time, I would love to get to know the code, but I have too
> much going on in my real job. 
> 
> 1. Intelligent support for !$ (and related - like !2$, !-N, etc) : This
> means - ignore the & at the end of the previous command.

This is part of the history library: lib/readline/histexpand.c. Keep in
mind that history expansion has never worked like this, so "intelligent"
is subjective.

> 2. Autocomplete for !$ (and related tokens) : This means - if I have typed
> !$ and now press TAB, I want the last word from the previous command
> substituted on the command-line immediately.

This is part of bash-specific word completion: bashline.c.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: LINENO is affected by sync

2021-09-01 Thread Chet Ramey
On 9/1/21 5:36 AM, David Collier wrote:
> Version:
> 
> GNU bash, version 5.0.3(1)-release (arm-unknown-linux-gnueabihf)
> 
> Raspberry Pi using Raspbian.
> 
> Installed from repo?
> 
> LINENO goes backwards when run sync

I can't reprodude this.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Squiggly heredoc - new feature request

2021-09-01 Thread Chet Ramey
On 8/31/21 3:25 PM, Lawrence Velázquez wrote:
> On Tue, Aug 31, 2021, at 4:02 AM, Lawrence Velázquez wrote:
>> ksh does not blindly remove all leading whitespace
> 
> For the curious, this is how ksh(1) describes it:
> 
>   If '#' is appended to '<<', then leading spaces and tabs
>   will be stripped off the first line of the document and up
>   to an equivalent indentation will be stripped from the
>   remaining lines and from 'word'.  A tab stop is assumed to
>   occur at every 8 columns for the purposes of determining
>   the indentation.

I will consider this for a future version.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: bash command alias has a problem in using brace.

2021-08-31 Thread Chet Ramey

On 8/27/21 9:47 PM, Dale R. Worley wrote:

Hyunho Cho  writes:



but if i use alias then '>' prompt does not appear and default bash
prompt appears

bash$ alias myalias='{ cat <<\@ > foo ;} 2> /dev/null'
bash$ myalias
bash$ 111
bash$ 222  # bash$ prompt
bash$ @





I don't know the details, but it must have something to do with what
event triggers the reading of the here-document.  That event isn't the
execution of the command containing it, I don't think, but more like
when the here-document-redirection is parsed. 


This is substantially correct. When the parser goes and collects the body
of the here-document, it's still parsing the contents of the alias.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: ${par@P} can cause segmentation faults

2021-08-28 Thread Chet Ramey

On 8/28/21 4:07 AM, Emanuele Torre wrote:


Bash Version: 5.1
Patch Level: 8
Release Status: release

Description:

 PS1=\${PS1@P}

   makes bash segfault in interactive shells.

 a=\${a@P};${a@P}

   makes bash segfault also in non-interactive shells.


Yes. If you set up and execute an infinitely recursive expansion, you're
eventually going to run out of stack or data space and crash.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: readline 'unix-filename-rubout' whitespace bug

2021-08-27 Thread Chet Ramey

On 8/26/21 10:18 PM, d...@dabe.com wrote:


Bash Version: 5.1
Patch Level: 8
Release Status: release

Description:
The manpage for bash(1) says:

   unix-filename-rubout
   Kill the word behind point, ***USING WHITE SPACE AND THE SLASH
  CHARACTER AS THE WORD BOUNDARIES***.  The killed text is saved
   on the kill-ring.   [Emphasis mine]

In certain circumstances, however, it gobbles up too much.


Thanks for the report. That circumstance is a pathname consisting solely of
one or more slashes, separated from the previous word by whitespace. I'll
fix it.



PS: I'm hopeful there might be some kind of workaround that will work even
on those dated releases!  [crossing fingers]


The code has been like this since January, 2004. That's pretty dated.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Defect in manual section "Conditional Constructs" / case

2021-08-25 Thread Chet Ramey
On 8/24/21 8:14 PM, Lawrence Velázquez wrote:
> On Tue, Aug 24, 2021, at 4:44 PM, Dietmar P. Schindler wrote:
>> Doesn't the example I gave above show that quotes are removed? If they
>> weren't, how could word aa with pattern a""a constitute a match?
> 
> The quotes are handled by the matching process itself, *not* as
> part of the usual shell expansions.  Otherwise these patterns would
> be equivalent, but they're not.
> 
> % cat /tmp/foo.sh
> case $1 in
> 'a?a') echo one ;;
> a?a) echo two ;;
> esac
> % bash /tmp/foo.sh 'a?a'
> one
> % bash /tmp/foo.sh aaa
> two

The literal quote characters are removed, and the characters between the
quotes are appropriately quoted for the pattern matcher, so they're forced
to match themselves. This happens AIBM.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Defect in manual section "Conditional Constructs" / case

2021-08-25 Thread Chet Ramey
On 8/24/21 1:47 AM, dietmar_schind...@web.de wrote:

> In the section 
> https://www.gnu.org/software/bash/manual/bash.html#Conditional-Constructs in 
> the description of the "case" command there is no mention (as far as I can 
> see, it doesn't follow from the documented expansions etc.) that a _pattern_ 
> undergoes quote removal, but it does [see e. g. case aa in a""a) echo match;; 
> esac]. (One might think it does self-evidently in the process of "Shell 
> Expansions" performed on the command line, but this expansion series is not 
> performed on the case command's _word_ and patterns - they for example don't 
> undergo brace expansion -; for _word_, it is explicitly said: "The _word_ 
> undergoes tilde expansion, parameter expansion, command substitution, 
> arithmetic expansion, and quote removal …"; for _pattern_: "Each _pattern_ 
> undergoes tilde expansion, parameter expansion, command substitution, and 
> arithmetic expansion." - quote removal is missing.)
> 
> To rectify it, I suggest to change and simplify these sentences from
> 
>   The _word_ undergoes tilde expansion, parameter expansion, command 
> substitution, arithmetic expansion, and quote removal (see Shell Parameter 
> Expansion) before matching is attempted. Each _pattern_ undergoes tilde 
> expansion, parameter expansion, command substitution, and arithmetic 
> expansion.
> 
> to
> 
>   The _word_ and each _pattern_ undergo tilde expansion, parameter 
> expansion, command substitution, arithmetic expansion, and quote removal (see 
> Shell Parameter Expansion) before matching is attempted.

Thanks for the report. This was changed in the devel branch back in May as
the result of

https://lists.gnu.org/archive/html/bug-bash/2021-05/msg00030.html

There was also a discussion on the POSIX mailing list about the wording.

It's tricky in the sense that quote removal, according to the strict shell
definition, is performed -- the literal quote characters are removed and
don't appear in the expanded pattern. The complication is that the shell
and pattern matcher have to arrange for the quoted characters to be
marked appropriately for the pattern matcher itself, so quoted special
characters lose their special meaning and match themselves. That has to
happen internally, using whatever API or function the pattern matcher
offers.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: efficient way to use matched string in variable substitution

2021-08-23 Thread Chet Ramey
On 8/23/21 3:13 PM, Oğuz wrote:
> 23 Ağustos 2021 Pazartesi tarihinde L A Walsh  yazdı:
> 
>> Starting with a number N, is there
>> an easy way to print its digits into an array?
>> I came up with a few ways, but thought this
>> would be nice (with '\1' or '$1' being what was matched
>> in the 1st part), this could be statement:
> 
> 
> If memory serves, the ampersand character will have this function in bash
> 5.2.

Yes, it's tagged for bash-5.2.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
     ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: [feature request] parameter transformation to safely add strings to PS1, .

2021-08-23 Thread Chet Ramey
On 8/22/21 5:11 PM, Emanuele Torre wrote:
> It would be nice to have a parameter transformation (e.g. "${par@p}")
> that expands $par to a string that will not be expanded by PS1, PS2, 

So you want it to be expanded at some point, but its value not subject to
any of the prompt string expansions (\a, \d, \t, and so on)?


> 
> example:
> 
>   tmp_var=$(blabla) # this variable will not exist when PS1 is expanded

This seemns to be the key requirement. Otherwise, you would be able to
simply write PS1="blabla \${tmp_var} blabla" as others have suggested.

I'm not sure that requires a new transformation (which would have to be
much more completely specified than it has been so far).


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: EPOCHREALTIME does not behave correctly before 1970

2021-08-22 Thread Chet Ramey

On 8/22/21 7:52 PM, Kerin Millar wrote:

On Sun, 22 Aug 2021 16:13:28 -0400
Chet Ramey  wrote:


On 8/21/21 1:28 AM, Emanuele Torre wrote:


Bash Version: 5.1
Patch Level: 8
Release Status: release




get_epochrealtime() casts tv.tv_sec (a time_t a.k.a. int) to unsigned
int causing EPOCHREALTIME to not behave correctly before the Unix Epoch.


The definition is seconds since the Unix epoch. It's not surprising that it
doesn't pay attention to dates before that.


The problem with this statement is that EPOCHREALTIME is permitted to get it 
wrong, yet EPOCHSECONDS is permitted to get it right. The discrepancy certainly 
came as a surprise to me.


It's whatever the kernel tells you is ok. EPOCHSECONDS uses time();
EPOCHREALTIME uses gettimeofday(). Both are defined to return the time
in seonds (and, in the latter case, microseconds) since the epoch. You can
argue that bash should check whether the return value of seconds is < 0,
but that isn't one of the defined failure modes.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: Issue declaring an array via a variable name

2021-08-22 Thread Chet Ramey

On 8/21/21 6:02 PM, Hunter Wittenborn wrote:

As an end user I would expect the unquoted `('



operator to cause a syntax error, just as it does in `echo ('.




Well I'm expecting '(' to be part of the shell's syntax (when unquoted; so 
likewise not cause a syntax error), but when looking at things like the left 
side of a variable assignment, I'm sure you'll agree that it should allow any 
string that fits a variable's normal specification (regardless of being an 
array or not).


Maybe this is best seen as a misunderstanding about order of operations.

Before the `declare' builtin is executed, the command has to be parsed.
The parser identifies the token `declare' as the first word of a simple
command, and further identifies `declare' as a declaration command, as
explained in the documentation.

One thing about the shell's metacharacters (of which `(' is one) is that
they have to be quoted when they appear somewhere outside where the shell's
grammar permits. One place bash allows `(' -- an extension to the POSIX
grammar -- is on the rhs of an assignment statement.

Given a declaration command, the shell parser allows assignment statements
as arguments. An assignment statement, as documented, takes the form

identifier=value

where `identifier' is a `name' (as defined in the bash documentation) or an
array subscript of the form `name[index]'.

The key to understanding this is that all of this must happen before any of
the shell's word expansions, including quote removal, take place.

If the shell parser can't recognize a word, or at least the word that's
been accumulated when it sees the `(' operator, as a valid assignment
statement, the exception to having to quote the left paren doesn't come
into play.

We can disregard arguments that contain `=' that may be treated as 
assignments when `declare' sees them as long as they don't contain any

shell metacharacters:



`declare "${value}"="x"`

`declare "y"="x"`


However, if you want the parser to allow an unquoted metacharacter, you
have to follow the rules that allow it to happen, and this does not:


`declare "${value}"=("x" "z")`


Because "${value}" is not a valid shell `name' and cannot appear on the
lhs of an assignment statement.

Given that, it should be obvious why the above is a syntax error.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: EPOCHREALTIME does not behave correctly before 1970

2021-08-22 Thread Chet Ramey

On 8/21/21 1:28 AM, Emanuele Torre wrote:


Bash Version: 5.1
Patch Level: 8
Release Status: release




get_epochrealtime() casts tv.tv_sec (a time_t a.k.a. int) to unsigned
int causing EPOCHREALTIME to not behave correctly before the Unix Epoch.


The definition is seconds since the Unix epoch. It's not surprising that it
doesn't pay attention to dates before that.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: @K transformation

2021-08-20 Thread Chet Ramey

On 8/19/21 6:37 AM, Léa Gris wrote:


#!/usr/bin/env bash

declare -A assoc=(
  [P]=piano
  [TB]='foldable table'
  ['CH AIR']=chair
)

options=("${assoc[@]@K}")


The best way to clone an associative array is:

declare -A options
eval options=\( "${assoc[@]@K}" \)

The quoting @K performs is eval-safe.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: EPOCHREALTIME

2021-08-19 Thread Chet Ramey
On 8/19/21 9:41 AM, Léa Gris wrote:

> This will fail because of questionable design decision of having a mutable
> argument format:
> 
> LC_NUMERIC=fr_FR@UTF-8; printf 'Pi: %2.4f\n` "$(bc -l <<<'4*a(1)')"
> 
> Note how the format indicator still use a dot, but the argument format's
> decimal separator is that of the system's locale.

That's not a decimal point. The `2.4' is not some sort of decimal or
floating point number. The dot just separates the field width from the
precision.

If the argument is, in fact, a floating point number with a radix
character, the radix character is appropriately locale-dependent.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



  1   2   3   4   5   6   7   8   9   10   >