Re: EOF while in parse_matched_pair closes interactive shell

[Eduardo A . Bustamante López]

On Tue, Feb 21, 2023 at 11:03:59AM -0500, Chet Ramey wrote:
(...)
> The shell should exit on EOF. Previous versions relied on undocumented
> bison behavior, which resulted in a token that wasn't handled by the
> grammar.

Thank you for reviewing this and for the explanation. I initially thought this
changed as a result of this change:

| parse.y
| - yylex: return YYUNDEF as current_token if read_token returns < 0.
|   Fixes parser reset issue reported by Todd Stein 
|   in https://savannah.gnu.org/support/index.php?110745

I changed `yylex' to return -1 the way it was, and it doesn't make a difference
in this case.

By luck, I found that removing this line reverts to the original behavior:

$ git diff -- parse.y
diff --git a/parse.y b/parse.y
index 9178e9a7..87d45ac1 100644
--- a/parse.y
+++ b/parse.y
@@ -3714,7 +3714,7 @@ parse_matched_pair (int qc, int open, int close, size_t 
*lenp, int flags)
  free (ret);
  parser_error (start_lineno, _("unexpected EOF while looking for 
matching `%c'"), close);
  EOF_Reached = 1;  /* XXX */
- parser_state |= PST_NOERROR;  /* avoid redundant error message */
+ /*parser_state |= PST_NOERROR;*/  /* avoid redundant error 
message */
  return (_pair_error);
}

My understanding of how this works is as follows. The unexpected EOF causes
`parse_matched_pair' to return an error, which eventually leads to a call to
`yyerror'. In `yyerror', we call `report_syntax_error' _only if_ the NOERROR
parser flag is off.  Despite its name, `report_syntax_error' does more than
that. It'll actually clear `EOF_Reached' when the shell is interactive:

  bash: syntax error: unexpected end of file
  6458  if (interactive && EOF_Reached)
  6459EOF_Reached = 0;
  6462  last_command_exit_value = (executing_builtin && 
parse_and_execute_level) ? EX_BADSYNTAX : EX_BADUSAGE;

The parser eventually makes it way back to `reader_loop'. In 5.2, EOF_Reached is
1 at that point. In 5.1, it's 0 as it was cleared by `report_syntax_error'.

Knowing this now, I was able to find the commit that changed this, and the note
explaining the change:

| commit b48c234286ae692c758306dac26a5bb72f782fef
| Date:   Mon Oct 31 10:08:36 2022 -0400
| (...)
| parse.y
|- parse_matched_pair: set PST_NOERROR if we read to EOF without finding
|  a closing match and call parser_error; avoids redundant error
|  message

As an aside, this change also restores the original behavior:

$ git diff -- parse.y
diff --git a/parse.y b/parse.y
index 9178e9a7..d2d20874 100644
--- a/parse.y
+++ b/parse.y
@@ -3713,7 +3713,7 @@ parse_matched_pair (int qc, int open, int close, size_t 
*lenp, int flags)
{
  free (ret);
  parser_error (start_lineno, _("unexpected EOF while looking for 
matching `%c'"), close);
- EOF_Reached = 1;  /* XXX */
+ /*EOF_Reached = 1;*/  /* XXX */
  parser_state |= PST_NOERROR;  /* avoid redundant error message */
  return (_pair_error);
}

In any case, I do understand why you've made this change. I've checked how bash
deals with an EOF while parsing other partial commands (e.g. `if', `for x in',
`echo $(') and these all behave in the same way. So I don't disagree. I just
thought I should share my troubleshooting notes.

> If there's a bug here, it's that interactive shells need to handle
> IGNOREEOF in this case.

Thanks for reminding me about IGNOREEOF. It does seem like `parse_matched_pair'
doesn't work with `ignoreeof' in this case. I suppose that's because the parser
doesn't see `yacc_EOF', so it never calls `handle_eof_input_unit'.

EOF while in parse_matched_pair closes interactive shell

[Eduardo A . Bustamante López]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -g -O2 -fstack-protector-strong -Wformat 
-Werror=format-security -Wall
uname output: Linux system76-pc 6.1.0-4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 
6.1.11-1 (2023-02-09) x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu

Bash Version: 5.2
Patch Level: 15
Release Status: release

Description:

The handling of EOF while parsing an (unmatched) quoted string changes
between 5.1 and 5.2.  In 5.1, typing [Ctrl-d] alone while in the context of
a quoted string in an interactive shell will report a syntax error, and
prompt for a new command.  In 5.2, the same action does not report a syntax
error, and instead the shell exits.

I used to rely on [Ctrl-d] for getting out of a multiline string typed 
in
error. I suppose I should be using [Ctrl-c] to that effect instead. Old
habits die hard though, and I got curious about whether this change was
intentional. The oldest version of bash I have access to is
3.00.22(1)-release, and it behaves the same way as 5.1.

Repeat-By:

Please find the transcript of 3 terminal sessions illustrating the
difference in behavior between 5.2.15(1)-release, 5.2.15(7)-maint and
5.1.0(3)-release.  Terminal input is denoted as [INPUT], e.g. [Ctrl-d]

$ bash --norc
bash-5.2$ echo $BASHPID
132773
bash-5.2$ 
GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu)
bash-5.2$ echo '
>[Ctrl-d]
bash: unexpected EOF while looking for matching `''
>[Enter]
(Shell exits)

$ ./bash --norc
bash-5.2$ echo $BASHPID
132768
bash-5.2$ 
GNU bash, version 5.2.15(7)-maint (x86_64-pc-linux-gnu)
bash-5.2$ echo '
>[Ctrl-d]
bash: unexpected EOF while looking for matching `''
(Shell exits)

$ ./bash --norc
bash-5.1$ echo $BASHPID
132662
bash-5.1$ 
GNU bash, version 5.1.0(3)-release (x86_64-pc-linux-gnu)
bash-5.1$ echo '
>[Ctrl-d]
bash: unexpected EOF while looking for matching `''
bash: syntax error: unexpected end of file
bash-5.1$ echo $BASHPID
132662

[PATCH] Add missing parenthesis to tputs declaration in readline's tcap.h

[Eduardo A . Bustamante López]

Signed-off-by: Eduardo A. Bustamante López 
---
 lib/readline/tcap.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/readline/tcap.h b/lib/readline/tcap.h
index 1121061b..a9c6322a 100644
--- a/lib/readline/tcap.h
+++ b/lib/readline/tcap.h
@@ -51,7 +51,7 @@ extern int tgetflag PARAMS((const char *));
 extern int tgetnum PARAMS((const char *));
 extern char *tgetstr PARAMS((const char *, char **));
 
-extern int tputs PARAMS((const char *, int, int (*)(int));
+extern int tputs PARAMS((const char *, int, int (*)(int)));
 
 extern char *tgoto PARAMS((const char *, int, int));
 
-- 
2.39.1

Re: obscure bug "extern void free (void *__ptr) __attribute__ ((__nothrow__ , __leaf__));"

[Eduardo A . Bustamante López]

On Fri, Jan 22, 2021 at 05:05:31AM +0100, Mathias Steiger wrote:
> 
> Here is the strace output.
> 
> https://filebin.net/9auqyreezma08z12/bug_bash.tar.gz?t=3bjx4xpd
> 
> It is very excessive due to the nature of Autoconf and I couldn't make a lot
> of sense of it.
> 
> 
> When I downgraded the package from bash-5.1.004-1 to bash-5.0.018-2 the bug
> disappeared.

This trace file is quite interesting (redacted for brevity):

$ cat ./strace_if-statement_removed/strace_second.1243868 
(...)
openat(AT_FDCWD, "./config.status", O_WRONLY|O_CREAT|O_APPEND, 0666) = 8
dup2(8, 1)  = 1
close(8)= 0
openat(AT_FDCWD, "/dev/null", O_RDONLY) = 8
dup2(8, 0)  = 0
close(8)= 0
execve("/bin/cat", ["cat"], [(...)]) = 0
(...)
fstat(1, {st_dev=makedev(0xfe, 0x3), st_ino=42548069, st_mode=S_IFREG|0775, 
st_nlink=1, st_uid=1006, st_gid=0, st_blksize=4096, st_blocks=32, 
st_size=16357, st_atime=1611282495 /* 2021-01-22T03:28:15.296486563+0100 */, 
st_atime_nsec=296486563, st_mtime=1611282560 /* 
2021-01-22T03:29:20.712747829+0100 */, st_mtime_nsec=712747829, 
st_ctime=1611282560 /* 2021-01-22T03:29:20.712747829+0100 */, 
st_ctime_nsec=712747829}) = 0
fstat(0, {st_dev=makedev(0, 0x5), st_ino=1728, st_mode=S_IFREG|0666, 
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=73, 
st_atime=1611282495 /* 2021-01-22T03:28:15.243153560+0100 */, 
st_atime_nsec=243153560, st_mtime=1611282549 /* 
2021-01-22T03:29:09.822815317+0100 */, st_mtime_nsec=822815317, 
st_ctime=1611282549 /* 2021-01-22T03:29:09.822815317+0100 */, 
st_ctime_nsec=822815317}) = 0
(...)
read(0, "extern void free (void *__ptr) __attribute__ ((__nothrow__ , 
__leaf__));\n", 131072) = 73
write(1, "extern void free (void *__ptr) __attribute__ ((__nothrow__ , 
__leaf__));\n", 73) = 73
read(0, "", 131072) = 0
(...)


This is in essence:

  cat ./config.status

These are worth highlighting:

- Notice that the value returned by `fstat(0, ...)' indicates that /dev/null in 
your system is a *regular* file (it should be `st_mode=S_IFCHR|0666', but 
instead it is `st_mode=S_IFCHR|0666'). It also indicates that its size is 73 
bytes (`st_size=73').
- Notice that `cat' reads from file descriptor 0 (i.e. /dev/null), and the 
return value is a string of 73 bytes in length.
- `cat' then writes that string out to `./config.status'

73 happens to be the length of the string that has been causing issues for you:

  $ echo "extern void free (void *__ptr) __attribute__ ((__nothrow__ , 
__leaf__));" | wc -c
  73

Can you run:

  stat /dev/null

And share with us what you get?

This is how it should look like:

  $ stat /dev/null
File: /dev/null
Size: 0 Blocks: 0  IO Block: 4096   character special 
file
  Device: 5h/5d Inode: 4   Links: 1 Device type: 1,3
  Access: (0666/crw-rw-rw-)  Uid: (0/root)   Gid: (0/root)
  Access: 2021-01-09 15:28:48.589801534 -0800
  Modify: 2021-01-09 15:28:48.589801534 -0800
  Change: 2021-01-09 15:28:48.589801534 -0800
   Birth: -

Re: Segmentation fault in pat_subst

[Eduardo A . Bustamante López]

On Sun, Jul 21, 2019 at 06:56:09PM -0400, Chet Ramey wrote:
(...)
> Thanks for the report. Look at match_wpattern and consider what happens if
> wmatchlen returns something longer than the string length. It will be fixed
> for the next devel branch push.

Aha! I see.

| Starting program: /tmp/bash/bash -c x=0\;\ :\ \$\{x/\#\[0�\\Z\[:\]\]\}
| 
| Breakpoint 1, match_wpattern (wstring=0x556c1d90 L"0", 
indices=0x556c1e20, wstrlen=1, wpat=0x556c1d00 L"[0�\\Z[:]]", mtype=1, 
sp=0x7fffd918, ep=0x7fffd910)
| at /home/dualbus/src/gnu/bash/subst.c:4963
| 4963  if (mlen > wstrlen)
| (gdb) p mlen
| $1 = 2
| (gdb) p wstrlen
| $2 = 1

Interesting, I can see how that `mlen = 2' can cause trouble. Thank you for the 
pointer and the fix!

Segmentation fault in pat_subst

[Eduardo A . Bustamante López]

Bash `devel' crashes under the following circumstances:

| dualbus@system76-pc:/tmp/build-bash-devel$ CFLAGS='-O0 -ggdb' 
~/src/gnu/bash/configure --with-bash-malloc
| (...)
| dualbus@system76-pc:/tmp/build-bash-devel$ make -j$(nproc)
| (...)
| dualbus@system76-pc:/tmp/build-bash-devel$ ./bash -c $'x=0; : 
${x/#[0\xef\xbf\xbd\\Z[:]]}'
| Segmentation fault (core dumped)


Here's the stack trace:

| dualbus@system76-pc:/tmp/build-bash-devel$ gdb ./bash --args ./bash -c $'x=0; 
: ${x/#[0\xef\xbf\xbd\\Z[:]]}'
| GNU gdb (Debian 8.2.1-2+b1) 8.2.1
| (...)
| Reading symbols from ./bash...done.
| (gdb) r
| Starting program: /tmp/build-bash-devel/bash -c x=0\;\ :\ 
\$\{x/\#\[0�\\Z\[:\]\]\}
| 
| Program received signal SIGSEGV, Segmentation fault.
| 0x555d1fae in pat_subst (string=0x5575f298 "0", 
pat=0x55764509 "[0�\\Z[:]]", rep=0x0, mflags=1) at 
/home/dualbus/src/gnu/bash/subst.c:8136
| 8136  if (str && *str)
| (gdb) bt
| #0  0x555d1fae in pat_subst (string=0x5575f298 "0", 
pat=0x55764509 "[0�\\Z[:]]", rep=0x0, mflags=1) at 
/home/dualbus/src/gnu/bash/subst.c:8136
| #1  0x555d250b in parameter_brace_patsub (varname=0x5575f248 "x", 
value=0x5575f288 "0", ind=0, patsub=0x55763f48 "#[0�\\Z[:]]", quoted=0, 
pflags=0, flags=0)
| at /home/dualbus/src/gnu/bash/subst.c:8306
| #2  0x555d47e2 in parameter_brace_expand (string=0x55763f28 
"${x/#[0�\\Z[:]]}", indexp=0x7fffe0d8, quoted=0, pflags=0, 
quoted_dollar_atp=0x7fffe1d4,
| contains_dollar_at=0x7fffe1cc) at 
/home/dualbus/src/gnu/bash/subst.c:9028
| #3  0x555d5ae8 in param_expand (string=0x55763f28 
"${x/#[0�\\Z[:]]}", sindex=0x7fffe1d8, quoted=0, 
expanded_something=0x7fffe374, contains_dollar_at=0x7fffe1cc,
| quoted_dollar_at_p=0x7fffe1d4, had_quoted_null_p=0x7fffe1d0, 
pflags=0) at /home/dualbus/src/gnu/bash/subst.c:9557
| #4  0x555d6ed8 in expand_word_internal (word=0x55763f68, 
quoted=0, isexp=0, contains_dollar_at=0x7fffe370, 
expanded_something=0x7fffe374)
| at /home/dualbus/src/gnu/bash/subst.c:10125
| #5  0x555da0b6 in shell_expand_word_list (tlist=0x55763f88, 
eflags=31) at /home/dualbus/src/gnu/bash/subst.c:11504
| #6  0x555da3bb in expand_word_list_internal (list=0x55763948, 
eflags=31) at /home/dualbus/src/gnu/bash/subst.c:11628
| #7  0x555d95b9 in expand_words (list=0x55763948) at 
/home/dualbus/src/gnu/bash/subst.c:11148
| #8  0x555a51d9 in execute_simple_command 
(simple_command=0x557639c8, pipe_in=-1, pipe_out=-1, async=0, 
fds_to_close=0x55763a88)
| at /home/dualbus/src/gnu/bash/execute_cmd.c:4334
| #9  0x5559ed6b in execute_command_internal (command=0x55763988, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x55763a88)
| at /home/dualbus/src/gnu/bash/execute_cmd.c:823
| #10 0x555a2116 in execute_connection (command=0x55763a48, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x55763a88) at 
/home/dualbus/src/gnu/bash/execute_cmd.c:2707
| #11 0x5559f134 in execute_command_internal (command=0x55763a48, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x55763a88)
| at /home/dualbus/src/gnu/bash/execute_cmd.c:996
| #12 0x55609e4d in parse_and_execute (string=0x557632c8 "x=0; : 
${x/#[0�\\Z[:]]}", from_file=0x5566b0f0 "-c", flags=4)
| at /home/dualbus/src/gnu/bash/builtins/evalstring.c:458
| #13 0x55585632 in run_one_command (command=0x7fffebdc "x=0; : 
${x/#[0�\\Z[:]]}") at /home/dualbus/src/gnu/bash/shell.c:1424
| #14 0x5558477d in main (argc=3, argv=0x7fffe8f8, 
env=0x7fffe918) at /home/dualbus/src/gnu/bash/shell.c:735
| 
| (gdb) p str
| $1 = 0xdfdfdfdfdfdfdfdf 

I have been looking around but I don't understand what's going on. I can see
that the value of `str' comes from `e', which in turn comes from
`match_pattern', but it's not clear to me why this is happening.

Re: Bug? Bash manual not indexable by search engines

[Eduardo A . Bustamante López]

On Sat, May 25, 2019 at 02:56:43PM -0400, Richard Marmorstein wrote:
> There was discussion on Twitter today
> (https://twitter.com/PttPrgrmmr/status/1132351142938185728) about how the
> Bash manual appears to not be indexable by search engines.
> 
> https://www.gnu.org/software/bash/manual/bashref.html
> redirects to
> https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html, and
> www.gnu.org/robots.txt
> has
> "Disallow: /savannah-checkouts/"
> 
> We reasoned that this probably wasn't deliberate and wanted to report it.

Hmmm, interesting. How did you get to 
?

I went to: , which is the "landing page" (?)
for Bash. That has:

> Documentation
> 
> *Documentation for Bash* is available online, as is documentation for most GNU
> software. You may also find more information about Bash by running info bash 
> or
> man bash, or by looking at /usr/doc/bash/, /usr/local/doc/bash/, or similar
> directories on your system. A brief summary is available by running bash 
> --help.

The "Documentation for Bash" text includes a link to:
, which then links to:
 (i.e. it's "bash.html", 
not "bashref.html").

Furthermore, if I search for "bash manual" in Google (i.e.
), the top three results (for me)
are:

1. 
2. 
3. 

So, it looks like the manual IS indexable?


I searched for "https://www.gnu.org/software/bash/manual/bashref.html; in Google
too, and I can see it's referenced from a couple of 

user submitted posts, but that's it.

Re: "here strings" and tmpfiles

[Eduardo A . Bustamante López]

On Mon, Mar 18, 2019 at 05:18:10PM -0400, Daniel Kahn Gillmor wrote:
> hi bash developers--
(...)
>  a) use socketpair(2) or pipe(2) instead of making a tmpfile.  this has
> the potential downside that the semantics of access to the remaining
> file descriptor would be subtly different from "regular file"
> semantics.

Disclaimer: not a bash developer, just a user.

I don't think the implementation details of herestrings are documented anywhere,
and I'm not too sure if they should (i.e. IMO if you need that degree of control
over the implementation details, then you should use something other than
shell).


Having said that, have you tried process substitution as an option?

You should be able to do something like:


  mycommand < <(printf %s 'super secret')


That will:

- not write the 'super secret' string to the file-system, nor
- show the mentioned string in the process tree (because printf is a bash
  built-in command, and thus, does not require a fork).

Re: General Associative Array problem -- Was: UUID as Array Keys strangely not possible

[Eduardo A . Bustamante López]

On Sat, Feb 23, 2019 at 01:48:08AM +, Robert White wrote:
(...)
> Your syntax checker is straight tripping on that SC1036 error dude. Array
> assignment using ARRAYNAME=( expression ) is completely legal and correct
> with or without the eval. The structure even allows for line continuation
> just like pipelines

To be fair, I didn't say the code was wrong. I just wanted to illustrate that 
the code you sent wasn't particularly easy
to work with (from my perspective). Which is fine, we found the problem 
anyways. But in the future it might help you
(and us), to reduce the test case to the most minimal script that still shows 
the behavior. That way we can easily run
it in our machines and see exactly what you see, and we can also focus on the 
stuff that matters (i.e. the `blkid',
`sed', `eval', etc. are all distractions from the actual problem).

I usually practice the code reduction approach myself and I think it's quite 
effective at helping me understand my own
mistakes :-).

configure: gl_PTHREAD_RWLOCK_RDLOCK_PREFER_WRITER: command not found

[Eduardo A . Bustamante López]

Commit `be4078d25ff3af268a6ef7ab56c8121f9a8dfb36' introduced the following 
issue:

| dualbus@system76-pc:~/src/gnu/bash$ ./configure --silent
| 
| Beginning configuration for bash-5.0-maint for x86_64-pc-linux-gnu
| 
| ./configure: line 9614: gl_PTHREAD_RWLOCK_RDLOCK_PREFER_WRITER: command not 
found
| config.status: creating po/POTFILES
| config.status: creating po/Makefile

There's no mention of these changes in the changelog, so I don't really 
understand what's broken or how to fix it.


I patched bash and ran `autoconf' to regenerate the `configure' file, and it 
/seems/ to work, but I don't really know
what I'm doing:


diff --git a/MANIFEST b/MANIFEST
index cf957d3e..e30536f6 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -502,6 +502,7 @@ m4/nls.m4   f
 m4/po.m4   f
 m4/printf-posix.m4 f
 m4/progtest.m4 f
+m4/pthread_rwlock_rdlock.m4f
 m4/size_max.m4 f
 m4/stdint_h.m4 f
 m4/threadlib.m4f
diff --git a/configure.ac b/configure.ac
index ab8bc23f..81816ce8 100644
--- a/configure.ac
+++ b/configure.ac
@@ -712,6 +712,7 @@ m4_include([m4/nls.m4])
 m4_include([m4/po.m4])
 m4_include([m4/printf-posix.m4])
 m4_include([m4/progtest.m4])
+m4_include([m4/pthread_rwlock_rdlock.m4])
 m4_include([m4/size_max.m4])
 m4_include([m4/stdint_h.m4])
 m4_include([m4/threadlib.m4])
diff --git a/m4/pthread_rwlock_rdlock.m4 b/m4/pthread_rwlock_rdlock.m4
new file mode 100644
index ..3c1d6456
--- /dev/null
+++ b/m4/pthread_rwlock_rdlock.m4
@@ -0,0 +1,165 @@
+# pthread_rwlock_rdlock.m4 serial 2
+dnl Copyright (C) 2017-2019 Free Software Foundation, Inc.
+dnl This file is free software; the Free Software Foundation
+dnl gives unlimited permission to copy and/or distribute it,
+dnl with or without modifications, as long as this notice is preserved.
+
+dnl From Bruno Haible.
+dnl Inspired by
+dnl 
https://github.com/linux-test-project/ltp/blob/master/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_rdlock/2-2.c
+dnl by Intel Corporation.
+
+dnl Test whether in a situation where
+dnl   - an rwlock is taken by a reader and has a writer waiting,
+dnl   - an additional reader requests the lock,
+dnl   - the waiting writer and the requesting reader threads have the same
+dnl priority,
+dnl the requesting reader thread gets blocked, so that at some point the
+dnl waiting writer can acquire the lock.
+dnl Without such a guarantee, when there a N readers and each of the readers
+dnl spends more than 1/Nth of the time with the lock held, there is a high
+dnl probability that the waiting writer will not get the lock in a given finite
+dnl time, a phenomenon called "writer starvation".
+dnl Without such a guarantee, applications have a hard time avoiding writer
+dnl starvation.
+dnl
+dnl POSIX:2017 makes this requirement only for implementations that support TPS
+dnl (Thread Priority Scheduling) and only for the scheduling policies 
SCHED_FIFO
+dnl and SCHED_RR, see
+dnl 
http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_rwlock_rdlock.html
+dnl but this test verifies the guarantee regardless of TPS and regardless of
+dnl scheduling policy.
+dnl Glibc currently does not provide this guarantee, see
+dnl https://sourceware.org/bugzilla/show_bug.cgi?id=13701
+AC_DEFUN([gl_PTHREAD_RWLOCK_RDLOCK_PREFER_WRITER],
+[
+  AC_REQUIRE([gl_THREADLIB_EARLY])
+  AC_CACHE_CHECK([whether pthread_rwlock_rdlock prefers a writer to a reader],
+[gl_cv_pthread_rwlock_rdlock_prefer_writer],
+[save_LIBS="$LIBS"
+ LIBS="$LIBS $LIBMULTITHREAD"
+ AC_RUN_IFELSE(
+   [AC_LANG_SOURCE([[
+#include 
+#include 
+#include 
+#include 
+
+#define SUCCEED() exit (0)
+#define FAILURE() exit (1)
+#define UNEXPECTED(n) (exit (10 + (n)))
+
+/* The main thread creates the waiting writer and the requesting reader threads
+   in the default way; this guarantees that they have the same priority.
+   We can reuse the main thread as first reader thread.  */
+
+static pthread_rwlock_t lock;
+static pthread_t reader1;
+static pthread_t writer;
+static pthread_t reader2;
+static pthread_t timer;
+/* Used to pass control from writer to reader2 and from reader2 to timer,
+   as in a relay race.
+   Passing control from one running thread to another running thread
+   is most likely faster than to create the second thread.  */
+static pthread_mutex_t baton;
+
+static void *
+timer_func (void *ignored)
+{
+  /* Step 13 (can be before or after step 12):
+ The timer thread takes the baton, then waits a moment to make sure
+ it can tell whether the second reader thread is blocked at step 12.  */
+  if (pthread_mutex_lock ())
+UNEXPECTED (13);
+  usleep (10);
+  /* By the time we get here, it's clear that the second reader thread is
+ blocked at step 12.  This is the desired behaviour.  */
+  SUCCEED ();
+}
+
+static void *
+reader2_func (void *ignored)
+{
+  int err;
+
+  /* Step 8 (can be before or after step 7):
+ The second reader 

AddressSanitizer: heap-use-after-free on (...) in rl_do_undo ../../../bash-devel/lib/readline/undo.c:188

[Eduardo A . Bustamante López]

I found another issue in rl_do_undo, but I haven't been successful in figuring 
out how it happens.

I've been working with the `devel' branch, commit 
`8a9718cfc93958b34e205d0507c3bbf64cba6db5'

Here's how I built the binaries I use below:

debian@debian-fuzz:~/tmp$ cat ~/build.sh 
#!/bin/bash
mkdir build-devel{,-asan,-gdb}
(cd build-devel  && CFLAGS='' CC='afl-clang-fast'   
   ../bash-devel/configure --silent --without-bash-malloc && make 
-sj$(nproc))
(cd build-devel-asan && CFLAGS='-O0 -ggdb -fno-omit-frame-pointer 
-fsanitize=address ' ../bash-devel/configure --silent --without-bash-malloc && 
make -sj$(nproc))
(cd build-devel-gdb  && CFLAGS='-O0 -ggdb -fno-omit-frame-pointer ' 
   ../bash-devel/configure --silent --without-bash-malloc && make 
-sj$(nproc))

This is the input to `read -e':

debian@debian-fuzz:~/tmp$ base64 < o
MBgFEBAQDhUwEBgoHx8wEDAYRRQbEDAYBTAYRQ4=

debian@debian-fuzz:~/tmp$ cat -A o
0^X^E^P^P^P^N^U0^P^X(^_^_0^P0^XE^T^[^P0^X^E0^XE^N


My poor attempt of a trace under GDB:

debian@debian-fuzz:~/tmp$ gdb --batch --command=trace_rl_undo.gdb --args 
~/build-devel-gdb/bash --noprofile --norc -c 'EDITOR=: PATH= read -e < o' > 
out.txt 2>&1
(no output)
---

dualbus@system76-pc:~/src/dualbus/bash-fuzzing/tmp/tmp$ cat out.txt 
0
0
/tmp/bash-fc.9lfkNE: line 1: 0: No such file or directory
000
000
/tmp/bash-fc.HkVuNJ: line 1: 000: No such file or directory
readline: maximum macro execution nesting level exceeded

munmap_chunk(): invalid pointer
---


dualbus@system76-pc:~/src/dualbus/bash-fuzzing/tmp/tmp$ cat gdb.txt 
Temporary breakpoint 1 at 0x2eea9: file ../bash-devel/shell.c, line 392.

Temporary breakpoint 1, main (argc=5, argv=0x7fffdfd8, env=0x7fffe008) 
at ../bash-devel/shell.c:392
392   code = setjmp_nosigs (top_level);
Breakpoint 2 at 0x5565004c: file ../../../bash-devel/lib/readline/undo.c, 
line 177.
[Detaching after fork from child process 588]

Breakpoint 2, rl_do_undo () at ../../../bash-devel/lib/readline/undo.c:177
(... snip ...)

Breakpoint 2, rl_do_undo () at ../../../bash-devel/lib/readline/undo.c:177
177   start = end = waiting_for_begin = 0;
$625 = "~~~ print_rl_undo_list>"
$626 = 0
$627 = (UNDO_LIST *) 0x556f9e50
$628 = (struct undo_list *) 0x556fa6f0
$629 = 0x0
$630 = 1
$631 = (struct undo_list *) 0x556fa6f0
$632 = (struct undo_list *) 0x0
$633 = 0x0
$634 = "~~~ print_the_history>"
$635 = 0
$636 = (HIST_ENTRY *) 0x556fa720
$637 = 0x556debd0 "000"
$638 = 0
$639 = (UNDO_LIST *) 0x556fa6f0
$640 = (struct undo_list *) 0x0
$641 = 0x0
$642 = 1
$643 = (HIST_ENTRY *) 0x556fa7b0
$644 = 0x556f89a0 "0"
$645 = 0
$646 = (UNDO_LIST *) 0x556fa330
$647 = (struct undo_list *) 0x556fa3d0
$648 = 0x0
$649 = 1
$650 = (struct undo_list *) 0x556fa3d0
$651 = (struct undo_list *) 0x556f9f00
$652 = 0x556de940 "\360\237oUUU"
$653 = 2
$654 = (struct undo_list *) 0x556f9f00
$655 = (struct undo_list *) 0x0
$656 = 0x556fa750 " "
$657 = 2
$658 = (HIST_ENTRY *) 0x556fa2c0
$659 = 0x556f85f0 ""
$660 = 0
$661 = (UNDO_LIST *) 0x556f90e0
$662 = (struct undo_list *) 0x0
$663 = 0x0
$664 = 3
$665 = (HIST_ENTRY *) 0x556f9130
$666 = 0x556fa6d0 ""
$667 = 0
$668 = (UNDO_LIST *) 0x556f90b0
$669 = (struct undo_list *) 0x0
$670 = 0x0
#0  rl_do_undo () at ../../../bash-devel/lib/readline/undo.c:177
#1  0x556504fd in rl_undo_command (count=1, key=31) at 
../../../bash-devel/lib/readline/undo.c:358
#2  0x55633b75 in _rl_dispatch_subseq (key=31, map=0x5569f200 
, got_subseq=0) at 
../../../bash-devel/lib/readline/readline.c:852
#3  0x556338ec in _rl_dispatch (key=1433265360, map=0x5569f200 
) at ../../../bash-devel/lib/readline/readline.c:798
#4  0x5563356f in readline_internal_char () at 
../../../bash-devel/lib/readline/readline.c:632
#5  0x556335ca in readline_internal_charloop () at 
../../../bash-devel/lib/readline/readline.c:659
#6  0x556335ea in readline_internal () at 
../../../bash-devel/lib/readline/readline.c:671
#7  0x55633008 in readline (prompt=0x556767bc "") at 
../../../bash-devel/lib/readline/readline.c:377
#8  0x5560c4cc in edit_line (p=0x556767bc "", itext=0x0) at 
../../bash-devel/builtins/../../bash-devel/builtins/read.def:1107
#9  0x5560b2b2 in read_builtin (list=0x0) at 
../../bash-devel/builtins/../../bash-devel/builtins/read.def:566
#10 0x555a3bc5 in execute_builtin (builtin=0x5560a45d 
, words=0x556c1030, flags=0, subshell=0) at 
../bash-devel/execute_cmd.c:4709
#11 0x555a4ae9 in execute_builtin_or_function (words=0x556c1030, 
builtin=0x5560a45d , var=0x0, redirects=0x556bf720, 
fds_to_close=0x556c0cb0, flags=0) at ../bash-devel/execute_cmd.c:5217
#12 

Re: RETURN trap will inherit last RETURN trap cmd though set +T

[Eduardo A . Bustamante López]

On Wed, Jan 23, 2019 at 03:59:12PM +, Shen Herbert wrote:
(...)
> Repeat-By:
> Run this script:
> ```
> set +T
> one { trap 'echo in one' RETURN; }
> all {
> trap 'echo in all' RETURN
> one
> }
> all
> ```

Hm, the script above is a bit odd, these aren't syntactically valid function 
definitions:

  dualbus@system76-pc:~$ cat > file
  set +T
  one { trap 'echo in one' RETURN; }
  all {
  trap 'echo in all' RETURN
  one
  }
  all
  
  dualbus@system76-pc:~$ bash file 
  file: line 2: syntax error near unexpected token `}'
  file: line 2: `one { trap 'echo in one' RETURN; }'


Do you have another example we can use?

Re: bug: illegal function name?

[Eduardo A . Bustamante López]

On Mon, Jan 21, 2019 at 09:26:59AM +0700, Robert Elz wrote:
(...)
> I think his point is that if unset "unset f" (no flags) works to unset 
> function f, if f is not a (set) variable, then it should work every time
> "f" is not a set variable, not only the times when the word "f" happens
> to be of the correct syntax to be a variable name, but happens not
> to be.
> 
> I think that is a good point.
> 
> In the NetBSD sh we avoid that issue completely, the only way to
> unset a function is with "unset -f", a simple "unset f" only ever
> unsets variables.

I think that's a good point too. But how would you fix this particular issue?

Changing the behavior of `unset f' to only ever unset variables means
potentially breaking existing scripts. Is the inconsistency reported severe
enough to make this change?

Re: [bug-bash] $RANDOM not Cryptographically secure pseudorandom number generator

[Eduardo A . Bustamante López]

On Sun, Jan 20, 2019 at 05:22:12PM -0500, Chet Ramey wrote:
> On 1/20/19 4:54 PM, Chet Ramey wrote:
> 
> >> As an aside, I can confirm the findings of a performance difference
> >> between 4.4 and 5.0 when running the script provided earlier in the
> >> discussion. At first glance it seems to be due to the switch from the
> >> old LCG to the current MINSTD RNG, 
(...)
> So I ran a quick test.
> 
> $ ./bash ./x3
> iterations: 100
> BASH_VERSION: 5.0.2(4)-maint
> time: 9.684
> $ ../bash-5.0/bash ./x3
> iterations: 100
> BASH_VERSION: 5.0.0(1)-release
> time: 9.749
> $ ../bash-5.0-patched/bash ./x3
> iterations: 100
> BASH_VERSION: 5.0.2(3)-release
> time: 9.840
> $ ../bash-4.4-patched/bash ./x3
> iterations: 100
> BASH_VERSION: 4.4.23(7)-release
> time: 11.365
> $ ../bash-4.4-patched/bash ./x3
> iterations: 100
> BASH_VERSION: 4.4.23(7)-release
> time: 11.235
> jenna.local(1)
> 
> Where the script is Eduardo's iterator that just expands $RANDOM
> N times.
> 
> The random number generator has been the same since bash-4.0.

I'm sorry, my tests were wrong. I built bash using the default `./configure'
behavior for the `devel' branch, which I always forget, uses the internal
allocator with debugging enabled, and thus, all of my times were off due to the
additional malloc overhead.

I rebuilt it with `../bash/configure --without-bash-malloc', which causes it to
use the system's allocator, and surely enough, the timings make more sense now:

(`build-bash-devel-malloc' is `configure --with-bash-malloc',
 `build-bash-devel' is `configure --without-bash-malloc')

| dualbus@system76-pc:~/src/gnu/build-bash-devel-malloc$ ./bash ~/test-speed.sh 
| iterations: 100
| BASH_VERSION: 5.0.0(1)-maint
| time: 8.765
| 
| dualbus@system76-pc:~/src/gnu/build-bash-devel-malloc$ 
../build-bash-devel/bash ~/test-speed.sh 
| iterations: 100
| BASH_VERSION: 5.0.0(1)-maint
| time: 3.431
| 
| dualbus@system76-pc:~/src/gnu/build-bash-devel-malloc$ bash ~/test-speed.sh 
| iterations: 100
| BASH_VERSION: 5.0.0(1)-release
| time: 3.435
| 
| dualbus@system76-pc:~/src/gnu/build-bash-4.4$ ./bash ~/test-speed.sh 
| iterations: 100
| BASH_VERSION: 4.4.0(1)-release
| time: 3.443

Re: bug: illegal function name?

[Eduardo A . Bustamante López]

On Sun, Jan 20, 2019 at 06:26:43PM +0300, Andrey Butirsky wrote:
> Andreas, I know it will work with the '-f' flag.
> But for others function names, the '-f' unset flag is not required.
> Moreover, it seem confronts with Open Group Base Specification.
> So I consider it as a bug still.
> 

It is intentional. If you want bash to run in POSIX mode, you should use the 
`POSIXLY_CORRECT' variable or the `--posix'
flag.

| dualbus@system76-pc:~$ POSIXLY_CORRECT=y bash -c 'function 1a() { :; }'
| bash: `1a': not a valid identifier

And as Andreas mentioned, in general you should use `unset -f' if you want to 
operate on functions. Otherwise you risk
conflict with a variable name.

Re: realloc: start and end chunk sizes differ - rl_extend_line_buffer in lib/readline/util.c

[Eduardo A . Bustamante López]

On Mon, Jan 07, 2019 at 01:30:46AM -0800, Eduardo A. Bustamante López wrote:
(...)
> I missed a spot, updated patch:
> 
> diff -ruN bash-5.0-rc1.orig/lib/readline/undo.c 
> bash-5.0-rc1/lib/readline/undo.c
> --- bash-5.0-rc1.orig/lib/readline/undo.c   2019-01-03 13:14:43.428392927 
> -0800
> +++ bash-5.0-rc1/lib/readline/undo.c2019-01-07 01:28:08.288255650 -0800
> @@ -196,6 +196,8 @@
> /* Undoing deletes means inserting some text. */
> case UNDO_DELETE:
>   rl_point = start;
> + if (rl_point > rl_end)
> +   rl_point = rl_end;
>   rl_insert_text (rl_undo_list->text);
>   xfree (rl_undo_list->text);
>   break;
> @@ -204,6 +206,8 @@
> case UNDO_INSERT:
>   rl_delete_text (start, end);
>   rl_point = start;
> + if (rl_point > rl_end)
> +   rl_point = rl_end;
>   break;
>  
> /* Undoing an END means undoing everything 'til we get to a BEGIN. */

Missed 2 actually, this is another case, in lib/readline/search.c now:

Hardware watchpoint 1: (rl_point > 0 && rl_point > rl_end + 1)

Old value = 0
New value = 1
_rl_nsearch_abort (cxt=0x557caf08) at 
../../../bash-5.0-rc1/lib/readline/search.c:258
258   rl_mark = cxt->save_mark;
(gdb) bt
#0  _rl_nsearch_abort (cxt=0x557caf08) at 
../../../bash-5.0-rc1/lib/readline/search.c:258
#1  0x5563fc73 in _rl_nsearch_dispatch (cxt=0x557caf08, c=3) at 
../../../bash-5.0-rc1/lib/readline/search.c:300
#2  0x5563fe2d in noninc_search (dir=1, pchar=0) at 
../../../bash-5.0-rc1/lib/readline/search.c:389
#3  0x5563fea9 in rl_noninc_forward_search (count=1, key=110) at 
../../../bash-5.0-rc1/lib/readline/search.c:405
#4  0x55639ed4 in _rl_dispatch_subseq (key=110, map=0x556ac220 
, got_subseq=0) at 
../../../bash-5.0-rc1/lib/readline/readline.c:852
#5  0x5563a399 in _rl_dispatch_subseq (key=27, map=0x556ab200 
, got_subseq=0) at 
../../../bash-5.0-rc1/lib/readline/readline.c:986
#6  0x55639c4b in _rl_dispatch (key=-135542668, map=0x556ab200 
) at ../../../bash-5.0-rc1/lib/readline/readline.c:798
#7  0x556398ce in readline_internal_char () at 
../../../bash-5.0-rc1/lib/readline/readline.c:632
#8  0x55639929 in readline_internal_charloop () at 
../../../bash-5.0-rc1/lib/readline/readline.c:659
#9  0x55639949 in readline_internal () at 
../../../bash-5.0-rc1/lib/readline/readline.c:671
#10 0x55639367 in readline (prompt=0x55680f84 "") at 
../../../bash-5.0-rc1/lib/readline/readline.c:377
#11 0x55611bcf in edit_line (p=0x55680f84 "", itext=0x0) at 
../../bash-5.0-rc1/builtins/../../bash-5.0-rc1/builtins/read.def:1107
#12 0x556108f8 in read_builtin (list=0x0) at 
../../bash-5.0-rc1/builtins/../../bash-5.0-rc1/builtins/read.def:566
#13 0x555a5afa in execute_builtin (builtin=0x5560fa73 
, words=0x55764cc8, flags=0, subshell=0) at 
../bash-5.0-rc1/execute_cmd.c:4706
#14 0x555a6aa2 in execute_builtin_or_function (words=0x55764cc8, 
builtin=0x5560fa73 , var=0x0, redirects=0x55764b08, 
fds_to_close=0x55764a28, flags=0)
at ../bash-5.0-rc1/execute_cmd.c:5214
#15 0x555a5365 in execute_simple_command 
(simple_command=0x55764908, pipe_in=-1, pipe_out=-1, async=0, 
fds_to_close=0x55764a28) at ../bash-5.0-rc1/execute_cmd.c:4476
#16 0x5559e9f4 in execute_command_internal (command=0x557648c8, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x55764a28) at 
../bash-5.0-rc1/execute_cmd.c:842
#17 0x5560858a in parse_and_execute (string=0x5574c2a8 "PATH= read 
-e < bar", from_file=0x556690f0 "-c", flags=4) at 
../../bash-5.0-rc1/builtins/evalstring.c:436
#18 0x5558564a in run_one_command (command=0x7fffe2a6 "PATH= read 
-e < bar") at ../bash-5.0-rc1/shell.c:1426
#19 0x55584789 in main (argc=5, argv=0x7fffe008, 
env=0x7fffe038) at ../bash-5.0-rc1/shell.c:741
(gdb) l
253 _rl_nsearch_abort (_rl_search_cxt *cxt)
254 {
255   rl_maybe_unsave_line ();
256   rl_clear_message ();
257   rl_point = cxt->save_point;
258   rl_mark = cxt->save_mark;
259   rl_restore_prompt ();
260
261   RL_UNSETSTATE (RL_STATE_NSEARCH);
262 }
(gdb) c
Continuing.

malloc: unknown:0: assertion botched
malloc: 0x55769408: allocated: last allocated from unknown:0
realloc: start and end chunk sizes differ
Aborting...
Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.


diff -ruN bash-5.0-rc1.orig/lib/readline/search.c 
bash-5.0-rc1/lib/readline/search.c
--- bash-5.0-rc1.orig/lib/readline/search.c 2019-01-03 13:14:43.428392927 

Re: Segmentation fault in lib/readline/undo.c - rl_do_undo

[Eduardo A . Bustamante López]

On Mon, Jan 07, 2019 at 01:16:05AM -0800, Eduardo A. Bustamante López wrote:
> I found this with AFL. I think it's related to the problem reported here:
> http://lists.nongnu.org/archive/html/bug-bash/2018-09/msg00045.html
> 
> debian@debian-fuzz:/mnt$ cat -A rl_do_undo
> ^RM-CM-!M-CM-CM-!M-C^[.^[^[0^P^@^P^Q0^[-^P^Q0^[^W0^@0&/^[^[^[--^W^_~0^@0^@-^L^D^@^@'/^[B^@0^B^@M-
>  
> ^[^[M-^T^[M-mM-^?^[F-^W^_0^[M-^@0^P^@^@^@^@^D^I^@^[M-UM-UM-UM-NM-U^@M-^@^@M-=$^@01^@01^["0^?M-^?M-^?M-^?0M-r0^@'0M-^?^@^@^@M-CM-CM-!M-C^[.^[^[--^W00^P^@00(-^P^Q;^[-^P^Q0^[^W0^@n&/^[^[^[--^W^_~0^@0^@-^L^D^@^@'/^[B^@M-^T^B^@M-
>  
> ^[^[M-^T^[M-mM-^?^[F-^W^_0^[M-^@0^P^@^@^@^@^D^I^@^[M-UM-UM-UM-NM-U^@M-^@^@M-=$^@01^@01^["0^?M-^?M-^?M-^?0M-r0^@'0M-^?^@^@^@@^N\0^[11#0-^P^@^@^@^@^D^I^@^[M-UM-=M-UM-NM-U^@M-^@^@M-=$^@J^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^PM-eM-KM-YM-@M-nM-n^ZM-{0M-xM-|^@}}^L0#^A^Cd^\#^@^X^E^X^E^@M-^?M-^?^[^I^I^@^@^@M-^?\^O^@000M-^R00M-,0^@^@M-^?\^O^@qq0M-^Dq^@0^P^I^[^I^I^E^@M-^?M-^?0M-v^P^P^P^P^P^P^PM-eM-KM-YM-@M-nM-n^ZM-{0M-xM-|^@}}^L0#^A^Cd^\#^@^X^E^X^E^@M-^?M-^?^[^I^@^@M-^@^@0^@^@^@M-^?\^O^@0^?0M-^R00M-,0^@^@M-^?\^O^@0^@0^P^I^[^I^Iu000^E^@M-^?M-^?0M-vM-Q^A^@0^P^I^]0^I00^@^@^@M-^?\^O^GM-^?\^Oq0q^[^I^I^I^@^@M-h^C@^N\0^[11#0-^P^@^@^@^@^D^I^@^[M-UM-UM-UM-NM-U^@M-^@^@M-=$^@0^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^PM-eM-KM-YM-@M-nM-n^ZM-{0M-xM-|^@00^L0#^A^Cd^\#^@^X^E^X^E^@M-^?M-^?^[^I^I^@^@^@M-^?\^O^@000M-^R00M-,0^@^@M-^?\^O^@qq0M-^Dq^@0^P^I^[^I^Iu000^E^@M-^?M-^?0M-v^P^P^P^P^P^P^PM-eM-KM-YM-@M-nM-n^ZM-{0M-xM-|^@}}^L0#^A^Cd^\#^@^X^E^X^E^@M-^?M-^?^[^I^I^@^@^@M-^?\^O^@00M-^?M-^R00M-,0^@^@M-^?\^O^@0^@0^P^I^[^I^I^E^@M-^?M-^?0M-vM-Q^A^@0^P^I^]0^I00^@^@^@M-^?\^O^GM-^?\^Oq0q^[^I^I^I^@^@M-h^C^E^@000M-v
> 

Heh, I forgot to minimize the test case:

debian@debian-fuzz:/mnt$ cat -A rl_do_undo 
0^X^E0^P^P^P^X^E\^O^P^P

debian@debian-fuzz:/mnt$ base64 < rl_do_undo 
MBgFMBAQEBgFXA8QEA==

Also, running it with ASAN provides more information:

debian@debian-fuzz:/mnt$ ~/build-asan/bash --noprofile --norc -c 'PATH= read -e 
< rl_do_undo'
hi
0
/home/debian/build-asan/bash: emacs: No such file or directory
0
/home/debian/build-asan/bash: emacs: No such file or directory
\
0
=
==29290==ERROR: AddressSanitizer: heap-use-after-free on address 0x60304018 
at pc 0x007d7508 bp 0x7ffe9530f5c0 sp 0x7ffe9530f5b8
READ of size 4 at 0x60304018 thread T0
#0 0x7d7507 in rl_do_undo 
/home/debian/build-asan/lib/readline/../../../bash-5.0-rc1/lib/readline/undo.c:188:25
#1 0x7d8682 in rl_revert_line 
/home/debian/build-asan/lib/readline/../../../bash-5.0-rc1/lib/readline/undo.c:341:2
#2 0x767dd3 in readline_internal_teardown 
/home/debian/build-asan/lib/readline/../../../bash-5.0-rc1/lib/readline/readline.c:471:7
#3 0x7678b0 in readline_internal 
/home/debian/build-asan/lib/readline/../../../bash-5.0-rc1/lib/readline/readline.c:672:11
#4 0x7676ba in readline 
/home/debian/build-asan/lib/readline/../../../bash-5.0-rc1/lib/readline/readline.c:377:11
#5 0x6fe637 in edit_line 
/home/debian/build-asan/builtins/../../bash-5.0-rc1/builtins/../../bash-5.0-rc1/builtins/read.def:1107:9
#6 0x6fa7d5 in read_builtin 
/home/debian/build-asan/builtins/../../bash-5.0-rc1/builtins/../../bash-5.0-rc1/builtins/read.def:566:16
#7 0x592620 in execute_builtin 
/home/debian/build-asan/../bash-5.0-rc1/execute_cmd.c:4706:13
#8 0x5910a7 in execute_builtin_or_function 
/home/debian/build-asan/../bash-5.0-rc1/execute_cmd.c:5214:14
#9 0x579877 in execute_simple_command 
/home/debian/build-asan/../bash-5.0-rc1/execute_cmd.c:4476:13
#10 0x5701d2 in execute_command_internal 
/home/debian/build-asan/../bash-5.0-rc1/execute_cmd.c:842:4
#11 0x6dd393 in parse_and_execute 
/home/debian/build-asan/builtins/../../bash-5.0-rc1/builtins/evalstring.c:436:17
#12 0x51d4f4 in run_one_command 
/home/debian/build-asan/../bash-5.0-rc1/shell.c:1426:12
#13 0x518ec9 in main /home/debian/build-asan/../bash-5.0-rc1/shell.c:741:7
#14 0x7f194ff8c09a in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
#15 0x43fa39 in _start (/home/debian/build-asan/bash+0x43fa39)

0x60304018 is located 24 bytes inside of 32-byte region 
[0x60304000,0x60304020)
freed by thread T0 here:
#0 0x4e7502 in __interceptor_free (/home/debian/build-asan/bash+0x4e7502)
#1 0x6c2bbf in xfree /home/debian/build-asan/../bash-5.0-rc1/xmalloc.c:150:5
#2 0x7d7036 in _rl_free_undo_list 
/home/debian/build-asan/lib/readline/../../../bash-5.0-rc1/lib/readline/undo.c:113:7
#3 0x7d7070 in rl_free_undo_list 
/home/debian/build-asan/lib/readline/../../../bash-5.0-rc1/lib/readline/undo.c:124:3
#4 0x767e6a in readline_internal_teardown 
/home/debian/build-asan/lib/readline/../../../bash-5.0-rc1/lib/readline/readline.c:485:5
#5 0x7678b0 in readline_internal 
/home/debian/buil

Buffer overflow in string_extract_double_quoted - subst.c

[Eduardo A . Bustamante López]

Found by fuzzing `read -e' with AFL:

debian@debian-fuzz:/mnt$ cat -A dispose_word 
"^[^EM-b^_M-u$$(M-J^^_^Q$
^[^E

debian@debian-fuzz:/mnt$ base64 < dispose_word 
IhsF4h/1JCQoyl4fEQobBQ==

debian@debian-fuzz:/mnt$ LC_ALL=zh_CN.gbk ~/build-gdb/bash --noprofile --norc 
-c 'PATH= read -e < dispose_word' 
hi
"��$$(
TRACE: pid 15530: xparse_dolparen:0: base[5] != RPAREN (40), base = `"��$$(
'
TRACE: pid 15530: xparse_dolparen:0: *indp (5) < orig_ind (6), orig_string = `
'

malloc: ../bash-5.0-rc1/dispose_cmd.c:249: assertion botched
malloc: 0x55d956dc4de8: allocated: last allocated from 
../bash-5.0-rc1/subst.c:866
free: start and end chunk sizes differ
Aborting...Aborted

(...)

Aborting...
Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x77df4535 in __GI_abort () at abort.c:79
#2  0x555b39b5 in programming_error (format=0x55686bd8 "free: start 
and end chunk sizes differ") at ../bash-5.0-rc1/error.c:175
#3  0x5566523d in xbotch (mem=0x55765da8, e=8, s=0x55686bd8 
"free: start and end chunk sizes differ", file=0x5566c268 
"../bash-5.0-rc1/dispose_cmd.c", line=249)
at ../../../bash-5.0-rc1/lib/malloc/malloc.c:354
#4  0x5566648e in internal_free (mem=0x55765da8, 
file=0x5566c268 "../bash-5.0-rc1/dispose_cmd.c", line=249, flags=1) at 
../../../bash-5.0-rc1/lib/malloc/malloc.c:960
#5  0x55667006 in sh_free (mem=0x55765da8, file=0x5566c268 
"../bash-5.0-rc1/dispose_cmd.c", line=249) at 
../../../bash-5.0-rc1/lib/malloc/malloc.c:1321
#6  0x556001f4 in sh_xfree (string=0x55765da8, file=0x5566c268 
"../bash-5.0-rc1/dispose_cmd.c", line=249) at ../bash-5.0-rc1/xmalloc.c:223
#7  0x5559d860 in dispose_word (w=0x55761da8) at 
../bash-5.0-rc1/dispose_cmd.c:249
#8  0x555d6ef5 in expand_word_internal (word=0x55761e08, quoted=0, 
isexp=0, contains_dollar_at=0x0, expanded_something=0x0) at 
../bash-5.0-rc1/subst.c:10189
#9  0x555c84dd in call_expand_word_internal (w=0x55761e08, q=0, 
i=0, c=0x0, e=0x0) at ../bash-5.0-rc1/subst.c:3684
#10 0x555c8b94 in expand_word (word=0x55761e08, quoted=0) at 
../bash-5.0-rc1/subst.c:3978
#11 0x555f35b6 in shell_expand_line (count=1, ignore=5) at 
../bash-5.0-rc1/bashline.c:2755
#12 0x55639ed4 in _rl_dispatch_subseq (key=5, map=0x556ac220 
, got_subseq=0) at 
../../../bash-5.0-rc1/lib/readline/readline.c:852
#13 0x5563a399 in _rl_dispatch_subseq (key=27, map=0x556ab200 
, got_subseq=0) at 
../../../bash-5.0-rc1/lib/readline/readline.c:986
#14 0x55639c4b in _rl_dispatch (key=-136275877, map=0x556ab200 
) at ../../../bash-5.0-rc1/lib/readline/readline.c:798
#15 0x556398ce in readline_internal_char () at 
../../../bash-5.0-rc1/lib/readline/readline.c:632
#16 0x55639929 in readline_internal_charloop () at 
../../../bash-5.0-rc1/lib/readline/readline.c:659
#17 0x55639949 in readline_internal () at 
../../../bash-5.0-rc1/lib/readline/readline.c:671
#18 0x55639367 in readline (prompt=0x55680f84 "") at 
../../../bash-5.0-rc1/lib/readline/readline.c:377
#19 0x55611bcf in edit_line (p=0x55680f84 "", itext=0x0) at 
../../bash-5.0-rc1/builtins/../../bash-5.0-rc1/builtins/read.def:1107
#20 0x556108f8 in read_builtin (list=0x0) at 
../../bash-5.0-rc1/builtins/../../bash-5.0-rc1/builtins/read.def:566
#21 0x555a5afa in execute_builtin (builtin=0x5560fa73 
, words=0x55761e68, flags=0, subshell=0) at 
../bash-5.0-rc1/execute_cmd.c:4706
#22 0x555a6aa2 in execute_builtin_or_function (words=0x55761e68, 
builtin=0x5560fa73 , var=0x0, redirects=0x55761bc8, 
fds_to_close=0x55761ba8, flags=0)
at ../bash-5.0-rc1/execute_cmd.c:5214
#23 0x555a5365 in execute_simple_command 
(simple_command=0x55761a88, pipe_in=-1, pipe_out=-1, async=0, 
fds_to_close=0x55761ba8) at ../bash-5.0-rc1/execute_cmd.c:4476
#24 0x5559e9f4 in execute_command_internal (command=0x55761a48, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x55761ba8) at 
../bash-5.0-rc1/execute_cmd.c:842
#25 0x5560858a in parse_and_execute (string=0x55761688 "PATH= read 
-e < dispose_word", from_file=0x556690f0 "-c", flags=4) at 
../../bash-5.0-rc1/builtins/evalstring.c:436
#26 0x5558564a in run_one_command (command=0x7fffe28c "PATH= read 
-e < dispose_word") at ../bash-5.0-rc1/shell.c:1426
#27 0x55584789 in main (argc=5, argv=0x7fffdfe8, 
env=0x7fffe018) at ../bash-5.0-rc1/shell.c:741

(gdb) frame 7
#7  0x5559d860 in dispose_word (w=0x55761da8) at 
../bash-5.0-rc1/dispose_cmd.c:249
249   FREE (w->word);
(gdb) l
244 /* How to free a WORD_DESC. */
245 void
246 

Re: realloc: start and end chunk sizes differ - rl_extend_line_buffer in lib/readline/util.c

[Eduardo A . Bustamante López]

On Sun, Jan 06, 2019 at 08:46:33PM -0800, Eduardo A. Bustamante López wrote:
> On Sun, Jan 06, 2019 at 07:18:27PM -0800, Eduardo A. Bustamante López wrote:
> (...)
> > malloc: unknown:0: assertion botched
> > malloc: 0x55769408: allocated: last allocated from unknown:0
> > realloc: start and end chunk sizes differ
> 
> OK, I think I know what the problem is.
(...)
> I still don't know how to trigger this with "human" input, but I think the
> problem is that rl_point should be bounded by the value of rl_end, thus the
> following patch makes the problem go away:
> 
> dualbus@system76-pc:~/src/gnu/bash$ git diff -- lib/readline/undo.c
> diff --git a/lib/readline/undo.c b/lib/readline/undo.c
> index ae65d380..12952555 100644
> --- a/lib/readline/undo.c
> +++ b/lib/readline/undo.c
> @@ -196,6 +196,8 @@ rl_do_undo (void)
> /* Undoing deletes means inserting some text. */
> case UNDO_DELETE:
>   rl_point = start;
> + if (rl_point > rl_end)
> +   rl_point = rl_end;
>   rl_insert_text (rl_undo_list->text);
>   xfree (rl_undo_list->text);
>   break;

I missed a spot, updated patch:

diff -ruN bash-5.0-rc1.orig/lib/readline/undo.c bash-5.0-rc1/lib/readline/undo.c
--- bash-5.0-rc1.orig/lib/readline/undo.c   2019-01-03 13:14:43.428392927 
-0800
+++ bash-5.0-rc1/lib/readline/undo.c2019-01-07 01:28:08.288255650 -0800
@@ -196,6 +196,8 @@
/* Undoing deletes means inserting some text. */
case UNDO_DELETE:
  rl_point = start;
+ if (rl_point > rl_end)
+   rl_point = rl_end;
  rl_insert_text (rl_undo_list->text);
  xfree (rl_undo_list->text);
  break;
@@ -204,6 +206,8 @@
case UNDO_INSERT:
  rl_delete_text (start, end);
  rl_point = start;
+ if (rl_point > rl_end)
+   rl_point = rl_end;
  break;
 
/* Undoing an END means undoing everything 'til we get to a BEGIN. */

Segmentation fault in lib/readline/undo.c - rl_do_undo

[Eduardo A . Bustamante López]

I found this with AFL. I think it's related to the problem reported here:
http://lists.nongnu.org/archive/html/bug-bash/2018-09/msg00045.html

debian@debian-fuzz:/mnt$ cat -A rl_do_undo
^RM-CM-!M-CM-CM-!M-C^[.^[^[0^P^@^P^Q0^[-^P^Q0^[^W0^@0&/^[^[^[--^W^_~0^@0^@-^L^D^@^@'/^[B^@0^B^@M-
 
^[^[M-^T^[M-mM-^?^[F-^W^_0^[M-^@0^P^@^@^@^@^D^I^@^[M-UM-UM-UM-NM-U^@M-^@^@M-=$^@01^@01^["0^?M-^?M-^?M-^?0M-r0^@'0M-^?^@^@^@M-CM-CM-!M-C^[.^[^[--^W00^P^@00(-^P^Q;^[-^P^Q0^[^W0^@n&/^[^[^[--^W^_~0^@0^@-^L^D^@^@'/^[B^@M-^T^B^@M-
 
^[^[M-^T^[M-mM-^?^[F-^W^_0^[M-^@0^P^@^@^@^@^D^I^@^[M-UM-UM-UM-NM-U^@M-^@^@M-=$^@01^@01^["0^?M-^?M-^?M-^?0M-r0^@'0M-^?^@^@^@@^N\0^[11#0-^P^@^@^@^@^D^I^@^[M-UM-=M-UM-NM-U^@M-^@^@M-=$^@J^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^PM-eM-KM-YM-@M-nM-n^ZM-{0M-xM-|^@}}^L0#^A^Cd^\#^@^X^E^X^E^@M-^?M-^?^[^I^I^@^@^@M-^?\^O^@000M-^R00M-,0^@^@M-^?\^O^@qq0M-^Dq^@0^P^I^[^I^I^E^@M-^?M-^?0M-v^P^P^P^P^P^P^PM-eM-KM-YM-@M-nM-n^ZM-{0M-xM-|^@}}^L0#^A^Cd^\#^@^X^E^X^E^@M-^?M-^?^[^I^@^@M-^@^@0^@^@^@M-^?\^O^@0^?0M-^R00M-,0^@^@M-^?\^O^@0^@0^P^I^[^I^Iu000^E^@M-^?M-^?0M-vM-Q^A^@0^P^I^]0^I00^@^@^@M-^?\^O^GM-^?\^Oq0q^[^I^I^I^@^@M-h^C@^N\0^[11#0-^P^@^@^@^@^D^I^@^[M-UM-UM-UM-NM-U^@M-^@^@M-=$^@0^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^PM-eM-KM-YM-@M-nM-n^ZM-{0M-xM-|^@00^L0#^A^Cd^\#^@^X^E^X^E^@M-^?M-^?^[^I^I^@^@^@M-^?\^O^@000M-^R00M-,0^@^@M-^?\^O^@qq0M-^Dq^@0^P^I^[^I^Iu000^E^@M-^?M-^?0M-v^P^P^P^P^P^P^PM-eM-KM-YM-@M-nM-n^ZM-{0M-xM-|^@}}^L0#^A^Cd^\#^@^X^E^X^E^@M-^?M-^?^[^I^I^@^@^@M-^?\^O^@00M-^?M-^R00M-,0^@^@M-^?\^O^@0^@0^P^I^[^I^I^E^@M-^?M-^?0M-vM-Q^A^@0^P^I^]0^I00^@^@^@M-^?\^O^GM-^?\^Oq0q^[^I^I^I^@^@M-h^C^E^@000M-v

debian@debian-fuzz:/mnt$ base64 < rl_do_undo
EsOhw8OhwxsuGxswMDAwMBAAMDAwMBARMBstEBEwGxcwADAmLxsbGy0tFx9+MAAwAC0MBAAAJy8b
QgAwAgCgGxuUG+3/G0YtFx8wG4AwEAAECQAb1dXVztUAgAC9JAAwMQAwMRsiMH8w8jAA
JzD/w8OhwxsuGxstLRcwMBAAMDAoLRAROxstEBEwGxcwAG4mLxsbGy0tFx9+MAAwAC0MBAAA
Jy8bQgCUAgCgGxuUG+3/G0YtFx8wG4AwEAAECQAb1dXVztUAgAC9JAAwMQAwMRsiMH8w
8jAAJzD/QA5cMBsxMSMwLRAABAkAG9W91c7VAIAAvSQAShAQEBAQEBAQEBAQEBAQEBAQ
EBAQ5cvZwO7uGvsw+PwAfX0MMCMBA2QcIwAYBRgFAP//GwkJMDAwMP9cDwAwMDCSMDCsMAAA
/1wPAHFxMIRxADAQCRsJCTAwMDAFAP//MPYQEBAQEBAQ5cvZwO7uGvsw+PwAfX0MMCMBA2QcIwAY
BRgFAP//GwkAAIAAMP9cDwAwfzCSMDCsMAAA/1wPADAwMDAwADAQCRsJCXUwMDAFAP//MPbR
AQAwEAkdMAkwMDAwMDD/XA8H/1wPcTBxGwkJCQAA6ANADlwwGzExIzAtEAAECQAb1dXV
ztUAgAC9JAAwEBAQEBAQEBAQEBAQEBAQEBAQEBDly9nA7u4a+zD4/AAwMAwwIwEDZBwjABgFGAUA
//8bCQkwMDAw/1wPADAwMJIwMKwwAAD/XA8AcXEwhHEAMBAJGwkJdTAwMAUA//8w9hAQEBAQ
EBDly9nA7u4a+zD4/AB9fQwwIwEDZBwjABgFGAUA//8bCQkwMDAw/1wPADAw/5IwMKwwAAD/
XA8AMDAwMDAAMBAJGwkJMDAwMAUA//8w9tEBADAQCR0wCTAwMDAwMP9cDwf/XA9xMHEbCQkJ
AADoAwUAMDAw9g==

debian@debian-fuzz:/mnt$ LC_ALL=zh_CN.gbk  ~/build/bash --noprofile --norc -c 
'PATH= read -e < rl_do_undo' >/dev/null 2>&1; echo $?
Segmentation fault
139


And the backtrace:

(gdb) bt
#0  0x55656672 in rl_do_undo () at 
../../../bash-5.0-rc1/lib/readline/undo.c:255
#1  0x55656807 in rl_revert_line (count=1, key=0) at 
../../../bash-5.0-rc1/lib/readline/undo.c:339
#2  0x5563956b in readline_internal_teardown (eof=0) at 
../../../bash-5.0-rc1/lib/readline/readline.c:471
#3  0x5563995c in readline_internal () at 
../../../bash-5.0-rc1/lib/readline/readline.c:672
#4  0x55639367 in readline (prompt=0x55680f84 "") at 
../../../bash-5.0-rc1/lib/readline/readline.c:377
#5  0x55611bcf in edit_line (p=0x55680f84 "", itext=0x0) at 
../../bash-5.0-rc1/builtins/../../bash-5.0-rc1/builtins/read.def:1107
#6  0x556108f8 in read_builtin (list=0x0) at 
../../bash-5.0-rc1/builtins/../../bash-5.0-rc1/builtins/read.def:566
#7  0x555a5afa in execute_builtin (builtin=0x5560fa73 
, words=0x55761ea8, flags=0, subshell=0) at 
../bash-5.0-rc1/execute_cmd.c:4706
#8  0x555a6aa2 in execute_builtin_or_function (words=0x55761ea8, 
builtin=0x5560fa73 , var=0x0, redirects=0x55761c08, 
fds_to_close=0x55761be8, flags=0)
at ../bash-5.0-rc1/execute_cmd.c:5214
#9  0x555a5365 in execute_simple_command 
(simple_command=0x55761ac8, pipe_in=-1, pipe_out=-1, async=0, 
fds_to_close=0x55761be8) at ../bash-5.0-rc1/execute_cmd.c:4476
#10 0x5559e9f4 in execute_command_internal (command=0x55761a88, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x55761be8) at 
../bash-5.0-rc1/execute_cmd.c:842
#11 0x5560858a in parse_and_execute (string=0x557616c8 "PATH= read 
-e < rl_do_undo", from_file=0x556690f0 "-c", flags=4) at 
../../bash-5.0-rc1/builtins/evalstring.c:436
#12 0x5558564a in run_one_command (command=0x7fffe284 "PATH= read 
-e < rl_do_undo") at ../bash-5.0-rc1/shell.c:1426
#13 0x55584789 in main (argc=5, argv=0x7fffdfe8, 
env=0x7fffe018) at ../bash-5.0-rc1/shell.c:741
(gdb) x/12xb 
0x7fffd7f8: 0x300x300x300x300x300x300x000x07
0x7fffd800: 0x080x3c0x76

Re: $RANDOM not Cryptographically secure pseudorandom number generator

[Eduardo A . Bustamante López]

On Mon, Jan 07, 2019 at 08:15:12AM +0100, Ole Tange wrote:
> On Mon, Jan 7, 2019 at 12:08 AM Chet Ramey  wrote:
> >
> > On 1/5/19 3:12 PM, Eduardo A. Bustamante López wrote:
> > > On Fri, Dec 28, 2018 at 10:24:50AM +0100, Ole Tange wrote:
> > > (...)
> > >> Patch attached.
> :
> > > - Does the new RNG generate uniformly distributed numbers? (Yes)
> > > - What is the performance impact (roughly 2X slower)
> 
> The impact is not from Salsa20/ChaCha20, though, if you compare
> patched vs. unpatched code.
> 
> On my system BASH_RANDOM_16 is a tiny bit faster than the original,
> while BASH_RANDOM_32 is a tiny bit slower.

I find that really hard to believe. The new RNG is clearly doing more work, so
it should be slower. And that's OK. I just wanted to know by how much.

What numbers are you seeing and how did you measure them?

Re: realloc: start and end chunk sizes differ - rl_extend_line_buffer in lib/readline/util.c

[Eduardo A . Bustamante López]

On Sun, Jan 06, 2019 at 07:18:27PM -0800, Eduardo A. Bustamante López wrote:
(...)
> malloc: unknown:0: assertion botched
> malloc: 0x55769408: allocated: last allocated from unknown:0
> realloc: start and end chunk sizes differ

OK, I think I know what the problem is.

I noticed that `rl_insert_text' writes past the end of the rl_line_buffer 
buffer:

| Hardware watchpoint 1: rl_line_buffer[255]
| 
| Old value = -33 '\337'
| New value = 48 '0'
| __strncpy_ssse3 () at ../sysdeps/x86_64/multiarch/strcpy-ssse3.S:2399
| 2399../sysdeps/x86_64/multiarch/strcpy-ssse3.S: No such file or directory.
| 
| (gdb) bt
| #0  __strncpy_ssse3 () at ../sysdeps/x86_64/multiarch/strcpy-ssse3.S:2399
| #1  0x55659522 in rl_insert_text (string=0x5576bc88 '0' ) at text.c:99
| #2  0x55655a9f in rl_yank (count=1, key=25) at kill.c:494
| #3  0x55639ef2 in _rl_dispatch_subseq (key=25, map=0x556aa200 
, got_subseq=0) at readline.c:852
| #4  0x55639c69 in _rl_dispatch (key=1433844896, map=0x556aa200 
) at readline.c:798
| #5  0x556398ec in readline_internal_char () at readline.c:632
| #6  0x55639947 in readline_internal_charloop () at readline.c:659
| #7  0x55639967 in readline_internal () at readline.c:671
| #8  0x55639385 in readline (prompt=0x556809dc "") at 
readline.c:377
| #9  0x55611bdd in edit_line (p=0x556809dc "", itext=0x0) at 
./read.def:1107
| #10 0x55610906 in read_builtin (list=0x0) at ./read.def:566
| #11 0x555a5b3f in execute_builtin (builtin=0x5560fa81 
, words=0x55762d08, flags=0, subshell=0) at execute_cmd.c:4706
| #12 0x555a6ae7 in execute_builtin_or_function (words=0x55762d08, 
builtin=0x5560fa81 , var=0x0, redirects=0x557629c8, 
fds_to_close=0x55762a08, flags=0)
| at execute_cmd.c:5214
| #13 0x555a53aa in execute_simple_command 
(simple_command=0x55762948, pipe_in=-1, pipe_out=-1, async=0, 
fds_to_close=0x55762a08) at execute_cmd.c:4476
| #14 0x5559ea39 in execute_command_internal (command=0x55762908, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x55762a08) at 
execute_cmd.c:842
| #15 0x55608586 in parse_and_execute (string=0x5574b268 "read -e < 
~/xxx", from_file=0x556690f0 "-c", flags=4) at evalstring.c:436
| #16 0x55585632 in run_one_command (command=0x7fffec14 "read -e < 
~/xxx") at shell.c:1416
| #17 0x5558477d in main (argc=3, argv=0x7fffe948, 
env=0x7fffe968) at shell.c:735
| 
| (gdb) frame 1
| #1  0x55659522 in rl_insert_text (string=0x5576bc88 '0' ) at text.c:99
| 99strncpy (rl_line_buffer + rl_point, string, l);
| 
| (gdb) l
| 94if (rl_end + l >= rl_line_buffer_len)
| 95  rl_extend_line_buffer (rl_end + l);
| 96
| 97for (i = rl_end; i >= rl_point; i--)
| 98  rl_line_buffer[i + l] = rl_line_buffer[i];
| 99strncpy (rl_line_buffer + rl_point, string, l);
| 100
| 101   /* Remember how to undo this if we aren't undoing something. */
| 102   if (_rl_doing_an_undo == 0)
| 103 {
| 
| (gdb) p rl_end
| $1 = 213
| (gdb) p l
| $2 = 30
| (gdb) p rl_end + l
| $3 = 243
| (gdb) p string
| $4 = 0x5576bc88 '0' 
| (gdb) p strlen(string)
| $5 = 30
| (gdb) p rl_point
| $6 = 227

So somehow rl_point ended up with an invalid value. Tracing that problem:

| Hardware watchpoint 5: (rl_point > 0 && rl_point > rl_end + 1)
| 
| Old value = 0
| New value = 1
| rl_do_undo () at undo.c:199
| 199   rl_insert_text (rl_undo_list->text);
| (gdb) p rl_end
| $12 = 0
| (gdb) p rl_point
| $13 = 14
| (gdb) bt
| #0  rl_do_undo () at undo.c:199
| #1  0x5565685c in rl_undo_command (count=1, key=31) at undo.c:356
| #2  0x55639ef2 in _rl_dispatch_subseq (key=31, map=0x556aa200 
, got_subseq=0) at readline.c:852
| #3  0x55639c69 in _rl_dispatch (key=1433812680, map=0x556aa200 
) at readline.c:798
| #4  0x556398ec in readline_internal_char () at readline.c:632
| #5  0x55639947 in readline_internal_charloop () at readline.c:659
| #6  0x55639967 in readline_internal () at readline.c:671
| #7  0x55639385 in readline (prompt=0x556809dc "") at 
readline.c:377
| #8  0x55611bdd in edit_line (p=0x556809dc "", itext=0x0) at 
./read.def:1107
| #9  0x55610906 in read_builtin (list=0x0) at ./read.def:566
| #10 0x555a5b3f in execute_builtin (builtin=0x5560fa81 
, words=0x55762d08, flags=0, subshell=0) at execute_cmd.c:4706
| #11 0x555a6ae7 in execute_builtin_or_function (words=0x55762d08, 
builtin=0x5560fa81 , var=0x0, redirects=0x557629c8, 
fds_to_close=0x55762a08, flags=0)
| at execute_cmd.c:5214
| #12 0x555a53aa in execute_simple_command 
(simple_command=0x55762948, pipe_in=-1, pipe_out=-1, async=0, 
fds_to_close=0x

realloc: start and end chunk sizes differ - rl_extend_line_buffer in lib/readline/util.c

[Eduardo A . Bustamante López]

Found by fuzzing with AFL

debian@debian-fuzz:/mnt$ cat -A rl_extend_line_buffer 
00^[^?000^?^X^E^_^Y^Y^Y^Y^Y^Y^Y^Y

debian@debian-fuzz:/mnt$ base64 < rl_extend_line_buffer 
MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwG38wMDAwMDAwMDAwMDAwMDB/GAUfGRkZMDAw
MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAZGRkZGQ==

(gdb) r
Starting program: /home/debian/build-gdb/bash --noprofile --norc -c PATH=\ 
read\ -e\ \<\ rl_extend_line_buffer
hi
00
[Detaching after fork from child process 21638]
/home/debian/build-gdb/bash: emacs: No such file or directory
^@^@0^@0^@0^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@0^@0^@0^@0
malloc: unknown:0: assertion botched
malloc: 0x55769408: allocated: last allocated from unknown:0
realloc: start and end chunk sizes differ
Aborting...
Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x77df4535 in __GI_abort () at abort.c:79
#2  0x555b39b5 in programming_error (format=0x55686c98 "realloc: 
start and end chunk sizes differ") at ../bash-5.0-rc1/error.c:175
#3  0x556651dc in xbotch (mem=0x55769408, e=8, s=0x55686c98 
"realloc: start and end chunk sizes differ", file=0x0, line=0) at 
../../../bash-5.0-rc1/lib/malloc/malloc.c:354
#4  0x55666993 in internal_realloc (mem=0x55769408, n=512, 
file=0x0, line=0, flags=0) at ../../../bash-5.0-rc1/lib/malloc/malloc.c:1091
#5  0x556670c8 in realloc (mem=0x55769408, nbytes=512) at 
../../../bash-5.0-rc1/lib/malloc/malloc.c:1381
#6  0x555fff94 in xrealloc (pointer=0x55769408, bytes=512) at 
../bash-5.0-rc1/xmalloc.c:135
#7  0x556547d8 in rl_extend_line_buffer (len=273) at 
../../../bash-5.0-rc1/lib/readline/util.c:169
#8  0x556594a5 in rl_insert_text (string=0x5576d888 '0' ) at ../../../bash-5.0-rc1/lib/readline/text.c:95
#9  0x55655a81 in rl_yank (count=1, key=25) at 
../../../bash-5.0-rc1/lib/readline/kill.c:494
#10 0x55639ed4 in _rl_dispatch_subseq (key=25, map=0x556ab200 
, got_subseq=0) at 
../../../bash-5.0-rc1/lib/readline/readline.c:852
#11 0x55639c4b in _rl_dispatch (key=-136275877, map=0x556ab200 
) at ../../../bash-5.0-rc1/lib/readline/readline.c:798
#12 0x556398ce in readline_internal_char () at 
../../../bash-5.0-rc1/lib/readline/readline.c:632
#13 0x55639929 in readline_internal_charloop () at 
../../../bash-5.0-rc1/lib/readline/readline.c:659
#14 0x55639949 in readline_internal () at 
../../../bash-5.0-rc1/lib/readline/readline.c:671
#15 0x55639367 in readline (prompt=0x55680f84 "") at 
../../../bash-5.0-rc1/lib/readline/readline.c:377
#16 0x55611bcf in edit_line (p=0x55680f84 "", itext=0x0) at 
../../bash-5.0-rc1/builtins/../../bash-5.0-rc1/builtins/read.def:1107
#17 0x556108f8 in read_builtin (list=0x0) at 
../../bash-5.0-rc1/builtins/../../bash-5.0-rc1/builtins/read.def:566
#18 0x555a5afa in execute_builtin (builtin=0x5560fa73 
, words=0x55764dc8, flags=0, subshell=0) at 
../bash-5.0-rc1/execute_cmd.c:4706
#19 0x555a6aa2 in execute_builtin_or_function (words=0x55764dc8, 
builtin=0x5560fa73 , var=0x0, redirects=0x55764b48, 
fds_to_close=0x55764a48, flags=0)
at ../bash-5.0-rc1/execute_cmd.c:5214
#20 0x555a5365 in execute_simple_command 
(simple_command=0x557648c8, pipe_in=-1, pipe_out=-1, async=0, 
fds_to_close=0x55764a48) at ../bash-5.0-rc1/execute_cmd.c:4476
#21 0x5559e9f4 in execute_command_internal (command=0x55764888, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x55764a48) at 
../bash-5.0-rc1/execute_cmd.c:842
#22 0x5560858a in parse_and_execute (string=0x55764008 "PATH= read 
-e < rl_extend_line_buffer", from_file=0x556690f0 "-c", flags=4)
at ../../bash-5.0-rc1/builtins/evalstring.c:436
#23 0x5558564a in run_one_command (command=0x7fffe294 "PATH= read 
-e < rl_extend_line_buffer") at ../bash-5.0-rc1/shell.c:1426
#24 0x55584789 in main (argc=5, argv=0x7fffdff8, 
env=0x7fffe028) at ../bash-5.0-rc1/shell.c:741

Recursive execution of expand_prompt_string

[Eduardo A . Bustamante López]

The expand_prompt_string function will call itself recursively in some
situations:

dualbus@system76-pc:~$ bash -c 'q="\${q@P}"; echo "${q@P}"'
Segmentation fault

My guess is that this is expected behavior. I'm reporting this because it's
affecting my fuzzing effort.

Re: Identical function names in bash source code

[Eduardo A . Bustamante López]

On Sat, Jan 05, 2019 at 08:19:38AM -0600, Peng Yu wrote:
> Hi,
> 
> It is not uncommon to see the same name is used to defined functions
> in different .c files in bash source code.
> 
> For example, sh_single_quote is defined in both lib/readline/shell.c
> and lib/sh/shquote.c with the exact same signature. The two pieces of
> code are slightly different. Do they do the exact same things or do
> something different?
> 
> In either case, is having the same name for different functions a good
> practice? This will make the linked binary dependent on the order of
> the corresponding .a files specified. Or if linked via .o files, then
> one function will shadow the others. See 1) and 2) below for minimal
> working examples. Neither cases seem to be good and could be avoided
> easily by giving the functions unique names.
> 
> So, should such functions with the same name be named differently? Thanks.

This mailing list (bug-bash) is for reporting BUGS. If you must, at least please
use help-bash for generic questions?

Although you should probably be asking this to a C programming forum...

[PATCH 3/3] Fix incompatible pointer type warning in unicode.c

[Eduardo A . Bustamante López]

The warning is raised by Clang (7.0.1-4) when sizeof(wchar_t) is 4

| dualbus@system76-pc:~/src/gnu/bash/lib/sh$ make unicode.o
| clang -c   -I. -I../.. -I../.. -I../../lib -I../../include -I.  
-DHAVE_CONFIG_H -DSHELL  -ggdb -O0 -Wno-parentheses -Wno-format-security   
unicode.c
| unicode.c:262:69: warning: incompatible pointer types passing 'wchar_t [3]' 
to parameter of type 'unsigned short *' [-Wincompatible-pointer-types]
|   else if (sizeof (wchar_t) == 2 && c <= 0x10 && u32toutf16 (c, ws))
| ^~
| 1 warning generated.

dualbus@system76-pc:~/src/gnu/bash$ clang -v 2>&1 | head -n2
clang version 7.0.1-4 (tags/RELEASE_701/final)
Target: x86_64-pc-linux-gnu

In practice, this isn't really a problem because the compiler should optimize
away the `sizeof(wchar_t) == 2' branch. Still, it's easy to fix.

I think that maybe there should be some sort of compile-time assertion inside
`u32toutf16`, to ensure it's only used when `sizeof(wchar_t) == 2', but I don't
know how to do that.
---
 lib/sh/unicode.c | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/sh/unicode.c b/lib/sh/unicode.c
index fe13c4a0..99c422ab 100644
--- a/lib/sh/unicode.c
+++ b/lib/sh/unicode.c
@@ -216,21 +216,21 @@ u32toutf8 (wc, s)
 int
 u32toutf16 (c, s)
  u_bits32_t c;
- unsigned short *s;
+ wchar_t *s;
 {
   int l;
 
   l = 0;
   if (c < 0x0d800 || (c >= 0x0e000 && c <= 0x0))
 {
-  s[0] = (unsigned short) (c & 0x);
+  s[0] = (wchar_t) (c & 0x);
   l = 1;
 }
   else if (c >= 0x1 && c <= 0x010)
 {
   c -= 0x01;
-  s[0] = (unsigned short)((c >> 10) + 0xd800);
-  s[1] = (unsigned short)((c & 0x3ff) + 0xdc00);
+  s[0] = (wchar_t)((c >> 10) + 0xd800);
+  s[1] = (wchar_t)((c & 0x3ff) + 0xdc00);
   l = 2;
 }
   s[l] = 0;
-- 
2.20.1

[PATCH 2/3] Fix `hashtest' target in Makefile

[Eduardo A . Bustamante López]

- Adds the dependency on the Bash's libmalloc
- Removes the libintl dependency, since it's unused
- Adds the `running_trap' global variable, since libmalloc depends on
  it when the `SHELL' macro is defined (see [1])
- Removes compiler warning due to missing return type in main function

After these changes, the `hashtest' target can be built with:

| dualbus@system76-pc:~/src/gnu/bash$ CC=gcc ./configure --silent 
--with-bash-malloc && make -j$(nproc) -s hashtest
|
| Beginning configuration for bash-5.0-rc1 for x86_64-pc-linux-gnu
|
| config.status: creating po/POTFILES
| config.status: creating po/Makefile
| ./parse.y: warning: 1 shift/reduce conflict [-Wconflicts-sr]
|
|   ***
|   * *
|   * GNU bash, version 5.0.0(1)-rc1 (x86_64-pc-linux-gnu)
|   * *
|   ***
|
| make[1]: warning: -j16 forced in submake: resetting jobserver mode.
|
| dualbus@system76-pc:~/src/gnu/bash$ echo hi | ./hashtest >/dev/null 2>&1; 
echo $?
| 0

It doesn't work if `--without-bash-malloc' is specified.

[1]

/usr/bin/ld: ./lib/malloc/libmalloc.a(malloc.o): in function `morecore':
/home/dualbus/src/gnu/bash/lib/malloc/malloc.c:602: undefined reference to 
`running_trap'
---
 Makefile.in | 4 ++--
 hashlib.c   | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/Makefile.in b/Makefile.in
index 5fcb44b0..76a51b19 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -967,8 +967,8 @@ depends: force
$(Program) $(SUPPORT_SRC)mkdep -c ${CC} -- ${CCFLAGS} ${CSOURCES}
 
  PRIVATE TARGETS 
-hashtest:  hashlib.c
-   $(CC) -DTEST_HASHING $(CCFLAGS) $(TEST_NBUCKETS) -o $@ 
$(srcdir)/hashlib.c xmalloc.o $(INTL_LIB)
+hashtest:  hashlib.c xmalloc.o $(MALLOC_LIBRARY)
+   $(CC) -DTEST_HASHING $(CCFLAGS) $(TEST_NBUCKETS) -o $@ 
$(srcdir)/hashlib.c xmalloc.o $(MALLOC_LIBRARY)
 
  DEPENDENCIES ###
 
diff --git a/hashlib.c b/hashlib.c
index 8adbe221..f8e3b09a 100644
--- a/hashlib.c
+++ b/hashlib.c
@@ -392,6 +392,7 @@ hash_pstats (table, name)
 HASH_TABLE *table, *ntable;
 
 int interrupt_immediately = 0;
+int running_trap = 0;
 
 int
 signal_is_trapped (s)
@@ -417,6 +418,7 @@ internal_warning (const char *format, ...)
 {
 }
 
+int
 main ()
 {
   char string[256];
-- 
2.20.1

[PATCH 1/3] Fix implicit declaration of abort()

[Eduardo A . Bustamante López]

gcc version 8.2.0 (Debian 8.2.0-13) x86_64-linux-gnu

```
malloc.c:333:3: warning: incompatible implicit declaration of built-in function 
‘abort’
malloc.c:333:3: note: include ‘’ or provide a declaration of ‘abort’
```
---
 lib/malloc/malloc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/malloc/malloc.c b/lib/malloc/malloc.c
index 5621adf4..8f641036 100644
--- a/lib/malloc/malloc.c
+++ b/lib/malloc/malloc.c
@@ -83,6 +83,7 @@
 #endif
 #include 
 #include 
+#include 
 
 #if defined (HAVE_MMAP)
 #include 
-- 
2.20.1

Re: "COMMAND 2>&1 > PATH" doesn't work

[Eduardo A . Bustamante López]

On Sun, Dec 30, 2018 at 10:10:42PM +0100, Dušan Kreheľ wrote:
> Hello.
> 
> If I try in bash this command ex. "rmdir somethingA > somethingA.out
> 2>&1" work right. But, if I try "rmdir somethingB 2>&1 >
> somethingB.out" that work wrong.
> 
> That work's wrong in Bash version 4.4.23(1) and too in 5.0.0(1)-rc1:

Why do you expect these two to behave in the same way? They are clearly
distinct operations.


 COMMAND >FILE 2>&1

is not the same as:

 COMMAND 2>&1 >FILE

The order of redirections matter, they are processed left-to-right. Let's look
at these examples in more detail.


Example 1.

 COMMAND >FILE 2>&1

Let's break it down into tokens and see what happens at each step.

a) COMMAND

file descriptor 0 -> terminal
file descriptor 1 -> terminal
file descriptor 2 -> terminal

This is the "normal" state in an interactive shell, the three standard file
descriptors (input: 0, output: 1, and error: 2) are pointing to the terminal
device.

b) >FILE

file descriptor 0 -> terminal
file descriptor 1 -> FILE
file descriptor 2 -> terminal

The `>FILE' redirection is processed, causing file descriptor 1 (standard
output) to be directed to the file named `FILE'.

c) 2>&1

file descriptor 0 -> terminal
file descriptor 1 -> FILE
file descriptor 2 -> FILE

The `2>&1' redirection is processed, causing file descriptor 2 (the `2>' bit) to
be directed to "wherever-fd1-points-to" (i.e. the `&1' bit). FD1 currently
points to `FILE', so FD2 is updated to point to `FILE' too.


VS

Example 2.

 COMMAND 2>&1 FILE 

a) COMMAND

file descriptor 0 -> terminal
file descriptor 1 -> terminal
file descriptor 2 -> terminal

same as example 1.

b) 2>&1

file descriptor 0 -> terminal
file descriptor 1 -> terminal
file descriptor 2 -> terminal * no change

The `2>&1' redirection is processed, causing file descriptor 2 to be directed to
"wherever-fd1-points-to". FD1 currently points to the terminal. FD2 points to
the terminal too, so it stays the same.

c) >FILE

file descriptor 0 -> terminal
file descriptor 1 -> FILE
file descriptor 2 -> terminal

The `>FILE' redirection is processed, causing file descriptor 1 (standard
output) to be directed to the file named `FILE'.



So, this is not a bug. It's the expected behavior of redirections. If you want
to read more about it, I recommend:

- https://mywiki.wooledge.org/BashFAQ/055
- 
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07

Re: difference between /tmp and other directory for loadable mkdir?

[Eduardo A . Bustamante López]

On Wed, Dec 26, 2018 at 12:40:09PM -0600, Peng Yu wrote:
> Hi,
> 
> I can not mkdir -p . in /tmp/ via the loadable mkdir. What is the
> difference between /tmp/ and other directories? I am on Mac OS X. Is
> this a bug in mkdir?
> 
> $ cd /tmp
> $ mkdir -p -- .
> -bash: mkdir: .: Operation not permitted

I can reproduce this in Linux
[Linux debian 4.18.0-3-amd64 #1 SMP Debian 4.18.20-2 (2018-11-23) x86_64 
GNU/Linux]

dualbus@debian:~/src/gnu/bash$ strace -e trace=%file -f ./bash -c 'enable -f 
examples/loadables/mkdir mkdir && mkdir -p -- /tmp'
execve("./bash", ["./bash", "-c", "enable -f examples/loadables/mkd"...], 
0x7fff7c9b1c18 /* 33 vars */) = 0
(...)
stat("/home/dualbus/src/gnu/bash", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/home/dualbus", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/home/dualbus/src", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/home/dualbus/src/gnu", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/home/dualbus/src/gnu/bash", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/home/dualbus/src/iovisor/bcc", {st_mode=S_IFDIR|0755, st_size=4096, 
...}) = 0
openat(AT_FDCWD, "examples/loadables/mkdir", O_RDONLY|O_CLOEXEC) = 3
getcwd("/home/dualbus/src/gnu/bash", 128) = 27
stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0
chmod("/tmp", 0755) = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/usr/share/locale/en_CA/LC_MESSAGES/libc.mo", O_RDONLY) = -1 
ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 
ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/share/locale/en_CA/LC_MESSAGES/bash.mo", O_RDONLY) 
= -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/share/locale/en/LC_MESSAGES/bash.mo", O_RDONLY) = 
-1 ENOENT (No such file or directory)
./bash: line 0: mkdir: /tmp: Operation not permitted
+++ exited with 1 +++

dualbus@debian:~/src/gnu/bash$ stat /tmp
  File: /tmp
  Size: 4096Blocks: 8  IO Block: 4096   directory
Device: fe01h/65025dInode: 25427969Links: 18
Access: (1777/drwxrwxrwt)  Uid: (0/root)   Gid: (0/root)
Access: 2018-12-26 11:08:30.062681076 -0800
Modify: 2018-12-26 11:37:56.826984422 -0800
Change: 2018-12-26 11:37:56.826984422 -0800
 Birth: -


If you look at the source, it calls the mkdir() function to create the 
directory.

dualbus@debian:~/src/gnu/bash$ cat -n examples/loadables/mkdir.c | sed -n 
'174,180p'
   174if (stat (npath, ) != 0)
   175  {
   176if (mkdir (npath, parent_mode))  <- this
   177  {
   178builtin_error ("cannot create directory `%s': %s", npath, 
strerror (errno));
   179umask (original_umask);
   180free (npath);

Ref: 
http://git.savannah.gnu.org/cgit/bash.git/tree/examples/loadables/mkdir.c?h=6870125961b85baa9628ce4c5fbbca94e8046656#n174


Comparing to GNU coreutils' mkdir (which uses gnulib):

dualbus@debian:~$ strace -e trace=%file -f mkdir -p /tmp/
execve("/bin/mkdir", ["mkdir", "-p", "/tmp/"], 0x7fff4f7a11b8 /* 33 vars */) = 0
(...)
mkdir("/tmp/", 0777)= -1 EEXIST (File exists)
stat("/tmp/", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0
+++ exited with 0 +++


My guess is that the "problem" has to do with how bash computes the 
`parent_mode':

dualbus@debian:~/src/gnu/bash$ cat -n examples/loadables/mkdir.c | sed -n 
'106,114p'
   106/* Make the new mode */
   107original_umask = umask (0);
   108umask (original_umask);
   109  
   110nmode = (S_IRWXU | S_IRWXG | S_IRWXO) & ~original_umask;
   111parent_mode = nmode | (S_IWUSR|S_IXUSR);  /* u+wx */
   112  
   113/* Adjust new mode based on mode argument */
   114nmode &= omode;

Ref: 
http://git.savannah.gnu.org/cgit/bash.git/tree/examples/loadables/mkdir.c?h=6870125961b85baa9628ce4c5fbbca94e8046656#n106


In gnulib, this is what is used to create the parent directories:

Ref: 
http://git.savannah.gnu.org/cgit/gnulib.git/tree/lib/mkdir-p.c?h=95c96b631f3676f72ed044d0c493ab5642d8#n116

Re: Memory continusely increase

[Eduardo A . Bustamante López]

On Mon, Dec 24, 2018 at 10:51:00AM +0800, chen liu wrote:
> Chet Ramey  于2018年12月22日周六 上午12:51写道：
(...)
> > What is the hard limit on the number of processes for a process started in
> > this environment? (The value of `ulimit -n'.)

Ah, interesting. Thanks for the pointer Chet.

(...)
> Below is all of the values of the bash ulimit optionals, this issue
> present in this environment
> --
> root@localhost:/root> cat ulimit.txt
(...)
> max user processes (-u) 516046
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> ---
> But found change the -u optionals value through the ( ulimit -u 1024)
> will let this issue disapeared.

As you probably found out already, the changes between 4.3 and 4.4 that make
this issue "disappear" are related to how bash stores the exit status of
child processes.

The threads that prompted this change provide some insight:

- https://lists.gnu.org/archive/html/bug-bash/2015-04/msg00069.html
- http://lists.gnu.org/archive/html/bug-bash/2015-04/msg00075.html

And the note in the changelog:

http://git.savannah.gnu.org/cgit/bash.git/tree/CWRU/old-changelogs/CWRU.chlog.v15?h=ab8ded9c30b51a6bc0c8145553263da044b565ea#n9811

> jobs.[ch]
>   - bgpids: new implementation from a patch from John Fremlin
> , uses an array for the list of the last CHILD_MAX
> terminated background pids, and a separate hash table to search it.
> The storage can be freed as a unit, and the size of the hash table
> (currently 4096) is independent of the size of the bgpids table

http://git.savannah.gnu.org/cgit/bash.git/tree/CHANGES-4.4?h=ab8ded9c30b51a6bc0c8145553263da044b565ea#n220

> e.  There is a new implementation of the code that saves the last CHILD_MAX
> exited background pids so their status can be queried later.

Bash build issues in `devel' branch due to -Werror compiler flag

[Eduardo A . Bustamante López]

Commit 9d80be9ab5cc17011c634e0348c64c15fcba95bf adds the following compiler 
flag:

  dualbus@debian:~/src/gnu/bash$ cat -n configure.ac | grep Werror -C3
1159CFLAGS="$CFLAGS -Wno-parentheses -Wno-format-security"
1160if test -n "$DEBUG"
1161then
  ~>1162CFLAGS="$CFLAGS -Werror"
1163fi
1164fi
1165

During build (with gcc):

  dualbus@debian:~/src/gnu/bash$ make -j$(nproc) -s
  make[1]: warning: -j4 forced in submake: resetting jobserver mode.
  ./parse.y: warning: 1 shift/reduce conflict [-Wconflicts-sr]
  make[1]: warning: -j4 forced in submake: resetting jobserver mode.
  expr.c:217:17: error: conflicting types for built-in function ‘exp2’ 
[-Werror=builtin-declaration-mismatch]
   static intmax_t exp2 __P((void));
   ^~~~
  cc1: all warnings being treated as errors


I know the `exp2' function here has nothing to do with the built-in exponential
function, and that it has had that name for a long time, but the build breaks 
due to that.

To hack around it, I renamed it from `exp2' to `exp22', and then it broke here:

  dualbus@debian:~/src/gnu/bash$ make -j$(nproc) -s
  make[1]: warning: -j4 forced in submake: resetting jobserver mode.
  
***
* *
* GNU bash, version 5.0.0(1)-rc1 (x86_64-pc-linux-gnu)
* *
***
  
  making lib/glob/libglob.a in ./lib/glob
  make[1]: warning: -j4 forced in submake: resetting jobserver mode.
  smatch.c: In function ‘_fnmatch_fallback_wc’:
  smatch.c:318:11: error: implicit declaration of function ‘fnmatch’; did you 
mean ‘gmatch’? [-Werror=implicit-function-declaration]
 return (fnmatch ((const char *)w2, (const char *)w1, 0));
 ^~~
 gmatch


Which I believe is an actual issue and I assume the fix is along the lines of:

  dualbus@debian:~/src/gnu/bash$ git diff -- lib/
  diff --git a/lib/glob/smatch.c b/lib/glob/smatch.c
  index 3826c93e..9150a28e 100644
  --- a/lib/glob/smatch.c
  +++ b/lib/glob/smatch.c
  @@ -25,6 +25,7 @@
  
   #include "strmatch.h"
   #include 
  +#include 
   
   #include "bashansi.h"
   #include "shmbutil.h"


I'm not sure what to do about `exp2' though. Personally, I'd love for these
functions to have slighly more meaningful names.


I also tried clang version 7.0.1-+rc3-2, which gives an additional error:

  making lib/sh/libsh.a in ./lib/sh
  (...)
  getenv.c:55:7: error: comparison of nonnull parameter 'name' equal to a null 
pointer is 'false' on first encounter [-Werror,-Wtautological-pointer-compare]
if (name == 0 || *name == '\0')
^~~~~
  /usr/include/stdlib.h:631:50: note: declared 'nonnull' here
  extern char *getenv (const char *__name) __THROW __nonnull ((1)) __wur;
   ^
  /usr/include/x86_64-linux-gnu/sys/cdefs.h:293:44: note: expanded from macro 
'__nonnull'
  # define __nonnull(params) __attribute__ ((__nonnull__ params))


This is due to the stdlib.h header being pulled by bashansi.h in
lib/sh/getenv.c, thus, providing a function signature that doesn't match Bash's
re-definition of getenv():

  dualbus@debian:~/src/gnu/bash$ grep bashansi.h lib/sh/getenv.c 
  #include 
  
  dualbus@debian:~/src/gnu/bash$ grep stdlib.h bashansi.h 
  #  include 
  #  include "ansi_stdlib.h"


~~~


Additional information:

gcc:

  dualbus@debian:~/src/gnu/bash$ cc -v
  Using built-in specs.
  COLLECT_GCC=cc
  COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/8/lto-wrapper
  OFFLOAD_TARGET_NAMES=nvptx-none
  OFFLOAD_TARGET_DEFAULT=1
  Target: x86_64-linux-gnu
  Configured with: ../src/configure -v --with-pkgversion='Debian 8.2.0-12' 
--with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs 
--enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr 
--with-gcc-major-version-only --program-suffix=-8 
--program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id 
--libexecdir=/usr/lib --without-included-gettext --enable-threads=posix 
--libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu 
--enable-libstdcxx-debug --enable-libstdcxx-time=yes 
--with-default-libstdcxx-abi=new --enable-gnu-unique-object 
--disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie 
--with-system-zlib --with-target-system-zlib --enable-objc-gc=auto 
--enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 
--with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic 
--enable-offload-targets=nvptx-none --without-cuda-driver 
--enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu 
--target=x86_64-linux-gnu
  Thread model: posix
  gcc version 8.2.0 

Re: Auto-update program cache feature

[Eduardo A . Bustamante López]

On Wed, Oct 03, 2018 at 04:45:44PM -0400, Jeffrey Walton wrote:
> Hi Everyone,
> 
> I noticed a fair number of new Linux users have trouble with stale
> program caches. Users install a package from a package manager or
> sources and then are confused when the new package is not used. They
> do not realize they need to run 'bash -r'; and most don't know where
> to begin searching.

You mean `hash -r' ?

> (...)
> so monitoring of common commands seems like a sensible way to
> implement the feature.

> A single terminal can monitor for a regex that looks for 'make
> install' and perhaps other common  installation commands. Multiple
> terminals seems like a trickier case, and could use a scheme where the
> source terminal broadcasts an 'update cache' message to other open
> terminals.

A shell is not a terminal. A terminal is a hardware (or emulated) device that
provides input/output capabilities. Nowadays most terminals are emulated
(gnome-terminal, xterm, urxvt, ...), but there are still physical terminals in
use. It seems quite complicated to have all terminal emulators and physical
terminals introduce this functionality.

And even if you manage to do that... how would that work? Terminals are only
aware of the input typed by the user, and the output provided by the programs
that run attached to that terminal device... that excludes a bunch of cases
like:

* Storing the command in a variable and then executing from the contents of that
  variable

* Running a script that installs said binaries in a silent way

...

Now, let's pretend you intention was to say that this logic should run in the
/SHELL/, not the terminal. A shell only knows about the input provided by the
user (through the terminal), for that specific process. So we still run into
similar problems:

* Binary is installed through a generic script

Also, consider that bash runs in a multitude of platforms, each using their own
package manager... it'd be a massive effort to gather all the possibilities into
a table of patterns that you test user input against... and even if you do it,
there are new platforms being introduced all the time, which means you'll have
to update the table all the time to capture the new cases... that seems like a
lot of work.



If you don't care about performance, why don't you just run `hash -r' through
PROMPT_COMMAND? That seems simple enough?

Segmentation fault in restore_tilde (bashline.c)

[Eduardo A . Bustamante López]

Found via fuzzing (of `read -e') using AFL,

To reproduce:

In a shell with `emacs' readline mode, type:

~/

e.g.

(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/dualbus/src/gnu/bash/bash 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
bash-5.0$ ~/
Program received signal SIGSEGV, Segmentation fault.
__memset_avx2_unaligned_erms () at 
../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:206
206 ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file 
or directory.

(gdb) bt
#0  __memset_avx2_unaligned_erms () at 
../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:206
#1  0x5575518e in internal_malloc (n=18446744073709551609, 
file=0x55762db7 "bashline.c", line=3015, flags=1) at malloc.c:870
#2  0x557568dc in sh_malloc (bytes=18446744073709551609, 
file=0x55762db7 "bashline.c", line=3015) at malloc.c:1302
#3  0x5569bf1f in sh_xmalloc (bytes=18446744073709551609, 
file=0x55762db7 "bashline.c", line=3015) at xmalloc.c:185
#4  0x556854f6 in restore_tilde (val=0x55990aa8 "~/.swt", 
directory_part=0x55990ac8 "~", '\"' ) at bashline.c:3015
#5  0x556855fe in maybe_restore_tilde (val=0x55990aa8 "~/.swt", 
directory_part=0x55990ac8 "~", '\"' ) at bashline.c:3031
#6  0x55682940 in command_word_completion_function 
(hint_text=0x55990a88 "~", '\"' , "/", state=0) at 
bashline.c:2157
#7  0x55718e12 in rl_completion_matches (text=0x55990a88 "~", '\"' 
, "/", entry_function=0x55680f3c 
) at complete.c:2185
#8  0x55714eb3 in gen_completion_matches (text=0x55990a88 "~", '\"' 
, "/", start=0, end=18, our_func=0x55680f3c 
, found_quote=2, quote_char=0)
at complete.c:1228
#9  0x55718352 in rl_complete_internal (what_to_do=9) at complete.c:2013
#10 0x55688187 in bash_specific_completion (what_to_do=9, 
generator=0x55680f3c ) at bashline.c:3812
#11 0x55687a6c in bash_complete_command_internal (what_to_do=9) at 
bashline.c:3690
#12 0x55687958 in bash_complete_command (ignore=1, ignore2=33) at 
bashline.c:3662
#13 0x55704df2 in _rl_dispatch_subseq (key=33, map=0x557cec60 
, got_subseq=0) at readline.c:852
#14 0x55705793 in _rl_dispatch_subseq (key=27, map=0x557cdc40 
, got_subseq=0) at readline.c:986
#15 0x557049e5 in _rl_dispatch (key=1437245440, map=0x557cdc40 
) at readline.c:798
#16 0x5570432d in readline_internal_char () at readline.c:632
#17 0x5570441a in readline_internal_charloop () at readline.c:659
#18 0x55704470 in readline_internal () at readline.c:671
#19 0x5570374f in readline (prompt=0x5598f3e8 "bash-5.0$ ") at 
readline.c:377
#20 0x555c5b2a in yy_readline_get () at ./parse.y:1476
#21 0x555c5945 in yy_getc () at ./parse.y:1409
#22 0x555c7648 in shell_getc (remove_quoted_newline=1) at ./parse.y:2334
#23 0x555ca34c in read_token (command=0) at ./parse.y:3230
#24 0x555c8e07 in yylex () at ./parse.y:2744
#25 0x555bfef6 in yyparse () at y.tab.c:1823
#26 0x555bf299 in parse_command () at eval.c:303
#27 0x555bf4a9 in read_command () at eval.c:347
#28 0x555bea6b in reader_loop () at eval.c:143
#29 0x555ba3e7 in main (argc=1, argv=0x7fffe7d8, 
env=0x7fffe7e8) at shell.c:805

(gdb) frame 4
#4  0x556854f6 in restore_tilde (val=0x55990aa8 "~/.swt", 
directory_part=0x55990ac8 "~", '\"' ) at bashline.c:3015
3015  ret = (char *)xmalloc (dl2 + 2 + l);

(gdb) p dl2
$3 = 1
(gdb) p l
$4 = -10
(gdb) p vl
$5 = 6
(gdb) p xl
$6 = 17

(gdb) p rl_line_buffer
$8 = 0x55a40e08 '\337' , ...



Due to the expansions performed in restore_tilde, xl ends up being larger than
vl, and thus, we end up requesting a negative amount of memory. I kind of
understand what the problem is, but I don't really know how to fix it.

Re: Which commit for a bug in 4.3.48 which is fixed in 4.4.23

[Eduardo A . Bustamante López]

On Fri, Sep 21, 2018 at 01:11:38PM +0200, Dr. Werner Fink wrote:
> Hi,
> 
> with 4.3.48 the line
> 
>   T="";echo ">${T//*/ }<"
> 
> leads to
> 
>   ><
> 
> but with 4.4.23 the correct result is given back
> 
>   > <
> 
> in the git repro I do not find any useful login entry for this

Check commit 34ec1876071b76d3654a418682e3f34ca9a72f1a:

+lib/glob/gmisc.c
+   - match_pattern_char, match_pattern_wchar: if passed an empty string,
+ return a match if the first character of the pattern is `*'
+
+subst.c
+   - pat_subst: change to allow empty strings to be replaced as long as
+ pattern matches empty string.  Report and fix from isabella parakiss
+ 

In particular, this change to match_pattern:

@@ -4435,7 +4437,7 @@ match_pattern (string, pat, mtype, sp, ep)
   size_t slen, plen, mslen, mplen;
 #endif

-  if (string == 0 || *string == 0 || pat == 0 || *pat == 0)
+  if (string == 0 || pat == 0 || *pat == 0)
 return (0);

Re: Special parameter ?

[Eduardo A . Bustamante López]

On Fri, Sep 21, 2018 at 04:17:11PM +0200, Enrique Soriano wrote:
> > You're seeing the status from the creation of the background job (which is
> > always 0), not from its completion.
> 
> Ah, I see.
> 
> Anyway, the behavior is not coherent with the manual page: in this
> case, $? has the status from the creation of the background job,
> that's not "the status of the most recently executed foreground
> pipeline".

The thing that creates the background job IS a foreground pipeline. It's a 
foreground pipeline that creates a background
job, if that makes sense.

dualbus@ubuntu:~$ cat t.sh
(exit 2)
(exit 3) & pid=$!
echo $?
wait "$pid"
echo $?

dualbus@ubuntu:~$ bash t.sh
0
3

I recommend to store the value of $? in a variable if you plan on using it 
afterwards, since it's extremely easy to end
up overriding its value.

Infinite loop on parse_and_execute command substituion

[Eduardo A . Bustamante López]

Found using AFL. The following causes bash 5.0 to get into an infinite loop:

(sorry for the junk reproducer, I don't know how to minimize non-crashing 
inputs)

bash-5.0$ IFS= read -r < <(base64 -d <<< 
'QCgusbGxw7GxsbGxsbGxsbGxKyQoKChzc3NzcwRpc3Nzc3Nzc3NzdnNzKSMpBymxsbGxsbGxsbGxsR4oGCgoVQ==');
 : ${REPLY@P}
(stuck)
^C

$ base64 -d <<< 
'QCgusbGxw7GxsbGxsbGxsbGxKyQoKChzc3NzcwRpc3Nzc3Nzc3NzdnNzKSMpBymxsbGxsbGxsbGxsR4oGCgoVQ=='
 | cat -A; echo
@(.M-1M-1M-1M-CM-1M-1M-1M-1M-1M-1M-1M-1M-1M-1M-1+$(((s^Disvss)#)^G)M-1M-1M-1M-1M-1M-1M-1M-1M-1M-1M-1^^(^X((U

0x0042f352 in discard_until (character=10) at ./parse.y:2665
2665  while ((c = shell_getc (0)) != EOF && c != character)
(gdb) bt
#0  0x0042f352 in discard_until (character=10) at ./parse.y:2665
#1  0x0042d176 in read_token (command=0) at ./parse.y:3253
#2  0x004287fe in yylex () at ./parse.y:2744
#3  0x004250de in yyparse () at y.tab.c:1823
#4  0x00424b8f in parse_command () at eval.c:303
#5  0x004bf238 in parse_and_execute (string=0x12a0a88 
"((s\004isvss)#)\a", from_file=0x53afa6 "command substitution", 
flags=20) at evalstring.c:352
#6  0x0047716b in command_substitute (string=0x12a0a88 
"((s\004isvss)#)\a", quoted=1, flags=0) at subst.c:6278
#7  0x0047bc51 in param_expand (string=0x13add88 
"@(.\261\261\261ñ\261\261\261\261\261\261\261\261\261\261+$(((s\004isvss)#)\a)",
 '\261' , "\036(\030((U", 
sindex=0x7ffea8aa1e6c, quoted=1, expanded_something=0x0, 
contains_dollar_at=0x7ffea8aa1e54, quoted_dollar_at_p=0x7ffea8aa1e68, 
had_quoted_null_p=0x7ffea8aa1e60, pflags=0) at subst.c:9415
#8  0x0047345a in expand_word_internal (word=0x7ffea8aa1f20, quoted=1, 
isexp=0, contains_dollar_at=0x0, expanded_something=0x0) at subst.c:9887
#9  0x004726ce in expand_prompt_string (string=0x13add08 
"@(.\261\261\261ñ\261\261\261\261\261\261\261\261\261\261+$(((s\004isvss)#)\a)",
 '\261' , "\036(\030((U", 
quoted=1, wflags=0) at subst.c:3804
#10 0x0042c80c in decode_prompt_string (string=0x13ad4c9 "A") at 
./parse.y:6065
#11 0x004851fc in string_transform (xc=80, v=0x12a0e48, 
s=0x13ad488 
"@(.\261\261\261ñ\261\261\261\261\261\261\261\261\261\261+$(((s\004isvss)#)\a)",
 '\261' , "\036(\030((U") at subst.c:7468
#12 0x0048224d in parameter_brace_transform (varname=0x12a05a8 "REPLY", 
value=0x13adc08 
"@(.\261\261\261ñ\261\261\261\261\261\261\261\261\261\261+$(((s\004isvss)#)\a)",
 '\261' , "\036(\030((U", ind=0, xform=0x1315e18 "P", 
rtype=64, quoted=0, 
pflags=0, flags=0) at subst.c:7616
#13 0x0047ebeb in parameter_brace_expand (string=0x129f808 
"${REPLY@P}", indexp=0x7ffea8aa34d4, quoted=0, pflags=0, 
quoted_dollar_atp=0x7ffea8aa3718, contains_dollar_at=0x7ffea8aa3704) at 
subst.c:8884
#14 0x0047b89a in param_expand (string=0x129f808 "${REPLY@P}", 
sindex=0x7ffea8aa371c, quoted=0, expanded_something=0x7ffea8aa389c, 
contains_dollar_at=0x7ffea8aa3704, quoted_dollar_at_p=0x7ffea8aa3718, 
had_quoted_null_p=0x7ffea8aa3710, pflags=0) at subst.c:9316
#15 0x0047345a in expand_word_internal (word=0x12a1848, quoted=0, 
isexp=0, contains_dollar_at=0x7ffea8aa3898, expanded_something=0x7ffea8aa389c) 
at subst.c:9887
#16 0x00487315 in shell_expand_word_list (tlist=0x12a17a8, eflags=31) 
at subst.c:11233
#17 0x00478b63 in expand_word_list_internal (list=0x12a02a8, eflags=31) 
at subst.c:11357
#18 0x004789aa in expand_words (list=0x12a02a8) at subst.c:10877
#19 0x004430e0 in execute_simple_command (simple_command=0x12a0448, 
pipe_in=-1, pipe_out=-1, async=0, fds_to_close=0x129f828) at execute_cmd.c:4278
#20 0x004412ab in execute_command_internal (command=0x12a0a08, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x129f828) at 
execute_cmd.c:840
#21 0x00445208 in execute_connection (command=0x12a1248, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x129f828) at 
execute_cmd.c:2689
#22 0x00441681 in execute_command_internal (command=0x12a1248, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x129f828) at 
execute_cmd.c:1013
#23 0x0044087b in execute_command (command=0x12a1248) at 
execute_cmd.c:394
#24 0x004247da in reader_loop () at eval.c:175
#25 0x00421aad in main (argc=1, argv=0x7ffea8aa3ed8, 
env=0x7ffea8aa3ee8) at shell.c:805
(gdb) n
2665  while ((c = shell_getc (0)) != EOF && c != character)
(gdb) 
2665  while ((c = shell_getc (0)) != EOF && c != character)
(gdb) p c
$1 = 32
(gdb) n
2665  while ((c = shell_getc (0)) != EOF && c != character)
(gdb) p c
$2 = 32

It seems to have been introduced by commit 
911ae06ca9e2b6e9778bfb2e0d8c04bfcc568beb, described as:

+  1/11
+  
+parse.y
+   - shell_getc: move code that decides whether to append a space to an
+ 

free: start and end chunk sizes differ - in _rl_isearch_cleanup

[Eduardo A . Bustamante López]

Found by fuzzing with AFL.

$ base64 -d >i
MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwEzAwMDAwMDAA
MDAwMDAwMDAwMDAwMDAwMDAw/zAwMDAwMDAwMDAwMDAwMDAwMDABExMZMDAw86Iw

$ cat -A i
^S000^@00M-^?000^A^S^S^Y000M-sM-"0

$ gdb -batch -ex=r -ex=bt --args ./bash --noprofile --norc -c 'set -o emacs; 
read -e < i'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
(failed i-search)`000': 
�000
malloc: unknown:0: assertion botched
malloc: 0x686408: allocated: last allocated from unknown:0
free: start and end chunk sizes differ
Aborting...
Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x779a7535 in __GI_abort () at abort.c:79
#2  0x004597bb in programming_error (format=0x55584b "free: start and 
end chunk sizes differ") at error.c:175
#3  0x00532422 in xbotch (mem=0x686408, e=8, s=0x55584b "free: start 
and end chunk sizes differ", file=0x0, line=0) at malloc.c:354
#4  0x005318aa in internal_free (mem=0x686408, file=0x0, line=0, 
flags=0) at malloc.c:960
#5  0x00531eaf in free (mem=0x686408) at malloc.c:1388
#6  0x00511b3b in _rl_scxt_dispose (cxt=0x686008, flags=0) at 
isearch.c:124
#7  0x0051331d in _rl_isearch_cleanup (cxt=0x686008, r=-1) at 
isearch.c:741
#8  0x00511c73 in rl_search_history (direction=1, invoking_key=19) at 
isearch.c:783
#9  0x00511c99 in rl_forward_search_history (sign=1, key=19) at 
isearch.c:144
#10 0x004fb3a0 in _rl_dispatch_subseq (key=19, map=0x575db0 
, got_subseq=0) at readline.c:852
#11 0x004fad39 in _rl_dispatch (key=19, map=0x575db0 
) at readline.c:798
#12 0x004faca9 in readline_internal_char () at readline.c:632
#13 0x004fc2d2 in readline_internal_charloop () at readline.c:659
#14 0x004fa5fe in readline_internal () at readline.c:671
#15 0x004fa4c0 in readline (prompt=0x55466f "") at readline.c:377
#16 0x004caa96 in edit_line (p=0x55466f "", itext=0x0) at 
./read.def:1104
#17 0x004c953a in read_builtin (list=0x0) at ./read.def:563
#18 0x0044b599 in execute_builtin (builtin=0x4c85e0 , 
words=0x628f88, flags=0, subshell=0) at execute_cmd.c:4677
#19 0x0044a96f in execute_builtin_or_function (words=0x628f88, 
builtin=0x4c85e0 , var=0x0, redirects=0x628b88, 
fds_to_close=0x628aa8, flags=0) at execute_cmd.c:5185
#20 0x004437c9 in execute_simple_command (simple_command=0x628b48, 
pipe_in=-1, pipe_out=-1, async=0, fds_to_close=0x628aa8) at execute_cmd.c:4449
#21 0x004412ab in execute_command_internal (command=0x628b08, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x628aa8) at 
execute_cmd.c:840
#22 0x00445208 in execute_connection (command=0x628c08, asynchronous=0, 
pipe_in=-1, pipe_out=-1, fds_to_close=0x628aa8) at execute_cmd.c:2689
#23 0x00441681 in execute_command_internal (command=0x628c08, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x628aa8) at 
execute_cmd.c:1013
#24 0x004bf587 in parse_and_execute (string=0x628408 "set -o emacs; 
read -e < i", from_file=0x5353fb "-c", flags=4) at evalstring.c:436
#25 0x00423845 in run_one_command (command=0x7fffeaaa "set -o 
emacs; read -e < i") at shell.c:1416
#26 0x00421920 in main (argc=5, argv=0x7fffe778, 
env=0x7fffe7a8) at shell.c:735

Re: expand_prompt_string segmentation faults

[Eduardo A . Bustamante López]

On Sun, Sep 16, 2018 at 07:24:13PM -0400, Chet Ramey wrote:
> On 9/16/18 5:15 PM, Eduardo A. Bustamante López wrote:
> > On Sun, Sep 16, 2018 at 02:39:07PM -0400, Chet Ramey wrote:
> >> On 9/16/18 4:12 AM, Eduardo A. Bustamante López wrote:
> >>> Here's another interesting crash:
> >>>
> >>> $ base64 < 53
> >>> MBgFEBAQMBgFEBAfEA==
> >>>
> >>> $ gdb -batch -ex=r -ex=bt --args ./bash --noprofile --norc -c 'PATH=; set 
> >>> -o emacs; IFS= read -re' < 53
> >>
> >> I can't reproduce this on Mac OS X or Fedora 28.
> 
> OK, let me look around some more with your debugging hints. If I find it,
> it will be in the next devel branch push.

Hah! Thanks for the fix to rl_do_undo. I appreciate the comments, it makes it
easier to understand what's going on.

I ran this latest push through the set of crashes I found. They're all fixed.

Thanks again!

Re: Bash-5.0-alpha available

[Eduardo A . Bustamante López]

On Sun, Sep 16, 2018 at 12:42:28PM -0400, Chet Ramey wrote:
(...)
> Bash-5.0 is about to go into beta test.

Interesting. Thanks.

> (...) While fuzzing has some
> interesting results, I haven't really seen any security problems or
> any serious bug that would cause me to hold up a release from the
> fuzzing-uncovered issues I've seen so far.

I agree. The parser became quite robust after shellshock. I haven't been able to
find any bugs there (and don't expect to find many anyways).

I do think there are still bugs around readline and I'm hoping to concentrate my
efforts there. It'd be nice to get rid of the multibyte character handling edge
cases and other oddities before the release. Although I'm not asking for you to
hold it, I just wanted to understand how much time I have left, and how to
organize my efforts around that schedule.

Re: expand_prompt_string segmentation faults

[Eduardo A . Bustamante López]

On Sun, Sep 16, 2018 at 02:39:07PM -0400, Chet Ramey wrote:
> On 9/16/18 4:12 AM, Eduardo A. Bustamante López wrote:
> > Here's another interesting crash:
> > 
> > $ base64 < 53
> > MBgFEBAQMBgFEBAfEA==
> > 
> > $ gdb -batch -ex=r -ex=bt --args ./bash --noprofile --norc -c 'PATH=; set 
> > -o emacs; IFS= read -re' < 53
> 
> I can't reproduce this on Mac OS X or Fedora 28.

Hm, I can reliably reproduce it with or without bash-malloc.

$ base64 -d <<< 'MBgFEBAQMBgFEBAfEA==' > 53

$ cat -v 53
0^X^E^P^P^P0^X^E^P^P^_^P

$ CC=clang CFLAGS='-O0 -ggdb' ./configure  && make -j4
(...)

$ ./bash --noprofile --norc -c 'PATH=; set -o emacs; IFS= read -re < 53' 
0
./bash: vim: No such file or directory
0rl_maybe_unsave_line: rl_undo_list=0x236c848
rl_maybe_unsave_line: rl_undo_list->next=(nil)
rl_maybe_unsave_line: rl_undo_list=0x236c848
rl_maybe_unsave_line: rl_undo_list->next=(nil)
0
./bash: vim: No such file or directory
rl_do_undo: rl_undo_list = (nil)
rl_do_undo: xfree(release = 0x236c848)
rl_maybe_unsave_line: rl_undo_list=0x2360188
rl_maybe_unsave_line: rl_undo_list->next=0x236c848
rl_do_undo: rl_undo_list = 0x236c848
rl_do_undo: xfree(release = 0x2360188)
rl_do_undo: rl_undo_list = 0x23601c0
rl_do_undo: xfree(release = 0x236c848)

malloc: unknown:0: assertion botched
malloc: 0x236c848: allocated: last allocated from unknown:0
free: called with already freed block argument
Aborting...Aborted


My interpretation of the issue is that the `^_' triggers an `undo', which then
causes bash to release that entry (i.e. `0x236c848' in the example above). Then
the `^P' somehow restores that free'd rl_undo_list entry and bash tries to free
it again during readline_internal_teardown.


Under GDB:

dualbus@ubuntu:~/src/gnu/bash$ gdb --args ./bash --noprofile --norc -c 'PATH=; 
set -o emacs; IFS= read -re < 53' 
GNU gdb (Ubuntu 8.2-0ubuntu1) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./bash...done.
(gdb) b rl_maybe_unsave_line 
Breakpoint 1 at 0x526128: file misc.c, line 354.
(gdb) b rl_do_undo 
Breakpoint 2 at 0x51e6a8: file undo.c, line 175.
(gdb) r
Starting program: /home/dualbus/src/gnu/bash/bash --noprofile --norc -c 
PATH=\;\ set\ -o\ emacs\;\ IFS=\ read\ -re\ \<\ 53
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0
[Detaching after fork from child process 20660]
/home/dualbus/src/gnu/bash/bash: vim: No such file or directory
0
Breakpoint 1, rl_maybe_unsave_line () at misc.c:354
354   if (_rl_saved_line_for_history)
(gdb) bt
#0  rl_maybe_unsave_line () at misc.c:354
#1  0x005266ea in rl_get_previous_history (count=1, key=16) at 
misc.c:618
#2  0x004fb350 in _rl_dispatch_subseq (key=16, map=0x575db0 
, got_subseq=0) at readline.c:852
#3  0x004face9 in _rl_dispatch (key=16, map=0x575db0 
) at readline.c:798
#4  0x004fac59 in readline_internal_char () at readline.c:632
#5  0x004fc282 in readline_internal_charloop () at readline.c:659
#6  0x004fa5ae in readline_internal () at readline.c:671
#7  0x004fa470 in readline (prompt=0x5546af "") at readline.c:377
#8  0x004caa46 in edit_line (p=0x5546af "", itext=0x0) at 
./read.def:1104
#9  0x004c94ea in read_builtin (list=0x0) at ./read.def:563
#10 0x0044b599 in execute_builtin (builtin=0x4c8590 , 
words=0x629428, flags=0, subshell=0) at execute_cmd.c:4677
#11 0x0044a96f in execute_builtin_or_function (words=0x629428, 
builtin=0x4c8590 , var=0x0, redirects=0x628d08, 
fds_to_close=0x628dc8, flags=0) at execute_cmd.c:5185
#12 0x004437c9 in execute_simple_command (simple_command=0x628c08, 
pipe_in=-1, pipe_out=-1, async=0, fds_to_close=0x628dc8) at execute_cmd.c:4449
#13 0x004412ab in execute_command_internal (command=0x628bc8, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x628dc8) at 
execute_cmd.c:840
#14 0x00445208 in execute_connection (command=0x628d88, asynchronous=0, 
pipe_in=-1, pipe_out=-1, fds_to_close=0x628dc8) at execute_cmd.c:2689
#15 0x00441681 in execute_command_internal (command=0x628d88, 
a

Re: expand_prompt_string segmentation faults

[Eduardo A . Bustamante López]

Here's another interesting crash:

$ base64 < 53
MBgFEBAQMBgFEBAfEA==

$ gdb -batch -ex=r -ex=bt --args ./bash --noprofile --norc -c 'PATH=; set -o 
emacs; IFS= read -re' < 53
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0
[Detaching after fork from child process 30991]
/home/dualbus/src/gnu/bash/bash: vim: No such file or directory
0
[Detaching after fork from child process 30992]
/home/dualbus/src/gnu/bash/bash: vim: No such file or directory

malloc: unknown:0: assertion botched
malloc: 0x630808: allocated: last allocated from unknown:0
free: called with already freed block argument
Aborting...
Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x779a7535 in __GI_abort () at abort.c:79
#2  0x004597bb in programming_error (format=0x5557be "free: called with 
already freed block argument") at error.c:175
#3  0x005322c2 in xbotch (mem=0x630808, e=1, s=0x5557be "free: called 
with already freed block argument", file=0x0, line=0) at malloc.c:354
#4  0x0053155c in internal_free (mem=0x630808, file=0x0, line=0, 
flags=0) at malloc.c:931
#5  0x00531d4f in free (mem=0x630808) at malloc.c:1388
#6  0x004b5150 in xfree (string=0x630808) at xmalloc.c:150
#7  0x0051e94c in rl_do_undo () at undo.c:240
#8  0x0051eb23 in rl_revert_line (count=1, key=0) at undo.c:312
#9  0x004fa7f9 in readline_internal_teardown (eof=1) at readline.c:471
#10 0x004fa591 in readline_internal () at readline.c:672
#11 0x004fa440 in readline (prompt=0x55466f "") at readline.c:377
#12 0x004caa16 in edit_line (p=0x55466f "", itext=0x0) at 
./read.def:1104
#13 0x004c94ba in read_builtin (list=0x0) at ./read.def:563
#14 0x0044b599 in execute_builtin (builtin=0x4c8560 , 
words=0x628f88, flags=0, subshell=0) at execute_cmd.c:4677
#15 0x0044a96f in execute_builtin_or_function (words=0x628f88, 
builtin=0x4c8560 , var=0x0, redirects=0x0, fds_to_close=0x628ce8, 
flags=0) at execute_cmd.c:5185
#16 0x004437c9 in execute_simple_command (simple_command=0x628c08, 
pipe_in=-1, pipe_out=-1, async=0, fds_to_close=0x628ce8) at execute_cmd.c:4449
#17 0x004412ab in execute_command_internal (command=0x628bc8, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x628ce8) at 
execute_cmd.c:840
#18 0x00445208 in execute_connection (command=0x628d48, asynchronous=0, 
pipe_in=-1, pipe_out=-1, fds_to_close=0x628ce8) at execute_cmd.c:2689
#19 0x00441681 in execute_command_internal (command=0x628d48, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x628ce8) at 
execute_cmd.c:1013
#20 0x004bf527 in parse_and_execute (string=0x628408 "PATH=; set -o 
emacs; IFS= read -re", from_file=0x5353fb "-c", flags=4) at evalstring.c:436
#21 0x00423845 in run_one_command (command=0x7fffea92 "PATH=; set 
-o emacs; IFS= read -re") at shell.c:1416
#22 0x00421920 in main (argc=5, argv=0x7fffe758, 
env=0x7fffe788) at shell.c:735


Bash compiled with:

`CC=clang CFLAGS='-O0 -ggdb' ./configure && make -j4' from `devel' 
(b52e30b8dd5bba5aed7b13040c9d4f7a0c1990cc) on Linux
(Ubuntu 18.04)

expand_prompt_string segmentation faults

[Eduardo A . Bustamante López]

Found the following two cases by fuzzing with AFL:

# Case #1: array_expand_index
bash <<'EOF'
x='${p[--b[?]]'; echo ${x@P}
EOF

# Case #1 backtrace
: <<'EOF'
Program received signal SIGSEGV, Segmentation fault.
0x0080e0d3 in __strchr_sse2 ()
#0  0x0080e0d3 in __strchr_sse2 ()
#1  0x006d954b in mbschr (s=0x0, c=91) at mbschr.c:90
#2  0x0058acdf in valid_array_reference (name=0x0, flags=0) at 
arrayfunc.c:899
#3  0x0049c4e9 in bind_int_variable (lhs=0x0, rhs=0xbb5228 "-1", 
flags=0) at variables.c:3371
#4  0x004c632c in expr_bind_variable (lhs=0x0, rhs=) at 
expr.c:333
#5  exp0 () at expr.c:1015
#6  exp1 () at expr.c:983
#7  0x004c54ae in exppower () at expr.c:938
#8  0x004c4cf8 in exp2 () at expr.c:863
#9  0x004c4695 in exp3 () at expr.c:837
#10 expshift () at expr.c:813
#11 0x004c3d95 in exp4 () at expr.c:783
#12 exp5 () at expr.c:761
#13 0x004c3a61 in expband () at expr.c:743
#14 expbxor () at expr.c:724
#15 0x004c3621 in expbor () at expr.c:705
#16 expland () at expr.c:678
#17 0x004c2e01 in explor () at expr.c:650
#18 expcond () at expr.c:603
#19 0x004c1f2b in expassign () at expr.c:488
#20 0x004be48e in expcomma () at expr.c:472
#21 subexpr (expr=0xbc9a48 "--b[?]") at expr.c:450
#22 0x004bdba0 in evalexp (expr=0xbc9a48 "--b[?]", flags=, validp=0x7fffce14) at expr.c:415
#23 0x00589d81 in array_expand_index (var=, s=, len=, flags=) at arrayfunc.c:952
#24 0x0058b7f5 in array_value_internal (s=0xbc9a08 "p[--b[?]]", 
quoted=, flags=1, rtype=0x7fffce9c, indp=) at 
arrayfunc.c:1133
#25 0x0053eed1 in parameter_brace_expand_word (name=0xbc9a08 
"p[--b[?]]", var_is_special=0, quoted=1, pflags=, 
indp=0x7fffcf40) at subst.c:6584
#26 0x00536c7b in parameter_brace_expand (string=, 
quoted=, pflags=, contains_dollar_at=, indexp=, quoted_dollar_atp=) at subst.c:8702
#27 param_expand (string=0xbc5fe8 "${p[--b[?]]", sindex=, 
quoted=, expanded_something=, 
contains_dollar_at=, quoted_dollar_at_p=, 
had_quoted_null_p=0x0, pflags=) at subst.c:9316
#28 0x00510893 in expand_word_internal (word=0x7fffd0b0, 
quoted=, isexp=, contains_dollar_at=, expanded_something=) at subst.c:9887
#29 0x0050f595 in expand_prompt_string (string=0xbc7ec8 "${p[--b[?]]", 
quoted=1, wflags=) at subst.c:3804
#30 0x00420e71 in decode_prompt_string (string=) at 
./parse.y:6065
#31 0x0055059c in string_transform (xc=, v=0xbc7dc8, 
s=0xbc5fc8 "${p[--b[?]]") at subst.c:7468
#32 0x0054a2b5 in parameter_brace_transform (varname=, 
value=, ind=, xform=, rtype=0, 
quoted=, pflags=0, flags=) at subst.c:7616
#33 0x0053bb17 in parameter_brace_expand (string=, 
quoted=, pflags=, contains_dollar_at=, indexp=, quoted_dollar_atp=) at subst.c:8884
#34 param_expand (string=0xbc7e68 "${REPLY@P}", sindex=, 
quoted=, expanded_something=, 
contains_dollar_at=, quoted_dollar_at_p=, 
had_quoted_null_p=, pflags=) at subst.c:9316
#35 0x00510893 in expand_word_internal (word=0xbc7828, 
quoted=, isexp=, contains_dollar_at=, expanded_something=) at subst.c:9887
#36 0x00529560 in shell_expand_word_list (tlist=, 
eflags=0) at subst.c:11233
#37 expand_word_list_internal (list=, eflags=) at 
subst.c:11357
#38 0x0046f341 in execute_simple_command (simple_command=, pipe_in=-1, pipe_out=-1, async=, fds_to_close=) at execute_cmd.c:4278
#39 execute_command_internal (command=, asynchronous=, pipe_in=, pipe_out=, 
fds_to_close=) at execute_cmd.c:840
#40 0x0046b5cb in execute_connection (command=, 
asynchronous=, pipe_in=, pipe_out=, fds_to_close=) at execute_cmd.c:2689
#41 execute_command_internal (command=0xbc5e48, asynchronous=, 
pipe_in=, pipe_out=, fds_to_close=) at execute_cmd.c:1013
#42 0x00605bcc in parse_and_execute (string=, 
from_file=, flags=4) at evalstring.c:436
#43 0x00409a8c in run_one_command (command=) at 
shell.c:1416
#44 0x004063a7 in main (argc=, argv=, 
env=) at shell.c:735
EOF

# Case #2
bash <<'EOF'
x='$[++K[+]]/'; echo ${x@P}
EOF

# Case #2 backtrace
: <<'EOF'
Program received signal SIGSEGV, Segmentation fault.
0x0080e0d3 in __strchr_sse2 ()
#0  0x0080e0d3 in __strchr_sse2 ()
#1  0x006d954b in mbschr (s=0x0, c=91) at mbschr.c:90
#2  0x0058acdf in valid_array_reference (name=0x0, flags=0) at 
arrayfunc.c:899
#3  0x0049c4e9 in bind_int_variable (lhs=0x0, rhs=0xbb5248 "1", 
flags=0) at variables.c:3371
#4  0x004c632c in expr_bind_variable (lhs=0x0, rhs=) at 
expr.c:333
#5  exp0 () at expr.c:1015
#6  exp1 () at expr.c:983
#7  0x004c54ae in exppower () at expr.c:938
#8  0x004c4cf8 in exp2 () at expr.c:863
#9  0x004c4695 in exp3 () at expr.c:837
#10 expshift () at expr.c:813
#11 0x004c3d95 in exp4 () at expr.c:783
#12 exp5 () at expr.c:761
#13 0x004c3a61 in expband () at expr.c:743
#14 expbxor () at 

Re: herestring brace expansion

[Eduardo A . Bustamante López]

On Sun, Sep 02, 2018 at 11:55:21PM +0300, Eliran Gonen wrote:
[...]
> Description:
>   “Here Strings” section in the bash(1) manual page claims that “word 
> undergoes brace expansion” while it does not.

Hi Eliran,

This was reported by Conor McCarthy back in February 2017:
http://lists.gnu.org/archive/html/bug-bash/2017-02/msg00018.html

and fixed in the `devel' branch of the git repository in commit 
ac495185aab17ef6030a02ddc0be33380a27a94d
(http://git.savannah.gnu.org/cgit/bash.git/commit/?h=devel=ac495185aab17ef6030a02ddc0be33380a27a94d).

It's also fixed in the 5.0 (alpha) version.

Re: Assignment of $* to a var removes spaces on unset IFS.

[Eduardo A . Bustamante López]

On Mon, Aug 13, 2018 at 10:52:20PM -0400, Bize Ma wrote:
(...)
> That version is not even beta, it is still alpha, are you asking that
> everyone should use
> non-released (and not yet tested as beta) alpha release ?
(...)

I didn't say that. My point is that, in the context of bug reports, it's
important that you always test against the *unreleased* version of the code,
since that's where most bug fixes are queued up (or at least the ones that break
backwards compatibility).

> So? It is not solved in that thread, it is even said that:
> 
>  > This is a bug in bash and it should be fixed, not excused.
> 
> To which I agree. After a year, nothing else have been said about it.
> 

For some reason the threading broke in the archive browser. Here's the response
from Chet:

http://lists.nongnu.org/archive/html/bug-bash/2017-10/msg00023.html

The fix for this issue is already in the Git repository (devel branch) and in
Bash 5.0. 

I cannot answer for Chet, but please consider this:

The fix for this bug breaks backwards compatibility, so that means it cannot be
a patch-level fix for 4.4. It should either be a minor version increase (4.5) or
go into the next major release, which is 5.0 (which was already declared alpha
state, and includes the fix).

> It seems about time to get it solved. Or say that it won't be.

Again, it's fixed already and the fix is scheduled to go out in the next
release (5.0).

Given that, do you think this bug is severe enough to have to issue a new minor
version just for it? (4.5),

or can we just wait for 5.0 to come out which fixes this and a bunch of other
bugs? (see the CHANGES file, in particular, entry `oo',
http://git.savannah.gnu.org/cgit/bash.git/diff/CHANGES?h=bash-5.0-testing=9a51695bed07d37086c352372ac69d0a30039a6b)

Re: Assignment of $* to a var removes spaces on unset IFS.

[Eduardo A . Bustamante López]

On Mon, Aug 13, 2018 at 09:36:23PM -0400, Bize Ma wrote:
(...)
> Please, do not waste our time in incorrect claims.
> 
> Do your homework and test !
(...)

Did you test Bash 5.0? Because that's where the current bug fixes are going to.
I don't see Chet releasing a new version of 4.4 to fix something that's already
fixed there.

dualbus@ubuntu:~/src/gnu/bash$ cat /tmp/script 
echo $BASH_VERSION
set -- "  foo  ""  bar  baz  "  "  quux  "
unset IFS
a=$*
b="$*"
printf '[%s]' "$a" "$b"; echo

dualbus@ubuntu:~/src/gnu/bash$ bash /tmp/script 
4.4.19(1)-release
[foo bar baz quux][  foo bar  baz quux  ]

dualbus@ubuntu:~/src/gnu/bash$ ./bash /tmp/script 
5.0.0(1)-alpha
[  foo bar  baz quux  ][  foo bar  baz quux  ]

Please read this thread:

* http://lists.nongnu.org/archive/html/bug-bash/2017-09/msg00058.html

Re: bash 4.3.48(1) and repo: processing of commands halts at first semicolon

[Eduardo A . Bustamante López]

On Mon, Jul 02, 2018 at 04:58:06PM -0700, toww wrote:
[...]
> Description:
> There seems to be a new incompatibility or incorrect string parsing in how
> the Google "repo" utility interacts with the bash shell. Previously I could
> invoke the "repo" command, passing to it a string to execute multiple
> commands separated by semicolons (';' characters), and all commands would
> execute. (...)

How is it a bug in GNU bash? The problem seems to be in the "repo"
utility. I suggest you direct this report to them instead.

> (...) Now with the latest bash 4.3.38, only the first command executes,
> and processing stops at the semicolon.

4.3.38 was released around 3 years ago, so it's not that "new". Bash is
currently at 4.4.23.

Re: command_not_found_handle strange behaviour

[Eduardo A . Bustamante López]

On Wed, Mar 28, 2018 at 06:42:21PM +0300, Кириллов Дима wrote:
[...]
> Bash Version: 4.4
> Patch Level: 19
> Release Status: release
> 
> Description:
>   I can't understand why read works without subshell before it, 
> but with subshell before read it stops
> 
> command_not_found_handle()
> {
> (true) # subshell call
> read line
> echo "$line"
> }
> 
>   bash-4.4$ foo
>   
>   [1]+  Stopped foo
> 
> And without subshell call it works fine:

I can reproduce this.


A few observations:

(1) `read' is stopped because it's in a different process group (i.e. it's a
background process), and background process that attempt to read from the
controlling terminal receive a SIGTTIN from the kernel.

(2) The subshell is a red herring. You can replace it with anything that causes
bash to fork a new process, e.g. /bin/true

(3) This behaviour was apparently introduced in bash-20151113 snapshot
(f9b024c839a3bbb9c6c2a98a16b1cf362010340a), although due to the nature of that
change, I suspect this issue has been present for far more time, and it was just
"uncovered" by this commit.

(4) This only affects interactive shells.

(5) The command_not_found_handle executes in a child process, i.e. the process
that was intended to be used by the not found program. Apparently it also
executes in its own process group, so it means that it should consistently
behave as a background process. 

Re: weird bash5 bug with ``for i; do echo; done; echo in''

[Eduardo A . Bustamante López]

On Sat, Mar 24, 2018 at 06:05:07PM +, Dennis Williamson wrote:
[...]
> >> $ bash5 -c 'for i; do echo; done; echo in'
> >> bash5: -c: line 0: syntax error near unexpected token `in'
> >> bash5: -c: line 0: `for i; do echo; done; echo in'
[...]
> bash5 -c 'for i; do echo in; done'
> 
> fails in the same way.

A shorter reproducer:

  for i do : in; done

f698849a75fc781472806182c3dc930077a5d828 is the last known good commit that
behaves correctly. It seems like the changes committed in
124d67cde092330257c69e8c9bd64af40ffd9b73 
(http://git.savannah.gnu.org/cgit/bash.git/commit/?h=devel=124d67cde092330257c69e8c9bd64af40ffd9b73)
introduced this problem:

|  3/25
|  
| parse.y
|   - read_token_word: if we read a WORD immediately after a CASE, FOR, or
| SELECT, increment `expecting_in_token' to note that we're going to
| be reading IN as the next non-newline token
|   - special_case_tokens: if we're expecting an IN (expecting_in_token > 0)
| and we have an "in", return IN and decrement expecting_in_token. This
| is a better implementation of Posix grammar rule 6 than the previous
| code in that function (which is still there for posterity, but should
| eventually be removed).  Fixes bug reported by Torka Noda
| 


The following is not a fix, but it does make the problem go away:

diff --git a/parse.y b/parse.y
index 29b59f81..9efcc4aa 100644
--- a/parse.y
+++ b/parse.y
@@ -5315,7 +5315,7 @@ got_token:
   if (word_top < MAX_CASE_NEST)
word_top++;
   word_lineno[word_top] = line_number;
-  expecting_in_token++;
+  //expecting_in_token++;
   break;
 }
 

The problem is that expecting_in_token is increased unconditionally for a `for'
statement, but the `in WORD' part of the loop is optional, so this confuses the
parser.

Re: Error: conflicting types for ‘sbrk’

[Eduardo A . Bustamante López]

On Wed, Mar 21, 2018 at 11:07:45AM -0300, Larissa Braz wrote:
> Hi,
> 
>  I found the following compilation error:
> 
> xmalloc.c:51:14: error: conflicting types for ‘sbrk’
>  extern char *sbrk();
>   ^
> In file included from xmalloc.c:29:0:
> /usr/include/unistd.h:1043:14: note: previous declaration of ‘sbrk’ was here
>  extern void *sbrk (intptr_t __delta) __THROW;
[...]
> #if defined (HAVE_SBRK) && !HAVE_DECL_SBRK
> extern char *sbrk();
> #endif

This declaration is conditional. It will only be evaluated if

  HAVE_SBRK=1
  HAVE_DECL_SBRK=0

These are set by the configure script based on what it can detect from your
system. It is strange that it detects a
working sbrk implementation, but no sbrk declaration.

* What source tree are you building? (i.e. git devel branch, git master branch,
  tarball?)
* Also, are you passing any special parameters to `configure' or `make'?
* Can you share the following blocks from your config.log and config.h after
  running the configure script?


$ grep -i 'checking for.*sbrk' config.log -A5
configure:10615: checking for sbrk
configure:10615: gcc -o conftest -ggdb -O0 -Wno-parentheses 
-Wno-format-security   conftest.c  >&5
configure:10615: $? = 0
configure:10615: result: yes
configure:10615: checking for fpurge
configure:10615: gcc -o conftest -ggdb -O0 -Wno-parentheses 
-Wno-format-security   conftest.c  >&5
--
configure:14469: checking for working sbrk
configure:14496: gcc -o conftest -ggdb -O0 -Wno-parentheses 
-Wno-format-security   conftest.c -ldl  >&5
configure:14496: $? = 0
configure:14496: ./conftest
configure:14496: $? = 0
configure:14506: result: yes

$ grep SBRK config.h
#define HAVE_DECL_SBRK 1
#define HAVE_SBRK 1

Re: Strange/incorrect behavior of a fake executable file

[Eduardo A . Bustamante López]

On Wed, Mar 14, 2018 at 08:43:45AM -0400, Greg Wooledge wrote:
> > 1) You do not provide a hash-bang (i.e. #!/bin/bash), which means that
> > /bin/sh will be used (
> 
> No, that's incorrect.  When bash tells the operating system to run
> this program, the operating system will return ENOEXEC (Exec format
> error).  Bash (and every other shell) will catch this and spawn a
> child copy of itself to try to interpret the program as a script.

Thank you for the correction.

What misled me is that the *p variants of exec() function do fallback to
/bin/sh,

> Special semantics for execlp() and execvp() 
> [...]
> If the header of a file isn't recognized (the attempted execve(2) failed with
> the error ENOEXEC), these functions will execute the shell (/bin/sh) with the
> path of the file as its first argument. (If this attempt fails, no further
> searching is done.) 

  from: https://linux.die.net/man/3/execvp

Of course, this is incorrect for bash's case, since it uses execve(), which
does not exhibit this special behavior.

Re: Bash's malloc does not work within qemu-user when compiled with PIE enabled

[Eduardo A . Bustamante López]

> On 2/14/18 3:38 AM, Raphael Hertzog wrote:
[...]
> > Isn't it possible that sbrk() returns that pointer to you and you treat
> > it as being an error instead of a valid address?

The problem is in QEMU:

dualbus@ubuntu:~$ cat sbrk.c 
#include 
#include 
int main() {
fprintf(stderr, "%ld\n", (long)sbrk(0));
fprintf(stderr, "%ld\n", (long)sbrk(4096));
return 0;
}

dualbus@ubuntu:~$ gcc -o sbrk sbrk.c

dualbus@ubuntu:~$ ./sbrk 
94870340603904
94870340603904

dualbus@ubuntu:~$ qemu-x86_64 ./sbrk 
warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
274880012288
-1

dualbus@ubuntu:~$ gcc -o sbrk sbrk.c -no-pie

dualbus@ubuntu:~$ qemu-x86_64 ./sbrk 
warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
6299648
6299648


i.e. the QEMU emulated `brk' system call returns -1 (ENOMEM) to a simple 4096
byte request. This causes bash's `pagealign()' function to fail, and thus the
error propagates into internal_malloc, and all the way up to xmalloc. See:

  
http://git.savannah.gnu.org/cgit/bash.git/tree/lib/malloc/malloc.c?h=devel#n765


I suspect the problem might be in the implementation of the `brk' system call in
QEMU, i.e. do_brk:

  https://github.com/qemu/qemu/blob/master/linux-user/syscall.c#L1091

Re: bug-bash

[Eduardo A . Bustamante López]

On Thu, Feb 08, 2018 at 05:51:06PM -0800, Nolan wrote:
> On 02/08/2018 05:04 PM, Eduardo Bustamante wrote:
> > On Thu, Feb 8, 2018 at 4:23 PM, Nolan <4030...@gmail.com> wrote:
> > > I have found a 'result' of a command that cannot be a feature.
> > > 
> > > Enter command, command executes, prints exit at the prompt.
> > > 
> > > Goes to next line in terminal showing the "#" prompt.
> > > 
> > > A "whoami" says root.
> > > 
> > > Is this known?
> > > Do you need screen captures of my terminal session?
> > > 
> > > Nolan
> > > 
> > 
> > What command are you running? Also run: type -a COMMAND_THAT_YOU_ARE_RUNNING
> > 
> > Most likely, you're running something that causes your shell to exit,
> > and if you logged in as root and then changed to your user, it
> > explains why you're left at a root owned shell.
> > 
> 
> I always log in a user, and only change to root as needed.
> 
> See the attached screen captures.
> 
> I was reading the Bash manual and practicing entering
> the commands. Thereby learning them.
> 
> If you need anything else just hollow.
> 
> Thanks for the reply.
> Nolan
> 
> ps: the 671 byte file is #1

> root@databank:~# su nolan
> nolan@databank:/root$ cd ~/bash
> nolan@databank:~/bash$ exec < ls.sh
> nolan@databank:~/bash$ ls -lA
> total 40
> -rw-r--r-- 1 nolan nolan   671 Feb  8 01:32 bug_in_bash.\?
> -rw-r--r-- 1 nolan nolan 7 Feb  8 01:25 ls.sh
> -rwxr-xr-x 1 nolan nolan   156 Feb  7 02:10 myCopy
> -rwxr-xr-x 1 nolan nolan   130 Feb  7 02:39 myHello
> -rw--- 1 nolan nolan 12288 Feb  7 02:39 .myHello.swp
> -rw-r--r-- 1 nolan nolan74 Jan 29 13:14 test8
> -rw-r--r-- 1 nolan nolan   275 Feb  7 02:20 testFile
> -rwxr-xr-x 1 nolan nolan   144 Feb  7 02:33 varassign
> nolan@databank:~/bash$ exit
> root@databank:~# whoami
> root
> root@databank:~# 
> 

> nolan@databank:~/bash$ echo ls -lA >> ls.sh
> nolan@databank:~/bash$ read < ls.sh
> nolan@databank:~/bash$ readline < ls.sh
> bash: readline: command not found
> nolan@databank:~/bash$ exec < ls.sh
> nolan@databank:~/bash$ ls -lA
> total 36
> -rw-r--r-- 1 nolan nolan 7 Feb  8 01:25 ls.sh
> -rwxr-xr-x 1 nolan nolan   156 Feb  7 02:10 myCopy
> -rwxr-xr-x 1 nolan nolan   130 Feb  7 02:39 myHello
> -rw--- 1 nolan nolan 12288 Feb  7 02:39 .myHello.swp
> -rw-r--r-- 1 nolan nolan74 Jan 29 13:14 test8
> -rw-r--r-- 1 nolan nolan   275 Feb  7 02:20 testFile
> -rwxr-xr-x 1 nolan nolan   144 Feb  7 02:33 varassign
> nolan@databank:~/bash$ exit
> root@databank:~# whoami
> root
> root@databank:~# 

Please keep the bug-bash mailing list in your replies.


There's no bug here.  You log into the machine as 'root', then 'su' into your
user; then, you run 'exec < ls.sh', which changes the input of the current
shell from terminal input to whatever is in 'ls.sh'. Once the shell consumes
the 'ls.sh' file, it detects the end-of-file, and exits, leaving you at the
'root' shell, where you started.


Also, for general inquiries, use help-bash,  not bug-bash.

Re: xmalloc crash

[Eduardo A . Bustamante López]

On Sat, Jan 06, 2018 at 09:13:17PM -0700, Bob Proulx wrote:
> > You can see that the brk() system call actually succeeds.
> 
> If you are running under the Linux kernel in the default configuration
> then memory overcommit is enabled.
> 
>   $ sysctl vm.overcommit_memory
> 
> With overcommit enabled brk(), and malloc(), will always succeed even
> if there isn't really enough memory.  Neither will fork().  It doesn't
> mean there actually is that much memory available.  Later if the
> kernel runs out of memory to implement the needs then it will fail and
> trigger the OOM Out Of Memory Killer to virtually kill -9 processes
> until enough memory is freed up to balance the books.  Processes
> killed by the OOM have no opportunity to clean up or log anything.

This is not related to Linux VM overcommit / OOM killer.


>From what I can see, Bash's internal memory allocator (lib/malloc/malloc.c,
internal_malloc) is actually unable to allocate the ~580 MiB of VM, even if
there's enough free physical memory to back it up.

See:
  
http://git.savannah.gnu.org/cgit/bash.git/tree/lib/malloc/malloc.c?h=devel=ce0469bfbe4176802d4ffd1f272010e14bc4ed81#n777

An easy way to compare is to compile bash with:

 ./configure --with-bash-malloc && make

 vs

 ./configure --without-bash-malloc && make

And then run the reported brace expansion on both.


In my case, the Bash internal allocator fails around 580 MiB. Unlike Bash
compiled to use the system's malloc (in my case, glibc malloc), which allocated
a few GiBs of memory before I had to kill the process (system started to swap).


So there's indeed a limitation in Bash's internal allocator which is enabled by
the configure script in some systems. I don't know if there's a reason for
having that artificial limitation in place though.

Re: xmalloc crash

[Eduardo A . Bustamante López]

On Sat, Jan 06, 2018 at 01:42:25AM +0200, Alfred Baroti wrote:
> Hi,
> I found this from long long time ago.
> Is this a serious bug?
[...]

This is not a serious bug at all. It's just a memory allocation failure.

> [root@n1x ~]#su nix
> nix@n1x:/root$ printf "%s\n"
> {{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}
> bash: xmalloc: .././braces.c:793: cannot allocate 7329062664 bytes
> (614682624 bytes allocated)

You are performing a brace expansion which generates 62^5 combinations and
requires bash to allocate 7329062664 bytes (6.83 GiB) from the operating
system. The allocation fails at 614682624 bytes (586 MiB).

Once bash is unable to allocate memory, it will crash (xmalloc is a wrapper
around malloc which terminates execution on allocation errors).

Looking at a program execution trace for bash 4.4.12(1)-release (as distributed
by Ubuntu):

dualbus@ubuntu:~$ LC_ALL=C strace -fo /dev/stdout bash -c ': 
{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}{{a..z},{A..Z},{0..9}}'
 | cat -n | tail -n6
bash: xmalloc: .././braces.c:793: cannot allocate 7329062664 bytes 
(613113856 bytes allocated)
116508  28672 brk(0x254ad000)   = 0x254ad000
116509  28672 brk(0x254ae000)   = 0x254ae000
116510  28672 brk(0x254af000)   = 0x254af000
116511  28672 write(2, "bash: xmalloc: .././braces.c:793"..., 95) = 95
116512  28672 exit_group(2) = ?
116513  28672 +++ exited with 2 +++

You can see that the brk() system call actually succeeds.

So the allocation fails, even if the system has enough memory to handle it.
Upon closer inspection:

dualbus@ubuntu:~/src/gnu/bash/lib/malloc$ cat -n malloc.c | sed -n 
'777,781p'
   777/* Silently reject too-large requests. */
   778if (nunits >= NBUCKETS) {
   779abort();
   780  return ((PTR_T) NULL);
   781}

The internal_malloc function that bash uses on some systems (instead of the
libc provided malloc), including Linux, has a special provision that rejects
memory requests that are too large (NBUCKETS = 30).

So you have two options:

(1) Compile bash by providing `--without-bash-malloc' to the `configure' script
(2) Find another option to generate all these permutations that doesn't require
requesting >580 MiB at once.


Question for Chet: Is there a reason we use bash-malloc in Linux systems by
default, instead of the typical glibc malloc which is available on most of
them?

Re: Integer overflow in command substitution

[Eduardo A . Bustamante López]

On Thu, Nov 16, 2017 at 06:50:59AM -0500, Siteshwar Vashisht wrote:
[...]
> Bash Version: 4.4
> Patch Level: 12
> Release Status: release
> 
> Repeat-By:
> $ bash -c 'true $(yes )'
> bash: xrealloc: cannot allocate 18446744071562067968 bytes

Interesting! I think this might explain the behavior reported in:
  https://lists.nongnu.org/archive/html/bug-bash/2017-11/msg00051.html

Re: Segfault: Lone surrogate followed by locale change

[Eduardo A . Bustamante López]

On Fri, Nov 10, 2017 at 01:59:46PM +0100, Egmont Koblinger wrote:
[...]
> On Ubuntu Artful (glibc-2.26), this tiny snippet reproducibly crashes bash:
> 
> LC_ALL=en_US.UTF-8 # or any other UTF-8 locale
> echo -e '\ud800'   # or any other lone high or low surrogate
> LC_ALL=en_US.UTF-8 # or any available locale

I'm able to reproduce it in the `devel' branch:

(gdb) r
Starting program: /home/dualbus/src/gnu/build-bash-devel/bash 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
dualbus@ubuntu:~/src/gnu/build-bash-devel$ LC_ALL=en_US.UTF-8
dualbus@ubuntu:~/src/gnu/build-bash-devel$ echo -e '\ud800' 
���
dualbus@ubuntu:~/src/gnu/build-bash-devel$ LC_ALL=en_US.UTF-8 

Program received signal SIGSEGV, Segmentation fault.
__gconv_close (cd=0x0) at gconv_close.c:35
35  gconv_close.c: No such file or directory.
(gdb) bt
#0  __gconv_close (cd=0x0) at gconv_close.c:35
#1  0x7662eb7f in iconv_close (cd=) at iconv_close.c:35
#2  0x5576dcb8 in u32reset () at ../../../bash/lib/sh/unicode.c:102
#3  0x556e9f7a in set_locale_var (var=0x603000171a00 "LC_ALL", 
value=0x602000207430 "en_US.UTF-8") at ../bash/locale.c:215
#4  0x556432e1 in sv_locale (name=0x603000171a00 "LC_ALL") at 
../bash/variables.c:5671
#5  0x55641c8c in stupidly_hack_special_variables (name=0x603000171a00 
"LC_ALL") at ../bash/variables.c:5280
#6  0x556759a8 in do_assignment_internal (word=0x602000204770, 
expand=1) at ../bash/subst.c:3225
#7  0x55675d08 in do_word_assignment (word=0x602000204770, flags=0) at 
../bash/subst.c:3263
#8  0x556a335e in expand_word_list_internal (list=0x602000205d70, 
eflags=31) at ../bash/subst.c:11080
#9  0x556a0b25 in expand_words (list=0x602000205d70) at 
../bash/subst.c:10635
#10 0x55628701 in execute_simple_command 
(simple_command=0x603000171940, pipe_in=-1, pipe_out=-1, async=0, 
fds_to_close=0x6020002073f0)
at ../bash/execute_cmd.c:4230
#11 0x556167b4 in execute_command_internal (command=0x603000171910, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x6020002073f0)
at ../bash/execute_cmd.c:821
#12 0x55614edb in execute_command (command=0x603000171910) at 
../bash/execute_cmd.c:393
#13 0x555e164f in reader_loop () at ../bash/eval.c:172
#14 0x555dc882 in main (argc=1, argv=0x7fffe138, 
env=0x7fffe148) at ../bash/shell.c:804

(gdb) frame 2
#2  0x5576dcb8 in u32reset () at ../../../bash/lib/sh/unicode.c:102
102   iconv_close (localconv);
(gdb) p localconv
$1 = (iconv_t) 0x0


The problem is that Bash treats UTF-8 as a special case, so it doesn't
initialize `localconv' to a proper value in `u32cconv', but then it calls
`iconv_close' on the uninitialized `localconv' value during the locale switch.

I think the fix looks something like this:


diff --git a/lib/sh/unicode.c b/lib/sh/unicode.c
index a6e3058f..2f64315e 100644
--- a/lib/sh/unicode.c
+++ b/lib/sh/unicode.c
@@ -272,6 +272,7 @@ u32cconv (c, s)
   if (u32init == 0)
 {
   utf8locale = locale_utf8locale;
+  localconv = (iconv_t)-1; /* initialize */
   if (utf8locale == 0)
{
 #if HAVE_LOCALE_CHARSET

Re: umask builtin

[Eduardo A . Bustamante López]

On Wed, Nov 08, 2017 at 12:04:08PM +0100, kalle wrote:
> in my version 4.4.0(1),
> `umask' gives me a numeric output of `0022',
> while `umask -S' gives me an output of `u=rwx,g=rx,o=rx'.
> Shouldn't umask -S also formulate the mask in a negative way, as does
> `umask'? Thus giving out `u=,g=w,o=w'?

Hm, is there a umask program that does this?

Looking at the shells I have installed, they all do pretty much the same:

dualbus@ubuntu:~$ for sh in bash dash ksh93 mksh posh zsh; do echo $sh $($sh -c 
'echo $(umask) $(umask -S)'); done
bash 0022 u=rwx,g=rx,o=rx
dash 0022 u=rwx,g=rx,o=rx
ksh93 0022 u=rwx,g=rx,o=rx
mksh 022 u=rwx,g=rx,o=rx
posh 022 u=rwx,g=rx,o=rx
zsh 022 u=rwx,g=rx,o=rx

Which is also what POSIX specifies: 
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/umask.html

| If -S is specified, the message shall be in the following format:
| 
| "u=%s,g=%s,o=%s\n", , ,
| 
| 
| where the three values shall be combinations of letters from the set 
| { r, w, x}; the presence of a letter shall indicate that the corresponding
| bit is clear in the file mode creation mask.

Re: Claim `declare' as a special command in manual?

[Eduardo A . Bustamante López]

On Wed, Nov 08, 2017 at 12:20:23PM +0800, Clark Wang wrote:
[...]
> It seems to me bash internally parses the `declare' command specially and
> (logically) convert `declare -a arr=()' to two statements `declare -a arr'
> and `arr=()'.

There is indeed special treatment for assignment builtins in the parser:

dualbus@ubuntu:~/src/gnu/bash$ ack -H -C3 ASSIGNMENT_BUILTIN parse.y 
parse.y
5222-{
5223-  struct builtin *b;
5224-  b = builtin_address_internal (token, 0);
5225:  if (b && (b->flags & ASSIGNMENT_BUILTIN))
5226-   parser_state |= PST_ASSIGNOK;
5227-  else if (STREQ (token, "eval") || STREQ (token, "let"))
5228-   parser_state |= PST_ASSIGNOK;

> So does it make sense to move `declare' out of the SHELL BUILTIN COMMANDS
> and put it in a separate section? Or at least explicitly state that
> `declare' is special?

But is it only `declare' that is special?

According to:

dualbus@ubuntu:~/src/gnu/bash$ ack -H ASSIGNMENT_BUILTIN builtins.h 
builtins.h
45:#define ASSIGNMENT_BUILTIN 0x10  /* This builtin takes assignment 
statements. */

A builtin is an assignment builtin if builtin->flags has the ASSIGNMENT_BUILTIN
bit enabled.

You can see which builtins are assignment builtins by looking at
builtins/mkbuiltins.c:

dualbus@ubuntu:~/src/gnu/bash$ ack -H -C3 'char \*assignment_builtins' 
builtins/mkbuiltins.c 
builtins/mkbuiltins.c
154-};
155-
156-/* The builtin commands that take assignment statements as arguments. */
157:char *assignment_builtins[] =
158-{
159-  "alias", "declare", "export", "local", "readonly", "typeset",
160-  (char *)NULL


Now, there's a slight complication. Loadable builtins are also subject to this
special treatment. That is, you can define a new loadable builtin which is an
assignment builtin. Which means that if you only document the 6 builtins above
in the manual, the information will not be complete.

I think the proper approach would be to mention in the manual that there are
"assignment builtins", and that you can see the complete list with `enable
-'



As a side note, this special treatment does lead to some weirdness around
shadowing assignment builtins with functions, or using them with the `builtin'
builtin:

dualbus@ubuntu:~$ typeset -fp declare
-bash: typeset: declare: not found
dualbus@ubuntu:~$ declare a=()
dualbus@ubuntu:~$ typeset -p a
declare -a a=()
dualbus@ubuntu:~$ builtin declare b=()
-bash: syntax error near unexpected token `('
dualbus@ubuntu:~$ typeset -p b
-bash: typeset: b: not found
dualbus@ubuntu:~$ declare() { builtin declare "$@"; }
dualbus@ubuntu:~$ declare b=()
dualbus@ubuntu:~$ typeset -p b
declare -a b=()
dualbus@ubuntu:~$ declare() { echo nope "$@"; }
dualbus@ubuntu:~$ declare c=()
nope c
dualbus@ubuntu:~$ typeset -p c
declare -a c=()

Re: New MILLISECONDS “special” variable?

[Eduardo A . Bustamante López]

On Wed, Nov 01, 2017 at 01:19:08PM -0700, Alan Dipert wrote:
[...]
> I can imagine many reasons not to add a new variable, including the fact
> that it could change the behavior of existing scripts. It may also be the
> case that there’s already a method to get sub-second times without shelling
> out, and so I’d be grateful for pointers to any existing solutions.
[...]

Probably the easiest option is to implement this as a loadable builtin:

$ src=~/src/gnu; gcc -fPIC -g -O0 -I $src/bash/ -I $src/bash/include -I 
$src/build-bash-devel/ -I $src/bash/examples/loadables/ -I $src/bash/builtins/ 
-c -o timer.o timer.c
$ gcc -shared -Wl,-soname,timer -o timer timer.o
$ enable -f ./timer timer
$ timer current delta
$ echo $current, $delta
1509589044.350954, 0.0
$ timer current delta
$ echo $current, $delta
1509589059.42946, 14.691992
$ timer current delta; sleep 3.14; timer current delta; echo $delta
3.143473

Given this, I don't see a reason to add this functionality to bash itself.

Below is the (buggy and ugly) implementation of the `timer' loadable builtin:

---
#include 
#include 
#include 
#include 

#include "loadables.h"

/* FIXME: do better validation */
static int parse_timeval (const char *s, struct timeval *tv) {
  char *ds;
  timerclear(tv);
  tv->tv_sec = strtol(s, , 10);
  if (ds && *ds == '.') {
tv->tv_usec = strtol(ds + 1, NULL, 10);
  }
  return 0;
}

int timer_builtin (WORD_LIST *list) {
  struct timeval tv1, tv2, tv3;
  char *s, *current, *delta;
  char b1[64], b2[64]; /* FIXME */

  if (0 == list) {
return (EXECUTION_FAILURE);
  }

  current = list->word->word;
  if (0 == current || 0 == *current) {
return (EXECUTION_FAILURE);
  }

  if (0 == list->next) {
return (EXECUTION_FAILURE);
  }

  delta = list->next->word->word;
  if (0 == delta || 0 == *delta) {
return (EXECUTION_FAILURE);
  }

  if (-1 == gettimeofday(, NULL)) {
return (EXECUTION_FAILURE);
  }

  s = get_string_value (current);
  if (0 == s) {
bind_variable (delta, "0.0", 0);
goto update_value;
  }

  parse_timeval (s, );
  timersub (, , );
  snprintf (b2, 64, "%ld.%ld", tv3.tv_sec, tv3.tv_usec);
  bind_variable (delta, b2, 0);

  update_value:
  snprintf (b1, 64, "%ld.%ld", tv1.tv_sec, tv1.tv_usec);
  bind_variable (current, b1, 0);

  return (EXECUTION_SUCCESS);
}

char *timer_doc[] = {
  "...",
  "",
  "...",
  (char *)NULL
};

struct builtin timer_struct = {
  "timer",
  timer_builtin,
  BUILTIN_ENABLED,
  timer_doc,
  "timer CURRENT DELTA",
  0
};

Re: Out-of-bounds read in parse_comsub()

[Eduardo A . Bustamante López]

On Thu, Oct 26, 2017 at 11:07:57PM +0200, Jakub Wilk wrote:
[...]
> Bash Version: 20171020 snapshot
> 
> Apparently an out-of-bounds read can happen in parse_comsub() when checking
> script sytnax. To reproduce, rebuild bash with AddressSanitizer enabled and
> run:
> 
>   $ printf 
> '$(00\177%012d\1%d00\177%090d\177%028d(%0267d\177%010d<<-\276%019d\n\n' | 
> bash -n

I can reproduce this error too. It seems `ret' is missing a NUL terminator, so
`STREQN' reads past the allocated buffer.


I'm not sure if this is the right fix, but at least it makes the issue go away 
for me:


dualbus@ubuntu:~/src/gnu/build-bash-devel$ CFLAGS='-g -O0 -fsanitize=address' 
../bash/configure --without-bash-malloc && make -j8
[...]
dualbus@ubuntu:~/src/gnu/build-bash-devel$ printf 
'$(00\177%012d\1%d00\177%090d\177%028d(%0267d\177%010d<<-\276%019d\n\n' | 
./bash -n
./bash: line 1: unexpected EOF while looking for matching `)'
./bash: line 3: syntax error: unexpected end of file


diff --git a/parse.y b/parse.y
index 623648c6..67d9b3c3 100644
--- a/parse.y
+++ b/parse.y
@@ -3838,6 +3838,7 @@ eof_error:
  tind = lex_firstind;
  while ((tflags & LEX_STRIPDOC) && ret[tind] == '\t')
tind++;
+ ret[retsize - 1] = '\0';
  if (STREQN (ret + tind, heredelim, hdlen))
{
  tflags &= ~(LEX_STRIPDOC|LEX_INHEREDOC|LEX_QUOTEDDOC);

Re: ctrl-w oddity on bash-4.4

[Eduardo A . Bustamante López]

On Wed, Oct 25, 2017 at 09:48:14AM -0400, Chet Ramey wrote:
> On 10/22/17 6:52 PM, Aron Griffis wrote:
> > I'm seeing some strange behavior and don't know if it's a bug or intended.
> > 
> > Reproducer:
> > 1. env INPUTRC=/dev/null bash --norc
> > 2. set -o vi
> > 3. true --foo=bar
> > 4. up arrow, then left arrow to put the cursor on the equals sign
> > 5. press ctrl-w, nothing happens
> 
> Posix says the word boundaries for ^W in insert mode are characters that
> aren't  or . So you deal with the character before the
> cursor (`o'), and delete to a character that isn't  or .
> Since the `o' is in neither character class, it's the word boundary, and
> you don't delete anything. FWIW, ksh93 behaves the same way (but beeps
> annoyingly).
[...]

Wow. I had to read the POSIX description of ^W a few times to finally
understand how this works. Thank you for the explanation Chet!



Patch for compatibility with ksh93 (just kidding!):

diff --git a/lib/readline/vi_mode.c b/lib/readline/vi_mode.c
index 3cb7e8c9..210d1b12 100644
--- a/lib/readline/vi_mode.c
+++ b/lib/readline/vi_mode.c
@@ -1615,7 +1615,14 @@ rl_vi_unix_word_rubout (int count, int key)
  rl_point--;
}
 
-  rl_kill_text (orig_point, rl_point);
+  if (orig_point == rl_point)
+{
+  rl_ding();
+}
+  else
+{
+  rl_kill_text (orig_point, rl_point);
+}
 }
 
   return 0;

Re: ctrl-w oddity on bash-4.4

[Eduardo A . Bustamante López]

On Sun, Oct 22, 2017 at 10:15:54PM -0400, Aron Griffis wrote:
[...]
> Thanks Eduardo. The clue there about bind-tty-special-chars is helpful,
> since previously I hadn't been able to make unix-word-rubout work either!
> 
> However this doesn't seem to explain the case where ctrl-w doesn't work at
> all, which is the bug I'm reporting.

Yes, you're right.

I'm looking at the POSIX specification for the shell vi-mode:

 
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html#tag_20_117_13_03

(emphasis is mine)

| -W
|  Delete the characters from /the one preceding the cursor/ to the preceding
|  word boundary. The word boundary in this case is the closer to the cursor of
|  either the beginning of the line or a character that is in neither the blank
|  nor punct character classification of the current locale.

And function `rl_vi_unix_word_rubout':

| dualbus@ubuntu:~/src/gnu/bash/lib/readline$ cat -n vi_mode.c | sed -n 
1601,1605p
|   1601  /* If we're at the start of a word, move back to word 
boundary so we
|   1602 move back to the `preceding' word */
|   1603  if (rl_point > 0 && (vi_unix_word_boundary 
(rl_line_buffer[rl_point]) == 0) &&
|   1604vi_unix_word_boundary (rl_line_buffer[rl_point 
- 1]))
|   1605rl_point--;

So if we have:

v-- cursor here
>>true --foo=bar<<
   ^-- character preceding the cursor

Then:

- rl_line_buffer[rl_point] -> `='  &  vi_unix_word_boundary(...) -> 1 
- rl_line_buffer[rl_point - 1] -> `o'  &  vi_unix_word_boundary(...) -> 0

So the test in lines 1603-1604 fails, and rl_point is not decreased.


According to my reading of the standard, it should decrease rl_point
unconditionally. See my proposed fix below.


diff --git a/lib/readline/vi_mode.c b/lib/readline/vi_mode.c
index 3cb7e8c9..238a77e9 100644
--- a/lib/readline/vi_mode.c
+++ b/lib/readline/vi_mode.c
@@ -1598,10 +1598,8 @@ rl_vi_unix_word_rubout (int count, int key)
while (--rl_point > 0 && whitespace (rl_line_buffer[rl_point]))
  ;
 
- /* If we're at the start of a word, move back to word boundary so we
-move back to the `preceding' word */
- if (rl_point > 0 && (vi_unix_word_boundary (rl_line_buffer[rl_point]) 
== 0) &&
-   vi_unix_word_boundary (rl_line_buffer[rl_point - 1]))
+ /* We start deleting at the character preceding the cursor */
+ if (rl_point > 0)
rl_point--;
 
  /* If we are at a word boundary (whitespace/punct), move backward


signature.asc
Description: PGP signature

Re: ctrl-w oddity on bash-4.4

[Eduardo A . Bustamante López]

On Sun, Oct 22, 2017 at 06:52:12PM -0400, Aron Griffis wrote:
> I'm seeing some strange behavior and don't know if it's a bug or intended.
[...]
> If this is intentional then I'd be curious about the rationale.

Please read:

 http://lists.gnu.org/archive/html/help-bash/2016-11/msg4.html

I believe this is the entry in the change log describing the change, although
I'm unable to find Carlos' original report in the archives:

| lib/readline/vi_mode.c
|- rl_vi_unix_word_rubout: new function, implements ^W as Posix 
specifies
|  it for vi mode, using whitespace and punctuation characters as word
|  boundaries. Prompted by question from Carlos Pita
|  

Re: Why are long functions slower?

[Eduardo A . Bustamante López]

On Sun, Oct 15, 2017 at 01:53:15PM -0500, Eduardo A. Bustamante López wrote:
[...]
> From what I can tell, most of the overhead should be in `execute_function' 
> (execute_cmd.c):
> 
> - calls `tc = (COMMAND *)copy_command (function_cell (var));' for every 
> function invocation. For large functions, this
>   means a lot of copying. 
> 
> - It also calls `dispose_command(tc)' to dispose of this temporary COMMAND 
> structure.

Just out of curiosity, I hacked bash to remove the copy_command /
dispose_command overhead (which AFAICT breaks the handling of errexit for 
functions).

I ran the script at the bottom (N = 100). The results:

- Unpatched bash, -O2 & bash malloc: ~263.78ms per function invocation
- Patched bash, -O2 & bash malloc: ~3.36ms per function invocation
- Unpatched bash, -O2 & glibc malloc (2.26-0ubuntu2): ~1.5ms per function 
invocation

So my current guess is that you're comparing a bash built with the internal 
malloc, vs bash built to use glibc's malloc.


BTW, Chet: Is there a reason for all the object copying / destruction, other 
than setting the `flags' attribute in the
COMMAND structure?


dualbus@ubuntu:~/src/gnu$ for bash in ./build/bash ./build-patched/bash; do 
time $bash ~/bash-func/script ; done

real0m26.378s
user0m26.368s
sys 0m0.004s

real0m0.336s
user0m0.332s
sys 0m0.000s

# I ran this after the two above; unpatched bash, --without-bash-malloc
dualbus@ubuntu:~/src/gnu$ for bash in ./build-libcmalloc/bash; do time $bash 
~/bash-func/script ; done

real0m0.150s
user0m0.148s
sys 0m0.000s


dualbus@ubuntu:~/src/gnu$ (cd bash; PAGER= git diff)
diff --git a/execute_cmd.c b/execute_cmd.c
index 8d38378e..cdb2ce6d 100644
--- a/execute_cmd.c
+++ b/execute_cmd.c
@@ -4724,7 +4724,7 @@ execute_function (var, words, flags, fds_to_close, async, 
subshell)
   GET_ARRAY_FROM_VAR ("BASH_LINENO", bash_lineno_v, bash_lineno_a);
 #endif
 
-  tc = (COMMAND *)copy_command (function_cell (var));
+  tc = function_cell (var);
   if (tc && (flags & CMD_IGNORE_RETURN))
 tc->flags |= CMD_IGNORE_RETURN;
 
@@ -4743,7 +4743,6 @@ execute_function (var, words, flags, fds_to_close, async, 
subshell)
   unwind_protect_int (function_line_number);
   unwind_protect_int (return_catch_flag);
   unwind_protect_jmp_buf (return_catch);
-  add_unwind_protect (dispose_command, (char *)tc);
   unwind_protect_pointer (this_shell_function);
   unwind_protect_int (funcnest);
   unwind_protect_int (loop_level);

Re: Why are long functions slower?

[Eduardo A . Bustamante López]

On Tue, Oct 10, 2017 at 03:46:38PM +0200, Bernhard M. Wiedemann wrote:
> Hi,
> 
> today I was debugging performance issues with a 200KB bash script [1]
> with bash-4.3 and 4.4
> and it seems that much of it came from a function call that took 0.1
> seconds (and it was done in a loop for 37000 files)
> even though it basically just consisted of an
>   if [[ 0 != 0 ]] ; then

Hi Bernhard,

I have some questions:

1. Which specific versions of 4.3 and 4.4?
2. Did you compile bash from source? (if so, what did you use for CFLAGS and 
the configure script parameters? are you
   using bash's malloc or the system malloc?)
3. ... Or did you use a distro package? (if so, can you provide links to the 
specific packages used?)
4. Do you have precise timing information? How specifically did you test? How 
many iterations? etc


From what I can tell, most of the overhead should be in `execute_function' 
(execute_cmd.c):

- calls `tc = (COMMAND *)copy_command (function_cell (var));' for every 
function invocation. For large functions, this
  means a lot of copying. 

- It also calls `dispose_command(tc)' to dispose of this temporary COMMAND 
structure.


There were no changes there between 4.3.30 and 4.4.12. The only noticeable 
change I can see is the addition of the
`restore_funcarray_state' function and `struct func_array_state', used to 
restore the contents of FUNCNAME between
function invocations. Although from my observation, this doesn't seem to have a 
noticeable effect in performance.


signature.asc
Description: PGP signature

Re: PDF Version - Page Numbers Incorrect

[Eduardo A . Bustamante López]

On Sat, Oct 14, 2017 at 09:25:26PM -0400, Josh Blagden wrote:
> Hi GNUs,  
> 
>     The page numbers in the Table of Contents are incorrect. For
> example, section 3.2.6, titled "GNU Parallel" is on page 21, not page
> 22. Also, section 3.6, titled "Redirections" is on page 38, not page 32.
> I assume that this is merely due to the author making additions to the
> manual without correcting the page numbers in the PDF.

Hi Josh,

I'm guessing you're talking about the Bash reference manual. What source file 
are you using? I'm looking at:
, and the TOC lists 3.2.6 in 
page 15, and 3.6 is listed as being in
page 32. I see that both sections begin in the advertised page number.

I also checked against the `devel' branch in git, by running:

$ ./configure --silent && make -s -C doc/ bashref.pdf

(NOTE: you need the `texinfo' and `texlive' packages to be able to generate the 
PDF file from source).

In the generated file (doc/bashref.pdf), 3.2.6 is listed in page 16, and 3.6 in 
page 33. Both indicate the correct page
number.


By the way, are you checking against the page number in the page header, or the 
"PDF page number"? I noticed that the
PDF page number doesn't match the page number in the page header, which seems 
to be a limitation of the tool used to
generate the file (i.e. texi2dvi). In both cases (bash.pdf from www.gnu.org and 
doc/bashref.pdf), the PDF page number is
offset by a constant factor of +6 (i.e. PDF's pg. 22 is doc's pg. 16; PDF's pg. 
38 is doc's pg. 32).

I read 
, 
but I don't see any mention of an
option to control the page number offset described above.

Looking around, I see that this is a common problem where the logical page 
number doesn't match the PDF physical page
number. It seems there are some PDF readers that allow you to switch between 
page numbering schemes [1]. Sadly, the ones
I've tried (evince: libevdocument3.so.4, libevview3.so.3; xpdf: 
libpoppler.so.68 [2], firefox: pdf.js). If this is the
problem you're seeing, I don't think there's a solution, other than using a PDF 
reader that supports using logical page
numbers.


[1] 
[2] 


signature.asc
Description: PGP signature

Re: command_not_found_handle documentation omission

[Eduardo A . Bustamante López]

On Sun, Oct 08, 2017 at 11:16:33AM -0500, Dan Douglas wrote:
[...]
> Thinking out loud some more... it does make sense that a user in an
> interactive session expects commands to not alter their shell environment,
> and a badly written command_not_found_handle could do that, possibly
> without the user's knowledge on systems that put a handler in a global
> bashrc (likely the most common scenario).
> 
> On the other hand a user that actually defines their own handler could
> have a million reasons to want to propagate some effect to the interactive
> process, e.g. defining an alias or altering PS1. Same for non-interactive
> scripts.
> 

I guess that instead of changing the semantics of
command_not_found_handle, a new special trap could be added that
executes in the context of the shell performing the command lookup.

Although I'm not sure how valuable it would be (the added complexity).
Are there any serious uses of the command_not_found_handle aside from
suggestions during interactive use?

Re: [PATCH] Fix hang if $OLDPWD points to inaccessible directory

[Eduardo A . Bustamante López]

On Tue, Oct 03, 2017 at 10:29:08PM +0200, Mikulas Patocka wrote:
> If $OLDPWD points to a non-existing directory, 'cd -' will fail.
> But if we clear $OLDPWD, 'cd -' will fail too (with just different message).
[...]

I performed the following tests:


dualbus@ubuntu:~$ for sh in mksh ksh93 dash zsh posh bash; do echo $sh \| 
$(OLDPWD=/invalid $sh -c 'echo "$OLDPWD | "; cd -; echo " | $?"' 2>&1); done
mksh | /invalid | mksh: cd: /invalid: No such file or directory | 2
ksh93 | /invalid | ksh93: cd: /invalid: [No such file or directory] | 1
dash | /invalid | dash: 1: cd: can't cd to /invalid | 2
zsh | /home/dualbus | | 0
posh | /invalid | posh: cd: /invalid - No such file or directory | 1
bash | | bash: line 0: cd: OLDPWD not set | 1

dualbus@ubuntu:~$ for sh in mksh ksh93 dash zsh posh bash; do echo $sh \| 
$(OLDPWD=/tmp $sh -c 'echo "$OLDPWD | "; cd -; echo " | $?"' 2>&1); done
mksh | /tmp | /tmp | 0
ksh93 | /tmp | /tmp | 0
dash | /tmp | /tmp | 0
zsh | /home/dualbus | | 0
posh | /tmp | /tmp | 0
bash | /tmp | /tmp | 0


So:

- Bash is the only major shell that performs stat() on OLDPWD to see if it's
  going to import it from the environment or initialize it to an empty
  value.

- Zsh doesn't import OLDPWD from the environment (bash's old behavior)

- The rest of the shells (mksh, ATT ksh, dash, posh) import OLDPWD from
  the environment unconditionally.

- Mikulas already made the argument above of why this change wouldn't
  break existing scripts that rely on bash performing the stat().

- I don't see a way around this behavior using existing bash features.
  The value of OLDPWD can't be reset via BASH_ENV or other
  initialization mechanisms. Sure, `unset OLDPWD; program' might work,
  but I wouldn't call that a solution.


Given the above, I've changed my mind, and I think this patch should be
fine :-)

Re: [PATCH] Fix hang if $OLDPWD points to inaccessible directory

[Eduardo A . Bustamante López]

On Fri, Sep 29, 2017 at 12:51:37AM -0700, L A Walsh wrote:
[...]
> Why does bash clear OLDPWD when a child script is started?
> 
> OLDPWD is exported and passed to any children, but bash apparently clears
> OLDPWD whenever a child script is started...
> 
> GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
> 
> Can bash be fixed to preserve the value of any OLDPWD in its initial
> environment, like it does with PWD?

Eh, 4.1? That's ~8 years old.

Check the commit I linked, there's the fix you're asking for.

> ---
> 
> I don't see a reason...
> I see a "want", but no real reason
> 
> I'd agree it's a rare event to hit though, but when you do...
> 
> Requires console access, but changing /etc/profile to
> insert a bad OLDPWD to a known down network location might not be
> considered a trivial occurrence to someone stressed out and trying to log
> in and find out why everyone logging in is hanging...

Why would logins hang? We're talking about OLDPWD here. OLDPWD is not
set if you're not logged in.

The reported scenario is:

- You are in a network filesystem (let's call it /net/myuser)
- chdir() somewhere else, e.g. /home/myuser 
  (OLDPWD=/net/myuser, PWD=/home/myuser)
- Then you unplug your computer, or the computer that provides the
  network share.
- Then you start a new bash shell, which will hang because it'll try to
  stat($OLDPWD).

To which the solution is: unmount the network share properly.

Re: [PATCH] Fix hang if $OLDPWD points to inaccessible directory

[Eduardo A . Bustamante López]

On Fri, Sep 29, 2017 at 11:12:59AM +0200, Mikulas Patocka wrote:
> This patch doesn't revert the above change. It only removes a test if 
> $OLDPWD is a directory. With this patch, if $OLDPWD points to a directory, 
> there will be no change, and if $OLDPWD points to non-directory, it will 
> not be cleared.

You're right. It doesn't revert it, but it does make it less useful.

> Hung mount shouldn't cause problems unless someone is accessing it.

My reasoning is that if stat()'ing a directory causes everything to
halt, then perhaps fix the source of that problem?

There's no reason why bash should make this a special case.

Re: [PATCH] Fix hang if $OLDPWD points to inaccessible directory

[Eduardo A . Bustamante López]

On Thu, Sep 28, 2017 at 11:44:02AM +0200, Mikulas Patocka wrote:
[...]
> +++ bash/variables.c
> @@ -899,7 +899,7 @@ set_pwd ()
>   don't find OLDPWD in the environment, or it doesn't name a directory,
>   make a dummy invisible variable for OLDPWD, and mark it as exported. */
>temp_var = find_variable ("OLDPWD");
> -  if (temp_var == 0 || value_cell (temp_var) == 0 || file_isdir (value_cell 
> (temp_var)) == 0)
> +  if (temp_var == 0 || value_cell (temp_var) == 0)
>  {
>temp_var = bind_variable ("OLDPWD", (char *)NULL, 0);

This patch would revert the change introduced in:
  
http://git.savannah.gnu.org/cgit/bash.git/commit/?h=devel=e6f5e0c858b7c0839d346d7d55e56894648c5a33

Which was prompted by the following bug report from John Wiersba:
  https://lists.gnu.org/archive/html/bug-bash/2015-11/msg00115.html

Correct me if I'm wrong, but a hanged sshfs mount will cause many more
issues, not only with OLDPWD.

I tried the following:

dualbus@ubuntu:~/test$ ls -l
total 4
drwxr-xr-x 2 dualbus dualbus 4096 sep 28 10:05 host
dualbus@ubuntu:~/test$ sshfs $host: host
dualbus@ubuntu:~/test$ ls -ld host
drwxr-xr-x 1 dualbus dualbus 20480 sep 28 08:15 host
dualbus@ubuntu:~/test$ cd host
dualbus@ubuntu:~/test/host$ pwd
/home/dualbus/test/host
dualbus@ubuntu:~/test/host$ cd ..
dualbus@ubuntu:~/test$ df host | sed "s/$host//"
Filesystem 1K-blocks Used Available Use% Mounted on
: 20511592 13003368   6452528  67% /home/dualbus/test/host
dualbus@ubuntu:~/test$ sudo iptables -A OUTPUT -p tcp -d $host -j DROP
dualbus@ubuntu:~/test$ ls
^C^C^C^C^C

Also tried:

cd ho # hangs
cd host # hangs
ls -l host # hangs

So in my opinion, it's not worth to revert this change given that you'll have
many more troubles with a hanged network filesystem.

Re: mapfile doesn't accept input from a pipe

[Eduardo A . Bustamante López]

On Wed, Jun 28, 2017 at 07:08:27PM -0700, Keith Thompson wrote:
[...]
> mapfile REDIRECT < /tmp/input.txt
> cat /tmp/input.txt | mapfile PIPE

The `mapfile PIPE' is a piece of a pipeline, and as such, it runs in a
subshell (different process).

See: http://mywiki.wooledge.org/BashFAQ/024

-- 
Eduardo Bustamante
https://dualbus.me/

Re: Fwd: Non-upstream patches for bash (2014)

[Eduardo A . Bustamante López]

On Sat, Jun 24, 2017 at 04:46:47PM -0400, George wrote:
[...]
> I'm not seeing the problem here (at least, not in Bash or ksh - mksh and zsh 
> seem to have gotten it wrong...)

You are right. I should get some sleep. FWIW, the original claim is that
having a locale-dependent parser is a problem.

-- 
Eduardo Bustamante
https://dualbus.me/

Re: Fwd: Non-upstream patches for bash (2014)

[Eduardo A . Bustamante López]

I was looking through this old thread:
http://seclists.org/oss-sec/2014/q3/851

It looks like the issue reported in there is still there:

  dualbus@debian:~$ LANG=zh_CN.GBK printf 'echo \u4e57\n' |LANG=zh_CN.GBK bash
  �\
  dualbus@debian:~$ LANG=en_US.UTF8 printf 'echo \u4e57\n' |LANG=en_US.UTF8 bash
  乗
  dualbus@debian:~$ LANG=zh_CN.GBK printf 'echo \u4e57\n' |LANG=zh_CN.GBK mksh
  �
  dualbus@debian:~$ LANG=zh_CN.GBK printf 'echo \u4e57\n' |LANG=zh_CN.GBK ksh
  �\
  dualbus@debian:~$ LANG=zh_CN.GBK printf 'echo \u4e57\n' |LANG=zh_CN.GBK zsh
  �
(In the case that your font doesn't render the glyph for U+4E57, it's:
http://www.unicode.org/cgi-bin/GetUnihanData.pl?codepoint=4e57)

  dualbus@debian:~$ LANG=zh_CN.GBK printf '\u4e57' | od -tx1 -An
   81 5c

It looks like it doesn't detect that \x81\x5c is a single character, and
instead treats the multibyte character as separate characters.

-- 
Eduardo Bustamante
https://dualbus.me/

Re: Bug in select-command

[Eduardo A . Bustamante López]

On Sat, Jun 24, 2017 at 01:17:23PM -0400, Chet Ramey wrote:
[...]
> You haven't exported these.  If you had, the subshell started to run the
> script would have the correct values.

Hm. I think this may be a documentation / usability problem.

The manual states the following:

  COLUMNS
 Used  by the select compound command to determine the terminal width
 when printing selection lists.  *Automatically set if the checkwinsize
 option is enabled* or in an interactive shell upon receipt of a
 SIGWINCH.

  [...]

  checkwinsize
  If set, bash checks the window size after each command and, if
  necessary, updates the values of LINES and COLUMNS.

Furthermore, POSIX [1] says the following:

  COLUMNS
  This variable shall represent a decimal integer >0 used to indicate
  the user's preferred width in column positions for the terminal
  screen or window; see Column Position. If this variable is unset or
  null, the implementation determines the number of columns,
  appropriate for the terminal or window, in an unspecified manner.
  When COLUMNS is set, any terminal-width information implied by TERM
  is overridden. Users and conforming applications should not set
  COLUMNS unless they wish to override the system selection and produce
  output unrelated to the terminal characteristics.

  Users *should not need to set this variable in the environment*
  unless there is a specific reason to override the implementation's
  default behavior, such as to display data in an area arbitrarily
  smaller than the terminal or window.

So, from the manual alone, I know the following:

- `select' uses the COLUMNS variable to determine the terminal width
- the value of COLUMNS is automatically set if the `checkwinsize' option is
  enabled (** restrictions apply).
- bash checks the window size after each command

I have the following problems with the documentation:

- It seems to imply that it's set automatically *when* `checkwinsize' is
  enabled. A careful reading uncovers that this is not true though, and that
  you have to run a command for the window size to be checked.
- It doesn't mention that the window size is NOT checked after *builtin*
  commands.
- POSIX recommends against exporting the COLUMNS variable.

I know that I can get the desired behavior by either:

- Running the script under an interactive shell
- Setting `checkwinsize' and running a dummy external command to trigger the
window size check.
- Exporting the COLUMNS variable from the parent interactive shell.

I think that's too much hassle, when bash could easily:

- If the `select' statement is used and `checkwinsize' is set and
  `COLUMNS' is unset, then call `get_new_window_size'
- Call `get_new_window_size' when `checkwinsize' is enabled.

[1] 
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08_03

-- 
Eduardo Bustamante
https://dualbus.me/

Re: Bug in select-command

[Eduardo A . Bustamante López]

Chet:

I think the patch below could be a useful addition, to make the behavior
in this case a little bit less surprising.


*** /tmp/BdLnYa_shopt.def   2017-06-24 10:21:15.029707643 -0500
--- builtins/shopt.def  2017-06-24 10:17:00.540600773 -0500
***
*** 134,139 
--- 134,140 
  static int shopt_set_complete_direxpand __P((char *, int));
  #endif
  
+ static int shopt_set_check_window_size __P((char *, int));
  static int shopt_set_debug_mode __P((char *, int));
  
  static int shopt_login_shell;
***
*** 164,170 
  #if defined (JOB_CONTROL)
{ "checkjobs", _jobs_at_exit, (shopt_set_func_t *)NULL },
  #endif
!   { "checkwinsize", _window_size, (shopt_set_func_t *)NULL },
  #if defined (HISTORY)
{ "cmdhist", _oriented_history, (shopt_set_func_t *)NULL },
  #endif
--- 165,171 
  #if defined (JOB_CONTROL)
{ "checkjobs", _jobs_at_exit, (shopt_set_func_t *)NULL },
  #endif
!   { "checkwinsize", _window_size, shopt_set_check_window_size },
  #if defined (HISTORY)
{ "cmdhist", _oriented_history, (shopt_set_func_t *)NULL },
  #endif
***
*** 593,598 
--- 594,606 
  #endif
  
  static int
+ shopt_set_check_window_size (char *option_name, int mode)
+ {
+   get_new_window_size (0, (int *)0, (int *)0);
+   return (0);
+ }
+ 
+ static int
  set_compatibility_level (option_name, mode)
   char *option_name;
   int mode;

-- 
Eduardo Bustamante
https://dualbus.me/

Re: Unexpected word splitting on $* when IFS is unset

[Eduardo A . Bustamante López]

On Sat, Jun 24, 2017 at 09:10:34PM +0700, Robert Elz wrote:
[...]
> The bugs in various implementations cause problems, yes, dealing with
> someone else's mistakes (and especially doing it in a way that things
> still work when the bugs get fixed) can be difficult.
> 
> But the rules, no, the rules for $* are actually trivial.
> 
> Unquoted, $* (or $@ which is identical) is exactly the same
> as the sequence
> 
>   $1 $2 $3 ...
> 
> would be, if we knew in advance how many of those we should write
> (which can be 0 of them of course.)
> 
> It is exactly that simple (ignoring implementation bugs.)

I think we are arguing about different things. My point (and which I
think is Greg's point) is that we should start recommending shell script
writers to stay away from the unquoted expansions of $* and $@.

Yes, the POSIX rules might be "trivial", but the actual implementations
are not, and shell script programmers care about running their scripts
on real implementations.

-- 
Eduardo Bustamante
https://dualbus.me/

Re: Bug in select-command

[Eduardo A . Bustamante López]

On Sat, Jun 24, 2017 at 01:46:00PM +0200, sky...@top-email.net wrote:
[...]
> 
> ### DEFINE FUNCTIONS ###
> _config-keymap() {
>   # Select keymap
>   # find keymap list, cut directory path and sort
>   local list_keymaps=( $(find /usr/share/kbd/keymaps/ 
> -type f -name 
> *.map.gz | grep -Eo [^/]*.map.gz | sort -V) )
>   # cut suffixes
>   list_keymaps=( ${list_keymaps[@]%%.*} )
> 
>   select locale_keymap in ${list_keymaps[@]}; do
>   if [[ "$REPLY" -ge 1 && "$REPLY" -lt 
> ${#list_keymaps[@]} ]]; then
>   echo ${locale_keymap}; break
>   fi
>   done
> }
> 
> ### MAIN PROGRAMM ###
> clear
> _config-keymap
> 
I don't have `/usr/share/kbd/keymaps/' in my system.

dualbus@debian:~$ ls -l /usr/share/kbd/keymaps/
ls: cannot access '/usr/share/kbd/keymaps/': No such file or directory

>From what I can tell, I get these keymaps by installing `console-data'
in my Debian system, although these are installed under `/usr/share/keymaps/',
and have different extensions (e.g.
`/usr/share/keymaps/i386/qwerty/pc110.kmap.gz')

~

I think this is an easier way to reproduce the problem. I have a terminal
window with the following dimensions:

  dualbus@debian:~$ declare -p COLUMNS LINES
  declare -- COLUMNS="191"
  declare -- LINES="49"

  dualbus@debian:~$ bash --version | head -n1
  GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu)

Running the script in `interactive' mode (which triggers calls to
`get_new_window_size' through `get_tty_state' when checkwinsize is
enabled), lays out the options on seven columns:

  dualbus@debian:~$ bash -i script 
  $- himB
  cols 191
  1) 8)   15)   
22)   29)   36)   
43) 
  2) 9)   16)   
23)   30)   37)   
44) 
  3)    10)   17)   
24)   31)   38)   
45) 
  4)    11)   18)   
25)   32)   39)   
46) 
  5)    12)   19)   
26)   33)   40)   
47) 
  6)    13)   20)   
27)   34)   41)   
48) 
  7)    14)   21)   
28)   35)   42) 
  #? q

Whereas running the same script in `non-interactive' mode results in three
columns.

  dualbus@debian:~$ bash  script 
  $- hB
  cols 191
   1)   17)   33) 
   2)   18)   34) 
   3)   19)   35) 
   4)   20)   36) 
   5)   21)   37) 
   6)   22)   38) 
   7)   23)   39) 
   8)   24)   40) 
   9)   25)   41) 
  10)   26)   42) 
  11)   27)   43) 
  12)   28)   44) 
  13)   29)   45) 
  14)   30)   46) 
  15)   31)   47) 
  16)   32)   48) 
  #? q

The script:

  dualbus@debian:~$ cat script 
  #!/bin/bash
  shopt -s checkwinsize
  
  options=(
     
   
 
     
   
 
     
  

Re: Unexpected word splitting on $* when IFS is unset

[Eduardo A . Bustamante López]

On Sat, Jun 24, 2017 at 09:35:34AM +0700, Robert Elz wrote:
[...]
> In cases where IFS is not a null string, the broken implementations mostly
> tend to work OK (sometimes perhaps only by a fluke), and even more work
> properly if IFS has its default value or something similar, that is
> where IFS[0] == ' ' to borrow an inappropriate way of writing it.
[...]
> These may be rare, but just saying "never do it" is too much.

Greg's advice is directed towards novice / intermediate shell script
programmers. I think we can all agree that the rules (and bugs) around
the expansion of $* are too complex. As a shell script writer trying to
make use $*, you'd have to verify that it works as expected across all
the shells (and even between different versions of the same shell).

IMO, it's easier to just document this kind of expansion as "undefined
behavior". Pretty much like we do with `errexit' already.

-- 
Eduardo Bustamante
https://dualbus.me/

Re: [PATCH] Unix domain socket filename redirection

[Eduardo A . Bustamante López]

On Tue, Jun 20, 2017 at 11:49:56AM -0400, tetsu...@scope-eye.net wrote:
[...]
> So implementing Unix Domain Sockets in the shell would give you the
> ability to connect to existing services on the machine that use named
> sockets for control (systemd, dbus, cups, etc.) or you could use the
> feature to implement new services and connect to them with shell
> scripts, or just use them to connect different scripts and programs
> together for various forms of IPC.

AFAICT, dbus uses a binary wire protocol [1], so we will also need a
new builtin that provides some of the functionality to read binary data
as described by Pierre in [2]. My point: in order to do something useful
with UNIX domain sockets, which in my experience are usually binary wire
protocols, bash would need a robust mechanism to read, manipulate, and
write binary strings. Why not implement both as loadable builtins?

> (As a side note, the shell somewhat defines the user's operating
> environment. I think that not providing a feature because of a
> perception that "no one uses it" becomes somewhat self-fulfilling: No
> one writes shell-friendly Unix Domain Socket servers because the shell
> doesn't support connecting to them, and the definition of what's
> "shell-friendly" is rather narrow.)

I didn't claim that this feature is not desired or useful. I just
claimed that it seems to be targetted to a very small set of users. And
if this is correct, it might be better to provide this functionality as
a loadable builtin. I'm happy to be proven wrong here.

> Though even with the benefits of redirection, personally I oppose the
> idea of spoofing more fake devices in /dev. (The shell spoofs some
> stuff in /dev already, and it's all long-established at this
> point...  But I don't think more such fake devices should be added.)
> In this patch, spoofing /dev grants us the ability to specify a single
> socket option. It's not a great way to set options, though. What if
> the API changed, and we suddenly wanted to be able to specify two
> options? Specifying options as command arguments is both clearer and
> more flexible.

I agree.

> I've given some thought to how the two approaches could be combined:
> use a command to open a file descriptor, but bind the file
> descriptor's lifetime to a redirection. For instance, consider this
> concept for a new redirection syntax:

I'm more in favor of a shell builtin, with the user having to explicity
close the descriptors, instead of bloating the parser. It is incredibly
complex as it is now. Please don't add more complexity.

[1] 
https://dbus.freedesktop.org/doc/dbus-specification.html#message-protocol-marshaling
[2] https://lists.gnu.org/archive/html/bug-bash/2017-06/msg00096.html

P.S. I enjoy reading your emails, but please, get a better MUA!

-- 
Eduardo Bustamante
https://dualbus.me/

AddressSanitizer: heap-use-after-free in readtok | PS1='$((b[x++}]))'

[Eduardo A . Bustamante López]

I don't know how to fix this.


dualbus@debian:~/readline$ 
ASAN_OPTIONS=disable_coredump=0:unmap_shadow_on_exit=1:abort_on_error=1:detect_leaks=0
 ~/src/gnu/bash-builds/devel-asan/bash
bash-4.4$ PS1='$((b[x++}]))'
bash: x++}: syntax error: invalid arithmetic operator (error token is "}")
=
==11490==ERROR: AddressSanitizer: heap-use-after-free on address 0x60201f77 
at pc 0x55c081f6ad7e bp 0x7ffd7429f4a0 sp 0x7ffd7429f498
READ of size 1 at 0x60201f77 thread T0
#0 0x55c081f6ad7d in readtok ../../bash/expr.c:1274
#1 0x55c081f6a32a in exp0 ../../bash/expr.c:1042
#2 0x55c081f6a08e in exp1 ../../bash/expr.c:982
#3 0x55c081f69f7d in exppower ../../bash/expr.c:937
#4 0x55c081f69c62 in exp2 ../../bash/expr.c:862
#5 0x55c081f69b76 in exp3 ../../bash/expr.c:836
#6 0x55c081f69b07 in expshift ../../bash/expr.c:812
#7 0x55c081f69a5a in exp4 ../../bash/expr.c:782
#8 0x55c081f699e3 in exp5 ../../bash/expr.c:760
#9 0x55c081f699a1 in expband ../../bash/expr.c:742
#10 0x55c081f69963 in expbxor ../../bash/expr.c:723
#11 0x55c081f69925 in expbor ../../bash/expr.c:704
#12 0x55c081f69896 in expland ../../bash/expr.c:677
#13 0x55c081f69803 in explor ../../bash/expr.c:649
#14 0x55c081f696c6 in expcond ../../bash/expr.c:602
#15 0x55c081f692f5 in expassign ../../bash/expr.c:487
#16 0x55c081f69240 in expcomma ../../bash/expr.c:467
#17 0x55c081f691cf in subexpr ../../bash/expr.c:449
#18 0x55c081f68f2a in evalexp ../../bash/expr.c:414
#19 0x55c081fb0527 in param_expand ../../bash/subst.c:9159
#20 0x55c081fb2ea4 in expand_word_internal ../../bash/subst.c:9655
#21 0x55c081f93a17 in expand_prompt_string ../../bash/subst.c:3785
#22 0x55c081f2199d in decode_prompt_string ../../bash/parse.y:5973
#23 0x55c081f1f71b in prompt_again ../../bash/parse.y:5484
#24 0x55c081f11c6a in yylex ../../bash/parse.y:2677
#25 0x55c081f068a1 in yyparse 
/home/dualbus/src/gnu/bash-builds/devel-asan/y.tab.c:1821
#26 0x55c081f05a72 in parse_command ../../bash/eval.c:294
#27 0x55c081f05cc7 in read_command ../../bash/eval.c:338
#28 0x55c081f04f03 in reader_loop ../../bash/eval.c:140
#29 0x55c081f006ad in main ../../bash/shell.c:794
#30 0x7f9bcc9c02b0 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
#31 0x55c081eff2a9 in _start 
(/home/dualbus/src/gnu/bash-builds/devel-asan/bash+0x842a9)

0x60201f77 is located 7 bytes inside of 8-byte region 
[0x60201f70,0x60201f78)
freed by thread T0 here:
#0 0x7f9bcd22ea10 in free (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1a10)
#1 0x55c081f689a9 in expr_unwind ../../bash/expr.c:311
#2 0x55c081f68e9d in evalexp ../../bash/expr.c:404
#3 0x55c081fde64e in array_expand_index ../../bash/arrayfunc.c:947
#4 0x55c081fdf169 in array_value_internal ../../bash/arrayfunc.c:1128
#5 0x55c081fdf917 in get_array_value ../../bash/arrayfunc.c:1198
#6 0x55c081f6aa51 in expr_streval ../../bash/expr.c:1179
#7 0x55c081f6b311 in readtok ../../bash/expr.c:1343
#8 0x55c081f691ca in subexpr ../../bash/expr.c:447
#9 0x55c081f68f2a in evalexp ../../bash/expr.c:414
#10 0x55c081fb0527 in param_expand ../../bash/subst.c:9159
#11 0x55c081fb2ea4 in expand_word_internal ../../bash/subst.c:9655
#12 0x55c081f93a17 in expand_prompt_string ../../bash/subst.c:3785
#13 0x55c081f2199d in decode_prompt_string ../../bash/parse.y:5973
#14 0x55c081f1f71b in prompt_again ../../bash/parse.y:5484
#15 0x55c081f11c6a in yylex ../../bash/parse.y:2677
#16 0x55c081f068a1 in yyparse 
/home/dualbus/src/gnu/bash-builds/devel-asan/y.tab.c:1821
#17 0x55c081f05a72 in parse_command ../../bash/eval.c:294
#18 0x55c081f05cc7 in read_command ../../bash/eval.c:338
#19 0x55c081f04f03 in reader_loop ../../bash/eval.c:140
#20 0x55c081f006ad in main ../../bash/shell.c:794
#21 0x7f9bcc9c02b0 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

previously allocated by thread T0 here:
#0 0x7f9bcd22ed28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0x55c08200fa54 in xmalloc ../../bash/xmalloc.c:112
#2 0x55c081f6912e in subexpr ../../bash/expr.c:438
#3 0x55c081f68f2a in evalexp ../../bash/expr.c:414
#4 0x55c081fb0527 in param_expand ../../bash/subst.c:9159
#5 0x55c081fb2ea4 in expand_word_internal ../../bash/subst.c:9655
#6 0x55c081f93a17 in expand_prompt_string ../../bash/subst.c:3785
#7 0x55c081f2199d in decode_prompt_string ../../bash/parse.y:5973
#8 0x55c081f1f71b in prompt_again ../../bash/parse.y:5484
#9 0x55c081f11c6a in yylex ../../bash/parse.y:2677
#10 0x55c081f068a1 in yyparse 
/home/dualbus/src/gnu/bash-builds/devel-asan/y.tab.c:1821
#11 0x55c081f05a72 in parse_command ../../bash/eval.c:294
#12 0x55c081f05cc7 in read_command ../../bash/eval.c:338
#13 0x55c081f04f03 in reader_loop 

Re: Unexpected word splitting on $* when IFS is unset

[Eduardo A . Bustamante López]

On Tue, Jun 20, 2017 at 01:13:07AM +0100, Kevin Brodsky wrote:
[...]
> When IFS is unset, unquoted $* undergoes word splitting as if IFS=' ',
> and not the expected IFS=$' \t\n'. All the other unquoted mass
> This is a regression that appeared in 4.3 and is still present on devel
[...]

AFAICT, the following patch fixes this.

diff --git a/subst.c b/subst.c
index 3093309f..bf73a35f 100644
--- a/subst.c
+++ b/subst.c
@@ -10158,7 +10158,7 @@ finished_with_string:
 or we expanded "$@" with IFS null and we need to split the positional
 parameters into separate words. */
   if (split_on_spaces)
-   list = list_string (istring, " ", 1);   /* XXX quoted == 1? */
+   list = list_string (istring, " \t\n", 1);   /* XXX quoted == 1? */
   /* If we have $@ (has_dollar_at != 0) and we are in a context where we
 don't want to split the result (W_NOSPLIT2), and we are not quoted,
 we have already separated the arguments with the first character of

The above code (`list_string' function call when split_on_spaces != 0)
seems to be executed when IFS is either unset or the empty string. The
comment above that block of code mentions that in either case, the
result should be "split on spaces", although it has a narrow definition
of space.

-- 
Eduardo Bustamante
https://dualbus.me/

devel: Builtin wait infinite loop when SIGINT received

[Eduardo A . Bustamante López]

There's something weird about the wait builtin + SIGINT handling in the latest
`devel' branch (i.e. 6374eecf232e70e45fe9c49cc8335e8779c07979)

If you interrupt the second `wait' with C-c, the shell will go into an infinite
loop.

  dualbus@debian:~/src/gnu/bash-builds/devel$ cat /tmp/s
  #!/bin/bash
  : &
  wait
  sleep inf &
  wait

  dualbus@debian:~/src/gnu/bash-builds/devel$ ./bash -x /tmp/s
  + wait
  + :
  + wait
  + sleep inf
  ^C^Z
  [1]+  Stopped ./bash -x /tmp/s
  dualbus@debian:~/src/gnu/bash-builds/devel$ kill -6 %%
  
  [1]+  Stopped ./bash -x /tmp/s
  dualbus@debian:~/src/gnu/bash-builds/devel$ fg
  ./bash -x /tmp/s
  Aborted (core dumped)

(I suspended the process and killed it with SIGABRT to get a nice stack trace).

  Core was generated by `./bash -x /tmp/s'.
  Program terminated with signal SIGABRT, Aborted.
  #0  wait_sigint_handler (sig=2) at ../../bash/jobs.c:2527
  2527{
  (gdb) bt
  #0  wait_sigint_handler (sig=2) at ../../bash/jobs.c:2527
  #1  
  #2  0x7fcbc52e4fba in __GI___waitpid (pid=-1, stat_loc=0x7ffc4d9ec21c, 
options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
  #3  0x563ed5620317 in waitchld (wpid=8312, block=1) at 
../../bash/jobs.c:3566
  #4  0x563ed561e6a7 in wait_for (pid=8312) at ../../bash/jobs.c:2772
  #5  0x563ed561db90 in wait_for_single_pid (pid=8312, flags=1) at 
../../bash/jobs.c:2393
  #6  0x563ed561df8b in wait_for_background_pids () at 
../../bash/jobs.c:2461
  #7  0x563ed5678220 in wait_builtin (list=0x0) at 
../../../bash/builtins/../../../bash/builtins/wait.def:171
  #8  0x563ed5609ad2 in execute_builtin (builtin=0x563ed56780c7 
, words=0x563ed6ec7b48, flags=0, subshell=0) at 
../../bash/execute_cmd.c:4609
  #9  0x563ed560aa32 in execute_builtin_or_function (words=0x563ed6ec7b48, 
builtin=0x563ed56780c7 , var=0x0, redirects=0x0, 
fds_to_close=0x563ed6ec3e08, flags=0)
  at ../../bash/execute_cmd.c:5107
  #10 0x563ed56093b4 in execute_simple_command 
(simple_command=0x563ed6ec7a08, pipe_in=-1, pipe_out=-1, async=0, 
fds_to_close=0x563ed6ec3e08) at ../../bash/execute_cmd.c:4395
  #11 0x563ed5602ac8 in execute_command_internal (command=0x563ed6ec7ec8, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x563ed6ec3e08) at 
../../bash/execute_cmd.c:811
  #12 0x563ed5601fe4 in execute_command (command=0x563ed6ec7ec8) at 
../../bash/execute_cmd.c:393
  #13 0x563ed55eb51c in reader_loop () at ../../bash/eval.c:172
  #14 0x563ed55e8fd1 in main (argc=3, argv=0x7ffc4d9ecb38, 
env=0x7ffc4d9ecb58) at ../../bash/shell.c:794

-- 
Eduardo Bustamante
https://dualbus.me/

Re: [PATCH] Unix domain socket filename redirection

[Eduardo A . Bustamante López]

On Mon, Jun 19, 2017 at 07:30:42PM -0400, Z wrote:
> This patch adds support for Unix domain sockets. A filename of the form
> /dev/unixstream/path used in a redirection makes bash attempt to open a
> connection to the Unix domain stream socket at /path. Similarly a filename
> of the form /dev/unixdgram/path makes bash attempt to open a connection to
> the Unix domain datagram socket at /path.

Excuse my ignorance:

- What would be an expected use case of this feature? /dev/tcp and
  /dev/udp are commonly used because it allows the shell script writer
  to interact with popular text based protocols like HTTP or SMTP. It
  would seem that this UNIX domain socket patch targets a niche instead?

- Why should this be integrated into the core of bash, and not provided
  as a loadable builtin instead? It makes more sense to me to provide
  this as an optional shell builtin, which also make the usage less
  awkward (not a fan of the /dev/unixstream/the/path/is/here thing).


By the way, the patch does not apply. It would seem like you copy-pasted
the diff, so it got wrapped at some points and git is rejecting it.

-- 
Eduardo Bustamante
https://dualbus.me/

Re: "realpath" loadable (in the examples/loadables dir) infinite loops

[Eduardo A . Bustamante López]

On Mon, Jun 19, 2017 at 07:18:11PM -0600, gaze...@xmission.com wrote:
[...]
> Description:
>   Command "realpath --help" causes infinite loop.

I can reproduce this in the `devel' branch.

The getopts loop is missing a `return' to break out of it. This should
fix it:

diff --git a/examples/loadables/realpath.c b/examples/loadables/realpath.c
index b19b87fb..7ea9a995 100644
--- a/examples/loadables/realpath.c
+++ b/examples/loadables/realpath.c
@@ -88,6 +88,7 @@ WORD_LIST *list;
break;
default:
builtin_usage();
+   return (EX_USAGE);
}
}

-- 
Eduardo Bustamante
https://dualbus.me/

Re: Read builtin -e (readline) oddities

[Eduardo A . Bustamante López]

On Mon, Jun 19, 2017 at 10:08:30AM -0400, Chet Ramey wrote:
[...]
> I can't reproduce this.  I suspect one of your startup files is somehow
> being read and setting PATH. It may also be an interaction with the
> command-not-found hook.

Hm, I can still reproduce it under Debian 9, using the `devel' branch, and I'm
sure no startup files are being sourced.

  dualbus@debian:~/src/gnu/bash-builds/devel$ ./bash --norc --noprofile
  bash-4.4$ declare -p command-not-found
  bash: declare: command-not-found: not found
  bash-4.4$ 
  GNU bash, version 4.4.12(3)-maint (x86_64-unknown-linux-gnu)
  bash-4.4$ PATH= read -e
  
  bash: vim: No such file or directory
  
  echo this is vim
  this is vim

I then patched bash like this:

  dualbus@debian:~/src/gnu/bash$ git diff -- execute_cmd.c
  diff --git a/execute_cmd.c b/execute_cmd.c
  index 0183a105..50f4e2f0 100644
  --- a/execute_cmd.c
  +++ b/execute_cmd.c
  @@ -5550,6 +5550,7 @@ shell_execve (command, args, env)
 char sample[HASH_BANG_BUFSIZ];
 int sample_len;
   
  +  if (strlen(get_string_value ("PATH")) > 0) abort();
 SETOSTYPE (0);   /* Some systems use for USG/POSIX semantics */
 execve (command, args, env);
 i = errno;   /* error from execve() */

And I got (the first `\C-x \C-e' gets to shell_execve, but PATH is empty):

  dualbus@debian:~/src/gnu/bash-builds/devel$ ./bash --norc --noprofile
  bash-4.4$ ulimit -c unlimited
  bash-4.4$ 
  GNU bash, version 4.4.12(9)-maint (x86_64-unknown-linux-gnu)
  bash-4.4$ PATH= read -e
  
  bash: vim: No such file or directory
  
  Aborted (core dumped)
  
  Core was generated by `./bash --norc --noprofile'.
  Program terminated with signal SIGABRT, Aborted.
  #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
  51  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
  (gdb) bt
  #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
  #1  0x7f30b14c93fa in __GI_abort () at abort.c:89
  #2  0x55a0df6527c4 in shell_execve (command=0x55a0e0a21020 
"/usr/bin/vim", args=0x55a0e0a64610, env=0x55a0e0a207f0) at 
../../bash/execute_cmd.c:5553
  #3  0x55a0df6525b1 in execute_disk_command (words=0x55a0e0a63c20, 
redirects=0x0, command_line=0x55a0e0a20f60 "vim /tmp/bash-fc.HTeaRz", 
pipe_in=-1, pipe_out=-1, async=0, 
  fds_to_close=0x55a0e0a42d80, cmdflags=0) at ../../bash/execute_cmd.c:5348
  #4  0x55a0df650d4f in execute_simple_command 
(simple_command=0x55a0e0a62780, pipe_in=-1, pipe_out=-1, async=0, 
fds_to_close=0x55a0e0a42d80) at ../../bash/execute_cmd.c:4466
  #5  0x55a0df64a465 in execute_command_internal (command=0x55a0e0a20e80, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x55a0e0a42d80) at 
../../bash/execute_cmd.c:811
  #6  0x55a0df6ad6a8 in parse_and_execute (string=0x55a0e0a61b10 "vim 
/tmp/bash-fc.HTeaRz", from_file=0x55a0df71cecd "fc", flags=4) at 
../../../bash/builtins/evalstring.c:430
  #7  0x55a0df6aee93 in fc_builtin (list=0x0) at 
../../../bash/builtins/../../../bash/builtins/fc.def:438
  #8  0x55a0df6511cb in execute_builtin (builtin=0x55a0df6ae4ae 
, words=0x55a0e0a625d0, flags=0, subshell=0) at 
../../bash/execute_cmd.c:4609
  #9  0x55a0df6520a7 in execute_builtin_or_function (words=0x55a0e0a625d0, 
builtin=0x55a0df6ae4ae , var=0x0, redirects=0x0, 
fds_to_close=0x55a0e0a63df0, flags=0)
  at ../../bash/execute_cmd.c:5107
  #10 0x55a0df650ad1 in execute_simple_command 
(simple_command=0x55a0e0a62540, pipe_in=-1, pipe_out=-1, async=0, 
fds_to_close=0x55a0e0a63df0) at ../../bash/execute_cmd.c:4395
  #11 0x55a0df64a465 in execute_command_internal (command=0x55a0e0a20400, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x55a0e0a63df0) at 
../../bash/execute_cmd.c:811
  #12 0x55a0df6ad6a8 in parse_and_execute (string=0x55a0e0a613b0 "fc -e 
\"${VISUAL:-${EDITOR:-emacs}}\"", from_file=0x55a0df70ccd5 "C-xC-e", flags=4)
  at ../../../bash/builtins/evalstring.c:430
  #13 0x55a0df696203 in edit_and_execute_command (count=1, c=5, 
editing_mode=1, edit_command=0x55a0df70cd10 "fc -e 
\"${VISUAL:-${EDITOR:-emacs}}\"") at ../../bash/bashline.c:977
  #14 0x55a0df6962f0 in emacs_edit_and_execute_command (count=1, c=5) at 
../../bash/bashline.c:1013
  #15 0x55a0df6dccd5 in _rl_dispatch_subseq (key=5, map=0x55a0df9461a0 
, got_subseq=0) at 
../../../../bash/lib/readline/readline.c:851
  #16 0x55a0df6dd19a in _rl_dispatch_subseq (key=24, map=0x55a0df944160 
, got_subseq=0) at 
../../../../bash/lib/readline/readline.c:985
  #17 0x55a0df6dca4c in _rl_dispatch (key=-1320386609, map=0x55a0df944160 
) at ../../../../bash/lib/readline/readline.c:797
  #18 0x55a0df6dc6d4 in readline_internal_char () at 
../../../../bash/lib/readline/readline.c:629
  #19 0x55a0df6dc72c in readline_internal_charloop () at 
../../../../bash/lib/readline/readline.c:656
  #20 0x55a0df6dc750 in readline_internal () at 

Read builtin -e (readline) oddities

[Eduardo A . Bustamante López]

I'm trying to figure out a way to fuzz >>read -e -d ""<<, without having the
fuzzer break due to the temporary files created by fc.

While doing this, I noticed the oddities described below.


#1
Hit `C-x C-e' twice.  The value of PATH seems to be ignored for the second
line.

  dualbus@debian:~$ PATH= read -e
  
  bash: vim: No such file or directory
  
  echo hi
  hi


#2
Spurious `;'s are inserted into the history.

  dualbus@debian:~$ history -c
  dualbus@debian:~$ PATH= read -e
  
  bash: vim: No such file or directory
  
  echo hi
  hi
  
  dualbus@debian:~$ history
  1  PATH= read -e; ; 
  2  echo hi
  3  history


#3
Hit `C-x C-e' thrice. Also, the mapping of `\C-m' became `self-insert' instead
of `accept-line'.

  dualbus@debian:~$ PATH= EDITOR=: read -e -d ''
  
  PATH= EDITOR=: read -e -d ''; 
  
  fc -e "${VISUAL:-${EDITOR:-$(command -v editor || echo emacs)}}"
  : /tmp/bash-fc.IZAzmF
  PATH= EDITOR=: read -e -d ''; ; 
  bash: syntax error near unexpected token `;'
  
  fc -e "${VISUAL:-${EDITOR:-$(command -v editor || echo emacs)}}"
  vim /tmp/bash-fc.35ePED
  ^M^M^C
  dualbus@debian:~$ ^M^M^M^M^M^M^M^M



In relation to my fuzzing goal, it seems this is what I wanted:

  dualbus@debian:~$ (enable -n fc; read -e -d '')
  
  bash: fc: command not found
  
  bash: fc: command not found
  
  bash: fc: command not found
  
  bash: fc: command not found
  
  bash: fc: command not found
  ^C

-- 
Eduardo Bustamante
https://dualbus.me/

Re: Regression -- can't read input w/stderr redirect

[Eduardo A . Bustamante López]

On Sun, Jun 18, 2017 at 02:02:05PM -0700, L A Walsh wrote:
[...]
>  int dpi=$(ord $(<"$pixels_path" 2>/dev/null))
> 
> This used to work but now works _unreliably_.

In what version does this used to work?

I tested on a couple of versions, and the behavior you describe didn't work:

  dualbus@debian:~/src/gnu/bash-builds$ for b in bash-*/; do $b/bash -c 'echo 
$(< <(echo x)) $BASH_VERSION'; done
  x 3.2.57(1)-release
  x 4.2.0(1)-release
  x 4.2.53(1)-release
  x 4.3.30(1)-release
  
  dualbus@debian:~/src/gnu/bash-builds$ for b in bash-*/; do $b/bash -c 'echo 
$(< <(echo x) >/dev/stdout) $BASH_VERSION'; done
  3.2.57(1)-release
  4.2.0(1)-release
  4.2.53(1)-release
  4.3.30(1)-release


And if you inspect the source code, you'll notice that this the command
substitution "cat file" functionality is implemented in here:

builtins/evalstring.c:

  413   /* See if this is a candidate for $( type == cm_simple && !command->redirects &&
  418   (command->flags & CMD_TIME_PIPELINE) == 0 &&
  419   command->value.Simple->words == 0 &&
  420   command->value.Simple->redirects &&
  421   command->value.Simple->redirects->next == 0 &&
  422   command->value.Simple->redirects->instruction == 
r_input_direction &&
  423   command->value.Simple->redirects->redirector.dest == 0)
  424 {
  425   int r;
  426   r = cat_file (command->value.Simple->redirects);
  427   last_result = (r < 0) ? EXECUTION_FAILURE : 
EXECUTION_SUCCESS;
  428 }

It's clear that the following conditions must be met:

- The redirection must be performed inside a command substitution (ln.
  415)
- The command substitution must be a simple command (ln. 417)
- The simple command must consist of a single input redirection (ln.
  420), i.e. no words (ln. 419), no "next" redirection (ln. 421), input
  redirection (ln. 422), and the target being file descriptor 0 (ln.
  423).

So that means that `echo $(Y)' is not a valid "cat file" command
substitution.

I think that you're looking for:

  $ bash -c 'printf "x\0y" > f; { a=$(/dev/null; declare -p a'
  declare -- a="xy"

Instead.

-- 
Eduardo Bustamante
https://dualbus.me/

Issues with `rl_vi_change_char' function in combination with `_rl_vi_redoing = 1'

[Eduardo A . Bustamante López]

# When bash is compiled with multibyte support, the following happens:

- set -o vi
- Type: <ñ><.>
- Bash writes \261 to `mb[0]' and '\0' to `mb[1]' (ñ is U+00F1, or in
  UTF-8, \303\261)
- `mb' is of type signed char, and not really suitable to store a wide
  character.

  dualbus@debian:~$ set -o vi
  dualbus@debian:~$ �
  bash: $'\261': command not found


# When bash is compiled w/o multibyte support, the following happens:

  1948 int
  1949 rl_vi_change_char (int count, int key)
  1950 {
  1951   int c;
  1952   char mb[MB_LEN_MAX]; /* MB_LEN_MAX == 1 */
  1953 
  1954   if (_rl_vi_redoing)
  1955 {
  1956   c = _rl_vi_last_replacement;
  1957   mb[0] = c;
  1958   mb[1] = '\0'; /* out of bounds write */
  1959 }

Reproduce with:

- set -o vi
- Type: <.>
- Bash writes '\0' to `mb[1]', which is out of bounds.

  dualbus@debian:~/src/gnu/bash-build$ ./bash
  bash-4.4$ set -o vi
  bash-4.4$ =
  ==7881==ERROR: AddressSanitizer: stack-buffer-overflow on address 
0x7ffe8416f471 at pc 0x55e4ba75d81a bp 0x7ffe8416f420 sp 0x7ffe8416f418
  WRITE of size 1 at 0x7ffe8416f471 thread T0
  #0 0x55e4ba75d819 in rl_vi_change_char 
../../../bash/lib/readline/vi_mode.c:1959
  #1 0x55e4ba74fe3d in _rl_dispatch_subseq 
../../../bash/lib/readline/readline.c:851
  #2 0x55e4ba74fa18 in _rl_dispatch 
../../../bash/lib/readline/readline.c:797
  #3 0x55e4ba752a41 in rl_vi_redo ../../../bash/lib/readline/vi_mode.c:285
  #4 0x55e4ba74fe3d in _rl_dispatch_subseq 
../../../bash/lib/readline/readline.c:851
  #5 0x55e4ba74fa18 in _rl_dispatch 
../../../bash/lib/readline/readline.c:797
  #6 0x55e4ba74f257 in readline_internal_char 
../../../bash/lib/readline/readline.c:629
  #7 0x55e4ba74f2e9 in readline_internal_charloop 
../../../bash/lib/readline/readline.c:656
  #8 0x55e4ba74f30d in readline_internal 
../../../bash/lib/readline/readline.c:670
  #9 0x55e4ba74e9c3 in readline ../../../bash/lib/readline/readline.c:374
  #10 0x55e4ba5ffd01 in yy_readline_get ../bash/parse.y:1464
  #11 0x55e4ba5ffed9 in yy_readline_get ../bash/parse.y:1495
  #12 0x55e4ba5ffb64 in yy_getc ../bash/parse.y:1397
  #13 0x55e4ba601ea4 in shell_getc ../bash/parse.y:2297
  #14 0x55e4ba6048d0 in read_token ../bash/parse.y:3146
  #15 0x55e4ba6032ad in yylex ../bash/parse.y:2683
  #16 0x55e4ba5f7f31 in yyparse 
/home/dualbus/src/gnu/bash-build/y.tab.c:1821
  #17 0x55e4ba5f7102 in parse_command ../bash/eval.c:294
  #18 0x55e4ba5f7357 in read_command ../bash/eval.c:338
  #19 0x55e4ba5f6593 in reader_loop ../bash/eval.c:140
  #20 0x55e4ba5f1d3d in main ../bash/shell.c:794
  #21 0x7ff2e9a7d2b0 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
  #22 0x55e4ba5f0939 in _start 
(/home/dualbus/src/gnu/bash-build/bash+0x7b939)
  
  Address 0x7ffe8416f471 is located in stack of thread T0 at offset 33 in frame
  #0 0x55e4ba75d72f in rl_vi_change_char 
../../../bash/lib/readline/vi_mode.c:1950
  
This frame has 1 object(s):
  [32, 33) 'mb' <== Memory access at offset 33 overflows this variable
  HINT: this may be a false positive if your program uses some custom stack 
unwind mechanism or swapcontext
(longjmp and C++ exceptions *are* supported)
  SUMMARY: AddressSanitizer: stack-buffer-overflow 
../../../bash/lib/readline/vi_mode.c:1959 in rl_vi_change_char
  Shadow bytes around the buggy address:
0x100050825e30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x100050825e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x100050825e50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x100050825e60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x100050825e70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  =>0x100050825e80: 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1[01]f4
0x100050825e90: f4 f4 f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00
0x100050825ea0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x100050825eb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x100050825ec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x100050825ed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable:   00
Partially addressable: 01 02 03 04 05 06 07 
Heap left redzone:   fa
Heap right redzone:  fb
Freed heap region:   fd
Stack left redzone:  f1
Stack mid redzone:   f2
Stack right redzone: f3
Stack partial redzone:   f4
Stack after return:  f5
Stack use after scope:   f8
Global redzone:  f9
Global init order:   f6
Poisoned by user:f7
Container overflow:  fc
Array cookie:ac
Intra object redzone:bb
ASan internal:   fe
Left alloca redzone: ca
Right alloca redzone:cb
  ==7881==ABORTING

[PATCH] Bash does not build when --disable-multibyte due to missing HAVE_MULTIBYTE guard in display.c

[Eduardo A . Bustamante López]

dualbus@debian:~/src/gnu/bash$ git diff -- lib/readline/display.c 
diff --git a/lib/readline/display.c b/lib/readline/display.c
index d5536211..1f281cd3 100644
--- a/lib/readline/display.c
+++ b/lib/readline/display.c
@@ -2127,7 +2127,7 @@ dumb_update:
  cpos_adjusted = 1;
}
 
-#if 1
+#ifdef HAVE_MULTIBYTE
  /* If we write a non-space into the last screen column,
 remove the note that we added a space to compensate for
 a multibyte double-width character that didn't fit, since

-- 
Eduardo Bustamante
https://dualbus.me/

Return code of coproc when NAME is not a legal_identifier

[Eduardo A . Bustamante López]

The manual is clear on this:

  Since the coprocess is created as an asynchronous command, the coproc
  command always returns success

I'd argue that it should return an error code for this particular case
though:

  $ bash -c 'coproc % { :; }; echo $?'
  bash: `%': not a valid identifier
  0


The `execute_coproc' function _seems_ to try to return an error code for this
case (by the way, the value of `invert' is undefined at this point):

  2301   /* XXX - expand coproc name without splitting -- bash-5.0 */
  2302   /* could make this dependent on a shopt option */
  2303   name = expand_string_unsplit_to_string (command->value.Coproc->name, 
0);
  2304   /* Optional check -- bash-5.0. */
  2305   if (legal_identifier (name) == 0)
  2306 {
  2307   internal_error (_("`%s': not a valid identifier"), name);
  2308   return (invert ? EXECUTION_SUCCESS : EXECUTION_FAILURE);
  2309 }
  2310   else
  2311 {
  2312   free (command->value.Coproc->name);
  2313   command->value.Coproc->name = name;
  2314 }


The following patch fixes this (coproc returns $? = 1 when `NAME' is not a
legal identifier), although I'm not convinced about the quality of this patch.


dualbus@debian:~/src/gnu/bash$ git diff
diff --git a/execute_cmd.c b/execute_cmd.c
index 0183a105..ab022bdb 100644
--- a/execute_cmd.c
+++ b/execute_cmd.c
@@ -597,8 +597,9 @@ execute_command_internal (command, asynchronous, pipe_in, 
pipe_out,
 return (execute_in_subshell (command, asynchronous, pipe_in, pipe_out, 
fds_to_close));
 
 #if defined (COPROCESS_SUPPORT)
-  if (command->type == cm_coproc)
-return (execute_coproc (command, pipe_in, pipe_out, fds_to_close));
+  if (command->type == cm_coproc) {
+return (last_command_exit_value = execute_coproc (command, pipe_in, 
pipe_out, fds_to_close));
+  }
 #endif
 
   user_subshell = command->type == cm_subshell || ((command->flags & 
CMD_WANT_SUBSHELL) != 0);
@@ -2298,6 +2299,8 @@ execute_coproc (command, pipe_in, pipe_out, fds_to_close)
   coproc_init (_coproc);
 #endif
 
+  invert = (command->flags & CMD_INVERT_RETURN) != 0;
+
   /* XXX - expand coproc name without splitting -- bash-5.0 */
   /* could make this dependent on a shopt option */
   name = expand_string_unsplit_to_string (command->value.Coproc->name, 0);
@@ -2313,7 +2316,6 @@ execute_coproc (command, pipe_in, pipe_out, fds_to_close)
   command->value.Coproc->name = name;
 }
 
-  invert = (command->flags & CMD_INVERT_RETURN) != 0;
   command_string_index = 0;
   tcmd = make_command_string (command);




This came up as a warning from cppcheck:

  [../bash/execute_cmd.c:2308]: (error) Uninitialized variable: invert

-- 
Eduardo Bustamante
https://dualbus.me/

Re: AddressSanitizer: heap-buffer-overflow in rl_kill_text

[Eduardo A . Bustamante López]

On Thu, Jun 15, 2017 at 09:42:41AM -0500, Eduardo Bustamante wrote:
> Found by fuzzing `read -e' with AFL. The stacktrace reported by Address
> Sanitizer is followed by the base64 encoded crashing input.
> 
> 
> ==11018==ERROR: AddressSanitizer: heap-buffer-overflow on address 
> 0x6070ccc0 at pc 0x559bb60f1be7 bp 0x7ffc36ec8710 sp 0x7ffc36ec8708
> READ of size 8 at 0x6070ccc0 thread T0
> #0 0x559bb60f1be6 in _rl_copy_to_kill_ring 
> (/home/dualbus/src/gnu/bash-build/bash+0x23cbe6)

Easy fix. When `rl_kill_ring_length == rl_max_kills (10)', all of the entries
in the kill ring are shifted. The loop has an off-by-one error though.

I also think that using `rl_max_kills' in the loop instead of `slot' makes the
code easier to read.

dualbus@debian:~/src/gnu/bash$ git difftool -y -x 'diff -c' -- 
lib/readline/kill.c 
*** /tmp/uLCFvH_kill.c  2017-06-16 10:04:43.472930262 -0500
--- lib/readline/kill.c 2017-06-16 10:04:20.048344312 -0500
***
*** 113,119 
{
  register int i;
  xfree (rl_kill_ring[0]);
! for (i = 0; i < slot; i++)
rl_kill_ring[i] = rl_kill_ring[i + 1];
}
  else
--- 113,119 
{
  register int i;
  xfree (rl_kill_ring[0]);
! for (i = 0; i < rl_max_kills - 1; i++)
rl_kill_ring[i] = rl_kill_ring[i + 1];
}
  else

-- 
Eduardo Bustamante
https://dualbus.me/

Re: AddressSanitizer: heap-buffer-overflow in rl_tilde_expand

[Eduardo A . Bustamante López]

On Thu, Jun 15, 2017 at 09:39:09AM -0500, Eduardo Bustamante wrote:
> Found by fuzzing `read -e' with AFL. The stacktrace reported by Address
> Sanitizer is followed by the base64 encoded crashing input.
> 
> 
> ==472==ERROR: AddressSanitizer: heap-buffer-overflow on address 
> 0x6110977f at pc 0x562befba4a14 bp 0x7ffdee172bb0 sp 0x7ffdee172ba8
> READ of size 1 at 0x6110977f thread T0
> #0 0x562befba4a13 in rl_tilde_expand 
> (/home/dualbus/src/gnu/bash-build/bash+0x23ba13)

This one looks like an easy fix. When `start = 0', the loop ends up
dereferencing `rl_line_buffer[-1]'. Changing the order of the test does the 
trick.


dualbus@debian:~/src/gnu/bash$ git difftool -y -x 'diff -c' -- 
lib/readline/util.c 
*** /tmp/zFXFei_util.c  2017-06-16 09:34:03.958088209 -0500
--- lib/readline/util.c 2017-06-16 09:33:09.384638705 -0500
***
*** 193,199 
  }
else if (start >= 0 && rl_line_buffer[start] != '~')
  {
!   for (; !whitespace (rl_line_buffer[start]) && start >= 0; start--)
  ;
start++;
  }
--- 193,199 
  }
else if (start >= 0 && rl_line_buffer[start] != '~')
  {
!   for (; start >= 0 && !whitespace (rl_line_buffer[start]); start--)
  ;
start++;
  }

-- 
Eduardo Bustamante
https://dualbus.me/

Re: AddressSanitizer: global-buffer-overflow in rl_filename_completion_function

[Eduardo A . Bustamante López]

On Thu, Jun 15, 2017 at 09:41:08AM -0500, Eduardo Bustamante wrote:
> Found by fuzzing `read -e' with AFL. The stacktrace reported by Address
> Sanitizer is followed by the base64 encoded crashing input.
> 
> 
> ==1098==ERROR: AddressSanitizer: global-buffer-overflow on address 
> 0x55e61a6b4c5c at pc 0x55e61a3426ca bp 0x7fff1820a300 sp 0x7fff1820a2f8
> READ of size 4 at 0x55e61a6b4c5c thread T0
> #0 0x55e61a3426c9 in bash_dequote_filename 
> (/home/dualbus/src/gnu/bash-build/bash+0x17a6c9)

Easy fix. `p' is a signed char pointer, therefore when `*p = 255', it tries to
dereference `sh_syntaxtab[-1]'.

dualbus@debian:~/src/gnu/bash$ git difftool -y -x 'diff -c' -- bashline.c
*** /tmp/mVc4sH_bashline.c  2017-06-16 09:52:56.471508904 -0500
--- bashline.c  2017-06-16 09:48:55.706503276 -0500
***
*** 3886,3892 
*r++ = *p;
  /* Backslashes are preserved within double quotes unless the
 character is one that is defined to be escaped */
! else if (quoted == '"' && ((sh_syntaxtab[p[1]] & CBSDQUOTE) == 0))
*r++ = *p;
  
  *r++ = *++p;
--- 3886,3892 
*r++ = *p;
  /* Backslashes are preserved within double quotes unless the
 character is one that is defined to be escaped */
! else if (quoted == '"' && ((sh_syntaxtab[(unsigned char)p[1]] & 
CBSDQUOTE) == 0))
*r++ = *p;
  
  *r++ = *++p;





Maybe it's a good idea to change these too. In locale.c there shouldn't be a
problem, because the loop is constrained to `x < sh_syntabsiz', but perhaps
just to silence compiler warnings :-)?

dualbus@debian:~/src/gnu/bash$ git difftool -y -x 'diff -c' -- locale.c parse.y 
*** /tmp/OyLWya_locale.c2017-06-16 09:55:15.854368199 -0500
--- locale.c2017-06-16 09:50:52.816950476 -0500
***
*** 552,565 
for (x = 0; x < sh_syntabsiz; x++)
  {
if (isblank ((unsigned char)x))
!   sh_syntaxtab[x] |= CSHBRK|CBLANK;
else if (member (x, shell_break_chars))
{
! sh_syntaxtab[x] |= CSHBRK;
! sh_syntaxtab[x] &= ~CBLANK;
}
else
!   sh_syntaxtab[x] &= ~(CSHBRK|CBLANK);
  }
  }
  
--- 552,565 
for (x = 0; x < sh_syntabsiz; x++)
  {
if (isblank ((unsigned char)x))
!   sh_syntaxtab[(unsigned char)x] |= CSHBRK|CBLANK;
else if (member (x, shell_break_chars))
{
! sh_syntaxtab[(unsigned char)x] |= CSHBRK;
! sh_syntaxtab[(unsigned char)x] &= ~CBLANK;
}
else
!   sh_syntaxtab[(unsigned char)x] &= ~(CSHBRK|CBLANK);
  }
  }
  
*** /tmp/qKFxWa_parse.y 2017-06-16 09:55:15.862368362 -0500
--- parse.y 2017-06-16 09:52:22.522808775 -0500
***
*** 4842,4848 
  
  /* If the next character is to be quoted, note it now. */
  if (cd == 0 || cd == '`' ||
! (cd == '"' && peek_char >= 0 && (sh_syntaxtab[peek_char] & 
CBSDQUOTE)))
pass_next_character++;
  
  quoted = 1;
--- 4842,4848 
  
  /* If the next character is to be quoted, note it now. */
  if (cd == 0 || cd == '`' ||
! (cd == '"' && peek_char >= 0 && (sh_syntaxtab[(unsigned 
char)peek_char] & CBSDQUOTE)))
pass_next_character++;
  
  quoted = 1;

-- 
Eduardo Bustamante
https://dualbus.me/

Re: AddressSanitizer: heap-buffer-overflow in rl_delete

[Eduardo A . Bustamante López]

On Thu, Jun 15, 2017 at 09:36:58AM -0500, Eduardo Bustamante wrote:
> Found by fuzzing `read -e' with AFL. The stacktrace reported by Address
> Sanitizer is followed by the base64 encoded crashing input.
> 
> 
> ==1736==ERROR: AddressSanitizer: heap-buffer-overflow on address 
> 0x61109880 at pc 0x7f464da3a063 bp 0x7ffe86032fe0 sp 0x7ffe86032790
> READ of size 115 at 0x61109880 thread T0
> #0 0x7f464da3a062  (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x3c062)
> #1 0x5634e38634c3 in _rl_find_next_mbchar_internal 
> (/home/dualbus/src/gnu/bash-build/bash+0x25d4c3)
> #2 0x5634e3864375 in _rl_find_next_mbchar 
> (/home/dualbus/src/gnu/bash-build/bash+0x25e375)
> #3 0x5634e3850c0e in rl_delete 
> (/home/dualbus/src/gnu/bash-build/bash+0x24ac0e)

OK. Here's an easy way to reproduce this.

- Start on an empty rl_line_buffer
- Call `set-mark' with a numeric argument (a large number, e.g. 500, is better).
- Call `exchange-point-and-mark' so that now rl_point = 500
- Call `delete-char'
- Bash crashes

The _rl_set_mark_at_pos function already checks for `position > rl_end',
so I'm not sure how to fix this.

-- 
Eduardo Bustamante
https://dualbus.me/

Re: memory leak in execute_simple_command when dofork is true

[Eduardo A . Bustamante López]

Actually, there are more leaks in execute_cmd.c, I'm just going to list them 
here.

---
hp% ./bash -c '(:)'

=
==7854==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 6 byte(s) in 1 object(s) allocated from:
#0 0x7fb0c2580d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0x562cb665d6cb in xmalloc /home/dualbus/local/src/gnu/bash/xmalloc.c:112
#2 0x562cb658ab1c in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:610
#3 0x562cb666e900 in parse_and_execute 
/home/dualbus/local/src/gnu/bash/builtins/evalstring.c:443
#4 0x562cb655797f in run_one_command 
/home/dualbus/local/src/gnu/bash/shell.c:1399
#5 0x562cb6555eaa in main /home/dualbus/local/src/gnu/bash/shell.c:724
#6 0x7fb0c1d132b0 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

SUMMARY: AddressSanitizer: 6 byte(s) leaked in 1 allocation(s).

---
hp% ./bash -c 'coproc :; sleep 1'

=
==8716==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x7fc17740ad28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0x55d4b48826cb in xmalloc /home/dualbus/local/src/gnu/bash/xmalloc.c:112
#2 0x55d4b47b72c6 in execute_coproc 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:2301
#3 0x55d4b47af935 in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:593
#4 0x55d4b47af0fb in execute_command 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:405
#5 0x55d4b47b8f54 in execute_connection 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:2609
#6 0x55d4b47b16e9 in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:971
#7 0x55d4b4893900 in parse_and_execute 
/home/dualbus/local/src/gnu/bash/builtins/evalstring.c:443
#8 0x55d4b477c97f in run_one_command 
/home/dualbus/local/src/gnu/bash/shell.c:1399
#9 0x55d4b477aeaa in main /home/dualbus/local/src/gnu/bash/shell.c:724
#10 0x7fc176b9d2b0 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

SUMMARY: AddressSanitizer: 16 byte(s) leaked in 1 allocation(s).

---
hp% ./bash -c '::; :'   
./bash: ::: command not found

=
==10176==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 3 byte(s) in 1 object(s) allocated from:
#0 0x7f003afe5d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0x561a310216cb in xmalloc /home/dualbus/local/src/gnu/bash/xmalloc.c:112
#2 0x561a30f641b3 in execute_disk_command 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:5194
#3 0x561a30f61b01 in execute_simple_command 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:4403
#4 0x561a30f4f914 in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:802
#5 0x561a30f4e0fb in execute_command 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:405
#6 0x561a30f57f54 in execute_connection 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:2609
#7 0x561a30f506e9 in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:971
#8 0x561a31032900 in parse_and_execute 
/home/dualbus/local/src/gnu/bash/builtins/evalstring.c:443
#9 0x561a30f1b97f in run_one_command 
/home/dualbus/local/src/gnu/bash/shell.c:1399
#10 0x561a30f19eaa in main /home/dualbus/local/src/gnu/bash/shell.c:724
#11 0x7f003a7782b0 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

SUMMARY: AddressSanitizer: 3 byte(s) leaked in 1 allocation(s).

-- 
Eduardo Bustamante
https://dualbus.me/

memory leak in execute_simple_command when dofork is true

[Eduardo A . Bustamante López]

I ran the configure script with the following arguments:

hp% ./configure CC=gcc-6 CFLAGS='-Wall -g -ggdb -O0 -fsanitize=address' 
LDFLAGS=-fsanitize=address --without-bash-malloc

Which enables the LeakSanitizer 
(https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer).

It detected the following memory leak:

hp% ./bash -c ': & wait'

=
==5784==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 2 byte(s) in 1 object(s) allocated from:
#0 0x7f0ec8737d28 in malloc 
(/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0x559e5096b6cb in xmalloc 
/home/dualbus/local/src/gnu/bash/xmalloc.c:112
#2 0x559e508aa246 in execute_simple_command 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:4105
#3 0x559e50899914 in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:802
#4 0x559e508a19a9 in execute_connection 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:2581
#5 0x559e5089a6e9 in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:971
#6 0x559e5097c900 in parse_and_execute 
/home/dualbus/local/src/gnu/bash/builtins/evalstring.c:443
#7 0x559e5086597f in run_one_command 
/home/dualbus/local/src/gnu/bash/shell.c:1399
#8 0x559e50863eaa in main /home/dualbus/local/src/gnu/bash/shell.c:724
#9 0x7f0ec7eca2b0 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

SUMMARY: AddressSanitizer: 2 byte(s) leaked in 1 allocation(s).

The leak is in line 4105, since the savestring function is called (which just a
strcpy/malloc wrapper), but the allocated memory is not FREE'd.

hp% cat -n execute_cmd.c|sed -n '4100,4110p'
  4100   vast majority of cases. */
  4101maybe_make_export_env ();
  4102
  4103/* Don't let a DEBUG trap overwrite the command string to be 
saved with
  4104   the process/job associated with this child. */
  4105if (make_child (savestring (the_printed_command_except_trap), 
async) == 0)
  4106  {
  4107already_forked = 1;
  4108simple_command->flags |= CMD_NO_FORK;
  4109
  4110subshell_environment = SUBSHELL_FORK; /* XXX */

The issue is more evident when running long commands:

# This will start chewing chunks of ~2MB
hp% bash -c 'while :; do : "$(printf '%.sx' {1..2097152})" & wait; done'

I think the fix is easy, since we just have to store the result of savestring
in a temporary pointer, so that we can later call FREE(p).

-- 
Eduardo Bustamante
https://dualbus.me/

devel branch fails to build when using bash malloc due to missing errno.h include

[Eduardo A . Bustamante López]

I'm running configure with the following flags:

|./configure CC=gcc-6

The error I get while building:

|make[1]: Entering directory '/home/dualbus/local/src/gnu/bash/lib/malloc'
|gcc-6  -I. -I../.. -I../.. -I../../include -I../../lib  -DHAVE_CONFIG_H 
-DSHELL  -Wall -g -ggdb -O0 -Wno-parentheses -Wno-format-security -DRCHECK 
-Dbotch=programming_error -DMALLOC_DEBUG  -c malloc.c
|malloc.c: In function ‘xsplit’:
|malloc.c:472:21: warning: unused variable ‘split_max’ [-Wunused-variable]
|   int nbuck, nblks, split_max;
| ^
|malloc.c: In function ‘posix_memalign’:
|malloc.c:1143:12: error: ‘EINVAL’ undeclared (first use in this function)
| return EINVAL;
|^~
|malloc.c:1143:12: note: each undeclared identifier is reported only once 
for each function it appears in
|malloc.c:1153:10: error: ‘ENOMEM’ undeclared (first use in this function)
|   return ENOMEM;
|  ^~
|malloc.c:1154:1: warning: control reaches end of non-void function 
[-Wreturn-type]
| }
| ^
|At top level:
|malloc.c:669:1: warning: ‘malloc_debug_dummy’ defined but not used 
[-Wunused-function]
| malloc_debug_dummy ()
| ^~
|Makefile:62: recipe for target 'malloc.o' failed
|make[1]: *** [malloc.o] Error 1
|make[1]: Leaving directory '/home/dualbus/local/src/gnu/bash/lib/malloc'
|Makefile:399: recipe for target 'lib/malloc/libmalloc.a' failed
|make: *** [lib/malloc/libmalloc.a] Error 1

This is fixed by adding an errno.h include inside lib/malloc/malloc.c

-- 
Eduardo Bustamante
https://dualbus.me/

Re: Assigning to BASHPID fails silently

[Eduardo A . Bustamante López]

On Wed, Oct 19, 2016 at 11:53:37PM +0200, Martijn Dekker wrote:
[...]
> Assigning to BASHPID most certainly does have an effect. Since you
> didn't quote that part, I think you might have missed my point that
> attempting this will silently exit the shell without any error message,
> causing the problem to be hard to track down. This is different from
> GROUPS and FUNCNAME, where the shell silently continues (causing the
> problem to be hard to track down in a completely different way -- if
> anything, that's worse!).

I think he did get your point. There's definitely a bug here. It should be
either:

1. BASHPID is readonly, therefore assignment to it is fatal and the script exits
(with an error message printed). That's what my previous patch did.

2. BASHPID is not read-only, but changes to it are discarded (with the null
assignement function). Assignments to BASHPID are non-fatal, and it's possible
to unset it. Once it's unset, it's magical meaning is lost. (I think this is
what Chet is proposing). noro_bashpid.patch

> In what possible context would assigning to any of these variables make
> sense, or be an indication of anything other than a fatal bug in the
> script? I think they should all be readonly, and produce a proper
> diagnostic error message upon exit if assigning is attempted.
[...]

I wonder the same thing. I don't understand the reasoning for picking (2).

-- 
Eduardo Bustamante
https://dualbus.me/
diff --git a/variables.c b/variables.c
index 2e8b38c..5120f2e 100644
--- a/variables.c
+++ b/variables.c
@@ -1462,7 +1462,7 @@ get_bashpid (var)
   p = itos (pid);
 
   FREE (value_cell (var));
-  VSETATTR (var, att_integer|att_readonly);
+  VSETATTR (var, att_integer);
   var_setvalue (var, p);
   return (var);
 }
@@ -1767,7 +1767,7 @@ initialize_dynamic_variables ()
   VSETATTR (v, att_integer);
 
   INIT_DYNAMIC_VAR ("BASHPID", (char *)NULL, get_bashpid, null_assign);
-  VSETATTR (v, att_integer|att_readonly);
+  VSETATTR (v, att_integer);
 
 #if defined (HISTORY)
   INIT_DYNAMIC_VAR ("HISTCMD", (char *)NULL, get_histcmd, 
(sh_var_assign_func_t *)NULL);