Re: Bash-5.0-alpha available

[Chet Ramey]

On 9/16/18 5:22 PM, Eduardo A. Bustamante López wrote:

> I do think there are still bugs around readline and I'm hoping to concentrate 
> my
> efforts there. It'd be nice to get rid of the multibyte character handling 
> edge
> cases and other oddities before the release. Although I'm not asking for you 
> to
> hold it, I just wanted to understand how much time I have left, and how to
> organize my efforts around that schedule.

Usually there are a couple of months between beta and release; I tend to
run through the release candidate fairly quickly, but the biggest time
consumer is testing on different platforms and doing release engineering.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: expand_prompt_string segmentation faults

[Chet Ramey]

On 9/16/18 5:15 PM, Eduardo A. Bustamante López wrote:
> On Sun, Sep 16, 2018 at 02:39:07PM -0400, Chet Ramey wrote:
>> On 9/16/18 4:12 AM, Eduardo A. Bustamante López wrote:
>>> Here's another interesting crash:
>>>
>>> $ base64 < 53
>>> MBgFEBAQMBgFEBAfEA==
>>>
>>> $ gdb -batch -ex=r -ex=bt --args ./bash --noprofile --norc -c 'PATH=; set 
>>> -o emacs; IFS= read -re' < 53
>>
>> I can't reproduce this on Mac OS X or Fedora 28.

OK, let me look around some more with your debugging hints. If I find it,
it will be in the next devel branch push.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Bash-5.0-alpha available

[Eduardo A . Bustamante López]

On Sun, Sep 16, 2018 at 12:42:28PM -0400, Chet Ramey wrote:
(...)
> Bash-5.0 is about to go into beta test.

Interesting. Thanks.

> (...) While fuzzing has some
> interesting results, I haven't really seen any security problems or
> any serious bug that would cause me to hold up a release from the
> fuzzing-uncovered issues I've seen so far.

I agree. The parser became quite robust after shellshock. I haven't been able to
find any bugs there (and don't expect to find many anyways).

I do think there are still bugs around readline and I'm hoping to concentrate my
efforts there. It'd be nice to get rid of the multibyte character handling edge
cases and other oddities before the release. Although I'm not asking for you to
hold it, I just wanted to understand how much time I have left, and how to
organize my efforts around that schedule.

Re: expand_prompt_string segmentation faults

[Eduardo A . Bustamante López]

On Sun, Sep 16, 2018 at 02:39:07PM -0400, Chet Ramey wrote:
> On 9/16/18 4:12 AM, Eduardo A. Bustamante López wrote:
> > Here's another interesting crash:
> > 
> > $ base64 < 53
> > MBgFEBAQMBgFEBAfEA==
> > 
> > $ gdb -batch -ex=r -ex=bt --args ./bash --noprofile --norc -c 'PATH=; set 
> > -o emacs; IFS= read -re' < 53
> 
> I can't reproduce this on Mac OS X or Fedora 28.

Hm, I can reliably reproduce it with or without bash-malloc.

$ base64 -d <<< 'MBgFEBAQMBgFEBAfEA==' > 53

$ cat -v 53
0^X^E^P^P^P0^X^E^P^P^_^P

$ CC=clang CFLAGS='-O0 -ggdb' ./configure  && make -j4
(...)

$ ./bash --noprofile --norc -c 'PATH=; set -o emacs; IFS= read -re < 53' 
0
./bash: vim: No such file or directory
0rl_maybe_unsave_line: rl_undo_list=0x236c848
rl_maybe_unsave_line: rl_undo_list->next=(nil)
rl_maybe_unsave_line: rl_undo_list=0x236c848
rl_maybe_unsave_line: rl_undo_list->next=(nil)
0
./bash: vim: No such file or directory
rl_do_undo: rl_undo_list = (nil)
rl_do_undo: xfree(release = 0x236c848)
rl_maybe_unsave_line: rl_undo_list=0x2360188
rl_maybe_unsave_line: rl_undo_list->next=0x236c848
rl_do_undo: rl_undo_list = 0x236c848
rl_do_undo: xfree(release = 0x2360188)
rl_do_undo: rl_undo_list = 0x23601c0
rl_do_undo: xfree(release = 0x236c848)

malloc: unknown:0: assertion botched
malloc: 0x236c848: allocated: last allocated from unknown:0
free: called with already freed block argument
Aborting...Aborted


My interpretation of the issue is that the `^_' triggers an `undo', which then
causes bash to release that entry (i.e. `0x236c848' in the example above). Then
the `^P' somehow restores that free'd rl_undo_list entry and bash tries to free
it again during readline_internal_teardown.


Under GDB:

dualbus@ubuntu:~/src/gnu/bash$ gdb --args ./bash --noprofile --norc -c 'PATH=; 
set -o emacs; IFS= read -re < 53' 
GNU gdb (Ubuntu 8.2-0ubuntu1) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./bash...done.
(gdb) b rl_maybe_unsave_line 
Breakpoint 1 at 0x526128: file misc.c, line 354.
(gdb) b rl_do_undo 
Breakpoint 2 at 0x51e6a8: file undo.c, line 175.
(gdb) r
Starting program: /home/dualbus/src/gnu/bash/bash --noprofile --norc -c 
PATH=\;\ set\ -o\ emacs\;\ IFS=\ read\ -re\ \<\ 53
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0
[Detaching after fork from child process 20660]
/home/dualbus/src/gnu/bash/bash: vim: No such file or directory
0
Breakpoint 1, rl_maybe_unsave_line () at misc.c:354
354   if (_rl_saved_line_for_history)
(gdb) bt
#0  rl_maybe_unsave_line () at misc.c:354
#1  0x005266ea in rl_get_previous_history (count=1, key=16) at 
misc.c:618
#2  0x004fb350 in _rl_dispatch_subseq (key=16, map=0x575db0 
, got_subseq=0) at readline.c:852
#3  0x004face9 in _rl_dispatch (key=16, map=0x575db0 
) at readline.c:798
#4  0x004fac59 in readline_internal_char () at readline.c:632
#5  0x004fc282 in readline_internal_charloop () at readline.c:659
#6  0x004fa5ae in readline_internal () at readline.c:671
#7  0x004fa470 in readline (prompt=0x5546af "") at readline.c:377
#8  0x004caa46 in edit_line (p=0x5546af "", itext=0x0) at 
./read.def:1104
#9  0x004c94ea in read_builtin (list=0x0) at ./read.def:563
#10 0x0044b599 in execute_builtin (builtin=0x4c8590 , 
words=0x629428, flags=0, subshell=0) at execute_cmd.c:4677
#11 0x0044a96f in execute_builtin_or_function (words=0x629428, 
builtin=0x4c8590 , var=0x0, redirects=0x628d08, 
fds_to_close=0x628dc8, flags=0) at execute_cmd.c:5185
#12 0x004437c9 in execute_simple_command (simple_command=0x628c08, 
pipe_in=-1, pipe_out=-1, async=0, fds_to_close=0x628dc8) at execute_cmd.c:4449
#13 0x004412ab in execute_command_internal (command=0x628bc8, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x628dc8) at 
execute_cmd.c:840
#14 0x00445208 in execute_connection (command=0x628d88, asynchronous=0, 
pipe_in=-1, pipe_out=-1, fds_to_close=0x628dc8) at execute_cmd.c:2689
#15 0x00441681 in execute_command_internal (command=0x628d88, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x628dc8) at 
execute_cmd.c:1013
#16 0x004bf557 in parse_and_execute (string=0x628408 "PATH=; set -o 
emacs; IFS= read -re < 53", from_file=0x5353fb "-c", 

Re: expand_prompt_string segmentation faults

[Chet Ramey]

On 9/16/18 2:55 AM, Eduardo Bustamante wrote:
> On Sat, Sep 15, 2018 at 8:15 PM Chet Ramey  wrote:
> (...)
>> Thanks for the report. They are the same problem.
> 
> Thanks Chet. Do you have a patch for the issue? I have a few other
> cases but I'm not sure if they're all due to the same cause.

Sure.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
*** ../bash-20180907/expr.c 2018-08-27 15:31:05.0 -0400
--- expr.c  2018-09-15 22:53:56.0 -0400
***
*** 326,329 
--- 326,332 
int aflags;
  
+   if (lhs == 0 || *lhs == 0)
+ return;   /* XXX */
+ 
  #if defined (ARRAY_VARS)
aflags = (assoc_expand_once && already_expanded) ? ASS_NOEXPAND : 0;
***
*** 1013,1017 
  else
  #endif
!   expr_bind_variable (tokstr, vincdec);
}
free (vincdec);
--- 1016,1021 
  else
  #endif
!   if (tokstr)
! expr_bind_variable (tokstr, vincdec);
}
free (vincdec);

Re: expand_prompt_string segmentation faults

[Chet Ramey]

On 9/16/18 4:12 AM, Eduardo A. Bustamante López wrote:
> Here's another interesting crash:
> 
> $ base64 < 53
> MBgFEBAQMBgFEBAfEA==
> 
> $ gdb -batch -ex=r -ex=bt --args ./bash --noprofile --norc -c 'PATH=; set -o 
> emacs; IFS= read -re' < 53

I can't reproduce this on Mac OS X or Fedora 28.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Bash-5.0-alpha available

[Chet Ramey]

On 9/16/18 3:00 AM, Eduardo Bustamante wrote:

> Out of curiosity, when / how will bash 5.x become a stable release?
> 
> The main reason I ask is that I'd like to ensure that there's
> reasonable fuzzing coverage of the parser, readline and other
> easy-to-fuzz areas of bash before that happens.

Bash-5.0 is about to go into beta test. While fuzzing has some
interesting results, I haven't really seen any security problems or
any serious bug that would cause me to hold up a release from the
fuzzing-uncovered issues I've seen so far.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: expand_prompt_string segmentation faults

[Eduardo A . Bustamante López]

Here's another interesting crash:

$ base64 < 53
MBgFEBAQMBgFEBAfEA==

$ gdb -batch -ex=r -ex=bt --args ./bash --noprofile --norc -c 'PATH=; set -o 
emacs; IFS= read -re' < 53
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0
[Detaching after fork from child process 30991]
/home/dualbus/src/gnu/bash/bash: vim: No such file or directory
0
[Detaching after fork from child process 30992]
/home/dualbus/src/gnu/bash/bash: vim: No such file or directory

malloc: unknown:0: assertion botched
malloc: 0x630808: allocated: last allocated from unknown:0
free: called with already freed block argument
Aborting...
Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x779a7535 in __GI_abort () at abort.c:79
#2  0x004597bb in programming_error (format=0x5557be "free: called with 
already freed block argument") at error.c:175
#3  0x005322c2 in xbotch (mem=0x630808, e=1, s=0x5557be "free: called 
with already freed block argument", file=0x0, line=0) at malloc.c:354
#4  0x0053155c in internal_free (mem=0x630808, file=0x0, line=0, 
flags=0) at malloc.c:931
#5  0x00531d4f in free (mem=0x630808) at malloc.c:1388
#6  0x004b5150 in xfree (string=0x630808) at xmalloc.c:150
#7  0x0051e94c in rl_do_undo () at undo.c:240
#8  0x0051eb23 in rl_revert_line (count=1, key=0) at undo.c:312
#9  0x004fa7f9 in readline_internal_teardown (eof=1) at readline.c:471
#10 0x004fa591 in readline_internal () at readline.c:672
#11 0x004fa440 in readline (prompt=0x55466f "") at readline.c:377
#12 0x004caa16 in edit_line (p=0x55466f "", itext=0x0) at 
./read.def:1104
#13 0x004c94ba in read_builtin (list=0x0) at ./read.def:563
#14 0x0044b599 in execute_builtin (builtin=0x4c8560 , 
words=0x628f88, flags=0, subshell=0) at execute_cmd.c:4677
#15 0x0044a96f in execute_builtin_or_function (words=0x628f88, 
builtin=0x4c8560 , var=0x0, redirects=0x0, fds_to_close=0x628ce8, 
flags=0) at execute_cmd.c:5185
#16 0x004437c9 in execute_simple_command (simple_command=0x628c08, 
pipe_in=-1, pipe_out=-1, async=0, fds_to_close=0x628ce8) at execute_cmd.c:4449
#17 0x004412ab in execute_command_internal (command=0x628bc8, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x628ce8) at 
execute_cmd.c:840
#18 0x00445208 in execute_connection (command=0x628d48, asynchronous=0, 
pipe_in=-1, pipe_out=-1, fds_to_close=0x628ce8) at execute_cmd.c:2689
#19 0x00441681 in execute_command_internal (command=0x628d48, 
asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x628ce8) at 
execute_cmd.c:1013
#20 0x004bf527 in parse_and_execute (string=0x628408 "PATH=; set -o 
emacs; IFS= read -re", from_file=0x5353fb "-c", flags=4) at evalstring.c:436
#21 0x00423845 in run_one_command (command=0x7fffea92 "PATH=; set 
-o emacs; IFS= read -re") at shell.c:1416
#22 0x00421920 in main (argc=5, argv=0x7fffe758, 
env=0x7fffe788) at shell.c:735


Bash compiled with:

`CC=clang CFLAGS='-O0 -ggdb' ./configure && make -j4' from `devel' 
(b52e30b8dd5bba5aed7b13040c9d4f7a0c1990cc) on Linux
(Ubuntu 18.04)

Re: Bash-5.0-alpha available

[Eduardo Bustamante]

On Wed, May 23, 2018 at 12:55 PM Chet Ramey  wrote:
>
> The first alpha release of bash-5.0 is now available with the URL
>
> ftp://ftp.cwru.edu/pub/bash/bash-5.0-alpha.tar.gz
>
> and from the bash-5.0-testing branch in the bash git repository
> (http://git.savannah.gnu.org/cgit/bash.git/log/?h=bash-5.0-testing).

Out of curiosity, when / how will bash 5.x become a stable release?

The main reason I ask is that I'd like to ensure that there's
reasonable fuzzing coverage of the parser, readline and other
easy-to-fuzz areas of bash before that happens.

Re: expand_prompt_string segmentation faults

[Eduardo Bustamante]

On Sat, Sep 15, 2018 at 8:15 PM Chet Ramey  wrote:
(...)
> Thanks for the report. They are the same problem.

Thanks Chet. Do you have a patch for the issue? I have a few other
cases but I'm not sure if they're all due to the same cause.