Re: Bug with && in piped functions in minimal-config bash

[Martijn Dekker]

Op 04-11-16 om 21:41 schreef Martijn Dekker:
> $ fn() { false && echo bug || echo ok; }
> $ fn
> ok
> $ fn
> ok
> $ fn | cat
> ok
> $ fn | cat
> bug
> $ fn | cat
> bug
> $ (fn) | cat
> ok

Another datapoint: the result and output is exactly as above even if you
use if/then/else.

fn() { if false; then echo bug; else echo ok; fi; }

Thanks,

- M.

'kill -l' outputs SIG prefix if job control not compiled in

[Martijn Dekker]

Normal bash:

$ kill -l 141
PIPE

Bash compiled with --enable-minimal-config --enable-alias:

$ kill -l 141
SIGPIPE

That SIG prefix shouldn't be there. (Enabling POSIX mode makes no
difference to the output.)

A look at the source tells me that, for some reason, the POSIXly correct
output of 'kill -l' was made dependent on job control being compiled in.
I don't see any reason for that, so unless I'm missing something, the
attached patch should fix it.

Thanks,

- M.
diff --git a/builtins/common.c b/builtins/common.c
index 0eabe06..9e06c1e 100644
--- a/builtins/common.c
+++ b/builtins/common.c
@@ -782,13 +782,9 @@ display_signal_list (list, forcecols)
 	  list = list->next;
 	  continue;
 	}
-#if defined (JOB_CONTROL)
 	  /* POSIX.2 says that `kill -l signum' prints the signal name without
 	 the `SIG' prefix. */
 	  printf ("%s\n", (this_shell_builtin == kill_builtin) ? name + 3 : name);
-#else
-	  printf ("%s\n", name);
-#endif
 	}
   else
 	{

Bug with && in piped functions in minimal-config bash

[Martijn Dekker]

My development with Modernish just exposed the following bug on bash
4.4.0(1)-maint, bash-20161028 snapshot, compiled with
--enable-minimal-config --enable-alias:

$ fn() { false && echo bug || echo ok; }
$ fn
ok
$ fn
ok
$ fn | cat
ok
$ fn | cat
bug
$ fn | cat
bug
$ (fn) | cat
ok

So apparently this only occurs if the function is piped into another
command, without explicitly declaring it to run in a subshell, and only
from the second time on.

Thanks,

- M.

Re: memory leak in execute_simple_command when dofork is true

[Chet Ramey]

On 11/4/16 2:02 AM, Eduardo A. Bustamante López wrote:
> Actually, there are more leaks in execute_cmd.c, I'm just going to list them 
> here.

Same.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: memory leak in execute_simple_command when dofork is true

[Chet Ramey]

On 11/4/16 12:35 AM, Eduardo A. Bustamante López wrote:
> I ran the configure script with the following arguments:
> 
> hp% ./configure CC=gcc-6 CFLAGS='-Wall -g -ggdb -O0 -fsanitize=address' 
> LDFLAGS=-fsanitize=address --without-bash-malloc
> 
> Which enables the LeakSanitizer 
> (https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer).
> 
> It detected the following memory leak:

It `leaks' in the child process.  The parent eventually stores that memory
in the jobs list and frees it when the job is deleted.  The leaks don't
accumulate; they are individual to each child process.  It's never been a
big deal in practice.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: bind -m vi-insert '"jj": "\e"' problem in 4.4

[Chet Ramey]

On 11/4/16 6:36 AM, Clark Wang wrote:
> For example, in vi-insert mode I input following in the command line:
> 
> # foo abc
> 
> The cursor is right after 'c'. Now if I press ESC then it'll go to
> vi-command mode and the cursor will move to under 'c'. But if I press jj
> the cursor would be still after 'c' and if then I press h the cursor will
> skip 'c' and move to under 'b' . In 4.3, ESC and jj works the same so there
> must be something changed in 4.4. Please take a look.

What changed is not really what you would think.  The key sequence you
have typed results in ESC, which is ambiguous: it's both a key binding
and a key prefix.  Readline is waiting until you type an additional
character to resolve the ambiguity.  For instance, if you were to type
`0' after this, readline would move the cursor to column 0 in vi
command mode.

There is special code in readline that uses a timeout to disambiguate for
the special case of ESC typed in vi insert mode.  I will have to look
and see whether or not it makes sense to use that same timeout code for
a keyboard macro that resolves to ESC.

Chet


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Severe IFS glob bug needs patch [was: Setting IFS='*' causes bash to freeze on tab completion]

[Chet Ramey]

On 11/4/16 3:05 AM, Martijn Dekker wrote:
> Op 04-11-16 om 01:27 schreef Eduardo Bustamante:
>> I agree with everything, except calling it severe. This is
>> self-inflicted harm, and easy to work around
> 
> It is not self-inflicted. The behaviour is a bug in bash. The user did
> nothing to cause that bug.

I don't think anyone is disputing that it's a bug in bash.  I agree with
Eduardo's point, though, which is that there doesn't appear to be anything
special enough about this particular problem to warrant releasing patches
for versions of bash that were released seven years ago.

So it's a bug, it's been present for at least 17 years, it's extremely
rare in practice, and I will fix it.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: [Help-bash] How to make vi-insert mode's ctrl-w work in the 4.3 way?

[Clark Wang]

On Fri, Nov 4, 2016 at 10:32 PM, Chet Ramey  wrote:

> On 11/4/16 6:30 AM, Clark Wang wrote:
> > For example, if I have inputted the following after the prompt:
> >
> > # foo "abc"
> >
> > In bash 4.3's vi-insert mode, when I press ctrl-w it'll delete the whole
> > "abc" (including quotes). But with 4.4 I have to press ctrl-w for 3 times
> > (one for the right " char, one for abc and one for the left " char).
>
> This was changed due to a bug report about readline not being Posix-
> conformat with its ^W binding in vi insert mode.  Posix specifies that
> word boundaries include whitespace and punctuation.  Apparently vi is
> the same, but I'm not enough of a vi user to say.
>

> The old binding (unix-word-rubout) is still there, but to avoid it being
> overwritten, you need to set the readline variable `bind-tty-special-chars'
> to `off'.  (Since ^W is your default stty werase character, readline will
> bind it to its vi-mode equivalent when that variable is enabled.)
>

This works great. Thanks Chet.

>
> You would have noticed the ^W binding being overwritten if you had done
> a `bind -m vi-insert -p' after one of your key binding commands, but I
> suppose there isn't any real reason to do that.
>
> Chet
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~
> chet/
>

Re: [Help-bash] How to make vi-insert mode's ctrl-w work in the 4.3 way?

[Greg Wooledge]

On Fri, Nov 04, 2016 at 10:32:17AM -0400, Chet Ramey wrote:
> On 11/4/16 6:30 AM, Clark Wang wrote:
> > For example, if I have inputted the following after the prompt:
> > 
> > # foo "abc"
> > 
> > In bash 4.3's vi-insert mode, when I press ctrl-w it'll delete the whole
> > "abc" (including quotes). But with 4.4 I have to press ctrl-w for 3 times
> > (one for the right " char, one for abc and one for the left " char).
> 
> This was changed due to a bug report about readline not being Posix-
> conformat with its ^W binding in vi insert mode.  Posix specifies that
> word boundaries include whitespace and punctuation.  Apparently vi is
> the same, but I'm not enough of a vi user to say.

Both vim (versions 6.1 and 7.4) and /usr/bin/vi (on HP-UX 10.20) act
this way.

Re: [Help-bash] How to make vi-insert mode's ctrl-w work in the 4.3 way?

[Chet Ramey]

On 11/4/16 6:30 AM, Clark Wang wrote:
> For example, if I have inputted the following after the prompt:
> 
> # foo "abc"
> 
> In bash 4.3's vi-insert mode, when I press ctrl-w it'll delete the whole
> "abc" (including quotes). But with 4.4 I have to press ctrl-w for 3 times
> (one for the right " char, one for abc and one for the left " char).

This was changed due to a bug report about readline not being Posix-
conformat with its ^W binding in vi insert mode.  Posix specifies that
word boundaries include whitespace and punctuation.  Apparently vi is
the same, but I'm not enough of a vi user to say.

The old binding (unix-word-rubout) is still there, but to avoid it being
overwritten, you need to set the readline variable `bind-tty-special-chars'
to `off'.  (Since ^W is your default stty werase character, readline will
bind it to its vi-mode equivalent when that variable is enabled.)

You would have noticed the ^W binding being overwritten if you had done
a `bind -m vi-insert -p' after one of your key binding commands, but I
suppose there isn't any real reason to do that.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: devel branch fails to build when using bash malloc due to missing errno.h include

[Chet Ramey]

On 11/4/16 12:17 AM, Eduardo A. Bustamante López wrote:
> I'm running configure with the following flags:
> 
> |./configure CC=gcc-6
> 
> The error I get while building:

Thanks for the report.  I'm not working on a system that uses the bash
malloc right now.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

bind -m vi-insert '"jj": "\e"' problem in 4.4

[Clark Wang]

For example, in vi-insert mode I input following in the command line:

# foo abc

The cursor is right after 'c'. Now if I press ESC then it'll go to
vi-command mode and the cursor will move to under 'c'. But if I press jj
the cursor would be still after 'c' and if then I press h the cursor will
skip 'c' and move to under 'b' . In 4.3, ESC and jj works the same so there
must be something changed in 4.4. Please take a look.

-clark

[bug] [[ $'\Ux' = $'\Ux' ]] returns false for some values of x in some locales

[Stephane Chazelas]

(reproduced with bash 4.3 or 4.4 on Debian unstable and Ubuntu 16.04).

perl -le "printf q([[ $'\U%X' = $'\U%X' ]] || echo %06X: $'\U%X').\"\n\",
  \$_,\$_,\$_,\$_ for (1..0xd7FF, 0xE000..0x10)" |
  LC_ALL=zh_HK.big5hkscs bash | LC_ALL=C sed -n l

Where the perl command outputs:

[[ $'\U1' = $'\U1' ]] || echo 01: $'\U1'
[[ $'\U2' = $'\U2' ]] || echo 02: $'\U2'
[[ $'\U3' = $'\U3' ]] || echo 03: $'\U3'
[[ $'\U4' = $'\U4' ]] || echo 04: $'\U4'


for all valid (albeit not necessarily assigned, let alone available in any
charset) Unicode codepoints.

Gives:

CA: $
CB: \\u00CB$
EA: $
EB: \\u00EB$
00011A: \210\\$
0003B1: \243\\$
000436: \310\\$
003075: \307\\$
003618: \234\\$
003661: \215\\$
0044C0: \226\\$
004A35: \232\\$
004AA4: \207\\$
004E48: \244\\$
004F62: \312\\$
004FDE: \253\\$
005045: \324\\$
00509C: \330\\$
00515D: \242\\$
00529F: \245\\$
005412: \246\\$
00542D: \247\\$
0056ED: \373\\$
00577C: \251\\$
0057A5: \316\\$
00587F: \341\\$
0058A6: \274\\$
0058F0: \211\\$
005A09: \256\\$
005A16: \321\\$
005A2B: \230\\$
005AF9: \345\\$
005B1E: \351\\$
005B40: \304\\$
005C10: \311\\$
005CA4: \314\\$
005D24: \261\\$
005E4B: \335\\$
005EC4: \264\\$
0060DD: \325\\$
006127: \267\\$
0063CA: \331\\$
0064FA: \302\\$
00669D: \272\\$
0067AF: \254\\$
0067E6: \317\\$
0069D9: \342\\$
006A9D: \375\\$
006B7F: \252\\$
006C7B: \313\\$
006C94: \250\\$
006D82: \322\\$
006DDA: \262\\$
006EDC: \336\\$
006F7F: \346\\$
007019: \362\\$
007035: \364\\$
00712E: \332\\$
0071E1: \355\\$
00727E: \326\\$
0072D6: \315\\$
007366: \352\\$
0073E2: \227\\$
0073EE: \257\\$
007435: \265\\$
00749E: \277\\$
0075B1: \236\\$
007667: \240\\$
007912: \360\\$
007A1E: \270\\$
007A40: \275\\$
007B0B: \216\\$
007BA4: \343\\$
007CED: \231\\$
007D85: \337\\$
007E37: \301\\$
007F61: \323\\$
0080D0: \320\\$
0080EC: \213\\$
00812A: \223\\$
0082D2: \255\\$
00833B: \333\\$
00838D: \327\\$
0084CB: \273\\$
00850C: \347\\$
00855A: \217\\$
00878F: \353\\$
0087B0: \356\\$
008A31: \263\\$
008C79: \260\\$
008D15: \367\\$
008D68: \340\\$
008DDA: \266\\$
008E0A: \344\\$
008E7E: \212\\$
008EA1: \306\\$
009103: \334\\$
009140: \363\\$
009145: \366\\$
009186: \350\\$
00923E: \271\\$
0093AA: \361\\$
0095B1: \276\\$
0097B8: \233\\$
009910: \300\\$
009924: \354\\$
0099F9: \357\\$
009A31: \365\\$
009ACF: \305\\$
009AE2: \221\\$
009AFF: \237\\$
009C4B: \370\\$
009C6D: \371\\$
009EE0: \303\\$
00FE4F: \241\\$
0205EB: \224\\$
020C3A: \376\\$
023600: \372\\$
0265AD: \225\\$
026C21: \222\\$
0270F8: \374\\$
02870F: \214\\$
02913C: \235\\$
02A014: \220\\$

$ LC_ALL=zh_HK.big5hkscs locale charmap
BIG5-HKSCS

Most of the problematic characters are the ones ending in 0x5c
(which happens to be backslash in ASCII (or in BIG5-HKSCS when
standing alone).

$ LC_ALL=zh_HK.big5hkscs bash -xc "[[ $'\u3b1' = $'\u3b1' ]]" 2>&1 | sed -n l
+ [[ \243\\ = \\\243\\ ]]$

Note that

bash -xc $'[[ \u3b1 = \u3b1 ]]'


also returns false in those locales.

There are similar problems for locales using BIG5, GB18030 or GBK charsets.

Same with "case" or

a=$'\u3b1'; [[ $a = $a ]]
or
[[ "$a" = "$a" ]]
or ${a#"$a"}

[ "$a" = "$a" ] is fine.

The CA and EA ones do look a lot like a bug in the glibc's
locale definition or gconv module (and the CB, EB ones are a
consequence of it)

$ LC_ALL=zh_HK.big5hkscs bash -xc "[[ $'\uca' = $'\uca' ]]" 2>&1 | sed -n l
+ [[ '' = \\\210f ]]$

A $'\uanything' following a $'\uca' always yields 0x88 0x66
(which happens to be the BIG5-HKSCS encoding of U+00CA) in
bash, zsh and ksh93 (though only for anything >= 0x80 in bash).

Those locales are problematic and should be avoided in general.
The problem  is that they are often *available*, so all those
corner cases caused by the fact that some characters contain
ASCII ones can be exploited (think of sudo or many sshd
deployments letting LC_* variables through for instance).

-- 
Stephane

Re: Severe IFS glob bug needs patch [was: Setting IFS='*' causes bash to freeze on tab completion]

[Eduardo Bustamante]

If you want to split with read, use a tempenv IFS:

dualbus@hp:~/local/src/gnu/bash$ bash -c 'IFS="?" read a b c <<<
"hello?world?xyz"; echo "$a"'
hello

If you want to split using word splitting:

dualbus@hp:~/local/src/gnu/bash$ bash -c 'v="hello?world?xyz";
IFS="?"; set -- $v; unset IFS; echo "$1"'
hello

If you want to use the $* parameter:

  dualbus@hp:~/local/src/gnu/bash$ bash -c 'set -- abc def ghi;
IFS="?"; echo "$*"; unset IFS'
  abc?def?ghi

I didn't say there's no bug. There is. And when I said,
"self-inflicted harm", I meant that leaving as IFS='][?*' is not
really needed. If you need to do some word splitting, set the IFS, and
then restore its original value or unset it.

Perhaps I'm missing something here, but I think for the relevant use
cases of IFS, there are sane work arounds.

What other use case would require of setting IFS to such value permanently?

Re: Severe IFS glob bug needs patch [was: Setting IFS='*' causes bash to freeze on tab completion]

[Martijn Dekker]

Op 04-11-16 om 01:27 schreef Eduardo Bustamante:
> I agree with everything, except calling it severe. This is
> self-inflicted harm, and easy to work around

It is not self-inflicted. The behaviour is a bug in bash. The user did
nothing to cause that bug.

Even if you think it's somehow reasonable to expect bash users to be
aware of this bug while using or coding for bash (though the bug has not
been publicised anywhere), bash is supposed to be a POSIX shell, and
POSIX is supposed to be a standard. You cannot assume that the code bash
executes is written with bash bugs in mind.

It is also not "easy" to work around. If your program for any reason
needs to split strings on any of the characters ?, *, [ and ], and uses
field splitting to achieve this (which in POSIX is the canonical
method), you will now need to program a workaround with something like
sed or parameter substitutions. Unless, of course, the easy workaround
is to simply use another shell.

Another big issue is that third-party library functions need to work for
arbitrary values of IFS. This bug makes that impossible for bash.

For instance, if you recall, the original bug reporter wanted to split
some string by "*" on an interactive shell. This caused a freeze upon
using auto-completion because the bash-completion package broke as a
result. Chet initially blamed it on bash-completion, but in fact the
fault was with bash itself. The resulting breakage was clearly severe.

- M.

Re: memory leak in execute_simple_command when dofork is true

[Eduardo A . Bustamante López]

Actually, there are more leaks in execute_cmd.c, I'm just going to list them 
here.

---
hp% ./bash -c '(:)'

=
==7854==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 6 byte(s) in 1 object(s) allocated from:
#0 0x7fb0c2580d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0x562cb665d6cb in xmalloc /home/dualbus/local/src/gnu/bash/xmalloc.c:112
#2 0x562cb658ab1c in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:610
#3 0x562cb666e900 in parse_and_execute 
/home/dualbus/local/src/gnu/bash/builtins/evalstring.c:443
#4 0x562cb655797f in run_one_command 
/home/dualbus/local/src/gnu/bash/shell.c:1399
#5 0x562cb6555eaa in main /home/dualbus/local/src/gnu/bash/shell.c:724
#6 0x7fb0c1d132b0 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

SUMMARY: AddressSanitizer: 6 byte(s) leaked in 1 allocation(s).

---
hp% ./bash -c 'coproc :; sleep 1'

=
==8716==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x7fc17740ad28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0x55d4b48826cb in xmalloc /home/dualbus/local/src/gnu/bash/xmalloc.c:112
#2 0x55d4b47b72c6 in execute_coproc 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:2301
#3 0x55d4b47af935 in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:593
#4 0x55d4b47af0fb in execute_command 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:405
#5 0x55d4b47b8f54 in execute_connection 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:2609
#6 0x55d4b47b16e9 in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:971
#7 0x55d4b4893900 in parse_and_execute 
/home/dualbus/local/src/gnu/bash/builtins/evalstring.c:443
#8 0x55d4b477c97f in run_one_command 
/home/dualbus/local/src/gnu/bash/shell.c:1399
#9 0x55d4b477aeaa in main /home/dualbus/local/src/gnu/bash/shell.c:724
#10 0x7fc176b9d2b0 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

SUMMARY: AddressSanitizer: 16 byte(s) leaked in 1 allocation(s).

---
hp% ./bash -c '::; :'   
./bash: ::: command not found

=
==10176==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 3 byte(s) in 1 object(s) allocated from:
#0 0x7f003afe5d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0x561a310216cb in xmalloc /home/dualbus/local/src/gnu/bash/xmalloc.c:112
#2 0x561a30f641b3 in execute_disk_command 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:5194
#3 0x561a30f61b01 in execute_simple_command 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:4403
#4 0x561a30f4f914 in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:802
#5 0x561a30f4e0fb in execute_command 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:405
#6 0x561a30f57f54 in execute_connection 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:2609
#7 0x561a30f506e9 in execute_command_internal 
/home/dualbus/local/src/gnu/bash/execute_cmd.c:971
#8 0x561a31032900 in parse_and_execute 
/home/dualbus/local/src/gnu/bash/builtins/evalstring.c:443
#9 0x561a30f1b97f in run_one_command 
/home/dualbus/local/src/gnu/bash/shell.c:1399
#10 0x561a30f19eaa in main /home/dualbus/local/src/gnu/bash/shell.c:724
#11 0x7f003a7782b0 in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

SUMMARY: AddressSanitizer: 3 byte(s) leaked in 1 allocation(s).

-- 
Eduardo Bustamante
https://dualbus.me/