Re: Anomaly

[Chet Ramey]

On 9/20/16 11:04 PM, PePa wrote:
> When sourcing this script (version 1), it will print y after receiving
> an interrupt, but not in the 2 different versions (2 and 3).
> 
> # version 1
> echo x
> sleep 99
> echo y
> 
> # version 2
> echo x; sleep 99
> echo y
> 
> # version 3
> echo x
> sleep 99; echo y
> 
> Is this a bug or expected behaviour??

Well, it's definitely a difference between three simple commands and a
command list.  I think the right behavior is that exhibited by versions 2
and 3: the interrupt causes the execution of the sourced script to halt.
I'll take a look at it.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: "HISTSIZE=999999999" cause bash failure after lastest upgrade

[Chet Ramey]

On 9/25/16 3:51 PM, Sean Zha wrote:

> Bash Version: 4.4
> Patch Level: 0
> Release Status: release
> 
> Description:
>   I use a huge value for HISTSIZE (=9) to enable infinite
>   history items. The actural size of ~/.bash_history is only 4MB now.
>   Everything worked fine before the lastest upgrade. Now bash refuse
>   me to login due to memory allocation failure. After choosing
>   a smaller HISTSIZE, bash still eatup too much unnesssary memory.
> 
> Repeat-By:
>   open a workable terminal with small HISTSIZE setting,
>   $ echo HISTSIZE=9 > ~/test.rc
>   $ bash --rcfile ~/test.rc
> bash: xmalloc: cannot allocate 88 bytes (114688 bytes 
> allocated)

Since you've specified the desired history size, bash tries to allocate
enough entries to hold all of the entries.  The assumption is that this
will reduce the number of allocations/reallocations and the number of
times the history list has to be copied as it's reallocated.  I suppose
I should cap the maximum value for which that occurs instead of trusting
the supplied max number of entries.

Chet


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: bash history with mixed timestamps slow and broken

[Chet Ramey]

On 9/24/16 2:17 PM, Hubert Schmid wrote:
> Configuration Information [Automatically generated, do not change]:
> Machine: x86_64
> OS: linux-gnu
> Compiler: gcc
> Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' 
> -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu' 
> -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL 
> -DHAVE_CONFIG_H   -I.  -I../. -I.././include -I.././lib  -Wdate-time 
> -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/bash-31ueiH/bash-4.4=. 
> -fstack-protector-strong -Wformat -Werror=format-security -Wall 
> -Wno-parentheses -Wno-format-security
> uname output: Linux vivo 4.7.0-1-amd64 #1 SMP Debian 4.7.4-2 (2016-09-19) 
> x86_64 GNU/Linux
> Machine Type: x86_64-pc-linux-gnu
> 
> Bash Version: 4.4
> Patch Level: 0
> Release Status: release
> 
> Description:
>   If the history file (`.bash_history`) starts with a timestamp
>   (`HIST_TIMESTAMP_START`), and contains lines that have been written
>   without timestamps, then reading the history file is (a) very slow
>   because (b) consecutive lines without timestamps are merged into a
>   single history entry with quadratic complexity.
> 
>   Apparently, this problem didn't exist in the previous version (4.3).

One of the most frequently-requested features for the bash and readline
history implementations is a way to preserve multi-line commands across
shell sessions.  It's been the subject of numerous previous discussions
on bug-bash.

There hasn't been a good way to do that, since there isn't good support
for it in the traditional readline (flat) history file format.

I decided to implement a heuristic: if the history file starts with a
timestamp, and the application using the history library -- in this case,
bash -- has indicated that it's interested in writing timestamps to the
history file (which is off by default), the history file reading code
assumes that timestamps appear consistently in the history file and it can
use them as markers to delimit commands.

The appending behavior isn't really quadratic: the code simply keeps
reallocating the line and appending to it.  You can imagine how long it
takes to append a million commands to a single buffer.  You've managed
to identify the most degenerate case.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: [BUG] false positive: [ ! -o optname ]

[Chet Ramey]

On 9/25/16 5:57 PM, Martijn Dekker wrote:
> Op 25-09-16 om 22:40 schreef isabella parakiss:
>> On 9/25/16, Martijn Dekker  wrote:
>>> The '!' operator in the legacy test/[ builtin does not invert the result
>>> of the -o operator. Consequently the command
>>>
>>> [ ! -o noclobber ]
>>>
>>> amounts to a no-op, always returning exit status 0.
> [...]
>> [ ! -o noclobber ] means
>> is '!' a non empty string?  if not, is 'noclobber' a non empty string?
> 
> Wow. Yeah, I suppose that's one possible interpretation. I was going for
> the POSIX one:
> 
> | These primaries can be combined with the following operators:
> |
> | !  expression
> | True if expression is false. False if expression is true.
> Ref:
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html#tag_20_128

You need to look at the part of the Posix description where the behavior is
described as being based on the number of arguments.  In particular:

3 arguments:

If $2 is a binary primary, perform the binary test of $1 and $3.

-a and -o are considered binary primaries for the purposes of a three-
argument test.  This is spelled out pretty clearly in the bash
documentation.

Chet


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: [BUG] false positive: [ ! -o optname ]

[Martijn Dekker]

Op 25-09-16 om 22:57 schreef Martijn Dekker:
> And indeed your interpretation does not apply to something like
> "[ ! -e /tmp ]":
> 
> $ [ -e /tmp ]; echo $?
> 0
> $ [ ! -e /tmp ]; echo $?
> 1
> 
> However, the supposed synonym -a acts differently:
> 
> $ [ -a /tmp ]; echo $?
> 0
> $ [ ! -a /tmp ]; echo $?
> 0

Which also makes sense now that I think about it, since -a means "and"
so it tests for the non-emptiness of both strings.

Bug report withdrawn.

Sorry for the noise,

- M.

Re: Bash bind bug: infinite loop with memory leak in Readline

[Chet Ramey]

On 9/22/16 11:23 AM, Christian Klomp wrote:

> Limiting the nesting level sounds like a straightforward solution.
> Although I'm not sure whether exposing a configurable level is
> necessary. Do people actually nest their macros that deep that no
> reasonably safe upper bound can be set?

There's no reason to expose the level and allow it to be configured.
I just meant capping execution at some maximum nesting level.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: [BUG] false positive: [ ! -o optname ]

[Martijn Dekker]

Op 25-09-16 om 22:40 schreef isabella parakiss:
> On 9/25/16, Martijn Dekker  wrote:
>> The '!' operator in the legacy test/[ builtin does not invert the result
>> of the -o operator. Consequently the command
>>
>>  [ ! -o noclobber ]
>>
>> amounts to a no-op, always returning exit status 0.
[...]
> [ ! -o noclobber ] means
> is '!' a non empty string?  if not, is 'noclobber' a non empty string?

Wow. Yeah, I suppose that's one possible interpretation. I was going for
the POSIX one:

| These primaries can be combined with the following operators:
|
| !  expression
| True if expression is false. False if expression is true.
Ref:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html#tag_20_128

And indeed your interpretation does not apply to something like
"[ ! -e /tmp ]":

$ [ -e /tmp ]; echo $?
0
$ [ ! -e /tmp ]; echo $?
1

However, the supposed synonym -a acts differently:

$ [ -a /tmp ]; echo $?
0
$ [ ! -a /tmp ]; echo $?
0

Well, one more entry in the long list of arguments to avoid test/[...

Thanks,

- M.

"HISTSIZE=999999999" cause bash failure after lastest upgrade

[Sean Zha]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu' 
-DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL 
-DHAVE_CONFIG_H   -I.  -I../. -I.././include -I.././lib  -Wdate-time 
-D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/bash-31ueiH/bash-4.4=. 
-fstack-protector-strong -Wformat -Werror=format-security -Wall 
-Wno-parentheses -Wno-format-security
uname output: Linux kali 4.6.0-kali1-amd64 #1 SMP Debian 4.6.4-1kali1 
(2016-07-21) x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu

Bash Version: 4.4
Patch Level: 0
Release Status: release

Description:
I use a huge value for HISTSIZE (=9) to enable infinite
history items. The actural size of ~/.bash_history is only 4MB now.
Everything worked fine before the lastest upgrade. Now bash refuse
me to login due to memory allocation failure. After choosing
a smaller HISTSIZE, bash still eatup too much unnesssary memory.

Repeat-By:
open a workable terminal with small HISTSIZE setting,
$ echo HISTSIZE=9 > ~/test.rc
$ bash --rcfile ~/test.rc
  bash: xmalloc: cannot allocate 88 bytes (114688 bytes 
allocated)

Re: [BUG] false positive: [ ! -o optname ]

[isabella parakiss]

On 9/25/16, Martijn Dekker  wrote:
> The '!' operator in the legacy test/[ builtin does not invert the result
> of the -o operator. Consequently the command
>
>   [ ! -o noclobber ]
>
> amounts to a no-op, always returning exit status 0.
>
> Proof:
>
> $ set -o noclobber && [ -o noclobber ] && [ ! -o noclobber ] && echo bug
> bug
>
> Interestingly, mksh has this bug as well, though not the original pdksh.
>
> Thanks,
>
> - M.
>
>

[ ! -o noclobber ] means
is '!' a non empty string?  if not, is 'noclobber' a non empty string?


---
xoxo iza

[BUG] false positive: [ ! -o optname ]

[Martijn Dekker]

The '!' operator in the legacy test/[ builtin does not invert the result
of the -o operator. Consequently the command

[ ! -o noclobber ]

amounts to a no-op, always returning exit status 0.

Proof:

$ set -o noclobber && [ -o noclobber ] && [ ! -o noclobber ] && echo bug
bug

Interestingly, mksh has this bug as well, though not the original pdksh.

Thanks,

- M.

Re: Magnitude of Order "For Loop" performance deltas based on syntax change

[Chet Ramey]

On 9/24/16 8:29 AM, Christian Franke wrote:

> - Method-2 could be significantly speed up if the order of the array
> accesses is reversed:
> 
>   for (( i=0; i   if (( -(Pi[i] - Pi[i+1]) < min )); then
>   min=$((-(Pi[i]-Pi[i+1])))
>   fi
>   done
> 
> Result: ~3 seconds
> (using 'let' in the 'min' assignment failed with syntax error - Bug?)

I assume you mean that you used something like

let min=-(Pi[i]-Pi[i+1])

That's a syntax error because `(' is a shell operator, and it needs to be
quoted if you want it to appear in a word.  Since `let' is a builtin, not
part of the shell syntax, you have to make sure that all arguments to it
are valid shell words.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Magnitude of Order "For Loop" performance deltas based on syntax change

[Chet Ramey]

On 9/23/16 7:15 PM, Tom McCurdy wrote:

> Bash Version: 4.3
> Patch Level: 11
> Release Status: release
> 
> Description:
> Two nearly identical "For Loop" setups have large deltas in
> performance. See test program below. Was confirmed in IRC chat room by
> multiple users. Input is very noticeable with around 100,000 values.
> 
> The script was originally used to take an input file where the first line
> would determine how many int values were to follow. The remaining lines
> each had values that were sorted, and then the smallest difference between
> any two values were found.
> 
> Repeat-By:
> 
> Using script below comment/uncomment each method to test.
> On my machine
> Method-1 finishes in around 2 seconds
> Method-2 finishes in around 72 seconds
> 
> 
>  Code Below (also attached script and test input file) ---
> START=$(date +%s.%N)
> 
> read -r N && mapfile -t Pi < <(sort -n)
> #echo "Done"
> 
> min=1000
> 
> #Find the smallest difference between two numbers in sorted array with max
> array element = 1000
> 
> # For-Loop Method-1: is super fast
>  for (( i=0; i  do
>  current=${Pi[$i]}
>  next=${Pi[$i+1]}
>  diff=$(( next - current))
>  if [ $diff -lt $min ]
>  then
>  min=$diff
>  fi
>  done

Yes, bash array access is heavily optimized for (increasing) sequential
access, the most common case.  Bash arrays don't have a size limit, and
they're allowed to be very sparse, so the internal implementation is a
doubly-linked list. You can make this very fast by keeping a roving
pointer that indicates the last-accessed element.

Now, what bash does as a consequence of this roving pointer optimization is
bias towards references that increase monotonically.  Christian Franke hit
on the reason that the first construct is so much faster: the reference to
Pi[i+1] in the second for loop construct increments the last-referenced
element, and the immediate reference of Pi[i] forces bash into a search for
it from the beginning of the array.  This potentially happens twice: once
for the test, and once for the assignment.

There are a couple of other ways to speed this up, but that doesn't help
you right now, of course.  I'll be looking at additional optimizations
before bash-5.0 gets released.  You'd be better off sticking with method 1
for the time being.  The other details don't really make all that much
difference, though I will look at the `let' anomaly Christian noticed.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: [PATCH] fix printf %b without an argument

[Chet Ramey]

On 9/23/16 8:10 PM, isabella parakiss wrote:
> here's a patch that fixes this problem
> 
> $ printf "<%3s><%3b>"
> <   ><>

Thanks for the report.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/