Re: Command substitution with null bytes generates warning

[Linda Walsh]

Greg Wooledge wrote:

Bash has only three choices that I can think of: it can silently drop the
NUL bytes  behavior), it can drop ALL of the bytes and return an
error, or it can drop the NUL bytes with a warning (4.4 behavior).
  

now who is being disingenuous?  It was silent not just in 4.3, but in all
previous versions.   There is far more weight behind the being silent (the
unix design philosophy) than being needlessly noisy. 


 Calling that an "illegally modified input" is disingenuous.


Claiming it needed a warning for something that is "impossible" is
equally ridiculous.  Why not warn that bash can't do your laundry?
If bash is to warn about everything it ***can't*** do, it would be
unusable.  So why make a special case to warn about the "impossible"?

Re: Command substitution with null bytes generates warning

[Linda Walsh]

Eric Blake wrote:

On 09/19/2016 11:58 AM, Greg Wooledge wrote:
  

wooledg@wooledg:~$ x=$(< /proc/$$/cmdline)
bash-4.4: warning: command substitution: ignored null byte in input
wooledg@wooledg:~$ x=$(< /proc/$$/cmdline 2>/dev/null)
wooledg@wooledg:~$ 



Or:

$ x=$(< /proc/$$/cmdline tr -d \\0)
  

Admittedly better, as it doesn't suppress valid messages, but still
requires tracking down source code mods on multiple machines and making
changes for something that wasn't a problem.  Why should people who
didn't expect them to make any difference (the valid behavior) be penalized
because others didn't know.

How about, w/r/t the new warning -- I complain because the null bytes
are missing after bash knowingly detected them and illegally modified
the input.  Putting out a warning about null bytes, doesn't mean it's
"ok" to drop them.  Now it's just compounded with an anti-unix (silence
is golden) warning message.

^^ see, you can still complain about the null bytes regardless of the
warning.  The warning is just adding salt to the wound -- like "nyaa nyaa,
I know about the null bytes, and am still refusing to return them to you!!!"

What's the purpose in making emit warnings about everything
that doesn't work?  Could just fix it and use the C++ definition
of Strings that have a builtin count and can handle nulls in the middle
of the data... ;^/

Re: change in OS defaults (was: bug in [ -f file ] test)

[Linda Walsh]

� wrote:
Yes, the can be overridden in /etc/sysctl.d. But I am going to modify 
the troublesome scripts in the $50k software and leave the security of 
the OS intact. Thanks.





Actually, setting them to "1", is not "leaving the OS security intact" --
it's adding new, non-default security in Redhat and Suse distro's. 
The default OS setting is 0 for those as it makes the way hard &

softlinks incompatible with past behavior.  I ran into the
new settings when upgrading from SUSE 13.1 -> 13.2.

Many new "security pollution" problems are being introduced due to
people forgetting about how groups can be used to control access.
Various programs (ssh, sendmail, sudo, lilo,  to name a few),
complain or fail, if you use groups to control access and try
allow access by group. 


The new settings broke a use case where where I use permissions,
different UID's and hardlinks to save considerable space.

Case in point.  I control access (*to myself*) in my linux source
trees.  Under my linux-source dir, I have "unmodified" dirs for each
version of linux I've been working with.  When a new version comes
out, I create a 2nd copy using 'cp -al', and, as root, apply the patch
necessary to create the new version.  I.e. Currently I have source trees
going back to 3.10.  Though I think I started with a new copy with
linux 4.0.

To upgrade to 4.0.2 or 4.0.4, I apply a patch to the 4.0 source tree
in a new directory.

This saves on disk space, as well as propagates local patches for my
build environment.  Looking at "virgin" sources with "du" for some
4.0 derived source dirs:


 du -sh linux-4.0{,.[0-9]}

638M  linux-4.0
2.6M  linux-4.0.0
14M linux-4.0.2
19M linux-4.0.4

Linux-4.0.2 and linux-4.0.4 share about 620M with the
linux-4.0 sources.  The linux-4.0.0 is an exact copy of
the 4.0 tree with the "2.6M" being space used for the
duplication of the directories which "cp -al" creates
as new copies, owned by the user (so they can write in them!).

Between main-releases:


 du -sh linux-4.[034]

638M  linux-4.0
220M  linux-4.3
231M  linux-4.4

About 220-240M changes.

Looking at 4.4 derived trees, we see the stand-alone size of
the linux-4.4 dir:


 du -sh linux-4.4{,.[13]}

699M  linux-4.4
6.5M  linux-4.4.1
17M linux-4.4.3

I.e. linux-4.4, by itself is 699M.  But it shares over 400M with
linux-4.0.

To ensure I don't accidently change my "reference" sources,
the dirs are writable by the owner or the group, with files only
being writable by the owner:


 llg -d linux-4.[034]

drwxrwsr-x 23 root root 4096 Aug 12  2015 linux-4.0/
drwxrwsr-x 24 root root 4096 Feb  1 21:42 linux-4.3/
drwxrwsr-x 24 root root 4096 Feb  1 21:52 linux-4.4/

To actually work & build, as non-root user, I do a "cp -al"
from the reference-dirs into work dirs, so du looks like:


 du -sh linux-4.[034] ish-4[034]*

638M  linux-4.0
396M  linux-4.3
231M  linux-4.4
3.0G  ish-400
3.5G  ish-400-nlnk
3.0G  ish-402
3.0G  ish-404
3.3M  ish-43
3.2G  ish-433
476M  ish-441
440M  ish-443

Note -- the "GB"-size dirs have object files in them.  Ish-441 & Ish-443
likely had a make-clean done in them which leaves stuff.  When I
copy them as a local user, the mode-bits get copied over to the
new tree, but where files are owned by root (if unchanged) or by
me (if changed).  Pruned ls output:


 llg ish-443

total 50876
-rw-r--r--  99 root root18693 Jan 19  2014 COPYING
-rw-r--r--   7 root root97181 Feb  1 21:41 CREDITS
drwxrwsr-x 112 lin  root 8192 Feb  1 21:52 Documentation/
-rw-r--r--  24 root root 2622 Sep 10  2015 Kbuild
-rw-rw-r--   1 lin  root  2943284 Feb 25 22:53 System.map
drwxrwsr-x   2 lin  root  178 Feb 25 17:51 usr/
-rwxrwxr-x   1 lin  root 24768740 Feb 25 22:53 vmlinux*


My script to download patches and create new 'linux' source
trees and then create a "work dir" for me, started failing
when I moved from SuSE 13.1->13.2, since SuSE copied Redhat's
new settings for the links.

The new setting prohibited me creating a links to root-owned
files -- even though I could read them, and was only creating
links in my user-owned directory.


I.e. the new protected links (at least for hard),
protect me from using the normal unix mode-bits and groups
to create secure, read-only copies of unmodified files in
my source tree. 


The links (in the case of hardlinks) are not protected,
but the ability to create links to a common source was restricted
to root -- which would defeat my userid's ability to create a copy
of root's source tree that I could work in without duplicating
all the files. 


The difference size for ~55 versions of the kernel is
==>> 6.2G linked, and 74G not linked.<<==

The hard-link protection doesn't give any added protection, but
does prevent getting easy protection using permissions & UID's
to disable accidental writes to "original sources".

Needless to say, on my systems, I don't modify those
settings from the OS's default.



On Thu, 2016-07-28 at 15:06 -0500, John McKown wrote:
On Thu, Jul 28, 2016 at 2:49 PM, 

Re: Mulit-line aliases and PROMPT_COMMAND

[Linda Walsh]

Dan Douglas wrote:

 The 4.4 changes will make aliases even more interesting.
  

---
Oh?   Why is that?  I.e. what's happening to aliases that
will make them more "interesting"? 


BTW, are you using "interesting" in the same way as the
saying "May you live in interesting times"?

Re: IGNOREEOF and POSIXLY_CORRECT don't ignore invisible vars

[Linda Walsh]

Grisha Levit wrote:
The manual references these values being set, not just declared, so 
maybe should check for invisible_p?

---
Why?  I.e. what were you wanting to happen?


Noticeable for example because this works:

$ set +o posix; f() { local POSIXLY_CORRECT=; shopt -p -o posix; }; f; 
shopt -p -o posix

set -o posix
set +o posix

---
Yeah, you've set P_C to a null string.


But the opposite does not:

$ set -o posix; f() { local POSIXLY_CORRECT; shopt -p -o posix; }; f; 
shopt -p -o posix

set -o posix
set -o posix

---
The "opposite"?  You aren't really showing the opposite.  I.e.
wouldn't the opposite be (alias used for printing P_C & output of shopt)

 alias shopos='printf "PC=%s; " $(if [[ -v POSIXLY_CORRECT ]]; then 

printf "  set"; else printf "unset"; fi) ;shopt -p -o posix'

 set -o posix; f() { local POSIXLY_CORRECT; unset POSIXLY_CORRECT; 
printf "   in f: "; shopos; }; printf "bfore f: "; shopos; f ; printf 
"after f: "; shopos

bfore f: PC=set; set -o posix
  in f: PC=unset; set -o posix
after f: PC=set; set -o posix

(and your first example:)

 set +o posix; f() { local POSIXLY_CORRECT=;  printf "   in f: "; 

shopos; }; printf "bfore f: "; shopos; f ; printf "after f: "; shopos
bfore f: PC=unset; set +o posix
  in f: PC=set; set -o posix
after f: PC=unset; set +o posix

Re: [minor] "precision" of $SECONDS

[Linda Walsh]

Stephane Chazelas wrote:

2016-02-25 03:03:41 -0800, Linda Walsh:

  

Stephane Chazelas wrote:


$ time bash -c 'while ((SECONDS < 1)); do :; done'
bash -c 'while ((SECONDS < 1)); do :; done'  0.39s user 0.00s system 99% cpu 
0.387 total
  


   Sorry I took "cpu xxx total" to be the total cpu time.  Silly me. 
(I do believe you, just the display format could be more clear).





  

Try:

TIMEFORMAT='%2Rsec %2Uusr %2Ssys (%P%% cpu)'



That would be for bash. In anycase, bash does already include
the elapsed time in its default time output like zsh.
  

---
but not as clearly, IMO... ;-)


But the problem here is not about the time keyword, but about the
$SECONDS variable.
  

---
   I realize that.

[...]
  

   With linux, one can read /proc/uptime to 100th's of a sec, or
use date to get more digits.  A middle of the road I used for
trace timing was something like:

function __age { declare ns=$(date +"%N"); declare -i
ms=${ns##+(0)}/100;
 printf "%4d.%03d\n" $SECONDS $ms
}


[...]

I'm not sure how that gives you the time since startup.
  

---
   The time since the bash script startup.
I was guessing that SECONDS recorded the integer value of 'seconds'
at the start of the script.  Thus it shows later times as the
later recorded time in seconds - the original time in seconds -- or
at least that would match current behavior -- initial "seconds"
param from gettime, (ignoring the nano or centi secs, depending on call).

going from an integer value of the time
at start

Currently, if bash is started at

00:00:00.7
  


   Well, ok, but time calls usually give back
seconds from 1970, with some giving back another param for
centi, or more modern calls giving back 2nd parm for nano's.

Theoretically, bash could never start when seconds=0, unless
it was started in 1970...  But I'm guessing you are using clock
time, whereas I was using the time from the start of the script.

I.e. @ start of script, SECONDS gets the # secs since 1970,
and (if done at the same time, the date call for nanosecs) would
be the #nanos above that number of secs.



After 0.4 seconds (at 00:00:01.1), $SECONDS will be 1 (the "bug"
I'm raising here). "ms" will be 100, so you'll print 1.100
instead of 0.600. And with my suggested fix, you'd print 0.100.
  

---
At bash startup, I'll see 0 seconds, and 7000,000 nanosecs.
after .4 secs, Ill see 1 sec & 1000,000 nanosecs.

So diff would be 1.1=0.7 = .4secs = correct answer, no?




Note that all of zsh, ksh93 and mksh have builtin support to get
elapsed time information with subsecond granularity.
  

That's not a POSIX requirement, and bash is hardly an ideal tool
to need or rely on sub-second granularity, especially since
it doesn't process signals in real-time, but only upon
a keypress in readline.

Re: [minor] "precision" of $SECONDS

[Linda Walsh]

Stephane Chazelas wrote:

$ time bash -c 'while ((SECONDS < 1)); do :; done'
bash -c 'while ((SECONDS < 1)); do :; done'  0.39s user 0.00s system 99% cpu 
0.387 total

That can take in between 0 and 1 seconds. Or in other words,
$SECONDS becomes 1 in between 0 and 1 second after the shell was
started.

The format you are using to display output of 'time' doesn't show
real time -- only CPU seconds.

Try:

TIMEFORMAT='%2Rsec %2Uusr %2Ssys (%P%% cpu)'

It shows the real, "clock" time, as well as
the standard % formula of cpu-secs/real-secs -- which can give
up to #Cores x 100"%" as value for %cpu  stretches the standard
def of "percent", but is at least more honest than grouping
all cpu's together as "1 cpu", and showing 100% usage of 1 core
on a 12-core machine as "8.3%".



mksh and zsh behave like bash (I'll raise the issue there as
well).

With zsh (like in ksh93), one can do "typeset -F SECONDS" to
make $SECONDS floating point, which can be used as a work around
of the "issue".
  


   With linux, one can read /proc/uptime to 100th's of a sec, or
use date to get more digits.  A middle of the road I used for
trace timing was something like:

function __age { declare ns=$(date +"%N"); declare -i 
ms=${ns##+(0)}/100;

 printf "%4d.%03d\n" $SECONDS $ms
}
Then I add that in my PS4 prompt:
## for *relative* script exec speed, only!
##   (slows down normal speed significantly):
#export 
PS4='>[$(__age)]${BASH_SOURCE:+${BASH_SOURCE[0]/$HOME/\~}}#${LINENO}${FUNCNAME:+(${FUNCNAME[0]})}> 
'


On cygwin, calling 'date' was a deal breaker, so I just used /proc/uptime
to get the relative time down to centiseconds. 


As you can see, I wanted the times
relative to the start of a given script, thus used SECONDS for that.

Given the overhead of calling 'date', the times printed are easily 
20-50x slower

than w/o the time -- BUT, as the comment says above, it's good for
determining relative differences -- not only times for each command, but
also algorithmic differences, as the overhead is relatively constant.

Re: Q: what is a fast way to see if an 'item' is present in an array?

[Linda Walsh]

Greg Wooledge wrote:

On Mon, Feb 15, 2016 at 07:59:21PM -0800, Linda Walsh wrote:
  

Obviously, If I could precook the array into
a hash, I'd think it would be faster... but
the "cooking part", I think, is the high
overhead part.



Redesign the entire script so that you use the associative array (hash)
from the beginning instead of, or in addition to, storing your information
in a "list" (indexed array).
  

It would also take throwing away previous cfg-file compat as well
as changing the linux kernel interface.  Somehow I don't think that
is going to be easier:


 (cd /sys/class/net/br0/brif/; echo *)

eth0 eth5

 grep  ^BRIDGE_PORTS "$CFG"

BRIDGE_PORTS='eth0 eth5'

---
In my case it's only 2, but other interfaces... still not that long
*usually*,

 (cd $PWD/holders; echo *)

dm-0 dm-1 dm-12 dm-2 dm-3 dm-4 dm-5 dm-6 dm-7 dm-8 dm-9

But I tend to think about worst case behaviors and at least
try to ask around if anyone has anything better... but in this case,
the impact is minimal.

Though when designing 'anew', I try to make things data/table
driven as much as possible for things that fit into that paradigm.

Thanks!
L.

Re: bug adding K,V pairs to existing hash with HASH+=([K]=V)

[Linda Walsh]

Dan Douglas wrote:

Ah so `arr+=([a]=x [b]=y)` will no longer be the same as `arr+=([a]+=x
[b]+=y)`? I never liked that for associative arrays because the only
workaround was to do multiple arr[key]= assignments to update or add
more than one element at a time.

My thinking was that this is due to bash's unique auto-incrementing of
indexed array keys. Since in ksh you're not allowed to specify an
index for one element of a compound assignment without specifying it
for all elements, bash has some additional things to consider.

 ~ $ bash-4.2 -c 'typeset -ia a=({0..5}); a+=([0]+=1 {2..4}); typeset -p a'
declare -ai a='([0]="1" [1]="3" [2]="5" [3]="7" [4]="4" [5]="5")'

Bash 4.4:
 ~ $ ./doc/programs/bash-build/bash -c 'typeset -ia a=({0..5});
a+=([0]+=1 {2..4}); typeset -p a'
declare -ai a=([0]="1" [1]="2" [2]="3" [3]="4" [4]="4" [5]="5")

I almost think it makes sense to treat ordered and unordered
collections differently.

Why?

 With an unordered collection an outer +=
should obviously mean "add or update for each assignment".

For an array, you mean? like
array a=( 1 2 3)
array a+=( 4 5 6)
then you get array a=(5 7 9). Or are you saying
for an ordered array you'd have to use indexes, like
array a=(1 2 3)
array a+=([0]=4 [1]=7 [2]=10), then that would do your
-- but wouldn't that be doing a vector operation of
sorts?

^^ You can make similar "logic" jumps in going the same way in
arrays as you can hashes.

... To me, if you want to add elements to an array or hash
and use '=' on the interior, that should assign, like scalars do.
Use '+=' on the interior to get an increment of the scalar.

Why would you assign special non-uniform rules for scalars in an array
vs. ones not in an array?

Q: what is a fast way to see if an 'item' is present in an array?

[Linda Walsh]

I has a little 4 line func that returns true or false
if a string was in an array

I came up with a different way of doing it, that I
though might be faster... but...
putting them in a file and running loops on them.
(times below).  Obviously, my feeling for what might
be faster was way wrong. w/my 2nd method
being 15-44X slower depending on where the match is.

Obviously, If I could precook the array into
a hash, I'd think it would be faster... but
the "cooking part", I think, is the high
overhead part.

The fast function was:
my fn1='() {
   my __
   my t="${2:?}[@]"
   for __ in "${!t}"; do [[ $1 == "$__" ]] && return 0;done
   return 1;
}
'
w/the slow func  being killed by a $() sub process, likely:
my fn2='() {   
 my t="${2:?}[*]"

   my arRE="^($(IFS="|"; echo "${!t}"))$"
   [[ $1 =~ $arRE ]]
}
'

Was wondering if anyone else had fast functions like this
to determine if a string is in a list?

I can attach the test script (doubles as an "include" or
"source" file), if people want to test their own...
The frameworks should be simple to copy in other test
functions...(or replace #2...)...

This was in bash-4.3.42.

---data results---



(-t [#loops] string WORDLIST):

so false case 1st:
 _in.shh -t 300 word {0..999} 

func(fn1): 1.49sec 1.48usr 0.00sys (100.02% cpu)
false
func(fn2): 22.78sec 18.38usr 4.32sys (99.69% cpu)
false

then true w/match @ end of the list

_in.shh -t 300 word {0..999} word
func(fn1): 1.49sec 1.49usr 0.00sys (100.03% cpu)
true
func(fn2): 22.85sec 18.46usr 4.31sys (99.67% cpu)
true

then true w/match @ beginning of list...


 _in.shh -t 300 word word {0..999}

func(fn1): 0.51sec 0.51usr 0.00sys (100.01% cpu)
true
func(fn2): 22.75sec 18.40usr 4.28sys (99.70% cpu)
true

Re: inconsistent function of "-v" use w/arrays and hashes

[Linda Walsh]

Linda Walsh wrote:

I seem to remember some discussion of this before --
not being able to use -v to check if a hash was defined,
but having it "work"(?) w/arrays?


So instead of "-v" on an array/hash name only looking at the
value of [0] (for either), can it be enhanced to check the
type of the "varname", and if it is an array or hash,
check to see if the count of members returns >0? and use
that as a t/f value for multi-valued objects?

Re: bug adding K,V pairs to existing hash with HASH+=([K]=V)

[Linda Walsh]

Chet Ramey wrote:

On 2/15/16 12:35 PM, Linda Walsh wrote:

  

Bash Version: 4.3
Patch Level: 42
Release Status: release

Description:
[Detailed description of the problem, suggestion, or complaint.]

When I create hash and later add keys using the form
"HASH+=([key]=VALUE)", if KEY already existed in the HASH, then
I get the += applied to the VALUE as well
(strings get appended, ints get added).

Whether '+=' or '=' is used in the VALUE assignment, the
effect is '+='.



http://lists.gnu.org/archive/html/bug-bash/2014-11/msg00169.html
http://lists.gnu.org/archive/html/bug-bash/2014-11/msg00179.html
  


Right... and?

Did it never make it in to 4.3?  (Note, the "auto generated info
included with this bug report was wrong -- as explained in next bug
report...)

Re: 'bashbug behavior and doc problem(s)...

[Linda Walsh]

Greg Wooledge wrote:

On Mon, Feb 15, 2016 at 10:31:24AM -0800, LA Walsh wrote:
  

Ended up looking at manpage, where it says under "ENVIRONMENT":

 EDITOR Specifies the preferred editor. If EDITOR is not set, bashbug
defaults to emacs.



This is not correct (probably it used to be).  bashbug defaults to
/usr/bin/editor if that exists, then ce, then emacs (it looks in
multiple places, but doesn't check the PATH variable), then jove,
then finally vi.
  

wow...

I'm guessing your operating system has a /usr/bin/editor symlink
that points to gvim.
  

nep

 whence editor ce emacs jove vi

-bash: type: editor: not found
-bash: type: ce: not found
emacs is /usr/bin/emacs
emacs is /usr/bin/X11/emacs
-bash: type: jove: not found
vi is /usr/bin/vi
vi is /bin/vi
vi is /usr/bin/X11/vi

 ll {,/usr}/bin/{editor,ce,emacs,jove,vi}

ls: cannot access /bin/editor: No such file or directory
ls: cannot access /bin/ce: No such file or directory
ls: cannot access /bin/emacs: No such file or directory
ls: cannot access /bin/jove: No such file or directory
ls: cannot access /usr/bin/editor: No such file or directory
ls: cannot access /usr/bin/ce: No such file or directory
ls: cannot access /usr/bin/jove: No such file or directory
lrwxrwxrwx  1  3 May 27  2015 /bin/vi -> vim*
-rwxrwxr-x+ 2 28 Sep  3  2011 /usr/bin/emacs*
lrwxrwxrwx  1  8 May 27  2015 /usr/bin/vi -> /bin/vim*
---

It would find 'emacs' first, hmmm 28 bytes? ah HA!:

 more /usr/bin/emacs

#!/bin/bash
exec gvim "$@"

Well, at least that makes some sense...still...
the idea of using your cmdline editing mode to select
a default seems a bit more userfriendly, IMO ;-)

bug adding K,V pairs to existing hash with HASH+=([K]=V)

[Linda Walsh]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc -I/home/abuild/rpmbuild/BUILD/bash-4.2 
-L/home/abuild/rpmbuild/BUILD/bash-4.2/../readline-6.2
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-suse-linux-gnu' 
-DCONF_VENDOR='suse' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL 
-DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib   -fmessage-length=0 
-grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector 
-funwind-tables -fasynchronous-unwind-tables -g  -D_GNU_SOURCE -DRECYCLES_PIDS 
-Wall -g -Wuninitialized -Wextra -Wno-unprototyped-calls -Wno-switch-enum 
-Wno-unused-variable -Wno-unused-parameter -ftree-loop-linear -pipe 
-DBNC382214=0 -DIMPORT_FUNCTIONS_DEF=0 -fprofile-use
uname output: Linux Ishtar 4.1.0-Isht-Van #2 SMP PREEMPT Tue Jun 23 07:52:09 
PDT 2015 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-suse-linux-gnu

Bash Version: 4.2
Patch Level: 53
Release Status: release

Description:
[Detailed description of the problem, suggestion, or complaint.]

When I create hash and later add keys using the form
"HASH+=([key]=VALUE)", if KEY already existed in the HASH, then
I get the += applied to the VALUE as well
(strings get appended, ints get added).

Whether '+=' or '=' is used in the VALUE assignment, the
effect is '+='.

  Can I suggest that, maybe, this "interesting" behavior be
limited to the in-parens, "[KEY][+]=VAL" part?

Repeat-By:
(Example)

> alias sv='my -p'  #(show_var)
> hash cfg=(); sv cfg
declare -A cfg='()' #empty hash

> cfg+=([one]=1 [two]=2) ; sv cfg
declare -A cfg='([one]="1" [two]="2" )' #two elements

> cfg+=([two]=2 [three]=3) ; sv cfg
declare -A cfg='([one]="1" [two]="22" [three]="3" )'  #Note key 'two'

> int cfg  # switch existing hash to integers.
   # explicitly use += in some
> cfg+=([one]+=1 [two]=2 [three]+=0-3 )# VALUE assignments
> sv cfg
declare -Ai cfg='([one]="2" [two]="24" [three]="0" )'


Fix:
Honor "=" to mean "set" and allow "+=" to continue to 
access this behavior.

inconsistent function of "-v" use w/arrays and hashes

[Linda Walsh]

I seem to remember some discussion of this before --
not being able to use -v to check if a hash was defined,
but having it "work"(?) w/arrays?

I think, though, the fact that it operates inconsistently
makes it a bug.  I.e. if always work or fail, at least it
is behaving consistently, feels wrong.

The rule for whether it works or not is whether or not
the there is a value for '[0]' -- whether it is a hash or an
array.  I.e:


 hash hash_cfg=() dummy_hash
 array array_cfg=() dummy_array

# define exists func to make life easier...
 exists () { see below...; } 
 type exists

exists ()
{
   (($#)) || {
   printf '"Null" is not set (no param)';
   return 1
   };
   printf 'var "%s" is' "$1";
   [[ -v $1 ]] || printf " not";
   printf ' set\n'
}

# 1st check what bash "thinks":

 my -p hash_cfg array_cfg dummy hash dummy array

declare -A hash_cfg='()'
declare -a array_cfg='()'
-bash: declare: dummy_hash: not found
-bash: declare: dummy_array: not found
# bash thinks only the assigned-to vars "exist" - "normal"

# assign some values:

 array_cfg+=([1]=one)
 hash_cfg+=([one]=1)
 exists hash_cfg

var "hash_cfg" is not set

 exists array_cfg

var "array_cfg" is not set
# "-v" doesn't think either is set

# but bash shows "set" content in each:

 my -p hash_cfg array_cfg

declare -A hash_cfg='([one]="1" )'
declare -a array_cfg='([1]="one")'


# assign some more values

 hash_cfg+=([zero]=0)
 array_cfg+=([0]=zero)
 exists hash_cfg; exists array_cg

var "hash_cfg" is not set
var "array_cfg" is set
# Now "-v" thinks the array is set, but not the hash

So for hashes and arrays, "-v" only looks at the [0] member,
which is more than a bit wrong... it would be like some
"newbie" thinking they could list the contents of an
array just by using "echo $NAME" , which, for an array lists
the "first" element[sic], but **NOT** for a hash:


 echo "h=$hash_cfg, a=$array_cfg"

h=, a=zero

# So lets give the hash a k,v pair of 0,zero:

 hash_cfg+=([0]=zero)
 exists hash_cfg

var "hash_cfg" is set

# how does echo work now?

 echo "h=$hash_cfg, a=$array_cfg"

h=zero, a=zero
# note that printing w/bash shows:

 my -p hash_cfg array_cfg

declare -A hash_cfg='([one]="1" [zero]="0" [0]="zero" )'
declare -a array_cfg='([0]="zero" [1]="one")'

i.e. if there is a _key_ or index of 0, it is printed, when
no subscript is given, whereas the manpage says:

  Referencing an array variable without a subscript is
  equivalent  to referencing the array with a subscript of 0.

It probably should say this is true for hashes as well (since
it is -- ;-).

Additionally, the manpage says:

  An array variable is considered set if a subscript has
  been assigned  a value.  The null string is a valid value.

But this isn't exactly true either:


 array new_ar=([1]=one)
 exists new_ar

var "new_ar" is not set

 echo "new_ar=$new_ar"

new_ar=

Only the assignment to element zero qualifies it as
existing w/-v, and the same holds true for a hash.


# Note, "-v" knows that other members in the hash and array
# are set, if it is asked about them:

 exists hash_cfg[one]; exists array_cfg[1]

var "hash_cfg[one]" is set
var "array_cfg[1]" is set



At the very least, the manpage should probably be updated
to talk about hash's.

Ideally, "-v" would be fixed to look at #members
of an array or hash and if >0, use that as "defined"
else not.  Would that be doable & not too difficult?
I enclosed a sample alpha-level function to sorta show
what I mean.  It needs to be 'sourced' to work properly,
then call it as:

  exists VARNAME

I don't see any possible compat probs -- only from the fact
that as it is now, it's not really usable to give useful
information about arrays and hashes...whereas returning
set only if the data object has members in it would seem
to be the most the closest parallel definition -- as for
vars -- it is supposed to return whether or not they are
defined and have been assigned to.

If people really felt strongly about it, though, I wouldn't
be totally adverse if it return "true" if it had been
assigned an empty list as well (i.e. array a=(); hash=h(); --
has those will show up as existing w/dcl's "-p".

Ooops... just noticed that it doesn't correctly report
on existence of functions...sigh... well later on that one!

;-)







#!/bin/bash -u 

# "proof-of-concept", "alpha level quality"
# -LAWalsh, 2016

export 
PS4='>${BASH_SOURCE:+${BASH_SOURCE[0]/$HOME/\~}}#${LINENO}${FUNCNAME:+(${FUNCNAME[0]})}>
 '

shopt -s expand_aliases
alias my=declare sub=function
alias int='my -i' array='my -a'  map='my -A'

sub _typ () {
my tp_fld="" jnk="" 
read -r jnk tp_fld jnk< <(
my -p $1 2>/dev/null || {
array fdef=()
readarray fdef< <(my -pf $1 2>/dev/null)
if [[ ${#fdef[@]} && ${fdef[0-1]} =~ ^declare ]]; then
printf "%s" ${fdef[0-1]}
else echo ""; fi; }
)

if [[ ${#tp_fld} 

Re: bash source question & suggestion?

[Linda Walsh]

Chet Ramey wrote:

On 2/9/16 7:30 PM, Linda Walsh wrote:
  

At various times I see people contributing code to bash.

I was wondering if it might be possible to use
C++ in some patches or new files?



I'm not opposed to looking at new functionality that uses C++, but I would
prefer patches to existing C code be in C.


  

Understandable.
Thanks!  ;-)

Re: bug when 'cd ..' to a directory who's parent has been deleted

[Linda Walsh]

Andreas Schwab wrote:

Linda Walsh <b...@tlinx.org> writes:

  

You can't open a file handle on a file.  The fd comes from some



  

OS call associating it with the file on disk (or some other connection).


  
You have to decide which sentence is true.
  

---
It depends on how you are defining file.  If you go by a standard
definition, like wikipedia's:

"A computer file is a resource for storing information, which is 
available to a computer program and is usually based on some kind of 
durable storage".


On linux and on unix, a fundamental difference between it and many other 
OS's, is that the data is pointed to in a file-system specific manner, 
but such that the OS can always return a "handle" or "FileDescriptor" to 
the file.


Processes that have the "handle" can alter or read the information on 
disk unless it has been physically overwritten (which is why, for 
security-sensitive applications, files are not "deleted", but 
overwritten with multiple layers of random garbage, before they are 
returned to the "free-disk-space pool").


On Unix, you can have multiple names for the same "data-collection" on disk.
Each is a separate name (a pathname) which points to the same 
"data-collection".
When you open a pathname, you get back an OS handle to an internal 
OS-structure

that fully describes where and how to access the data on disk.  This is
the "in-memory" version of the disk-inode that identifies the data. 


So whether you have multiple pathnames pointing to an inode, or processes
holding "file descriptors", the actual data isn't released until all the
references to that data are "released".  Deleting pathnames simply wipes
the disk pointers to the data, but as long as the OS references the file
with "in memory" pointers to the data, it is not deleted.  Of special 
note --

this isn't true on Windows, which locks data-areas on disk associated
with pathnames so they cannot be removed if some process has a "writable"
handle open to the Data.  This,  often requires the "Windows reboot" 
solution

to fully release the "write-options" to the data by all processes.

The linux method makes data storage and sharing easier for users --
and even updating executables as long as they own them.  On Windows,
a file's content cannot be updated as long as some process holds
open the Windows-type file-handle in  Shared-read mode (used to
read an executable from disk). 


On linux, when a executes, parts of the program that are marked
for execution, can be set to be read from disk only when the program
on-demand.  Much work has gone into preventing "race" conditions
on linux -- where on process is able to write to a file's data
area on disk while another process is executing it.  When done
with malevolent intent, it can be a security flaw.  This is all
done because long-lived processes can hold open executables for
a long time, either to execute or write from.  Note: many modern
CPU's have protections against this problem by defaulting to
disabling "write-access" to areas in memory that are executing or
paged-in directly from disk.

That's why I wrote the "deleted_procs" script -- since after
a Suse update, old versions can still continue running for
weeks or months unless the program is restarted (zypper has
a similar option to find this out: "zypper ps".

Linda.

Note -- this is using the wikipedia definition of file.  If you
have some other agenda to create confusion by using different
terminology, all bets are off.

bash source question & suggestion?

[Linda Walsh]

At various times I see people contributing code to bash.

I was wondering if it might be possible to use
C++ in some patches or new files?

I know that the kernel, and the glib include files go to
extra pain to declare 'C' functions when using C++ where
those C function need to be called from C++.

It seems like there are various parts and features in bash
that would be ideal for C++, that are more verbose,
and less easy to understand & maintain, and certainly
less easy to extend than similar code written in C.

As an example (I'm sure many here know C++, but for those
that might not knows some of the newer features (or some
old features that weren't as useful when mostly writing in
'C') was a routine to print out Human or SI prefixed numbers.

The header file:

char * Binary_Scale (char buf[], double value);
char * Binary_Scale (char buf[], uint64_t value);
char * Binary_Scale (char buf[], int64_t value);
char * SI_Scale (char buf[], uint64_t value);
string Binary_Scale (double value);
string SI_Scale (uint64_t value);

provides different interfaces to return allow
either standard C-strings or newer C++ strings. 
At link time, the correct form is chosen by which

args are used and by the type of the result.

All of the above call use only 1 function to calculate
and return the desired value and for each function,
only wrappers over the main function
are used to get the various forms like these 2 wrappers:

 string Binary_Scale (double value) { return Scale(value); }
 char * Binary_Scale (char buf[], double value) { return Scale(buf, 
value); }


that call a "string" and "char *" version of "scale":

 string Scale(double value, const int scale = 1024) {
  char buf[16]; return string(Scale(buf, value, scale)); }

which calls the workhorse:

 static char * Scale(char buf[], double value, const int scale = 1024) {
  ...; return buf;  }

--

With "compat functions, the same algorithms can be easily
reused in old and new code allowing for more flexibility in
algorithm design, and, as time progresses, greater reliability.

Not being a veteran C++ programmer, when I started wanting to make
changes in a C program,  but seeing the ways C++ could benefit, I
started out by creating alot of "C" compat functions that didn't
have the C++ alternates, but as I grew more comfortable with the C++
data types and features, I went back and implemented C++ interfaces
around the original 'C' funtions to do the same thing.  Later, I
more often started with C++ funcs and providing backwards compat
only where necessary.

The main benefit I found -- a *large* reduction in code size where
multiple C-files of hundreds to 1000 or more lines, shrank down using
'common functions' to <100 lines for many with at least 30-40%
code savings (and increase in clarity) for scores of files.

Maybe the above is already policy or maybe it's pointless to suggest,
but either way, I think it's a good way to go to be able to extend
bash, while shrinking code and boosting reliability.








one of my favorites -- that helps in transitioning
or providing compatibility going from 'C' to C++.

Re: bug when 'cd ..' to a directory who's parent has been deleted

[Linda Walsh]

Chet Ramey wrote:

On 2/8/16 9:59 AM, Andreas Schwab wrote:
  

Chet Ramey  writes:



`cd ..' should fail, since the parent no longer exists, and the pathname
canonicalization should fail, since there's presumably no longer a valid
path to reach the current directory.  No value for $PWD is correct.
  

${PWD%/*} would be a reasonable value.  FWIW, this is what ksh uses in
this case, it doesn't fail.



Why would that be more reasonable than anything else?  It references a
path that doesn't exist.

  

Um...Not exactly.  As long as there's a handle open to the previous path,
it still exists (at least on linux and unix).

manpage question re: '--debugger': what/where is debugger *profile*

[Linda Walsh]

Under the bash-arg "--debugger" it says:

   Arrange for the debugger profile to be executed before the shell
   starts.   Turns  on extended debugging mode (see the description
   of the extdebug option to the shopt builtin below).


When I think of a "profile" in bash, I think of things
like /etc/profile or ~/.{bash_,}profile.

Is there something like that for debugging
(ex: /etc/bash_debugging or ~/.bash_debugging)?

If not, what does it all mean, er, that is
if the only thing it does is turn on the extdebug
option, then what is the 1st sentence talking about
in addition the the extdebug option mentioned in the
2nd sentence?

Re: bug when 'cd ..' to a directory who's parent has been deleted

[Linda Walsh]

Dave Rutherford wrote:



On Mon, Feb 8, 2016 at 11:34 AM, Linda Walsh <b...@tlinx.org 
<mailto:b...@tlinx.org>> wrote:

>> Why would that be more reasonable than anything else?  It references a
>> path that doesn't exist.
> Um...Not exactly.  As long as there's a handle open to the previous path,
> it still exists (at least on linux and unix).

Really?
11:40:24 dave@Vger:~$ mkdir -p a/a
11:40:30 dave@Vger:~$ cd a/a
[remove a in another window]
11:40:49 dave@Vger:~/a/a$ touch file
touch: cannot touch ‘file’: No such file or directory



Yes!  Really.  You and Andreas have to read what was written.

Pretty much the same setup as you have, but I'm using an "fd" to
the previous path. Are you using an "fd" in your example, I seem
to be missing it.

Ish:lw> ll a
ls: cannot access a: No such file or directory
Ish:lw> mkdir -p a/a
Ish:lw> cd a/a
Ish:lw/a/a> cp /tmp/filesave.txt file.txt
Ish:lw/a/a> ll
total 4
-rw-rw-r-- 1 415 Feb  8 13:39 file.txt
Ish:lw/a/a> rm -f /tmp/ctl; Delayed_save_file_via_fd  ...Reading...

[ Other term>
Ish:lw> rm -fr "a/" ; echo -e ~/a/a/file.txt\\n.\\n >/tmp/ctl
] Original term:

saved 415 bytes in file "file.txt", dir "/home/lw/a/a"
cd "$PWD"
[1]+  Exit 1  Delayed_save_file_via_fd /tmp/ctl < file.txt
Ish:lw/a/a> ll
total 4
-rw-rw-r-- 1 415 Feb  8 15:16 file.txt

Delayed_save_file_via_fd is attached.




Andreas Schwab wrote:

You cannot have handles on file names, only on files.

---
I said:
"As long as there's a handle open to the previous path..."


You can't open a file handle on a file.  The fd comes from some
OS call associating it with the file on disk (or some other connection).

Not only that, but you can **run** from those deleted paths --
the kernel says the "fd" is associate with deleted path:

Ish:lw/a/a> cp /usr/bin/sleep .
Ish:lw/a/a> ll
total 48
-rw-rw-r-- 1   415 Feb  8 15:16 file.txt
-rwxr-xr-x 1 41676 Mar  2  2015 sleep*
Ish:lw/a/a> sleep 999&
[1] 41987
Ish:lw/a/a> cd ../..
Ish:lw> rm -fr a
Ish:lw> deleted_procs

Deleted programs  Pid(s)
-- --
/home/lw/a/a/sleep41987


Looks like a path to me.  Will attach "deleted_procs" too.




#!/bin/bash -u

#include stdalias
shopt -s expand_aliases
alias sub=function my=declare
alias int='my -i'  string=my  intConst='my -ir' array='my -a'

export 
PS4='>${BASH_SOURCE:+${BASH_SOURCE[0]/$HOME/\~}}#${LINENO}${FUNCNAME:+(${FUNCNAME[0]})}>
 '

shopt -s extdebug

my cd=$PWD
int tries=-1
string h1="" h2=""
intConst EPERM=1 EINVAL=22 ENOTDIR=20
int verbose=0


sub inf () {
(($#)) || return 0
int newline=1
if [[ $1 == "-n" ]]; then newline=0; shift; fi
(($#)) || return 0
my out=""
printf -v out "$@"
if ((newline)) && [[ ${out:0-1} != $'\n' ]]; then
out="$out"$'\n'
fi
printf >&2 "%s" "$out"
((verbose)) && sleep .1
return 0
}

sub err () {
int stat=$EINVAL
if (($#)); then
if [[ $1 =~ ^[-0-9]+$ ]]; then
stat=$1; shift
fi
if (($#)); then
inf "$@"
else
inf 'Err status=%d\n' "$stat";
fi
return $stat
else
return 0
fi
}

sub get_sp() {

while [[ ! -f $control ]];do sleep 1; echo -n . ;  done
while [[ ! $savepath ]]; do
read -t 1 inp  <"$control"
if [[ $inp ]]; then
if [[ $inp != . ]]; then
savepath=$inp
fi
break
fi
done

[[ ${savepath:0:1} != / ]] && savepath="$cd$savepath"
}

sub get_content () {
inf -n "Reading..."
while read inp;do
my out=""
printf -v out '%s\n' "$inp"
content+=( "$out" )
content_len+=${#out}
done
}



sub saveworker() {
my control=${1:?"need place to get fn"}
array content=()
int content_len=0
my inp="" savepath=""
string h1=""
string h2=""

get_sp

((verbose)) && inf 'Got save path as:%s' "$savepath"

get_content

((verbose)) && inf 'Attempting to save %d lines (%d chars) in %s\n' \
"${#content[@]}" "$content_len" "$savepath"

my d=${savepath%/*}
my f=${savepa

capturing in ${VAR//(*(\/)(+([^\/]))?(\/)/_${BASH_REMATCH[1]}_}

[Linda Walsh]

When you are doing a var expansion using the
replacement format ${VAR//./.}, is there some way to
put parens around some part of the expression and reference
them as in the [[V~re]] RE-matches?

Couldn't find anything in the manpage, but I could easily
have missed something...?

Thanks...

Re: doesn't bash do variable subst. or quote removal on function statement

[Linda Walsh]

Chet Ramey wrote:

On 1/12/16 4:33 PM, Linda Walsh wrote:
  

---
   But why?  It's less noisy than any alternative.
It's not as if I can type echo $(1+2) and expect it to work.



Because it only ever appeared in a Posix draft standard, and there's no
reason not to use the standard syntax.
  


   All of the linux security related to capabilities "only" appeared
in a Posix draft standard.  That hasn't stopped it from being adopted in
various forms in many *nix's implementing capabilities.  $() is already
used for command execution and () is already used for expression
grouping.  Putting the two of them together takes  2 operators from 2
disparate areas and claiming it's a new operator vs. using $[] which AFAIK,
isn't used for anything, seems deliberately Machiavellian, not to mention
noisier.  If  both were available as a standard, which would make for
clearer expressions?  I.e. What can $[] be confused with?  V. $() and ()
already having alternate uses, not to mention creating tortured,
ambiguous syntax like:

declare -i a=0;((a=1+2));(echo $(echo $a))

Now if I wanted to do the assignment in the echo, I have to do this?:

declare -i a=0; (echo $(echo $((a=1+2   Vs.
declare -i a=0; (echo $(echo $[a=1+2]))

Of course the 2nd form using brackets, w/o the '$'
can be used as an array subscript -- where it gets the same
evaluation.  I.e.

declare -i a=0; (echo $(echo ${ar[a=1+2]})) -- i.e. the language
already uses [] to hold an arithmetic expression for arrays, but you
can't take $((a=1+2)) and use the expression after the $ directly as
an array as in:
declare -i a=0; (echo $(echo ${ar((a=1+2))})) (not that such usage would
be that desirable -- but the point being that [] is already used for
arith evaluation in arrays, so adding the sigil on front for stand-alone
arithmetic evaluation seems to be a logical progression -- Not something
one has with $(()).

It doesn't seem very consistent to choose (()) over [] -- why come up with
a special longer operator for arith eval, when [] was already used for
arith evaluation in array indexing?




*bleh*

Re: doesn't bash do variable subst. or quote removal on function statement

[Linda Walsh]

Greg Wooledge wrote:

On Tue, Jan 12, 2016 at 11:52:47AM -0800, Linda Walsh wrote:
  

Also, from a documentation standpoint, not all behaviors are documented
in the manpage, like:

echo $[1+2]

as far as I can tell, isn't documented, yet not supporting
its behavior would break many older scripts



That syntax has been deprecated for at least 15 years.

---
   But why?  It's less noisy than any alternative.
It's not as if I can type echo $(1+2) and expect it to work.



  Nobody is
supposed to be using it.  It's undocumented ON PURPOSE, to try to make
people stop doing it.

If you have older scripts that use it, fix them now.
  

---
   Only in 3rd party scripts where scripts come with some product to be 
used
on *nix, which is why I worry about it.  I wouldn't want those scripts 
to become

my responsibility to "fix".

   Though, at one point, I seem to remember some wondering when systemd 
might

integrate a shell-service as it has many other *nix utils (somewhere around
the time it was integrating code to replace 'su' or 'sudo', I think).  
If that

happens, then it's clear where compatibility will go.

Re: doesn't bash do variable subst. or quote removal on function statement

[Linda Walsh]

Chet Ramey wrote:

No.  The shell doesn't perform any word expansions on the `name' when
executing a function definition command.  Since the documentation doesn't
say it performs any expansions, why would you assume it does?
  

Because it does quote quote removal on other statements that _seem_ to be
of similar syntax.


I.e.:

declare  "ab=1"   "ab"=1   ab"=1"  a"b=1"  "a"b=1   a"b"=1   ab"=1"
 all appear to do quote removal, making them equivalent to
declare ab=1

vs. for function, where
one has:
 function a {(echo hi)}

then one might think, _at least_, that some similar syntax would
work like:

function "a" {(echo hi)}  function "a" "{(echo hi)}"

or even

declare -f a="{(echo hi)}"

_might_ work.

Also, from a documentation standpoint, not all behaviors are documented
in the manpage, like:

echo $[1+2]

as far as I can tell, isn't documented, yet not supporting
its behavior would break many older scripts

doesn't bash do variable subst. or quote removal on function statement

[Linda Walsh]

Reuti wrote:

Am 08.01.2016 um 21:36 schrieb Piotr Grzybowski:

  

hello Linda,

On Fri, Jan 8, 2016 at 9:14 PM, Linda Walsh <b...@tlinx.org> wrote:


For what it's worth, It might be useful to have something like
'command' [..]
  

that would be useful, a keyword like:

function p params;


But 'function p'... is already a lead-in to a function definition, i.e:
function xyz { :; }  #(create function foo)


AFAICS putting the name in quotes (single or double) will prefer the function 
over the alias, but it will fall back to a command if there is no function.

'P' params
  


   Not in a function definition (i.e. this is broken AFAIAC in bash).
I.e.

 'func' () { :; }

-bash: `'func'': not a valid identifier

 function 'func' () { : ; }

-bash: `'func'': not a valid identifier

 function "func" () { :; }

-bash: `"func"': not a valid identifier
---
That's weird.  while, _maybe_ I could see the 1st one not working
if looking for a bash keyword, but the 2nd & third forms seems odd.
I thought variable substitution and quote-removal was supposed
to happen before the statement is executed ??

I.e.: this _seems_ broken:


 shopt -s expand_aliases; alias my=declare
 declare fn=myfunc## function name in variable 

doesn't work

 function $fn { echo $fn ; }

-bash: `$fn': not a valid identifier

my -pf myfunc

-bash: declare: myfunc: not found


 def="function $fn () { echo $fn ; }"  ## but same statement, eval'd works
 eval $def   
 my -f myfunc

myfunc ()
{
   echo myfunc
}

I.e. while having the function name in a variable doesn't work
interactively, it does work when used with eval.

Re: aliases v. functions; command extension? (-f & -F)?

[Linda Walsh]

For what it's worth, It might be useful to have something like
'command' that (or maybe a flag to 'command', like "-f & -F" that would
search for functions for the 'command' either before ("-f")
the PATH search, or after ("-F"), for the case of the command not
being found in the PATH.


Reuti wrote:

 Seems like using 'function' is safer.

Perhaps the idea of leaving off "function" at the beginning of a function
isn't such a good practice?



Even if you use it: someone could define an alias named function - and this 
stands for all commands.
  

---
   My scripts are designed to assist.  I rarely need to design them to be
"invulnerable" against malicious actions.  I usually define an alias or, 
when

I want "elaboration", a function, that uses '$(type -P cmd)' as the path
(elaboration = things like handling error conditions or running command as
root).

In some cases it might be possible to check the exit code of `alias P` resp. `type P` 
beforehand, or to remove with "unalias -a" all aliases.
  

"unalias P 2>&- || ((1))" seemed best for this case.

"unalias -a" would usually not be the best thing to do, in my 
environment, as aliases are usually more helpful than not.  There are 
several places in my shell

scripts where functions don't work (like in this case).  Seems like the best
solution, usually, is to define your own alias in the script and use 
that.  The down-side of that is if you later want to introduce a 
function in place of the

alias -- I guess _that_'s a good reason to use an include file like:

 "include stdalias"

so all of one's aliases are in one place.  Although that requires that 
an 'include' command be available and defined going into a script (I use a

function to define that in my system "/etc/bash_env").

=

Greg Wooledge wrote:

Yup. That's one of the many evils of aliases. They need to be removed
(unalias) before you can redefine them as functions. 


   Not if you include a prefix indicating it is a function (e.g. 
"function").

It was my trying to use 'recommended' format that got me in trouble in this
case.

   I usually define my own alias, "sub", for function.  Besides being 
shorter,
it also would seem to protect against this type of problem and is 
another case

where a function can't do the job of an alias.

-l

'official function declaration format' doesn't work if alias defined

[Linda Walsh]

I had an alias referring to printf that I wanted to replace
with a function.

instead of using the function declarator
'function' (or my alias 'sub'), I remembered that the official
way was to use:

 P () {
   ...
 }

But then ran into problems with the alias taking precedence over the
function.

Even in the file where the function was defined and exported, I got
an error in the export statement about "P" not being a function.
I also tried unaliasing it:

 unalias P >& /dev/null || ((1))

 export -f P Pe

But still got the "not a function" ... then I realized
I had used the official, POSIX format for function declarations,
above, but guess it had silently been changed into
 printf () {
   ...
 }
by the 'alias'. 


Sure enough, using the old form:

 function P () {
   ...
 }

caused "P" to be defined as a function -- so when it was
unaliased and exported, the export statement found the function.

I am not sure how one would catch a problem like that other than
by make sure the defined function is never the 1st argument on the line
(by adding function).  Seems like using 'function' is safer.

Perhaps the idea of leaving off "function" at the beginning of a function
isn't such a good practice?

(bash, version 4.3.39)

-l

Re: bash variable names do not comply w/POSIX character set rules

[Linda Walsh]

Eduardo A. Bustamante López wrote:

 This definition (


http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap03.html#tag_03_230

 ) states:

  3.230 Name

  In the shell command language, a word consisting solely of underscores,
  digits, and alphabetics from the portable character set. The first 

character

  of a name is not a digit.


  (1) -- It appears you /accidently/ left out part of the text under
section 3.230.  The full text:


 3.230 Name

 In the shell command language, a word consisting solely of
 underscores, digits, and alphabetics from the portable character
 set. The first character of a name is not a digit.

 Note: The Portable Character Set is defined in detail in
 P̲o̲r̲t̲a̲b̲l̲e̲ ̲C̲h̲a̲r̲a̲c̲t̲e̲r̲ ̲S̲e̲t̲⁽¹⁾

[§̲⁽¹⁾ 
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap06.html#tag_06_01 
]

 3.231 ...[next section]



   Thank-you.  This slightly clarifies matters as it only
requires the POSIX source.  At the location pointed to by
the hyper-link for "Portable Character Set" under section 6.1
sentences 2-4, it states:


 Each supported locale shall include the portable character set,
 which is the set of symbolic names for characters in Portable
 Character Set. This is used to describe characters within the text
 of IEEE Std 1003.1-2001. The first eight entries in Portable
 Character Set are defined in the ISO/IEC 6429:1992 standard and
 the rest of the characters are defined in the ISO/IEC 10646-1:2000
 standard. 


   FWIW, in full disclosure, in the last dotted paragraph before the
last sentence of section 6.1, there is a requirement that the alphabetic
character fit within 1 byte -- i.e. only characters in what is commonly
called the "Extended ASCII character set" (ex. ISO-8859-1) seem to be
required.  Note, the character 'Ø' is 1 byte.  So, as the quoted
section above mentions using [basically], the Unicode table for "symbolic
names", it doesn't prescribe a specific encoding. I.e. - While the
reference is to ISO-10646 (Unicode), it does not require a
specific encoding. 


   For Unicode values 0-255, ISO-8859-1 encodes the first 256
bytes of Unicode with 1 byte (satisfying the 1-byte posix constraint,
though it is not able to encode Unicode values >=256, which makes
posix's reference to ISO-10646 somewhat specious as only the 1st
256 values can be encoded in 1 byte (that I am aware of).

   Nevertheless, the symbolic name "LATIN CAPITAL LETTER O WITH STROKE
(o slash)" or 'U+00D8' is classified as an alphabetic, which is a subset
of the "alphanumeric" requirement of POSIX. 


   Note under section 9.3.5 "RE Bracket Expression", subsection 6:


 The following character class expressions shall be supported in
 all locales:

 [:alnum:]   [:cntrl:]   [:lower:]   [:space:]
 [:alpha:]   [:digit:]   [:print:]   [:upper:]
 [:blank:]   [:graph:]   [:punct:]   [:xdigit:]

 In addition, character class expressions of the form:

 [:name:]

 are recognized in those locales where the name keyword has been
 given a charclass definition in the LC_CTYPE category.


Note that "aØb" is classified as fully "alphabetic" by bash's
character-class matching facility -- whether in UTF-8 or ISO-8859-1:


 echo $LC_CTYPE

en_US.ISO-8859-1
LC_CTYPE=en_US.UTF-8
...

 declare -xg a=$(echo -n $'\x61\xd8\x62')
 declare -xg b=${a}c
 [[ $a =~ ^[[:alpha:]]+$ ]] && echo alpha

alpha
 [[ $a =~ ^[[:alnum:]]+$ ]] && echo alnum   

alnum

 [[ $b =~ ^[[:alpha:]]+$ ]] && echo alpha

alpha

 [[ $b =~ ^[[:alnum:]]+$ ]] && echo alnum

alnum

Notice bash classifies the string "aØb" as an alphanumeric AND
as an alphabetic character.  I.e.  bash, itself, claims that
"aØb" is a valid identifier.

Also note, it accepts "aØb" as a var and as an environment var
when used indirectly:


 declare -xg $a='a"slash-O"b'
 declare -xg $b='ab"slash-O"c'
 env|/usr/bin/grep -P '^[ab]...?'|hexdump -C

  61 d8 62 63 3d 61 62 22  73 6c 61 73 68 2d 4f 22  |aab"slash-O"|
0010  63 0a 61 d8 62 3d 61 22  73 6c 61 73 68 2d 4f 22  |c.a"slash-O"|
0020  62 0a 61 3d 61 d8 62 0a  62 3d 61 d8 62 63 0a |b.a=a=a|
002f

===



...
 So no, it does not mandate arbitrary unicode alphabetics. Only the 

ones listed

 there.


   Thank-you.  This better makes the case, as it only refers to
the POSIX reference pages.  But it seems that it boils down to the
allowed definition of envirionment variables:
(http://pubs.opengroup.org/onlinepubs/9699919799/)


 2.5.3 Shell Variables



 Variables shall be initialized from the environment (as defined by
 XBD Environment Variables and the exec function in the System
 Interfaces volume of POSIX.1-2008) and can be given new values
 with variable assignment commands.



The XBD interface is a description of API facilities for programs to use --
not an end-user-interface.  In particular, it says: (under section 8.1)

(http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08)


 Environment variable names used by the utilities in the 

bash variable names do not comply w/POSIX character set rules

[Linda Walsh]

Under section 2.5.3, Shell Variables, it mentions:

LC_CTYPE
   Determine the interpretation of sequences of bytes of text data as 
characters (for example, single-byte as opposed to multi-byte 
characters), which characters are defined as letters (character class 
alpha) and  characters (character class blank), and the behavior 
of character classes within pattern matching.


If I have an LC_CTYPE set to UTF-8, then the rules in unicode as
to how the character is defined (alpha, numeric, alphanumeric, etc...)
seem appropriate to use.

In the bash man page, there is a definition of 'name':
  name   A word consisting only of  alphanumeric  characters  and  under-
 scores,  and beginning with an alphabetic character or an under-
 score.  Also referred to as an identifier.

However, I was looking for a char to visually separate
a "class" and a var in the class (would have liked something
like a.b, but "." isn't alpha numeric), but
"LATIN CAPITAL LETTER O WITH STROKE" (U+00D8), is alphabetic,
but doesn't work:

 aØb=1

-bash: aØb=1: command not found

The POSIX portable character set:
6. Character Set
6.1 Portable Character Set

Conforming implementations shall support one or more coded character 
sets. Each supported locale shall include the portable character set, 
which is the set of symbolic names for characters in Portable Character 
Set. This is used to describe characters within the text of 
POSIX.1-2008. The first eight entries in Portable Character Set are 
defined in the ISO/IEC 6429:1992 standard and the rest of the characters 
are defined in the ISO/IEC 10646-1:2000 standard.


ISO10646 = Unicode -- I.e. Posix appears to base its definition of
alphanumeric characters, for example, on the Unicode character set.

So, theoretically, any alphanumeric class char from Unicode should work
as described in the bash manpages, to compose a "name" (variable or
subroutine name), but this doesn't seem to be the case.

I know this isn't a trivial POSIX requirement to meet, but given
Gnu and bash's changes in the shell and unix command behavior, it
seems support of the character set would be the foundation of POSIX
compatibility.

It it were me, I'd probably try to look at the perl-handling (imperfect
as it may be) for unicode -- which has had alot of work put into it and
may be one of the more complete and up-to-date implementations for unicode
character handling.  I'd try to see if there was any part that might
either give ideas for bringing bash into compliance or any code that
might provide a pattern for implementation.  But investigating it further
might yield other, better options for bash.  Dunno.

Is this something that's even been thought about or is planned for?

Thanks!
-Linda

Re: Installation of PDF/PS/DVI and HTML files

[Linda Walsh]

Mathieu Lirzin wrote:

Hi,

Greg Wooledge  writes:


On Thu, Dec 03, 2015 at 01:08:13PM +0200, Ludovic Courtès wrote:

Given that the GCS suggests installing only the Info version of the
manual by default (info "(standards) Standard Targets")
What do you think?

I think that's a stupid suggestion.


It would be nice to express your ideas in a polite way.  The point was
about achieving a reproducible build, not about droping the man page.


Nevertheless, Gnu and others have made similarly well-suggested
ideas regarding unix utils and architecture as to possibly leave some
people a bit short on patience.



The de facto standard for "make" followed "make install" on a
_Unix_-like system is to install man pages.  


For more _GNU_ standards [_emphasis_ mine]

---
Notice  you are comparing 'GNU' (new) standards compared to
older and established standards.  Many of the newer standards fly in
the face of good design and throw-out previous 'best-practices' standards
for no other reason than to follow new standards that are not justifiable
based on user-utility nor sound computer science.

L. Walsh

Re: prompt with \w incl. long UTF string messes up command line

[Linda Walsh]

Your prompt might be too long.

Your 'dir' string is awfully long...
echo "$dir"|wc -c
270
---

I tried a shorter version on linux:
sdir='△_ >'
pre='Ishtar:/tmp/' ## prefix of my dir
Ishtar:/tmp/△> 
012345678901234567890123456

-bash: 012345678901234567890123456: command not found
Ishtar:/tmp/△> echo ${#pre}+${#sdir}=$[${#pre}+${#sdir}]
12+13=25
My cursor started in col 27 with 26 chars preceding it.
As you can see above adding the length in chars
of the prefix and the UTF chars together, I get 25, + the '>'
at the end=26.  and together :
Ishtar:/tmp/△> echo "$pre$sdir>"|wc -c
53
are only 53 bytes long.






Ryo Furue wrote:

When PS1 includes \w and when \w contains long path including UTF
characters, the cursor position or the command line is messed up.

Repeat-By:
1. Source the attached script:
  $ . bash-prompt-bug.sh

2.1. You'll find the prompt is messed up: shorter than the
   real path name, garbage character at the top.
2.2. If you use the tput commands, which are commented out
   in bash-prompt-bug.sh, the prompt is fine, but the cursor
   position is messed up.

3. After 2.2, try to complete filenames by pressing TAB,
  and sometimes you'll find the cursor position is
  even more badly messed up.


Bash doesn't honor tabs as set on your terminal
at all.  It always expands them to '8' spaces and it's concept
of where it is vs. using actual tab chars which would have
output looking like what you typed.



I tested this both on the latest versions of iTerm and Terminal on Mac
OS 10.11.1 .


I tested this on bash 4.3.39.  Maybe the MAC doesn't
handle the expansions right?

Upgrading shell vars to allow for possibility of FD redirection into a var.

[Linda Walsh]

konsolebox wrote:

On Thu, Nov 19, 2015 at 4:29 AM, Linda Walsh <b...@tlinx.org> wrote:

However one cannot
categorically say that a NUL byte can't be used as an argument.


Likely only in an escaped format, or just as a single byte or character.


Solving
other places where it doesn't work might make it so that it *would* work...


Most parts of bash interpret, store and handle parameters (internal
data; not shell arguments) as C-strings (null-terminated) (as an
example, see make_variable_value() in variables.c, or savestring() in
general.h).  It's the key to understanding why NUL bytes doesn't work
most of the time in bash.  If you would want it to interpret them as
strings like C++ strings or Ruby strings, it would need a big
overhaul.  You would also think about possible compliance issues,
compatibility issues, complexities and vulnerabilities.  It's great
but not easy.


Easiest way to explore -- see what it would take to get
bash to compile under g++.

Then everywhere you use strings, you could declare it
with "string".  It includes a count of #chars, so embedded nulls
are allowed.  If you need the C-version (0 terminated), you can
use the string.cstr() method -- which will return the whole
string (including embedded NULs, if any), terminated by a NUL.

Of course thats all dependent on how hard it is to get
to compile under g++ (1), and (2) a long, but able-to-be-broken-apart
piecemeal approach of converting 1 subsystem at a time.  I.e.
by subsystem, some small part of 'bash', that has few interfaces
with the rest of the code, that could be converted, -- with 
compatibility provided at the interfaces.  The neat thing about

C++ is you create 2 copies of a function -- one that works with
old "char *" strings, and one that works with 'strings'.
Ex:a sub to convert a number to a computer-metric prefix

.h:

 char * Binary_Scale (char buf[], double value);
 string Binary_Scale (double value);

.cc:

 char * Binary_Scale (char buf[], double value) { return Scale(buf, value); }
 string Binary_Scale (double value) { return Scale(value); }

later, since the funcs are so small, one might want to inline them,
so move the 'source lines from 'cc' to the ".h" to replace them, and
add 'inline'.  In the new .h:

 inline char * Binary_Scale (char buf[], double value) { return Scale(buf, 
value); }
 inline string Binary_Scale (double value) { return Scale(value); }


Either way, any code that include the header will automatically
call the "right" version of "Binary_Scale", based on the argument
types -- so you can have new and old code living together in the same
program without it being excessively painful...  ;-)

If the library types don't work for one, you can roll your own.
I wanted to create a 'ratio' data type, that took 2-64bit integers.
I then made all the basic functions work with it (+-*/).  As data
was entered, it reduced the integers by the GCD and stored it as
well (in case the original number(s) were needed).  But for subsequent
calculations, they'd always try to use the reduced format to reduce
the possibility of overflow/underflow -- which was checked for and flagged.

I didn't want to use floating point, since, while modern floating point
performance is fast, integer ops are still 10-100x faster and I was
working with all 64-bit data counters -- so wanted to keep all the
intermediate calculations in integer ratios so I wouldn't lose precision
(which is lost on final output, as it was usually converted to a
metric-scaled float).

Anyway -- that's how I'd approach an upgrade like this that would
eventually enable shell vars to contain binary data, plus going to
a 'str+len' from zero terminated, _should_ make the code more robust
and less subject to buffer overruns. 

Re: redirecting a file descriptor to an array variable? Possible? How? RFE?

[Linda Walsh]

Greg Wooledge wrote:

On Sun, Nov 15, 2015 at 05:26:26AM -0800, Linda Walsh wrote:
This can be used to serialize an array into a file
and to read it back:

printf '%s\0' "${array[@]}" > file


So just like you can have [<>|]&[-0-9][0-0] as redirection ops that work
with files or FD's, why not have something like:

out=()
printf '%s\0' "${array[@]}" >&{out[@]} (**)

**: or ">@out" or ">&@out"



using 4.3,   But how is this:

  > mapfile -td '' array < file

That's fine for reading files, but I'm wanting this to be doable
in the absence of a file system, so again, instead of '>' used in printf,
above, use '<' -- but have the object it reads from represent
an array -- so you could have your file in an 'array' with the
nul's trimmed off (or nl's).

As for not wanting to use files... they are an unnecessary
dependency in most cases and file opens are expensive on some
OS's -- more so than linux (ex. cygwin).

If I have GigaBytes of memory, why should I have to use a 
file system to store transient, tmp information (unless it exceed 
all of memory), BUT -- that has often happened when "/tmp" hasn't been

large enough -- with cheaper memory, note the situation on my system
as it's been for a couple of years now...


df /tmp  /dev/mem

Filesystem  Size  Used Avail Use% Mounted on
/dev/sdc2   7.8G  3.3G  4.6G  42% /tmp
devtmpfs 48G   44K   48G   1% /dev


So if you are writing to a tmp file, which will die of
ENOMEM first?

If I thought adding 4.6G to '48G' (statistically), enough
of the the time, I'd certainly arrange for a spill.

But at run time, wouldn't it be smart to check resources --
like disk space in /tmp or free memory?  If free memory is

10x the disk space, maybe using the memory should be given
priority in automatic mode, thought for corner cases, 
'tmp' should allow "auto_largest", "mem", "/tmp" or "~/tmp", or 
auto_besteffort (do what one can -- but intially, start w/auto_largest,

then add a spell-to-disk "/tmp", if there is <90% space on it...




But contrary to the manpage under printf:
" The -v option causes the  output  to  be
 assigned  to  the  variable var rather than being printed to the
 standard output. "

printf does not print the output of the above into a var instead
of stdout.  


Seems like that would be a bug?


The command you used above does not have '-v var' in it.

---
1) printf '1\000\n'  | hexdump -C
  31 00 0a  |1..|

2) printf '1%c\n' '\x00' | hexdump -C  
  31 5c 0a  |1\.|


with/"-v"
1) printf -v v '1\000\n' 

declare -p v

declare -- v="1"  <<-- note that $v does not have '1\000\n' in it.

Ex2:

1)  printf '"%c"\n' '' | hexdump -C  
  22 00 22 0a 
1b) printf '"%c"\n' $'\x00' | hexdump -C  
  22 00 22 0a  
# Showing, indeed, '' == $'\x00', in this case, however, if we

try printing the result using "-v v" and look at it:

1st, assignment:

unset v; printf -v v '"%c"\n'  $'\x00' ;declare -p v

declare -- v="\""<<-- again note that "$v" doesn't contain $'"\x00"'.




If I am in a locale using UTF-16, there will be lots of 'NUL'
chars if I view that output as binary -- so there has to be one
or more ways to encode NUL bytes in a var such that they don't
terminate the string.  


I do not have access to a UTF-16 environment.  Does bash even WORK
in such an environment?  At all?


Actually, on this one, I'll have to find a set of 'locale'
files to describe the charset.  Several files that work with the
"LC_" files end up working due to the way they are specified ..
for example, the 1st go-around for the Chinese to encode their stuff
and everything else ended up considerably larger than UTF-8:

/usr/share/i18n/charmaps> ll --sort size -r | tail -2
-rw-rw-r-- 1 1951778 Aug 11 11:55 UTF-8
-rw-rw-r-- 1 4181789 Nov 17 15:12 GB18030


GB18030 _does_ provide a 1:1 mapping of Unicode codepoints there and
back again (same as UTF-8 and UTF-16).  
But since UTF-16 & UTF-8 have the same # of codepoints, they'd

likely be similar in size.



I don't see the relation between these two issues.  "Being able to
store NUL bytes in a string" has nothing to do with "being able to
capture stdout and stderr separately in shell variables" as far as
I can see.

---
As I pointed out above, you cannot safely create, save or restore
output from printf that will be redirected into a variable, which
contradicts the manpage specification saying:

The -v option causes the  output  to  be
assigned  to  the  variable var rather than being printed to the
standard output.


Chet would have to introduce a new feature for the latter to be possible.


	I'm sure it's not trivial to handle all the different 
surprise cases.

Re: redirecting a file descriptor to an array variable? Possible? How? RFE?

[Linda Walsh]

konsolebox wrote:

On Fri, Nov 13, 2015 at 8:45 AM, Linda Walsh <b...@tlinx.org> wrote:

I'd like to be able to record stdout and stderr
without using any temp files into bash array files, AND
record the status of the command executed.



You can use coproc processes that would act as buffers.
Obvious Note: Avoid it if you're conservative about some undocumented
areas that would look too hacky to you.


More often people think my code looks hacky because
I haven't used alot of Integrated GUI+Devel platforms, 


Re... using coprocs hadn't even though of those
(probably since I haven't had good luck with them working the way
I expected).

Maybe I'll work w/that, or maybe will try a refactor
of my current code to combine the output streams in 1 sub and that
might make my return of the 'exit code'($?) easier.



# TODO: Not sure how a proper coproc cleanup is done.  Closing
FDs does funny things.  Perhaps it's automatic, but not sure how
disown affects it.  We could examine the source code but not sure if
that would be reliable enough for all versions of bash including the
upcoming ones.


One thing I found useful is where you have
2 procs talking over a pipe, and the reader needs time to finish the pipe
read before parent closes it.  So instead of just closing a pipe, I
open a pair of pipes w/1 going in reverse direction. Then parent can send
various control commands to child while the child is generating output
so the child won't end prematurely.

As mentioned, was close but wasn't feeling right about
it.  But some things that simplify my life.  I'm running
commands that talk to the terminal.  None of them _should_ be putting
out NULs... and going off on red-herring rat-holes isn't the best
way to get something that works.  


One thing that seems to be another problem.  Greg called
my attention to another printf format bug:

 %q causes  printf  to output the corresponding argument in a
format that can be reused as shell input.

Which it doesn't when nuls are involved.

Re: redirecting a file descriptor to an array variable? Possible? How? RFE?

[Linda Walsh]

Greg Wooledge wrote:

You should stop using irrelevant examples, and return to your core
question.  I believe your initial goal was to run a command, and capture
the command's stdout in one variable, and the command's stderr in another
variable, without using any temp files.


Actually, wanted them already line split  into 2 arrays.

That's why I didn't care too much if the lines were separated
by NUL's instead of NL's.



Come up with some syntax 


The syntax isn't the hard part -- really, I think there are several
formats that could be used that wouldn't conflict w/current uses.

I'm more interested in some of the features...like the ability
to separate input datums w/NUL's read by readAR/mapfl seems perfect for this
type of thing.

The opposite side -- printing things out with a nul or NUL-sep
is missing -- but WOULDN'T be missing if bash's implementation followed
the man page.

I.e. If "$v" could really store '"\x00"\n', then another huge lump would
"go away".


I would steer away from > where "punc" is some third punctuation
characters.  We already have > for FD duplication and closing,
where "word" can be an integer FD to duplicate, or - to mean close.


---
Except that would be the wrong thing to do.

You have:
1) optional 'fd' to work with -- that, currently has to be blank or evaluate to 
an integer; so (after quote removal) it has to start with [$0-9].

2) "direction": into something (<); out of something (>);
a not-so-useful way to open a file with RW access(<>) (but no way to 
seek, skip, truncate or use use mapfile/readarray because

of the embedded nul problem;  the "through" something (|).

3) then comes the 2nd operand which says "word", but where is
'word' defined.  Is that any varname or funcname or alias name? 
Words normally don't start with "$" though varnames do.
Words, also, don't normally start with '(' but a subprocess 
can -- e.g. process substitution).


Since you are wanting to retain field '1' and field '2', it seems
a specification for '3' indicating array usage  would be a perfect
fit for the language... Since '@' is already used as a subscript meaning
all of an array -- using '@' before a 'word'  would store multiple,
separated, 'lines' as members in the array specified by 'word' & after the
'@'.

Something that parallels the syntax would be a better solution, I 
believe,  than coming up with something that looked entirely unrelated,

no?

Re: redirecting a file descriptor to an array variable? Possible? How? RFE?

[Linda Walsh]

Greg Wooledge wrote:

On Wed, Nov 18, 2015 at 10:46:57AM -0800, Linda Walsh wrote:

One thing that seems to be another problem.  Greg called
my attention to another printf format bug:

 %q causes  printf  to output the corresponding argument in 
 a

format that can be reused as shell input.

Which it doesn't when nuls are involved.


An argument cannot contain a NUL byte.  So it's moot.

---
As in:

printf '"%c"\n'  $'\x00'|hexdump -C

  22 00 22 0a   |".".|

	I see 2 arguments being passed to printf. That puts the 
NUL byte between 2 double quotes and terminates the line w/a newline.


What I'm pointing out is that a NUL byte can be used
and processed as an argument in some cases.  The fact that it doesn't work
in most places, I will agree, is a short-coming.  However one cannot
categorically say that a NUL byte can't be used as an argument.  Solving
other places where it doesn't work might make it so that it *would* work...

Maybe a "-b" (binary) option could be added to 'declare'?

Re: redirecting a file descriptor to an array variable? Possible? How? RFE?

[Linda Walsh]

Greg Wooledge wrote:

On Thu, Nov 12, 2015 at 04:45:49PM -0800, Linda Walsh wrote:

I'm coming up with a null-record in my brain when I think
about this:  I'd like to be able to record stdout and stderr
without using any temp files into bash array files, AND
record the status of the command executed.


There is no way to do this currently.  Storing stdout and stderr
separately requires at least one temp file.  (Actually in the most
general case, where those output streams may contain NUL bytes,
it requires two temp files, because you can't store arbitrary data
streams in bash variables at all.)


In the simple case, presume there are no NUL bytes.

In the more complex case, presume
readarray has a "-0" switch which uses null bytes (trimmed)
as line separators.  


Note that printf *can* print out nul bytes:
printf "\"%c\"\n" $'\x00' |hexdump -C  

  22 00 22 0a   |".".|
0004

But contrary to the manpage under printf:
" The -v option causes the  output  to  be
 assigned  to  the  variable var rather than being printed to the
 standard output. "

printf does not print the output of the above into a var instead
of stdout.  


Seems like that would be a bug?

If I am in a locale using UTF-16, there will be lots of 'NUL'
chars if I view that output as binary -- so there has to be one
or more ways to encode NUL bytes in a var such that they don't
terminate the string.  


Nevertheless, printf doesn't follow the documented behavior on the
manpage and print stdout to the variable.

If the issue of varables being able to contain any sequence of
binary bytescould be solved, it might make it easier to solve
the problem of streaming std[out/err] to a variable or an array.

No?

redirecting a file descriptor to an array variable? Possible? How? RFE?

[Linda Walsh]

I'm coming up with a null-record in my brain when I think
about this:  I'd like to be able to record stdout and stderr
without using any temp files into bash array files, AND
record the status of the command executed.

Something along the lines of

Array stderr=() stdout=()
runprog () {
   "RUN" cmd 2>@stderr   1>${stdout[@]}; status=$?
 ##^-pseudo or ^-syntax
   return $status
}



If there is too much output to fit into an array,
the same thing would happen as if one did something like
readarray ARRAY < <(while :;do echo "lots of lines";done)
(i.e. it would eventually overflow something).

---
But for the most part, I'm thinking smaller output commands
where one just wants to catch fd 1 & 2 and status and have
those returned in parameterized arrays and a return value of
the cmd or function.

I  cobbled together something that returned stderr/stdout, and it
worked for the simple cases I tried, but it seemed awfully overblown
and inflated for what I was trying to do.
Then tried to pass back status, and the whole thing fell apart -- at
which point, I was thinking -- there's gotta be an easier way that I'm
missing -- I seem to be making this a whole heck of a lot more complicated
than it needs to be.

One thing to remember -- I don't want to use data structures external
to bash (like external files), as I can't be sure where I'd be allowed
to write them without going through some undesirable kludges...

Thanks!
-l

Dynamic variable failure & equiv-const strings compare unequal

[Linda Walsh]

Greg Wooledge wrote:

 Also I think you are completely misrepresenting the dynamic variable
 scope system that bash uses.  Variables are not just global or local.
 There's an entire stack of them.  When you reference a
 variable (let's say i) inside a function, bash searches up
 through the call stack looking for a variable named
 i until it finds one.


  So where in the manpage is the behavior documented -- I'd like
to reread it to ensure I don't have any misconceptions.

   The reason being is that out of 4 tests below,
2 fail.  It appears test 2 failing is a failure of the
dynamic scoping system.

   Test 3 was meant to be an extension, as I have some
code where I change a member in a hash, and try to compare
the final result with a known correct value, but as in test-2
all the other members of the hash disappear

Notice that 'parts' is defined at the global level as an array.
in tst0 & tst1, I locally change the value of IFS, and
call "assignparts" that splits the value "$ip" based on
the "localized" values of IFS -- tst0 + tst1 work as I'd
expect with dynamic scoping.  Since parts was declared 'global',
the different splitting caused by differing 'IFS' values is
reflected in the global.

However, in tst3, I only change 1 member in the parts
array, via "parts[1]=222".

before the assignment, parts=(1 .2 .3 .4)
but after the assignment "parts[1]=222",
the entire array was wiped.

I.e. test output was:
Case 2 got/Expected:
"222"
"1\ 222\ .3\ .4"

instead of it only replacing the 2nd member of parts,
the entire array was wiped.

The same happens if parts is a 'hash' -- trying
to only replace 1 member of the hash results in the
entire hash being wiped.

I don't see how this is correct function for
"dynamically scoped" variables -- which is why
I need to reread (or read) the manpage descriptions of
how dynamic vars are *supposed* to work, as they don't
seem to be working by "unwinding" the call stack and using
the last definition of 'parts' (which is at the global level).

test 3 --- was supposed to check the values
of the hash field-by-field, but first I thought to check
them when printed out in a specific order as 2 identical
strings.  The strings look identical, yet don't compare equal.
so it seemed pointless to try to break down test 3's output
into a field-by-field comparison when I couldn't even get
the identical strings that contained the fields to compare.

tests 2 & 3 work backward from a program where I try
to change 1 value in a hash (vs. the array in test2), but
end up with the local assignments of values->[keys] not
being propagated, at all, to the global frame where the has
is declared.

I.e. these examples of dynamic variable declarations and usage
don't seem to work as I understand the concept, but there is
nothing about bash's 'dynamic variables' in the manpage. 



-l



-
#!/bin/bash

shopt -s expand_aliases ; alias my=declare
alias array='my -a' int='my -i'

ip=10.20.30.40
array  parts=()  
array answers=(  '10 20 30 40'   '1 .2 .3 .4'  '1 222 .3 .4' 
'192.168.0.1/24::{ [ipaddr]="192.168.0.1/24" [address]="192.168.0.1" 
[prefixlen]="24" ; }'

)


array addr_fields=(ipaddr address prefixlen)
   
assignparts() { parts=( $ip ) ; }


tst0 () { my IFS=. ; assignparts ; echo -E "${parts[@]}"; }
tst1 () { my IFS=0 ; assignparts ; echo -E "${parts[@]}"; }
tst2 () { parts[1]=222; echo -E "${parts[@]}" ; }

tst3 () {
my ipaddr="$1"; shift;
int status=0
my pat='^([^/]+)/([0-9]+)\s*$'
my address prefixlen
if [[ ! $ipaddr =~ $pat ]]; then
  echo >&2 "Error in ip/netsize format: \"$ipaddr\""
  status=1
else
  address=${BASH_REMATCH[1]}
  prefixlen=${BASH_REMATCH[2]}
  my out=""
  for flds in "${addr_fields[@]}"; do
out+="[$flds]=\"${!flds}\" "
  done
  printf "{ %s; }" "$out"
fi
return $status
}



int passno=0 tests=0
my fmt="Case %d got/Expected:\n \"%q\"\n \"%q\"\n"


testor () {
int tstno
my out=""
for tstno in {0..3}; do
  tests+=1
  exp="${answers[tstno]}"
  if [[ $exp =~ :: ]]; then
my args="${exp%::*}"
exp="${exp#*::}"
out="$(tst$tstno $args)"
  else
out="$(tst$tstno)"
  fi
  if [[ $out != $exp ]]; then
printf >&2 "$fmt" "$tstno" "$out" "$exp"
continue
  fi
  passno+=1
done
}

testor
echo "Passed $passno/$tests tests."


output:
Case 2 got/Expected:
"222"
"1\ 222\ .3\ .4"
Case 3 got/Expected:
"\{\ \[ipaddr\]=\"192.168.0.1/24\"\ \[address\]=\"192.168.0.1\"\ 
\[prefixlen\]=\"24\"\ \;\ \}"
"\{\ \[ipaddr\]=\"192.168.0.1/24\"\ \[address\]=\"192.168.0.1\"\ 
\[prefixlen\]=\"24\"\ \;\ \}"

Passed 2/4 tests.

The outputs for case 3 look identical -- I was using %s to print
them out, but switched to "%q", to ensure no hidden chars...

case 2 --
???  Why didn't it only change the 1 member?

Re: Dynamic variable failure & equiv-const strings compare unequal

[Linda Walsh]

Oleg Popov wrote:

On Thu, Oct 22, 2015 at 03:01:06AM -0700, Linda Walsh wrote:

[cut]
I.e. test output was:
Case 2 got/Expected:
"222"
"1\ 222\ .3\ .4"
[cut]


You didn't initialize the array. By the time you do "parts[1]=222" it's 
still empty. And in your previous message you tried to initialize it in 
a subshell. Variables don't retain their values after returning from 
subshells.  


	I was testing if dynamic scoping included subshells, I 
didn't think so, but that doesn't mean I don't test it.  I removed

it though, as it confused the example.

ip and 'parts' are both initialized in global.

testor calls (tst0, tst1, tst2 & tst3).

tst0 & tst1 both call "assignparts" which uses the global
value of $ip to set the global value of parts.  I.e. since
neither "ip" nor 'parts' are declared inside of any of the 
functions, they should use the top-level global values, no?


tst2, using the last global value set in tst1, only tries to
change 1 value in 'parts'... i.e. why would 'ip' reference the
global value of 'ip', but not parts?

ip and parts are declared at the same scope (global), so why
wouldn't the global 'parts' be initialized as well?



Bash uses unquoted characters on the right side of == and != as wildcard 
patterns. For example, [ipaddr] in $exp means "any of i, p, a, d, r".  
You should quote the right operand.

---
	Ahh... or if I quote both sides 
using 'printf "%q"' first, that should do the same, no?


So, ok, I get that one -- but the first looks sketchy,
as ip and parts are both, only defined at the global level, 
thus my expectation that just like it used the global value of

'ip' for tst0 & tsts1 -- it should have stored the split version
of it in the global value of parts...  I really don't get why
it would use the global value as a dynamic in 1 case but not
the other...?

Re: Dynamic variable failure & equiv-const strings compare unequal

[Linda Walsh]

Oleg Popov wrote:
$(...) is a subshell. Variables cannot be passed back from a subshell, 
no matter how and where they are declared.

---
Um... oh.. in testor, still calling that way.  I
missed that.

This is even more annoying for passing back results
than I thought.  Grrr.

Re: language inconsistency(wart) & RFE

[Linda Walsh]

Greg Wooledge wrote:

On Sat, Oct 17, 2015 at 08:55:31AM -0700, Linda Walsh wrote:

If I needed a way to declare something global, yes...
But what I am wanting is a way to allow changing the defaults
of the implicit variable creation (which could still be
explicitly declared with "-g" if one wanted their result to be
made global.


So you are basically saying you want all of your function variables
to be local

---
No...  only ones where 'shopt -s auto_local' was in effect.

but you are too lazy to write 'local i j k' and you want

bash to do it for you?


local i j k doesn't define j as an array or k as a hash
or g (w/-g ) as a global or any case modification vars.
Not to mention your statement doesn't work:

function nint {
 local i j k
 for ((i=0; i<10; ++i)) ; do j+=i; k+=j ;done
 echo "i=$i, j=$j, k=$k"
}


nint && echo "($i, $j, $k)"

i=10, j=ii, k=jj
(, , )

You declared them all the same, but at the end of the
function, j and k do not have integer values.

Trying to init them:
function nint {
 local i=0 j=0 k=0

 for ((i=0; i<10; ++i)) ; do
 j+=i
 k+=j
done
my -p i j k; echo "i=$i, j=$j, k=$k"
}

nint && echo "($i, $j, $k)"
declare -- i="10"
declare -- j="0ii"
declare -- k="0jj"
i=10, j=0ii, k=0jj
(, , )
---
Now we see that i & j & k all have the same
attributes ... yet only 'i' has an integer value.

If you could declare all the vars in 1 statement w/different types:

declare -i INT -a ARRAY -A HASH
---
That would be a huge improvement.  But instead,if you
use a 'for' statement, without pre-declaring all the vars
used, you end up leaking variables:

function nint {

for ((i=0; i<10; ++i)) ; do
j+=i   
k+=j

done
my -p i j k; echo "i=$i, j=$j, k=$k"
}

nint && echo "($i, $j, $k)"

declare -- i="10"
declare -- j="ii"
declare -- k="jj"
i=10, j=ii, k=jj
(10, ii, jj)
---
"leaking variables" into the external environment
is almost always considered bad-practice.

The default that bash encourages with its default behavior
is for all implicitly used vars in a function to be
leaked to the global level.  From a program maintenance
and development standpoint, having such a horrible default
with no way to override it just seems really icky.













Also I think you are completely misrepresenting the dynamic variable
scope system that bash uses.  Variables are not just global or local.
There's an entire stack of them.  When you reference a variable (let's
say i) inside a function, bash searches up through the call stack
looking for a variable named i until it finds one.

Since functions cannot return values to their callers, the entire system
of "put values into an upper-scope variable so the caller can see them"
would break if your proposal of automatic localization were to be
adopted.


# Pick unbiased random number from 0 to N-1 ($1 = N)
# Returns value in variable r.
rand() {
  local max=$((32768 / $1 * $1))
  while (( (r=$RANDOM) >= max )); do :; done
  r=$(( r % $1 ))
}

foo() {
  local r
  rand 6
  echo "I rolled $((r+1))"
}

foo
# r is not visible here


Under your proposal, the variable r which is defined locally in foo, and
is up-scope-visible to rand (so that rand can put a return value into
it), would also be defined locally within r, so there would be no way to
return a value from rand to foo.

(If you want to attack "language warts", start with the inability to
return values from functions to their callers!)

Re: language inconsistency(wart) & RFE

[Linda Walsh]

Greg Wooledge wrote:
--- please note locations of 'auto_local'

# at beginning
shopt -s auto_local
# Pick unbiased random number from 0 to N-1 ($1 = N)
# Returns value in variable r.
rand() {
  shopt -u auto_local
  local max=$((32768 / $1 * $1))
  while (( (r=$RANDOM) >= max )); do :; done
  r=$(( r % $1 ))
  shopt -s auto_local
}

foo() {
  local r
  rand 6
  echo "I rolled $((r+1))"
}

foo
# r is not visible here


Under your proposal, the variable r which 

is defined locally in foo, and

is up-scope-visible to rand (so that rand can put a return value into
it), would also be defined locally within r, 

so there would be no way to

return a value from rand to foo.


Under my proposal, it is a shopt option that can be turned
on or off.  When it is off, things behave as they do no.  When it is
on, vars w/o a specific declarator, or declarator+"-g switch" would
be default to be local. 




(If you want to attack "language warts", start with the inability to
return values from functions to their callers!)

---
They can -- unfortunately, it is by printing to, usually,
to stdout, and reading it in the parent (I agree this is a rather
roundabout and inefficient way to send return vals... 


I've thought about that, but running into inconsistencies in the
dynamic variable propagation.  It's hard to define an
"alternate syntax" when the primary syntax doesn't consistently
yield the same answer.  So need to figure that out first.

Hopefully the toggling of 'auto_local' isn't hard to understand:
it's like adding a -i or -l flag to var declaration -- doesn't
affect current values in vars, but only new assignments.

Re: language inconsistency(wart) & RFE

[Linda Walsh]

Here is an example of what I consider to be
strange and/or inconsistent behavior (  tests 
2 & 3 fail).


test 3's failure is a real puzzler:

-
#!/bin/bash

shopt -s expand_aliases ; alias my=declare
alias array='my -a' int='my -i'

dmp () { printf "%q\n" "${*:?}" ; }

ip=10.20.30.40
array  parts=()
(IFS=.; parts=( $ip ) )
array answers=( 
 '10 20 30 40'  
 '1 .2 .3 .4'
 '1 222 .3 .4' 
 '192.168.0.1/24::{ [ipaddr]="192.168.0.1/24" [address]="192.168.0.1" [prefixlen]="24" ; }'

)
#


array addr_fields=(ipaddr address prefixlen)
 
assignparts() { parts=( $ip ) ; }


tst0 () { my IFS=. ; assignparts ; echo -E "${parts[@]}"; }
tst1 () { my IFS=0 ; assignparts ; echo -E "${parts[@]}"; }
tst2 () { parts[1]=222; echo -E "${parts[@]}" ; }

tst3 () {
 my ipaddr="$1"; shift;
 int status=0
 my pat='^([^/]+)/([0-9]+)\s*$'
 my address prefixlen
 if [[ ! $ipaddr =~ $pat ]]; then
   echo >&2 "Error in ip/netsize format: \"$ipaddr\""
   status=1
 else
   address=${BASH_REMATCH[1]}
   prefixlen=${BASH_REMATCH[2]}
   my out=""
   for flds in "${addr_fields[@]}"; do
 out+="[$flds]=\"${!flds}\" "
   done
   printf "{ %s; }" "$out"
 fi
 return $status
}



int passno=0 tests=0
my fmt="Case %d got/Expected:\n \"%q\"\n \"%q\"\n"


testor () {
 int tstno
 my out=""
 for tstno in {0..3}; do
   tests+=1
   exp="${answers[tstno]}"
   if [[ $exp =~ :: ]]; then
 my args="${exp%::*}"
 exp="${exp#*::}"
 out="$(tst$tstno $args)"
   else
 out="$(tst$tstno)"
   fi
   if [[ $out != $exp ]]; then
 printf >&2 "$fmt" "$tstno" "$out" "$exp"
 continue
   fi
   passno+=1
 done
}

testor
echo "Passed $passno/$tests tests."


output:
Case 2 got/Expected:
"222"
"1\ 222\ .3\ .4"
Case 3 got/Expected:
"\{\ \[ipaddr\]=\"192.168.0.1/24\"\ \[address\]=\"192.168.0.1\"\ 
\[prefixlen\]=\"24\"\ \;\ \}"
"\{\ \[ipaddr\]=\"192.168.0.1/24\"\ \[address\]=\"192.168.0.1\"\ 
\[prefixlen\]=\"24\"\ \;\ \}"
Passed 2/4 tests.

The outputs for case 3 look identical -- I was using %s to print
them out, but switched to "%q", to ensure no hidden chars...

case 2 --
???  Why didn't it only change the 1 member?

Re: Design question(s), re: why use of tmp-files or named-pipes(/dev/fd/N) instead of plain pipes?

[Linda Walsh]

Bob Proulx wrote:

Linda Walsh wrote:

Where does the OS claim it is a file descriptor?

---

1) in the kernel documentation:
file 'devices.txt:
/dev/fd /proc/self/fd   symbolicFile descriptors

2) ls -Ll shows the type of what the symlink points to:

ls -Ll /dev/fd/

total 0
crw--w 1 law tty  136, 0 Oct 21 19:49 0
crw--w 1 law tty  136, 0 Oct 21 19:49 1
crw--w 1 law tty  136, 0 Oct 21 19:49 2
---

and 3)

YOU claim it is a descriptor further down in this note:

  tmpname=<(sort /dev/null); ls -l "$tmpname"
ls: cannot access /dev/fd/63: No such file or directory

But as soon as the first commend is finished at the ';' then the
descriptor will be closed and the path will evaporate.  Therefore the

 ^

path won't exist for the second command.

---
If you close a file-descriptor to a normal file
it doesn't disappear.  If it was deleted via 'rm' the name disappears
immediately, and the only access to it is through the descriptor.

In this case, closing the descriptor deletes the temporary
name in the process's 'file descriptor' "/fd/".  Normal filenames
are not "auto-deleted" when you close an I/O handle to them.



But if you use it for the ls command itself then it exists for that
command.

  ls -l <(sort /dev/null)
lr-x-- 1 rwp rwp 64 Oct 19 15:56 /dev/fd/63 -> pipe:[102059434]


But only as a pointer to something one can do I/O on.
You can't set any file attributes or metadata on "pipe:[]" 
It's not a real file somewhere.

Re: Design question(s), re: why use of tmp-files or named-pipes(/dev/fd/N) instead of plain pipes?

[Linda Walsh]

Greg Wooledge wrote:

On Sat, Oct 17, 2015 at 05:43:15PM -0700, Linda Walsh wrote:

Chet Ramey wrote:

I think you're missing that process substitution is a word expansion
that is defined to expand to a filename.

-
??? I've never seen a usage where it expands to a filename and
is treated as such.


A simple example:

diff -u <(sort file1) <(sort file2)


You claim <(sort file1) is a filename?


touch a
mv <(sort a) <(sort a) 
mv: cannot move ‘/dev/fd/63’ to ‘/dev/fd/62’: Operation not permitted 


The OS claims that <() generates a file descriptor -- not a filename.

"<()" is not the same, nor equivalent to a filename.  It represents
a *file descriptor* that you can perform stream I/O on.  


It's the same with 'FILENAME'a used with "read < FILENAME >FILENAME"


read < FILE > FILE

-bash: FILE: No such file or directory   # no file is created by '>'

echo content >FILE
read mycontent < FILE >FILE
echo $mycontent # empty: file truncated before read
a=<(sort a); chmod +x "$a"

chmod: cannot access ‘/dev/fd/63’: No such file or directory

echo aaa > a
read var <(sort a) && echo "var=$var" >a


## the above hangs: if <(sort a), was a filename, read would
## have read 'aaa' ## from it, and returned immediately.  
## Then the 'echo' command would have put 'var=aaa' 
## in 'a': instead:

cat a

aaa

The above operations show that <() is not equivalent to a filename.
It only returns a 'path' to a *device* that doesn't have file semantics
and can only be opened for read... and even that doesn't
appear to work all the time.

I observe similar inconsistencies and confusion around
the use of 'dynamic vars' -- not really being global, not really
being local, but supposedly on some dynamic frame, "somewhere",
but not anywhere they've been declared or used.

Re: Design question(s), re: why use of tmp-files or named-pipes(/dev/fd/N) instead of plain pipes?

[Linda Walsh]

Chet Ramey wrote:

On 10/16/15 7:52 PM, Linda Walsh wrote:


As I mentioned, my initial take on implementation was
using standard pipes instead of named pipes (not having read
or perhaps having glossed over the 'named pipes' aspect).


I think you're missing that process substitution is a word expansion
that is defined to expand to a filename.  When it uses /dev/fd, it
uses pipes and exposes that pipe to the process as a filename in
/dev/fd.  Named pipes are an alternative for systems that don't support
/dev/fd.

-
??? I've never seen a usage where it expands to a filename and
is treated as such.

Are you meaning:
readarray foo is creating a process ... that puts the input file on some "/dev/fd/xx", 
and that readarray (or read)

is then "handling that" as though it were a normal file?

That's what you are meaning by process substitution?... 


I wasn't understanding the literalness of the wording...(?)

But read or readarray, read from "file handle"s, not filenames --
as in:
"cat /etc/passwd |readarray foo"

The only conceptual difference between that and
readarray foo < <(cat /etc/passwd) is whether or not the readarray
is done in the parent or the child, .. i.e. 


from a semantic point of view,  how is:

  readarray foo < <(cat /etc/passwd)

different from

  shopt -s lastpipe
  cat /etc/passwd |readarray foo

Is there something in the semantics that would require they
be implemented differently?



While using pipes for here-documents is possible, the original file-based
implementation (and I mean original -- it's almost 28 years old now) is
simple and works.  Inserting another process just to keep writing data to
a pipe (many here documents are larger than the OS pipe buffer size), deal
with write errors and SIGPIPE asynchronously, and add a few pipe file
descriptors to manage, seems like more trouble than benefit.  Files are
simple, can handle arbitrary amounts of data, and don't require an
additional process or bookkeeping.


 But they do require a tempstore that wasn't present in a system-start
bash script I was writing -- kept having problems with the different
bash communication methods all needing some external-fs names to be
present to handle what could have been done in memory.

 BTW, Dealing with the SIGPIPE is why I kept the output pipe open in
the reader process, and use a child-sig handler to close the specific
pipe -- I.e. when I wrote similar code in perl, I kept track of child
processes->pipe pairs, so my foreground process could close the
"writer-pipe" only after the reader-pipe had everything.  Alternatively,
Could use a pair of pipes, and parent could send a message to child
process for more data, or close the pipe) -- a handshake, as it were.


 As far as may docs being larger than the buffer size -- shouldn't be
an issue unless you are trying 2-way communication -- but that has lots
of warnings about each reader waiting for output from the other as
a writer and the need for asynch/non-blocking design.

 Writing to a tmp file requires enough room for the entire tmp doc on
disk.  My tmp right now only has 4.6Available.  If some heredoc tried to
encode the contents of a video, or Blue-ray data disk, and then
something tried to unpack it to tmp first, it would die --- but with
pipes, you don't have to send the whole thing at once -- that requires
basically full space on sender and receiver or 2x the HEREDOC space.  If
it could hold it in memory... my linux box has 2x48G (2 NUMA nodes) or
~96G + ~8G Swap.  So I'd say it really is case dependent -- ideally if
bash was going to be enhanced in this area, it allow the user to set the
defaults to use processes/pipes or tmp files.

 The problem I had with bash & tmps, was that bash scripts started
execution off of 'root' before a separate FS for /tmp was mounted. The
root partition has even less space available on it than /tmp.  So the
use of a /tmp file when /tmp wasn't really available yet was an issue.





here-strings are similar, and there's no reason to have here strings use
a different mechanism than here documents.

There was also a backwards compatibility issue.  Since historical shells
had used temp files for here-documents, there was a set of applications
that assumed stdin was seekable as long as it wasn't a terminal, and it was
important to support them.


Oh?  What POSIX compatible applications?



Yes.  You are missing the fundamental nature of process substitution:
a word expansion that results in a filename.

---
hopefully my q's at top were more on track?



If you think that you can improve the implementation of here
documents and here strings by using a pipe instead of a temp
file, by all means take a shot at it.  You might want to get
ahold of some old Unix `shar' archives to test your implementation
with very large here-documents.


Or, I might read those same shar achives today directly
into memory -- depends on system resources.

Re: language inconsistency(wart) & RFE

[Linda Walsh]

isabella parakiss wrote:


Maybe you can just use this?alias declare='declare -g'


If I needed a way to declare something global, yes...
But what I am wanting is a way to allow changing the defaults
of the implicit variable creation (which could still be
explicitly declared with "-g" if one wanted their result to be
made global.

For example, I use some aliases in documenting the
intended usage of vars...  especially with respect to 
integers, arrays and hashes.  Declaring a hash by calling

it a 'hash' is somewhat generic, but if I call it a 'map', it
is more clear from a documentation standpoint that I am
trying to use something as a way to map names to a value.  


So I will tend to pre-declare many of my vars, but
some are just an annoyance to predeclare.

Like readarray -t config <~/.toolrc -- read a config
file into an array, and then parse it for relevant options &&
then throw the "config array" out when done parsing it.  
Having to "declare" a temporary, _first_, before usage is

a bit backwards...

Obvious example is using throw-away names as
index or iterator vars.  I believe it to be rare, that
people use "for nam in "${names[@]}"; or
"for ((i=0;i<10;++i));" as a means to declare nam or 'i' as
"global variables.  They are  too meaningless for use as
a global, and much more likely to collide with usage of such
'tmp' names in other functions that may be called in a nested
fashion.  


Allowing the option to default to private state,
would take care of accident collisions where one loop uses 'i',
but calls some util function elsewhere that also uses 'i'
and ending up with strange side-effects.  It removes default
behavior to suppress accidental "spooky actions at a distance",
where one or both functions didn't pre-declare 'i' in a func.

It seems to me, that pre-declaring a variable draws
attention to it -- which is more wanted if it is a global,
but drawing attention to 'i' in a forloop by pre-declaring
it makes the code more confusing, as you are drawing attention
to a temporary who's value you usually won't care about after
the function has exit'ed.  


Allowing an option to change this default, allows for
more portable, more reliable and less confusing code to be
written, which would seem like a desirable goal.

language inconsistency(wart) & RFE

[Linda Walsh]

Ok, thinking this from a different way.

shopt -s implicit_vars_local
or
shopt -s localize_func_implicit_vars whatever...

Right now, in a function, you *can* use local in a function
to make a local var.  Thing is, both 'declare' and 'typeset' also
make a *local* var in a function, unless the "-g" switch is used.

I.e. All standard, overt ways (local declare typeset) of creating
a var in a function all result in it being local, BUT, 
(and I think this is an ugly wart), 
any *implicit vars* without local, or the 
misleading declare or typeset, become global.


examples:  In these two for statements, used in functions, 'i'
becomes global:

 for((i=0; i<10; ++i)); do : done
 for i in {1..10}; do : done

And same with 'myarray':
 readarray myarray=$(echo "one";echo "two")

and reads and assignments, and 'lets'
 read ln < <(echo "one"; echo "two")
 ln2="one two"
 read ln3 <<< "one two"

but if this isn't a potential for confusion:

function aa {   

read ln < <(echo "one"; echo "two")
ln2="12"
read ln3 <<< "one two"
ar1=(one two) 
typeset -i ln2=34

typeset -a ar1=(three four)
}

whence aa

aa is a function
aa () 
{ 
   read ln < <(echo "one"; echo "two");

   ln2="12";
   read ln3 <<< "one two";
   ar1=(one two);
   typeset -i ln2=34;
   typeset -a ar1=(three four)
}

aa
declare -p ln ln2 ln3 ar1

declare -- ln="one"
declare -- ln2="12"
declare -- ln3="one two"
declare -a ar1='([0]="one" [1]="two")'

!!!  -- sure looks like I was trying to set the "type" of ln2 
and ar1... boy could that be confusing...  



So, how about a "shopt" 
to declare that **what's implicity declared in funcs, stays in funcs**

maybe shopt -s vegasvars ?.

but seriously -- it's so odd that anything you declare explicitly 
becomes local, while implicit vars default to global --

I know standards and compat must keep it that way... BUT
it would have made more sense to have
implicit vars in a function always be 'local'
and maybe have explicit declarators be global
(which has effectively been done with -g)...but
originally, I also thought it strange that 'declare/typeset'
were equivalent to 'local' inside a function.

This way, you wouldn't have to change any syntax
parsing functions and there certain isn't ANYTHING
that would look like perl, even though perl was
originally designed to be like shell with many shell standard
functions built-in.

???

Design question(s), re: why use of tmp-files or named-pipes(/dev/fd/N) instead of plain pipes?

[Linda Walsh]

In another note about different implementations associated
with the pattern "[read[array]] LVALUE [< $(]+ RVALUE [)]?"...

I gave a rough syntax for how I initially thought process
substitution might have been implemented before I got that
it used named pipes (like /dev/fd/99, for example).

repeated, here (minus blank lines) for ref:

 int savein,saveout;
 int pid;
 dup2(0, savein);
 dup2(1, saveout);
 int inout[2];
 #define stdin inout[0]
 #define stdout inout[1]
 pipe(,O_NONBLOCK);
 dupto(stdin,0);
 dupto(stdout,1);
 setup_childsighandler(to close 0 when child exits);  #semi-trivial to 

write...

 if ($pid=fork()) {  #parent
   dupto(saveout,1);
   shell("readarray -t uservar("xyz")");   #reads from pipe:inout[0]
   #child handler closes 0
   dupto(savein,0);
 } else if (pid==0) {  #child
   close(0);
   shell("echo $'a\nb\nc'");   #output goes out on pipe:inout[1]
   exit(0);
 }
 ##parent continues -- no tmpfiles or named fifo's needed.
 ---


As I mentioned, my initial take on implementation was
using standard pipes instead of named pipes (not having read
or perhaps having glossed over the 'named pipes' aspect). 


And, similarly, when using "read a b c" from "<<<(rvalue-expr)" or
<

Re: updating shopt compat settings to include current version

[Linda Walsh]

Mike Frysinger wrote:

with bash-4.0, new compat options were introduced:
shopt -s compat32
and with bash-4.3, a variable was added:
export BASH_COMPAT=3.2



shopt -p|grep compat43
# (no output)

---
Maybe if you could check if there is a compat setting for
a given version, since compat's aren't usually defined until
the following version --- and THEN, only if 4.4 changed something
that it needed a compat43 option.

AFAIK, compats were introduced because of newer versions being incompat
in some way  to the old.  But if there is no incompat, there wouldn't
be a need for a compat43 option...

Else shopt's options will just grow/grow/grow.
??

Running procs in bg, +Ctl-C, then Ctl-C again after they complete + NL==?

[Linda Walsh]

Was just playing around seeing how the sigint
being ignored in children acted.  wrote short
function:  This behavior seems limited to testing
this as a foregnd function, not a shell script.
I.e, sourcing the function, then
execute the function:

 lnch


function lnch {
 set -o monitor # fyi makes no difference if present
 for((i=0;i++<3;)); do
   declare -i l=1+i
   (sleep $l ; echo "$l exiting")&
 done
 echo Waiting...
 wait
 echo wait finished with $?
}

If I let it finish, nothing interesting.
If I press control C after I see the "wait finished with 0", I get
another cmd prompt as normal, and pressing 'enter' does nothing special

If I press ^C after seeing "Waiting, ...",
but before "wait finished", then
I still see the "num exiting"... as previously mentioned
being normal behavior..., but then:

After the 3 sleepers exit, if I
either
press control-c again, then enter
or
enter, Ctl-c, enter
the shell will exit.

It seems the first control-c messes something up
such that entering another one after the children *really* finish
then enter (NL), exits the fg shell...  Reproducible?
Normal behavior?


 echo ${BASH_VERSION[@]}

4.3.39(1)-release

my confusion on various I/O redirections syntaxes and indirect methods

[Linda Walsh]

Chet Ramey wrote:

On 10/12/15 7:39 PM, Linda Walsh wrote:

Does it also use a tmp file and use process-substitution, or is
that only when parens are present?


Here-documents and here-strings use temporary files and open them as
the standard input (or specified file descriptor) for the command.


read a < <( echo x)

I'm under the impression, uses a tmp file.


Why would you think that? 


	Well, we have 
"<< xxx"
as a HERE DOC using a tmp file, Some time ago, the ability to do 
"multiple assignments" at the same time was added (when I asked how to 
do that) that was told to use:


"read x y z <<< "one two three"

  (which I initially equated to something like:
(x y z)=(one two three)

   That would be like the regular assignment:
xyz=(one two three)

but with the array syntax on the left, would do word
splitting on the left and assign to the individual vars; 
as I was searching for a way to do multiple assignments 
in the same statement).


Then came along a way to do a process in background and end up
with being able to read & process its data in the main (foreground) 
process w/this syntax:


readarray -t foregnd < <(echo  $'one\ntwo\nthree')

Which I envisioned as 
as implemented something like (C-ish example

off top of head using a perl-code I wrote to do the same):

 int savein,saveout;
 int pid;
 dup2(0, savein);
 dup2(1, saveout);

 int inout[2];

 #define stdin inout[0]
 #define stdout inout[1]

 pipe(,O_NONBLOCK);
 dupto(stdin,0);
 dupto(stdout,1);

  setup_childsighandler(to close 0 when child exits);

 if ($pid=fork()) {  #parent

   dupto(saveout,1);
   shell("readarray -t uservar("xyz")");   #reads from pipe:inout[0]
   #child handler closes 0
   dupto(savein,0);

 } else if (pid==0) {  #child

   close(0);
   shell("echo $'a\nb\nc'");   #output goes out on pipe:inout[1]
   exit(0);

 }

 ##parent continues -- no tmpfiles or named fifo's needed.
---

So I didn't realize instead of doing it simply using
native pipes like above, it was implemented some other way.

didn't understand the complexity of the need
for < <( to need a named pipe or fifo)

These examples and concepts came up when I 
was trying to write a bash script that threw

out some error cases like /dev/fd/99 not found... or such
or implemented some other way /tmp/foobar not found...
(both of which I thought were mounted, but wasn't sure --
but then wondered why a tmpfile or named pipe was
needed for EITHER implementation... both can 
be done with native pipes like above, no external

"linux-isms" or files needed -- and I'm certain
the execution speed would be no worse (likely
better than using tmp files or named pipes/fifos).



The documentation clearly says it uses a named
pipe or a file descriptor associated with a /dev/fd filename (which happens
to be a pipe in this case).


yeah with the clear and unambiguous syntax
of :

  <<  xxx
  <<< xxx
  <<< $(echo 'xxx')
  < < (xxx)

I can't imagine why'd I ever have been confused -- or,
given the pipe example above -- why any of the above
had to use "pathname" based io.

So the fact that I get confused about what extra-complex
is used for which syntax isn't that surprising to me --
is it that surprising to you that given the complexities
chosen for implementation, why some people might be
confused about remembering the details of each when
they all could have been done without any pathname
confusions??

Re: ! in PS1 in posix mode

[Linda Walsh]

Chet Ramey wrote:

On 10/11/15 1:37 AM, isabella parakiss wrote:

In posix mode, bash replaces ! in my PS1 with the history number.


http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_03

"The shell shall replace each instance of the character '!' in PS1 with
the history file number of the next command to be typed."


I've never seen that -- even when my prompt has
! in it...maybe it's cause I have histexpand 'off'?

Re: read and env variables + POSIX => SEGFAULT

[Linda Walsh]

(Cc: Chet Ramey... forgot to send it to list...oop)


Chet Ramey wrote:

On 10/10/15 11:01 PM, Linda Walsh wrote:


a= read a <<< x;echo $?

0

declare -p a

declare -- a="x"
#  the manpage claims "one line is read from [the input], and the result
#  is split by words and assigns 1st word to 1st var and so forth, but
#  apparently the reading of 1 line is optional -- though this is 
consistent

#  with the fact that read can be told to read some number of characters
and #  return when the limit is reached.  So technically, read doesn't
"read one line",
#  but read whatever is on 'input' up to 1 line.  (DOC clarification?)


This is terribly wrong.

The command in question is `a= read a <<< x'.

The here-string construct takes the following word and, like a here
document, makes it the standard input to the command.  The standard
input is then a file consisting of a single line: x\n.

It's basically shorthand for

read a <

I wasn't sure if it put the "\n" at the end in a 1-line example.

Does it also use a tmp file and use process-substitution, or is
that only when parens are present?  I.e.

read a < <( echo x)

I'm under the impression, uses a tmp file.

does the read a <<< x
also use a tmp file?

I.e. is

readarray -t a < <( echo -e 'x\ny\n')
declare -p a

declare -a a='([0]="x" [1]="y")'

implemented the same way as

a=(x y)
b=$(printf "%s\n" ${a[@]})
readarray -t ar <<< "${b[@]}"
declare -p a

declare -a a='([0]="x" [1]="y")'

Re: ! in PS1 in posix mode

[Linda Walsh]

Chet Ramey wrote:

On 10/12/15 12:28 PM, Linda Walsh wrote:


Chet Ramey wrote:

On 10/11/15 1:37 AM, isabella parakiss wrote:

In posix mode, bash replaces ! in my PS1 with the history number.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_03


"The shell shall replace each instance of the character '!' in PS1 with
the history file number of the next command to be typed."

I've never seen that -- even when my prompt has
! in it...maybe it's cause I have histexpand 'off'?


It happens in posix mode.

---
How lame!  It's a '!' in a quoted string.  Another example
of posix braindeath -- how long before people realize that the
thing branded 'posix' now isn't the real posix that was started
to *describe* behaviors, not prescribe or proscribe behaviors.
They just bought the name -- like PacBell swallowed SW Bell and
then bought the AT name (not the labs... just the name).  So that
AT today is really the old PacBell in new shoes...

It's like its really tweaky watching a first run anime from
Japan (in japanese) with a Universal logo on it... and words that it is
an NBC company.  (Who is now owned by Comcast).

Oh well when I role my own, I think I usually leave
histexpand out of it.  Having to remember line numbers seems
ridiculous... and I can search by hitting ESC /string (in vi
mode anyway).

-l

Re: posix handling of mapfile: SEGFAULT

[Linda Walsh]

Chet Ramey wrote:

Thanks for the report.  I fixed this, and the fix is in the devel branch.



`mapfile' only takes a single variable name argument, and ignores others.
This is identical to the first example.

---
I see... but it pointed out that declare doesn't
actually die until it touch's b, otherwise we wouldn't
have seen the "a: not found"...  not that you needed
that info to isolate the problem.

Re: ! in PS1 in posix mode

[Linda Walsh]

Chet Ramey wrote:

On 10/12/15 7:02 PM, Linda Walsh wrote:


It happens in posix mode.

---
How lame!  It's a '!' in a quoted string.  Another example
of posix braindeath -- how long before people realize that the
thing branded 'posix' now isn't the real posix that was started
to *describe* behaviors, not prescribe or proscribe behaviors.


Do you realize that this prompt expansion was in ksh since at least
ksh-86?  ksh-88 was one of the Posix base documents, and that
expansion was estabished behavior by the time Posix started its work.

---
Haven't used ksh for some timeI thought the '!' stuff came
from csh?  It seemed so pointless, since having to look up
things by command number I thought, was way too much work... searching
via a string in the line seemed so much faster...

For stuff I typed last week or earlier, I'd use grep on the hist files.

I thought to have all my hist files merged and deduped... but the resulting
master hist file -- .. I used it for a while (somewhat accidently  as it was
renamed to a normal hist file @ around 2-3M in length).

command response was quite slow (took a while to figure out it was
the history updating every command that caused a delay of about
2-3 seconds/command).  
I don't think I have all the kinks in the combining routines worked

out yet either... but certainly wouldn't want to switch to the newest
master @ 6.4M until history updating is sped up a bit... (I'm not
holding my break... at least I can still browse/grep it by tty#)...

Re: read and env variables + POSIX => SEGFAULT

[Linda Walsh]

Geir Hauge wrote:

On Sat, Oct 10, 2015 at 08:01:05PM -0700, Linda Walsh wrote:

# this is odd: 2vars with content for 2:

unset a b
a= b= read a b <<< x y
declare -p a b

declare -- a="x"
declare -- b=""

# -- where did "y" go?


read a b <<< x y
is the same as
read a b y <<< x

If you escape the space, to make it literal instead of syntactical, you'll get
the expected result:

$ a= b= read a b <<< x\ y
$ declare -p a b
declare -- a="x"
declare -- b="y"

---
Was sorta meant as a rhetorical Question in the batch
of examples that seem to have differing behavior -- but especially
in contrast with the POSIX versions that dump core.

Bad-syntax shouldn't yield Segmentation Violations...

posix handling of mapfile: SEGFAULT

[Linda Walsh]

This was in the 2nd half of the note in the read+POSIX=>SEGFAULT,
but think it got missed by focus on the 1st part.

# I was doing some syntax testing and decided to try posix mode
# (as it disallows various  vague or unclear constructs)
# in the working cases yielded the same results, but
# in 2 cases it returned a SEGFAULT.


 b= mapfile b <<< x; echo $?

0

 declare -p b
Segmentation fault (core dumped)  


# another example of bash not returning consistent errno results
# ( |;>|  ducking while smirking)  ok, I vote for disallowing this
# type of result return.

The 2nd flavor:


 b= mapfile b a <<< x
 declare -p a b

bash: declare: a: not found
Segmentation fault (core dumped) 


# odd, this time it dumped on printing 'b'; still not so great
# status returning there...

# hmmm... I'm guessing that fixing these to not return
# segfaults won't be likely to get me better status
# returns for my type -P problems*sigh*


# vitals:

$0 --version

GNU bash, version 4.3.39(1)-release (x86_64-unknown-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


whence bash

bash is /bin/bash
Ishtar:/Torrents/Library/2014Q1> ldd /bin/bash
linux-vdso.so.1 (0x7fff396eb000)
libdl.so.2 => /lib64/libdl.so.2 (0x00300140)
libc.so.6 => /lib64/libc.so.6 (0x003000c0)
/lib64/ld-linux-x86-64.so.2 (0x55da52cbc000)

uname -a
Linux Ishtar 4.1.0-Isht-Van #2 SMP PREEMPT Tue Jun 23 07:52:09 PDT 2015 
x86_64 x86_64 x86_64 GNU/Linux

Re: Want way to run background processes with SIGINT unignored

[Linda Walsh]

Chet Ramey wrote:

On 10/10/15 4:19 PM, Linda Walsh wrote:

Chet Ramey wrote:

On 10/9/15 2:42 PM, Ian Jackson wrote:

However, it would be very easy for bash to provide an option (via `set
-o' perhaps) to disable this behaviour.  That is, to allow SIGINT to
be delivered normally to child processes.

I'm restricting non-standard options to `shopt' to avoid any possible
conflict with future posix changes.

---
Thats a straw-man excuse if I ever heard one: use a "bash_"
prefix to any new options.  But, still, if you think even that
that might collide w/future posix choices, then don't use shopt
at all: use "bashopt" with bash-defined options and usage.
Either way, that should allay your worries on that front.


That's a fairly impressive misunderstanding.  I'm talking about adding
options to `shopt', which is not standardized and specific to bash,
instead of adding the `set -o' option Ian requested because I'm leaving
the `set -o' namespace to Posix.

---
Oh... that explains the itch in the back of my head as
I wrote that... but your wording could be taken either way if
one wasn't remembering shopt was already 'bash-only'.

As far as it being an impressive misunderstanding, thanks, er,
I mean, I've done better!...er...yeah, yeah, yeah...






With such an option, scripts which run on modern systems and which
attempt to parallelise their work, would be able to arrange that ^C
properly cleans up the whole process group, rather than leaving the
background tasks running (doing needless work and perhaps causing
lossage).


I pointed this out as a serious regression in 4.3 as bash
makes steps to really screw up async/realtime children handling
work in:


This really doesn't have anything to do with what Ian is talking about.

-
	no?  they want to run background processes and not have 
the sig handlers mangled (and presumably that means getting the 
signals in real time -- not waiting for the users next keypress!)

Requiring timely delivery of said signals, would be required for
their use case of " a bash script which invokes a
number of subprocesses in parallel and collects the output. ".  If
the foreground process is also waiting for user input to provide
interactivity, will the SIGCHLD events come in when the children
terminate or when the foreground process leaves readline?

Re: read and env variables + POSIX => SEGFAULT

[Linda Walsh]

isabella parakiss wrote:

$ a= read a <<< x   # this creates a variable in the current shell
$ declare -p a
declare -- a="x"

$ b= mapfile b <<< x# this doesn't
$ declare -p b
bash: declare: b: not found


Very good point... more interesting is adding posix mode
to the mix:
# first verify a and b are not defined

set -o posix
declare -p a

bash: declare: a: not found

declare -p b

bash: declare: b: not found

a= read a <<< x;echo $?

0

declare -p a

declare -- a="x"
#  the manpage claims "one line is read from [the input], and the result
#  is split by words and assigns 1st word to 1st var and so forth, but
#  apparently the reading of 1 line is optional -- though this is consistent
#  with the fact that read can be told to read some number of characters and 
#  return when the limit is reached.  So technically, read doesn't "read one line",

#  but read whatever is on 'input' up to 1 line.  (DOC clarification?)

# assigning & trying to read 2 vars but only having content for 1:


a= b= read a b <<< x
declare -p a b

declare -- a="x"
declare -- b=""

# this is odd: 2vars with content for 2:

unset a b
a= b= read a b <<< x y
declare -p a b

declare -- a="x"
declare -- b=""

# -- where did "y" go?

# doing the same thing posix mode (as it disallows various
# vague or unclear constructs) yields mostly the same results
# except for your mapfile case (didn't try any further testing of
# variations because of the results (2 flavors)):


b= mapfile b <<< x; echo $?

0

declare -p b
Segmentation fault (core dumped)   


# another example of bash not returning consistent errno results
# ( |;>|  ducking while smirking)  ok, I vote for disallowing this
# type of result return.

The 2nd flavor:


b= mapfile b a <<< x
declare -p a b

bash: declare: a: not found
Segmentation fault (core dumped)  


# odd, this time it dumped on printing 'b'; still not so great
# status returning...

# hmmm... I'm guess fixing these to not return segfaults won't
# get me any better status returns in 'type -P'

# vitals:

$0 --version

GNU bash, version 4.3.39(1)-release (x86_64-unknown-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 

whence bash

bash is /bin/bash
Ishtar:/Torrents/Library/2014Q1> ldd /bin/bash
 linux-vdso.so.1 (0x7fff396eb000)
 libdl.so.2 => /lib64/libdl.so.2 (0x00300140)
 libc.so.6 => /lib64/libc.so.6 (0x003000c0)
 /lib64/ld-linux-x86-64.so.2 (0x55da52cbc000)

uname -a

Linux Ishtar 4.1.0-Isht-Van #2 SMP PREEMPT Tue Jun 23 07:52:09 PDT 2015 x86_64 
x86_64 x86_64 GNU/Linux

Re: Want way to run background processes with SIGINT unignored

[Linda Walsh]

Chet Ramey wrote:

On 10/9/15 2:42 PM, Ian Jackson wrote:


However, it would be very easy for bash to provide an option (via `set
-o' perhaps) to disable this behaviour.  That is, to allow SIGINT to
be delivered normally to child processes.


I'm restricting non-standard options to `shopt' to avoid any possible
conflict with future posix changes.

---
Thats a straw-man excuse if I ever heard one: use a "bash_"
prefix to any new options.  But, still, if you think even that
that might collide w/future posix choices, then don't use shopt
at all: use "bashopt" with bash-defined options and usage.
Either way, that should allay your worries on that front.



With such an option, scripts which run on modern systems and which
attempt to parallelise their work, would be able to arrange that ^C
properly cleans up the whole process group, rather than leaving the
background tasks running (doing needless work and perhaps causing
lossage).


I pointed this out as a serious regression in 4.3 as bash
makes steps to really screw up async/realtime children handling
work in:

 Subj: REGRESSION: signal handlers not running in real-time
 Url:   https://lists.gnu.org/archive/html/bug-bash/2015-07/msg00138.html

I warned that as computing is going towards more parallelism, his
breakage of real-time handling was going to be more costly and 
harmful for using bash as system-control scripting language.



 2. In the child, reset SIGINT and SIGQUIT to the values found at
shell startup.  That is, uninstall trap handlers.  This is what
most ordinary scripts will want, because they don't want the trap
handler running in both parent and child.  It's the same as is
done for all other signals, and for all signals in non-background
subshells and subprocesses.


The problem of unreliable signals and handling them had (at least)
two branches of solutions: one was a BSD solution adopted by
POSIX, who's behavior you see in bash, that resets signal-handlers
to default values before calling the handler.  


The other solution steered around legacy compatibility
issues and created the 'sigaction' commands.  It was designed
by mostly SysV descendants who were looking to support
reliable and real-time signal handling and delivery, similar
to requirements placed on interrupt handling.  They mostly
left signal alone for compatibility reasons.

Unfortunately the BSD crowd also wanted a sigaction -- 
but built it as a wrapper around signal -- resulting in

incompatible semantics and behaviors. Later lobbing by BSD
in an environment mostly bereft of remaining SysV based
vendors, voted their version into the new POSIX standard.
Linux still kept the SysV standards, because Linus has been
heavy-handed in supporting technical correctness over
popular standard and vendor wants that he felt would 
degrade linux quality.  


However, Gnu has, over the years, coincident with waning
patience toward many of Stallman's rigid attitudes has moved
toward more corporate acceptance via sponsorship.  One of
those conformance points was conforming the new POSIX's
mandatory and changing "compatibility"[sic] standards.
Thus with glibc2, linux users got the broken wrapper
version, by default on linux -- as the wrapper was the
default in any code that had _BSD_ or _GNU_ SOURCE defined
in the headers or code -- which includes bash.

Too many race conditions in the older standard led to the
current reversion to default POSIX behavior of "unsetting"
(or resetting) default SIG handlers before calling the handler.
This lead to the tactical-nuke fix on Signals in 4.3 of
resetting, disabling and postponing all of them.  As chet
stated on this list, there were too many restrictions in what
you could do in a BSD/POSIX/GNU signal handler, so the only
way to wipe out the current and future problems was by 
removing options for reliable signal handling and the

realtime semantics.

It does open up a new, but remote possibility for a different
class of vulnerabilities, with a known problems on
some systems that could result in system hangs and D-of-S
problems.

In short -- not only is only another bad step for 
bash in its usability as a system control script language.

Inconsistent and unreliable error handling is already present
in the current bash, as I mentioned only yesterday in regards
to the unreliability of the 'type' command in finding commands
and designed to be functional, yet, with so many 
examples to the contrary (not to mention, NOT being

POSIX conforming in regards to err-status codes return
on failure.


bash as I posted recently, So it's growing harder and harder
to work around its increasing limitations.

bad handling of error conditions in "type -P'

[Linda Walsh]

I've run across several problems with how type -P returns errors.

1) if a file isn't executable, type returns it anyway in

 ls -l /sbin/scat

-r--r--r-- 1 root root 245663 Nov 19  2013 /sbin/scat
 type -P scat

/sbin/scat

2) if a file is inaccessible, type still returns it an answer for
 the path of an executable named 'scat1':

 ls -l /sbin/scat1

-- 1 root root 245663 Nov 19  2013 /sbin/scat1

 type -P scat1

/sbin/scat1

3) bash "knows better" because it doesn't do this in "posix mode"

4) if it doesn't find the file it returns a status
 code meaning 'EPERM' rather than 'ENOENT'.
 (ENOENT  No such file or directory (POSIX.1))
 This is true in normal  mode or posix mode.

5) if the file is executable for root, it is still return as
 an answer for 'type -P':

 ls -l /sbin/scat2

---x-- 1 root root 245663 Nov 19  2013 /sbin/scat2

 type -P scat2

/sbin/scat2

6) if bash is in posix mode it will find '/sbin/scat2'
 only if the owner is root (good), BUT for a non-root
 user, a return code of '1' is return whether it the
 file exists or not. NOTE: by 'coincidence' on linux,
 1=EPERM, which would be correct for /sbin/scat2, but
 it also returns '1' for the "ENOENT" case.

7) if the file is NOT owned by root, type -P returns
 the alien-owned file (this seems like it would be a security
 risk -- but it is also in the kernel, so bash behaving
 differently, though correct, would be inconsistent with
 the insecure behavior of the kernel:

 ls -l /sbin/ucat2

---x--x--- 1 nobody nogroup 245663 Nov 19  2013 /sbin/ucat2

 type -P ucat2 #(normal user)

# type -P ucat2 #(root user is unprotected)
/sbin/ucat2

Proposals:
1) It seems the non-posix mode should parallel the posix mode in
this case.
2) type should return 'EPERM' if it finds an executable owned
 by someone else that isn't allowed execution by the caller.
3) if no file with any executable bits is set it should return
 status 'ENOENT'.
4) Ideally root would not behave differently from the normal
 user case, since ownership by a non-priviledged user might
 indicate a security problem, HOWEVER, this should be brought
 to the attention of the kernel folks for an explanation why
 root can execute files owned by suspect users.  Perhaps
 Bash being different in this case would be a best course,
 as it is doing a path seach, while in the kernel case,
 it should only be allowed if an absolute path was given
 (with no PATH search).

I regard this as rather broken, as it gives useless, wrong
and insecure answers depending on the case. I also think
bash, having had it's behavior changed due to posix rules should
be using posix standard errno names, doesn't that make sense?

Cheers,
L. Walsh

bad handling of error conditions in "type -P'

[Linda Walsh]

There are several problems with how type -P returns errors.

1) if a file isn't executable, type returns it anyway in

 ls -l /sbin/scat

-r--r--r-- 1 root root 245663 Nov 19  2013 /sbin/scat
 type -P scat

/sbin/scat

2) if a file is inaccessible, type still returns it an answer for
  the path of an executable named 'scat1':

 ls -l /sbin/scat1

-- 1 root root 245663 Nov 19  2013 /sbin/scat1

 type -P scat1

/sbin/scat1

3) bash "knows better" because it doesn't do this in "posix mode"

4) if it doesn't find the file it returns a status
  code meaning 'EPERM' rather than 'ENOENT'.
  (ENOENT  No such file or directory (POSIX.1))
  This is true in normal  mode or posix mode.

5) if the file is executable for root, it is still return as
  an answer for 'type -P':

 ls -l /sbin/scat2

---x-- 1 root root 245663 Nov 19  2013 /sbin/scat2

 type -P scat2

/sbin/scat2

6) if bash is in posix mode it will find '/sbin/scat2'
  only if the owner is root (good), BUT for a non-root
  user, a return code of '1' is return whether it the
  file exists or not. NOTE: by 'coincidence' on linux,
  1=EPERM, which would be correct for /sbin/scat2, but
  it also returns '1' for the "ENOENT" case.

7) if the file is NOT owned by root, type -P returns
  the alien-owned file (this seems like it would be a security
  risk -- but it is also in the kernel, so bash behaving
  differently, though correct, would be inconsistent with
  the insecure behavior of the kernel:

 ls -l /sbin/ucat2

---x--x--- 1 nobody nogroup 245663 Nov 19  2013 /sbin/ucat2

 type -P ucat2 #(normal user)

# type -P ucat2 #(root user is unprotected)
/sbin/ucat2

Proposals:
1) It seems the non-posix mode should parallel the posix mode in
this case.
2) type should return 'EPERM' if it finds an executable owned
  by someone else that isn't allowed execution by the caller.
3) if no file with any executable bits is set it should return
  status 'ENOENT'.
4) Ideally root would not behave differently from the normal
  user case, since ownership by a non-priviledged user might
  indicate a security problem, HOWEVER, this should be brought
  to the attention of the kernel folks for an explanation why
  root can execute files owned by suspect users.  Perhaps
  Bash being different in this case would be a best course,
  as it is doing a path seach, while in the kernel case,
  it should only be allowed if an absolute path was given
  (with no PATH search).

I regard this as rather broken, as it gives useless, wrong
and insecure answers depending on the case. I also think
bash, having had it's behavior changed due to posix rules should
be using posix standard errno names, doesn't that make sense?

Cheers,
L. Walsh

Re: minor language RFE(s)

[Linda Walsh]

Chet Ramey wrote:
These change the syntax of the shell in incompatible ways. 

---
I think I had the feeling of ensuring compatibility as being important
and changing things in incompatible ways . "That's not easily changed w/o potential 
chaos..."


The arithetic `for' command takes arithmetic expressions, not shell
commands, and the `for' command takes a name (identifier), not a
shell command.  Aside from any syntactic sugar (`int', `my'), these
are not consistent with how the shell grammar is formed, and this
isn't a good enough reason to change the grammar that dramatically.

---
Yeah, I think I mentioned that case:

  I've no idea of the difficulty level to do this, but
  was thinking if not too difficult...  and if it is...
  well keep it on a pile of ideas if bash ever got
  refactored such that implementation became easier..?

I understand the problems of working with 10+ year old code
that's been patched through the roof and trying to add _anything_
to the design.  Thus the proposal of keeping the idea around
if bash was ever refactored such that implementing a change like
this wouldn't be a big deal


Andreas Schwab wrote: 

If you want perl you know where to get it.

---
Actually I have no idea where to get a version of perl
that could even process POSIX compatible shell-script, let
alone bash's extensions (not to mention having a mechanism
to write in perl as well).

Perhaps you could enlighten me.

There are things about perl that I don't fancy as well -- 
much of what I want to change in bash or perl involves 
reducing repetitive typing -- as in this minor language RFE,

but perl is way too fossilized with maintenance being dominated
by an even more conservative core team.

So to think that Perl might be extensible to allow for 
greater language efficiency or compatibility is very unlikely, as

they can't even keep new versions of perl backward compatible with
older versions.

Bash has several fairly good reasons for slow evolution -- one
maintenance of POSIX compatibility, and the fact that maintenance and
development are still being coordinated by the original author.
Compared to the perl case that has a constantly changing cast, no
standards to adhere to (though some have been published, they aren't 
followed), and a coordination effort that is reminiscent of trying

to herd cats.

It's obvious that suse is moving away from simplicity with most of
the commands to manage the system being 2-3 times as long as previously.
So I wouldn't expect language efficiency to rate high on your priority 
list.


However, consider this:  

"the number of bugs in code is proportional to the number of source 
lines, not the number of ideas expressed."  --  So being able to
express ideas in 1 line are maybe 2x more reliable than a 
language that forces splitting.


There are several articles on programming language conciseness the
benefits to the programmer of being able to express and write more
concepts in less space and faster time (APL, for example, is thrown
out as being to slow to program in due to needing a specialized 
character set -- so conciseness down to special symbols isn't considered

a good thing).

Besides doing your own search, I thought this article did a fairly
good job of comparing various factors:

http://redmonk.com/dberkholz/2013/03/25/programming-languages-ranked-by-expressiveness/

*cheers*
-l

minor language RFE(s)

[Linda Walsh]

If I am using a var as an index in a for loop
as in:

  for((i=0; i<10; ++i)); do : done

or 2) as an iterator as in

  for i in {1..10}; do : done

**and** such usage is in a function,

the default is to promote 'i' to 'global' status,
which is usually not needed nor desired (that doesn't mean
there aren't cases where one wants it global,
but I'm referring to the most common case).

The common workaround is to put the onus on the user
of 'i' to first declare it as local.  That's not easily
changed w/o potential chaos... however,

I was thinking ... lets say we had 1 or 2 abbreviation
keywords, at least 1 being "int=declare -i",
and ease-of-use "my=declare"

that could then allow the "declare" of the 'for' iterator
as local, in-line.

i.e. instead of predeclaring them w/'declare -i' or 'declare'
one could write:

  for((int i=0; i<10; ++i)); do : done

or 2)

  for int i in {1..10}; do : done
  for my i in {a..z}; do : done

---
Note: "int" and "my" would be
equivalent to "declare -i" and "declare" -- NOT
"local -i" or "local" so that such statements
could be moved in or out of functions w/out removing
the declaration.

Note2: 'my' is for convenience -- so one wouldn't
have to spell out
   for declare i in {a..z}...
which looks a bit ugly IMO...

Anyway was just thinking about this when writing
such loops, and renaming the iterator a few times -- and
having to make sure the declaration was changed in line
with the 'for' usage: requiring 2 separate lines of code
(that may or may not be adjacent to each other) in sync
is always a pain, not to mention occasionally error prone.

Anyway -- just a thought?

I've no idea of the difficulty level to do this, but
was thinking if not too difficult...  and if it is...
well keep it on a pile of ideas if bash ever got
refactored such that implementation became easier..?

indirection on '$1, $2...' ?

[Linda Walsh]

Is it documented that ${!1} won't give you indirection
of "$1"?  I guess I missed that, and sorta expected it
to do indirection the same as any other var, but doesn't
seem to work in 4.3.39.

Should it?  Or why shouldn't it?

tnx...
-l

Re: indirection on '$1, $2...' ?

[Linda Walsh]

Never mind for now... there obviously something
else going on ... (had problems w/it buried in
a function, but in simple echo test, works fine...)
so ... likely operator error...so please ignore this
until  I figure out a proper example...(which I'm not
assuming I will -- i.e. more likely some other problem)...
sorry for line noise.

Linda Walsh wrote:

Is it documented that ${!1} won't give you indirection
of "$1"?  I guess I missed that, and sorta expected it
to do indirection the same as any other var, but doesn't
seem to work in 4.3.39.

Should it?  Or why shouldn't it?

tnx...
-l

Re: remaking bash, trying static, glibc refuses static...?

[Linda Walsh]

I sent this in response about 12 hours ago but haven't seen it
come though on the list, so thought I'd resend it.



 Original Message 
Subject: Re: remaking bash, trying static, glibc refuses static...?
Date: Wed, 19 Aug 2015 14:55:24 -0700
From: Linda Walsh 
To: bug-bash@gnu.org
References: 55d106b2.9000...@tlinx.org 20150817044932.GB1584@vapier 
55d396ef.8030...@tlinx.org 20150819031852.GU1584@vapier 55d408e4.9000...@tlinx.org 
20150819145815.GF1584@vapier



Mike Frysinger wrote:

On 18 Aug 2015 21:41, Linda Walsh wrote:

Mike Frysinger wrote:

On 18 Aug 2015 13:34, Linda Walsh wrote:
(2) it's using the nss system which lets people drop modules into the system
at anytime and change the overall lookups to use that.  statically linking a
specific subset would block that ability.

---
The linux kernel is a perfect example of a statically linked program that
can dynamically load plugins to provide authorization data from external
sources.  Static doesn't mean you can't support 3rd party plugins/libs --
like LDAP.


which is what the current glibc code already does and why you get a warning
when you link things staticlly.  i'm not sure how you can argue both sides
of the fence on this.
-mike

---
The current glibc code doesn't allow static linking (apart
from a special static glibc build that I'd have to build myself).  If the
kernel can be linked statically, and yet load 3rd party plugins dynamically,
why can't programs be linked statically with glibc (this usually pops up as
an Error -- though that could be because many or most build procedures treat
warnings as Errors).  Also, you don't need a dynamic linker at run time
to load plugins -- but glibc does because it patches addresses in the code
to make direct calls into a dynamically loaded lib transparent.  With the kernel
it performs callouts to 'helpers' -- usually giving 1 key to a helper with
the helper returning the associated value(s).  It doesn't require dynamic
linking -- but a helper script.

Examples
cifs.idmap (8)   - Userspace helper for mapping ids for Common Internet F...
cifs.upcall (8)  - Userspace upcall helper for Common Internet File Syste...
cifscreds (1)- manage NTLM credentials in kernel keyring
getcifsacl (1)   - Userspace helper to display an ACL in a security descr...
idmapwb (8)  - winbind ID mapping plugin for cifs-utils
pam_cifscreds (8)- PAM module to manage NTLM credentials in kernel keyring
setcifsacl (1)   - Userspace helper to alter an ACL in a security descrip...

None of those are required to be present when the kernel runs and
the kernel doesn't die if something that would use one of those helpers
is used but it isn't there. 


There are two types of dynamic linking... one where all libs must be
present when the program loads, the other -- linux makes far too little
use of.  I got spoiled at SGI, as they used runtime dynamic linking
to support different feature sets with different products.  


If you have an admin tool to add users, it might try to dlopen various
independently offered products -- if they are not there, the program
didn't show them on the menu -- it wasn't an error to have a dl-failure as
it usually is w/glibc.  Right now, if you try to load 'vim/gvim' on opensuse,
you also have to have perl libified and openable at program invocation.

You can't edit w/o the perl lib installed, but gvim doesn't require perl 
any more than it requires ruby, tcl or python that it can script with if

such libs are present.  If the libs are not present -- the core feature(s)
of vim are still there -- you can edit.  You may not be able to use alternate
scripting languages while you edit, but you can at least start the editor
and use it's standard features.

W/glibc, if you wanted to do the same, you'd have it link and programs 
runnable with whatever was statically built in -- just like gvim --

if I don't have ruby/tcl/whatever installed, it says the language isn't
available.  glibc could return a list of available modules for a specific
function and return soft-errors for missing components, with things
like falling back to /etc/passwd /| shadow being hard coded alternatives.

But even if the dlopen call *can* be done right, I've yet to see a
linux prog do it right and dynamically configure itself -- try an ldap
auth w/no ldap plugin script? -- get a permission denied.  glibc could use
scriptable plugins like the kernel, but uses the more arcane dlopen for
runtime loading -- that most projects get wrong -- and fall back to 
load-time (non-lazy) loading of dynamic libs.


If you wanted to convince me glibc could be distributed statically, 
you could have it interface (as an example or proof of concept) with

the userspace helpers the kernel uses for alternate authentication and
permission mechanisms.  But with the current mechanisms, if I'm not using
NIS or LDAP, but winbind for login, I still need to have the NIS
and LDAP libs available to run login

Re: -e does not take effects in subshell

[Linda Walsh]

Chet Ramey wrote:

The earlier spec had -e only exit a script if a *simple* (external)
command failed.  It didn't include builtins nor functions.


This is not; builtins and functions are simple commands.

---
The builtins are _complex_ binary blobs that replace external commands.

Functions are a collection of many commands.  They are not a single,
simple statement.  I remember using their return value in some cases.

With the change, I couldn't run a func and have it return the
value in $? (1 byte, I know) -- but about 128x as flexible 
as ok/die (i.e. any non-zero value triggers the behavior

now, but before, it didn't).

My simplistic view was that -e was there to auto-exit if an external
command failed because they are out of your control.  But if 
I write a function -- I design the behavior.  Even if I designed in

a bug -- it's still my code that has the problem not some
-_e_xternal command

  cf. perl's option autodie -- you can say you want the open
builtin to just die on errors, then for simple scripts you don't have
to put extra code to handle each problem.  I.e. -- it's not really 
something you want in production code, but I write far more throw
away quick and dirty scripts than production ones.  


I tried to write a more complex bash script one time -- that met my
expectations.  W/o error testing the code was about 150 lines.  With
error testing and _helpful_ error messages it was over 1000.
But that's the difference between quickdirty vs. production.

-e was useful for QD, IMO

Re: remaking bash, trying static, glibc refuses static...?

[Linda Walsh]

Mike Frysinger wrote:

On 18 Aug 2015 21:41, Linda Walsh wrote:

Mike Frysinger wrote:

On 18 Aug 2015 13:34, Linda Walsh wrote:
(2) it's using the nss system which lets people drop modules into the system
at anytime and change the overall lookups to use that.  statically linking a
specific subset would block that ability.

---
The linux kernel is a perfect example of a statically linked program that
can dynamically load plugins to provide authorization data from external
sources.  Static doesn't mean you can't support 3rd party plugins/libs --
like LDAP.


which is what the current glibc code already does and why you get a warning
when you link things staticlly.  i'm not sure how you can argue both sides
of the fence on this.
-mike

---
The current glibc code doesn't allow static linking (apart
from a special static glibc build that I'd have to build myself).  If the
kernel can be linked statically, and yet load 3rd party plugins dynamically,
why can't programs be linked statically with glibc (this usually pops up as
an Error -- though that could be because many or most build procedures treat
warnings as Errors).  Also, you don't need a dynamic linker at run time
to load plugins -- but glibc does because it patches addresses in the code
to make direct calls into a dynamically loaded lib transparent.  With the kernel
it performs callouts to 'helpers' -- usually giving 1 key to a helper with
the helper returning the associated value(s).  It doesn't require dynamic
linking -- but a helper script.

Examples
cifs.idmap (8)   - Userspace helper for mapping ids for Common Internet F...
cifs.upcall (8)  - Userspace upcall helper for Common Internet File Syste...
cifscreds (1)- manage NTLM credentials in kernel keyring
getcifsacl (1)   - Userspace helper to display an ACL in a security descr...
idmapwb (8)  - winbind ID mapping plugin for cifs-utils
pam_cifscreds (8)- PAM module to manage NTLM credentials in kernel keyring
setcifsacl (1)   - Userspace helper to alter an ACL in a security descrip...

None of those are required to be present when the kernel runs and
the kernel doesn't die if something that would use one of those helpers
is used but it isn't there. 


There are two types of dynamic linking... one where all libs must be
present when the program loads, the other -- linux makes far too little
use of.  I got spoiled at SGI, as they used runtime dynamic linking
to support different feature sets with different products.  


If you have an admin tool to add users, it might try to dlopen various
independently offered products -- if they are not there, the program
didn't show them on the menu -- it wasn't an error to have a dl-failure as
it usually is w/glibc.  Right now, if you try to load 'vim/gvim' on opensuse,
you also have to have perl libified and openable at program invocation.

You can't edit w/o the perl lib installed, but gvim doesn't require perl 
any more than it requires ruby, tcl or python that it can script with if

such libs are present.  If the libs are not present -- the core feature(s)
of vim are still there -- you can edit.  You may not be able to use alternate
scripting languages while you edit, but you can at least start the editor
and use it's standard features.

W/glibc, if you wanted to do the same, you'd have it link and programs 
runnable with whatever was statically built in -- just like gvim --

if I don't have ruby/tcl/whatever installed, it says the language isn't
available.  glibc could return a list of available modules for a specific
function and return soft-errors for missing components, with things
like falling back to /etc/passwd /| shadow being hard coded alternatives.

But even if the dlopen call *can* be done right, I've yet to see a
linux prog do it right and dynamically configure itself -- try an ldap
auth w/no ldap plugin script? -- get a permission denied.  glibc could use
scriptable plugins like the kernel, but uses the more arcane dlopen for
runtime loading -- that most projects get wrong -- and fall back to 
load-time (non-lazy) loading of dynamic libs.


If you wanted to convince me glibc could be distributed statically, 
you could have it interface (as an example or proof of concept) with

the userspace helpers the kernel uses for alternate authentication and
permission mechanisms.  But with the current mechanisms, if I'm not using
NIS or LDAP, but winbind for login, I still need to have the NIS
and LDAP libs available to run login. 


Right now it's like most progs require static linking at *load time*,
which is the least flexible linking (programs don't auto-config down
services that aren't available -- but you have to have all the kerberos
and SASL, and every other possible security lib installed -- and that
also allows separation of binaries and libraries to happen as people
use symlinks in /bin/ls - /usr/bin/ls when /usr isn't mounted and 
won't work without it's load-time dynamic libraries.


Right now most

Re: -e does not take effects in subshell

[Linda Walsh]

Greg Wooledge wrote:


(Wow, how did we get here from -e does not take effects in subshell?)


---
because the POSIX spec changed and bash's handling of -e
changed to follow the new spec.

The earlier spec had -e only exit a script if a *simple* (external)
command failed.  It didn't include builtins nor functions.

Re: -e does not take effects in subshell

[Linda Walsh]

Andreas Schwab wrote:

Linda Walsh b...@tlinx.org writes:


Ex: rmx -fr (alias to rm --one-file-system -fr, since rm lacks the
-x switch like 'find, cp, mv, et al.) no longer works to clean
out a directory  stay on *one* file system.

Now rm will delete things on any number of file systems, as long
as they correspond to a cmdline argument.


That's the only sensible way to implement it.  Which, incidentally,
works exactly like find -xdev.

---
with 'find', you can specify -xdev with a starting path of .

with 'rm' functionality to remove '/' '.' and '..' was prohibited
by POSIX, though the coreutils version still allows the choice 
of the more dangerous removal of '/' with with the --[no-]preserve-root.


But the more useful rm -fr . or the variant rm -fr dir/. so you 
know you are removing the contents of dir, no matter where or what

dir is... with find, that doesn't work -- if 'dir' is a symlink
to /tmp/dir/.., find won't remove anything.  Any other solution 
from POSIX added complication.  I was told by a BSD fanatic that 'rm'

was changed because after the SysV companies left POSIX (as most of
them had disappeared), BSD'ers gained a majority and could redirect
the standard as they pleased.  Disallowing students playing around with
rm -fr {/,dir/,}{.,..} apparently was a big thing @Berkeley.  Being
able to force the removal of such options from everyone's rm was
an huge win, they considered (this is from a discussion w/one fanatic, 
but boy, was it memorable).  Disallowing any option or ENV(POSIX_CORRECTLY)

option to re-allow the feature has been continuously shot down by
'rm' maintainers (even though they keep in their own alias-able switches
to all removal of '/'). 


Now please explain what this has anything to do with POSIX.



 It apparently was the POSIX 2008 standard that prohibited the historical
behavior (on linux -- removed dir contents, and failed on current-dir
because it made no sense -- but did so *quietly* and after following
the depth first design.

Re: remaking bash, trying static, glibc refuses static...?

[Linda Walsh]

Mike Frysinger wrote:

On 18 Aug 2015 13:34, Linda Walsh wrote:

Then can you give any technical reason why a static
lib that uses no network services (i.e. running
on a mini-root ) couldn't be made available for
the various calls that currently claim dynamic library
support is necessary.


(1) http://www.akkadia.org/drepper/no_static_linking.html

---
	I've seen this -- much of it not applicable to a 
miniroot recovery situation.  However one of his issues is

he lists as a benefit is 'Security'.  With a statically linked binary
you have no hope of waking up to a non-working system because new
shared libraries have changed all the behaviors on your system.  Just
ask how many MS users have this problem.

He mentions security fixes -- If my system is in single user, it's not
likely that I need them.  

I *like[d]* the old-school method of putting static binaries in /bin 
and and using /usr/[s]bin alternatives after boot -- and after /usr 
is mounted.  But now, with the benefits of shared libraries for 
/bin/mount and /sbin/xfs_restore being in /usr/lib64, when the system 
boots it can't mount anything.  Lovely -- shared binaries -- hate them
all the more with making mount use shared libs only located in /usr.  
Brilliant.  Oh, yes, I can copy them to /lib64 -- but if they wanted to

do a root and /usr (bin+lib) merge, why not put them in /[s]bin  /lib[64]
and put the compatibility symlinks in the /usr dirs pointing at their
corresponding root dirs.  But with dynamic linking, they are putting binaries
and libraries in /usr/ while leaving symlinks in the /bin  /lib dirs.  


Yup... dynamic linking -- beautiful concept being used by in all the wrong
ways.



(2) it's using the nss system which lets people drop modules into the system
at anytime and change the overall lookups to use that.  statically linking a
specific subset would block that ability.

---
The linux kernel is a perfect example of a statically linked program that
can dynamically load plugins to provide authorization data from external
sources.  Static doesn't mean you can't support 3rd party plugins/libs --
like LDAP.


Tools like keyboard monitors, and background auditing would no longer work
without LD_PRELOAD/PROFILE/AUDIT.  Gee, now that's a shame.  Most
of my binaries I build shared -- I don't really care, but having a set
of core binaries on a rescue partition makes sense.  





which means people using providers

like ldap would be stuck with static binaries that don't work.


	Wrong -- the linux kernel is statically linked.  It can use 
3rd party security plugins for user auth  privs.



https://www.gnu.org/software/libc/manual/html_node/Name-Service-Switch.html


It's a description of how they did it to be opaque to users -- so
developers, admins, hackers, crackers or law enforcement can easily put in
new shivs in the next dynamic-lib update.  Lovely.  It Has happened with
MS, you can't tell me it can't happen w/linux.



it's already been provided.  build glibc w/--enable-static-nss.

---
Funny, my distro must have forgotten that option...

I wonder if glibc is as easy to build as the kernel?

Re: remaking bash, trying static, glibc refuses static...?

[Linda Walsh]

Mike Frysinger wrote:

it is not political, nor is it related to bash at all
-mike


Then can you give any technical reason why a static
lib that uses no network services (i.e. running
on a mini-root ) couldn't be made available for
the various calls that currently claim dynamic library
support is necessary.

I know it is not just 'bash'.  Googling for the subject
shows it's a problem for many projects, so I find it
very odd that such a static lib couldn't be provided.

If an upstream DB provider (NSS, say), refuses to provide
a static lib, then the static lib Gnu provided would exclude
them, stating the reason why.

Seems simple enough to provide such a widely asked for 
feature -- even if it has to be less functional/flexible

than the dynamic version (i.e. Gnu would have done the best
they could under the circumstances). 


But the bash option for static even lists the reason for
such -- but with no way to actually use the option.  *sigh*.

Re: -e does not take effects in subshell

[Linda Walsh]

Eric Blake wrote:

Like it or not, it is the historical behavior standardized by POSIX.


This is not true.  POSIX no longer documents historical behavior,
but now dictates new, historically-incompatible behaviors for a
variety of features in a variety of products (not just BASH).

As such, since the original mission statement of POSIX was to be
*descriptive* of what was (so a compatible standard could be provided),
and that is NOT what the new POSIX (post 2001-2003) has as a mission
statement, I assert the new POSIX is simply a new organization that
got the rights to use the name but use it as a club to force 
products to their new, dumbed-down and maladaptive behaviors.


Ex: rmx -fr (alias to rm --one-file-system -fr, since rm lacks the
-x switch like 'find, cp, mv, et al.) no longer works to clean
out a directory  stay on *one* file system.

Now rm will delete things on any number of file systems, as long
as they correspond to a cmdline argument.  Many people said to use
rm -xfr * to delete contents... but each object in 'rm' can be
on a different file system.  Worse rm -xfr **.

The workaround -- to use non-posix options of 'find' (or have find
call 'rm' for each qualified object.  


Please don't spread the lies that the *current* POSIX specs only
reflect historical behavior because it is not true.



It
is NOT intuitive, and our advice is DON'T USE set -e - IT WON'T DO WHAT
YOU WANT.  We can't change the behavior, because it would break scripts
that rely on the POSIX-specified behavior.

===
	I used the old behavior for over 10 years in various 
SH-compat shells, and it *WAS* useful.  POSIX changed it to be unuseful.

Re: -e does not take effects in subshell

[Linda Walsh]

Greg Wooledge wrote:

On Tue, Aug 18, 2015 at 01:49:53PM -0700, Linda Walsh wrote:

Ex: rmx -fr (alias to rm --one-file-system -fr, since rm lacks the
-x switch like 'find, cp, mv, et al.) no longer works to clean
out a directory  stay on *one* file system.


When did POSIX or any historical Unix rm have a --one-file-system option?
You say no longer works as if it had EVER worked in the past.

---
	Historically, linux had it going back to early 2000's 
(linux being a *nix platform) -- but historically, it wasn't

so easy to have features like 'bind/rbind', snapshots, multiple
virtual machines that need their own root (or chroot), etc.  If
you go back far enough symlinks weren't even around.

I'm only talking about POSIX ~2001 or before.  After that
it started changing.  So it depends on how historical you are talking.
POSIX cmd language started with POSIX.2 in 1992, before that it
was purely a programming API.  It started including the cmd's as
a way of providing portable shell scripts.  Not as a way of
restricting users.

While POSIX changed the 'rm' algorithm to no longer do
depth-first removal (now it's 2-pass, depth-first permissions check,
then depth-first removal).  But that's not the behavior of the historical
'rm'.

Various one-file-system cp -x, find -x, du -x were added after
it became common to allow more complicated mount structures.

I remember an early version of cygwin-coreutils-rm on Win7 that
didn't recognize symlinks or mountpoints (linkd/junctions) wandering
up out of the C:\recycle bin over to a documents folder on another
computer...

Daily-backups do come in handy.

And yes, the standard way to do this (the only way with traditional
tools) would use find ... -xdev ... -exec rm {} +

---
Which won't reliably work if your starting path is pathname/.

but would with an rm -frx (or rmx -fr path/.).

Re: why is 'direxpand' converting relative paths to absolute?

[Linda Walsh]

Clark Wang wrote:



I had the same problem months ago. See Chet's answer: 
http://lists.gnu.org/archive/html/bug-bash/2014-03/msg00069.html

===
Yep though I'm not sure about the reasoning in providing
different behaviors based on default-dir-expansion == convert
all relative paths to absolute.

Thanks for the explanation though..

linda

Re: bash expanding relative paths in auto-complete

[Linda Walsh]

Andreas Schwab wrote:

Linda Walsh b...@tlinx.org writes:


in bash 4.3.39,
if I type a command, (like .(source)) and a relative path
like : ../confcomplete,
it expands the relative pathname to absolute pathnames.


Worksforme.  Make sure to run complete -r first.

---
You didn't say what version of bash you were using... but
that may not be relative:

It seems to be direxpand -- do you have it on or off.

It is supposed to perform word expansion.  It doesn't
say it will convert relative paths to absolute.

Has it always been that way? (I thought direxpand expanded
variables in a path?)

remaking bash, trying static, glibc refuses static...?

[Linda Walsh]

bash-4.3/lib/readline/complete.c:2193: warning: Using 'getpwent' in 
statically linked applications requires at runtime the shared libraries 
from the glibc version used for linking


Other syms: getpwnam, setpwent, endpwent.

Is providing a static lib that reads the local machine's /etc/passwd
file really beyond the capabilities of the glibc people?

The configure comment says one would build bash static for 'miniroot
operations by 'root' (formatting other partitions, restoring, etc...).

I don't really see why dynamic libs are needed at a time that no other
file systems may be mounted -- operating in like run level 1 (single user).

This has a political feel to it (only because I can't imagine any techncial
reason to not provide it).  Does anyone else ever try a static link
on bash?

My 'bash' is 'bigger than most distro bashes (1.2M), but everything but
the libc stuff is builtin (all the builtins  help files -- given that's
all in there, 1.2M doesn't seem that large these days


 ldd /bin/bash

 linux-vdso.so.1 (0x7ffd09d86000)
 libdl.so.2 = /lib64/libdl.so.2 (0x00300140)
 libc.so.6 = /lib64/libc.so.6 (0x003000c0)
 libgcc_s.so.1 = /lib64/libgcc_s.so.1 (0x00300400)
 /lib64/ld-linux-x86-64.so.2 (0x5612dbf11000)

why is 'direxpand' converting relative paths to absolute?

[Linda Walsh]

From the manpage I wouldn't have guess it changed

paths to absolute -- but would expand variables and wildcards
in the path.

It doesn't seem to make alot of sense when there are other
ways to go from rel-abs, but not so many that just expand
vars or aliases on a dir.  


Does anyone know why path conversion was built into it?



Linda Walsh wrote:



Andreas Schwab wrote:

Linda Walsh b...@tlinx.org writes:


in bash 4.3.39,
if I type a command, (like .(source)) and a relative path
like : ../confcomplete,
it expands the relative pathname to absolute pathnames.


Worksforme.  Make sure to run complete -r first.

---
You didn't say what version of bash you were using... but
that may not be relative:

It seems to be direxpand -- do you have it on or off.

It is supposed to perform word expansion.  It doesn't
say it will convert relative paths to absolute.

Has it always been that way? (I thought direxpand expanded
variables in a path?)

Re: bash expanding relative paths in auto-complete

[Linda Walsh]

Andreas Schwab wrote:

Linda Walsh b...@tlinx.org writes:


in bash 4.3.39,
if I type a command, (like .(source)) and a relative path
like : ../confcomplete,
it expands the relative pathname to absolute pathnames.


Worksforme.  Make sure to run complete -r first.

---
Nope -- didn't change anything.

But throwing in a complete -D prevented any expansions 
until I typed in complete -r again.


complete -r says to remove all completion specifications.

I tried complete -p ../lcomplete -- and that printed
out all the files starting w/'l' in my 'bin' dir
(currently in ~/bin/sys;  so from there, I can type 
ls ../lib and see all the lib supporting stuff under ~/bin/lib, but

it changes a typed ls ../l - ls /home/law/bin/l

Also tried a clean environment:
env bash --norc --noprofile -- and that undefined my
aliases until I typed a new command:

Ishtar:law/bin/sys env bash --norc --noprofile
bash: _pf: command not found
...elided 22 copies of same ...
bash: _pf: command not found
Ishtar: which bash ##short prompt
/bin/bash
Ishtar:law/bin/sys
=
Notice the 1st prompt before I type which bash doesn't
show the 'cwd' (_pf, BTW, is an alias for printf for when I'm
writing out my spwd (short pwd) to print out a shortened
version of my current directory for my prompt and window title).

I do have expand_tilde, but I'm not using a tilde and the
same happens in:

Ishtar:/etc/local ls ../rc.
rc.d@  rc.splash  rc.status.orig rc.status.rpmsave*
rc.include rc.status  rc.status.rpmorig* rc.status.systemd
Ishtar:/etc/local ls /etc/rc.  -- .. changed to /etc

I ran into this, BTW, when I was looking at a src-tree that used
symlinks w/abs. paths, so when I ran it with make clean,
the symlinks pointed at the real root-based binaries rather than
the tree-based binaries.  


This resulted in rmx -fr * (rmx=rm --one-file-system) not seeing that the
symlinks (or mount-points) in the starting dir were on different
file systems and rm only checks to see that *descendants* of cmdline-args
are on the same FS -- which used to not be a problem when you could
do an rmx -fr . which would delete the contents before failing on .
at the end (w/msg suppressed by -f).

Sigh.

bash expanding relative paths in auto-complete

[Linda Walsh]

in bash 4.3.39,
if I type a command, (like .(source)) and a relative path
like : ../confcomplete,
it expands the relative pathname to absolute pathnames.

This is bad .. since I reuse relative commands in different
absolute places.

It didn't used to do this, and it's a potential security problem,
since if you reuse a command to configure something from before, and
don't catch that it's not using the relative path, it could run
the wrong command...

Re: BUG?: (or what's going on?) test for NL == NL fails (bash-4.3.39(3)-release)

[Linda Walsh]

Greg Wooledge wrote:

On Sun, Aug 02, 2015 at 01:30:14PM -0700, Linda Walsh wrote:

isatty () { test -c /proc/self/fd/1 ; }


This can be done portably with test -t 1 if you want the body of the
function to do what the name implies.  (Testing for a character device
gives false positives if stdout is a tape drive or similar device.)

---
-t is portable?   I vaguely remember -t -- but I know I didn't when
I wrote that function...  thanks for the update!

Re: BUG?: (or what's going on?) test for NL == NL fails (bash-4.3.39(3)-release)

[Linda Walsh]

Charles Daffern wrote:

On 02/08/15 20:30, Linda Walsh wrote:

 if [[ $(printf \n) == $'\n' ]]; then echo T; else echo F; fi

F


The command substitution operators ($(...) and `...`) strip all trailing
linefeeds. You are left with an empty string here.

The usual workarounds I see (where necessary) are hacks like this:

variable=$(command; echo x); variable=${variable%?x}


I see what you mean --- erk...
I think the process substitution doesn't, since you need
to explicitly use a -t switch if you want to strip the \n's off

Thanks!

Re: BUG?: (or what's going on?) test for NL == NL fails (bash-4.3.39(3)-release)

[Linda Walsh]

also noticed this similar strange behavior:

used 'ord'** for this test (trying to compare decimal val of chars)

isatty () { test -c /proc/self/fd/1 ; }
ord () { local nl=; isatty  nl=\n; printf %d$nl '$1 ; } 


ord $(printf \n)

0
ord $(printf \t)
9

ord $(printf  )

32

ord $(printf \r)

13
-- but this DOES work:

ord $'\n'

10

Traditional separators and CR work, -- just \n.
Am just looking for an efficient way to test for 0x0a
as last char in a var, ... after another 20 minutes of
trying things at random, found this:

  printf %d\n '${a:0-1:1}'
 10

Oddly, the final single quote in the string seems unnecessary.
But that (or into a var: printf -v var) both work...but I
feel a bit contorted trying so many odd ways to get something
that I thought would be simple, to work.

Still... why so many problems with \n?
I tried unsetting IFS  even setting both eol chars,
explicitly (I wondered if both eol chars were 'undef'
w/stty:


stty

speed 38400 baud; line = 0;
kill = ^X; eol = #; eol2 = #; susp = ^Y;   ## yes, I set both eol chars to '#'
only to find it didn't change anything (LF was still end of line).
:-(

BUG?: (or what's going on?) test for NL == NL fails (bash-4.3.39(3)-release)

[Linda Walsh]

 printf \n |hexdump -C

  0a

 printf %s $'\n'|hexdump -C

  0a

 if [[ $(printf \n) == $'\n' ]]; then echo T; else echo F; fi

F


just noticed the problem doesn't happen if I try to use '\t' (tab)

 printf \t |hexdump -C

  09

 printf %s $'\t'|hexdump -C

  09

 if [[ $(printf \t) == $'\t' ]]; then echo T; else echo F; fi

T

REGRESSION: signal handlers not running in real-time

[Linda Walsh]

I decided to give 4.3 a try again and ended up writing a 100 line script
to take the version# as input and auto-apply all the patches in the 
patch dir.


ARG going down a rathole... oh well.

I think I got a build pretty much as I want it -- w/everything builtin
statically except for a few key system libs that are unable to issue
a .a version of the lib:

 ldd /bin/bash

 linux-vdso.so.1 (0x7fc4c3137000)
 libdl.so.2 = /lib64/libdl.so.2 (0x00300140)
 libc.so.6 = /lib64/libc.so.6 (0x003000c0)
 libgcc_s.so.1 = /lib64/libgcc_s.so.1 (0x00300300)
 /lib64/ld-linux-x86-64.so.2 (0x55e08a982000)

 echo $BASH_VERSION

4.3.39(1)-release

But -- one initial bug came up -- I have a signal handler that
is called on window resizing -- but now, it no longer is called when
the window is resized -- but potentially hours later when I press a
key in that window.

Yes Chet, you warned me you broke real-time sig-handlers... but I
think this is a bigger bug than you thought it was.  I was thinking
about other places one wants real-time sig-handling --
like in handling command timeouts, and being able to manage
child processes as I do in a few scripts.

If I want to keep X children running, I would have background procs
increment a semaphore that would be initialized with the number of
background children I would want to run.  When a child came in, it
can increment semaphore, but now  my foreground control proc, instead
of being stuck in select or such, will be stuck in readline's
interrupts disabled read routine.  And it won't be able to
wake up via SIG_ALARMs, children dying.  Basically, you've now
changed BASH to be POSIX-undefined in  its behaviors: signal.2
manpage:

NOTES
  The effects of signal() in a multithreaded process are unspecified.

  According  to  POSIX,  the  behavior of a process is undefined after it
  ignores a SIGFPE, SIGILL, or SIGSEGV signal that was not  generated  by
  kill(2)  or  raise(3).   Integer division by zero has undefined result.
  On some architectures it will generate a SIGFPE signal.  (Also dividing
  the  most  negative  integer by -1 may generate SIGFPE.)  Ignoring this
  signal might lead to an endless loop.

  See sigaction(2) for details on what happens when  SIGCHLD  is  set  to
  SIG_IGN.

So now if someone gets a div/0 and can't handle the signal, it could lock
up the machine -- only because the results are undefined.

I.e. you've take away the ability to process sigs in real-time -- and
apparently *several* of these result in undefined and/or nasty behaviors.

It seems that POSIX solved that problem w/sigaction that auto-resets
a signal's disposition so as to prevent getting into a situation of being
nested, handling the same sigs.

Though apparently BSD's implementation further screwed up signal handling
-- and why is that important?  Linux has different libs -- some providing
the stable SysV interface, and others the BSD interface.  Problem there is
that after around 2003, most of the SysV vendors were out of business, and
now the BSD'ers had enough clout to change POSIX's charter from
descriptive to proscriptive -- so they can push through mandatory
changes that anyone who wants to continue to claim POSIX compat has to
comply with.

The Gnu people are supporting the new posix compat so they can claim posix
compat as a feature of gnu SW (and added marketing bullet). So -- more from
the signal manpage:


  The situation on Linux is as follows:

  * The kernel's signal() system call provides System V semantics.

  * By default, in glibc 2 and later, the signal() wrapper function  does
not  invoke  the  kernel system call.  Instead, it calls sigaction(2)
using flags that supply BSD semantics.  This default behavior is pro-
vided  as  long as the _BSD_SOURCE feature test macro is defined.  By
default, _BSD_SOURCE is defined; it is also implicitly defined if one
defines _GNU_SOURCE, and can of course be explicitly defined.
...

  * The signal() function in Linux  libc4  and  libc5  provide  System  V
semantics.   If one on a libc5 system includes bsd/signal.h instead
of signal.h, then signal() provides BSD semantics.

That last one is -- can really be bad if an autoconfig ends up linking
a program to the BSD semantics and not the SysV semantics.

It's quite possible that depending on how bash is configured, someone
might pull in one or the other -- but many of the signal problems are
there because of gnu not using linux OS calls but re-implementing their
own stuff using BSD.

Anyway -- not being able to respond to signals in ***real-time*** is
a problem that is only going to get worse as machines become more
*parallel*.  Not being able to load-balance and respond to sigs in any
real-time is only going to become more costly as cpu resources continue
growing, but only in parallel.

I'm guessing that readline would have to be turned inside-out to
become event driven and thread-safe?  


*sigh*

Re: auto-complete going on a diagonal -- set print-completions-horizontally on = equals diagonally

[Linda Walsh]

Chet Ramey wrote:

On 7/14/15 7:49 PM, Linda Walsh wrote:


---
Yeah, figures it would work right in the BASH dir... BUT...

Ishtar:tools/bash/bash-4.3 cd /suse132
--- both of the below print diagonally because they have no line endings.


I can't reproduce any of this on SUSE 13.  What does `stty size' say your
screen size is?

---
	13.1 and 13.2 are rather different.  
size:

40 80
-a:

stty -a

speed 38400 baud; rows 40; columns 80; line = 0;
intr = ^C; quit = ^\; erase = ^H; kill = ^X; eof = ^D; eol = undef;
eol2 = undef; swtch = undef; start = ^Q; stop = ^S; susp = ^Y; rprnt = ^R;
werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff
-iuclc -ixany -imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt
echoctl echoke

TERM=linux
-
I could see a program relying on wrap and not putting out CRLF, but that
doesn't seem to be the case.

Also from my experience, so far, I've only seen the stairstepping
alignment happen when completion is limited to 1 column (one or more items
are =40).

Don't stress too much about it -- I'll probably uncover the problem
eventually -- was just wondering if anyone knew of a case where such
oddness happened.  Sounds like I have another mystery behavior.  ;-)

-l

auto-complete going on a diagonal -- set print-completions-horizontally on = equals diagonally

[Linda Walsh]

When I go for auto complete now, the filenames no longer line up:

Ishtar:/suse132 ls e17-
e17-0.17.6-2.1.10.x86_64.rpm 
e17-branding-openSUSE-0.1-10.1.
2.x86_64.rpm  e17-branding-upstream-0.17.6-2.1.10.x86_64.rpm   
e17-devel-0.1

7.6-2.1.10.x86_64.rpm   e17-doc-html-0.17.6-2.1.10.x86_64.rpm
e17-profiles-openSUSE-20131019-2.1.4.noarch.rpm  
e17-theme-a-os-agust-v3-1-
2.1.9.noarch.rpm   e17-theme-a-os-detour-1-2.1.9.noarch.rpm 
e17-them
e-a-os-green-1-2.1.9.noarch.rpm  
e17-theme-a-os-miguel-v3-1-2.1.9.noarch
.rpm  e17-theme-a-os-vision-v3-1-2.1.9.noarch.rpm  
e17-theme-darkness-81
195-2.1.9.noarch.rpm
e17-theme-default-0.17.6-2.1.10.x86_64.rpm   e17
-theme-edjy-1-2.1.9.noarch.rpm
e17-theme-openSUSE-20131101.1-2.1.

9.noarch.rpm
Ishtar:/suse132 ls e17-
---
A bit more investigating, and it looks like it streaming out the filenames
with NO LF's (it's all one big long string if I copy it from a window that
supports wrapped lines as contiguous -- vs. windows cmd, above.


Either way, it's not putting out any line endings.

From the number of spaces, it looks like it is thinking my
terminal is 98 or 49 spaces wide (i.e. resizing the window
makes the names line up at those values).  But it doesn't
like my old fashion 80-wide term.

I'm thinking maybe some library is messed up, since I don't recall changing
anything in my bash setup recently.

-- Closest thing was changing the
set print-completions-horizontally to 'on' in .input.rc 

Uhoh, that does affect it, but then the sort order is messed up.

i.e.

ls complete
a  f   k   p u   
  z
b  g   l   q v   



instead of
a  b   c   d  
e  f



So how can I get horizontal sorting order without it being all one line?

Re: Is this right? test -v $hash{defined_value} == false?

[Linda Walsh]

Chet Ramey wrote:

On 7/12/15 11:58 PM, Linda Walsh wrote:

 set -A hash


This is an error

---
a brain-o -- at least I got the capitalized-A   ;-)

(original code probably had a

hash hash

   or

map hash

)


declare -A hash...
(I rarely type out the full command due to my aliases
and lazinesses*ouch*)





 hash=([defined_value]=22)


This creates an indexed array.


Um?  Did you really assume I meant 'set'.. well
 I better not ask... I _have_ been so spacey, which is
why I try to write aliases and trivial helper routines

I should stick to my aliases.. ;-)

Ishtar:/suse132 declare -A has
Ishtar:/suse132 has=([defined_value]=22)
Ishtar:/suse132 declare -p has
declare -A has='([defined_value]=22 )'



test/[/[[ -v didn't understand array variables and subscripts until bash-4.3.


Ahh that's what I was wondering... u

Thanks...

Re: auto-complete going on a diagonal -- set print-completions-horizontally on = equals diagonally

[Linda Walsh]

Chet Ramey wrote:
Am still on 4.2, but in the bashdir, I get:

reset ... ^L

Erase is control-H (^H).
Kill is control-X (^X).
Ishtar:law hhsfshsdfhl^C
Ishtar:law bind
Ishtar:law cd ../tools/bash/bash-4.3
Ishtar:tools/bash/bash-4.3 ls bcomplete
bash*bashintl.h   bashversion* buildversion.o
bashansi.h   bashjmp.hbracecomp.c  builtins/
bashbug* bashline.c   bracecomp.o  builtins.h
bashhist.c   bashline.h   braces.c 
bashhist.h   bashline.o   braces.o 
bashhist.o   bashtypes.h  buildsignames.o  
Ishtar:tools/bash/bash-4.3 bind 'set print-completions-horizontally on'

Ishtar:tools/bash/bash-4.3 ls bcomplete
bash*bashansi.h   bashbug* bashhist.c
bashhist.h   bashhist.o   bashintl.h   bashjmp.h
bashline.c   bashline.h   bashline.o   bashtypes.h
bashversion* bracecomp.c  bracecomp.o  braces.c
braces.o buildsignames.o  buildversion.o   builtins/
builtins.h
---
Yeah, figures it would work right in the BASH dir... BUT...

Ishtar:tools/bash/bash-4.3 cd /suse132 

--- both of the below print diagonally because they have 
no line endings.


Ishtar:/suse132 ls e17complete
e17-0.17.6-2.1.10.x86_64.rpm 
e17-branding-openSUSE-0.1-10.1.2.x86_64.rpm  
e17-branding-upstream-0.17.6-2.1.10.x86_64.rpm   
e17-devel-0.17.6-2.1.10.x86_64.rpm   
e17-doc-html-0.17.6-2.1.10.x86_64.rpm
e17-profiles-openSUSE-20131019-2.1.4.noarch.rpm  
e17-theme-a-os-agust-v3-1-2.1.9.noarch.rpm   
e17-theme-a-os-detour-1-2.1.9.noarch.rpm 
e17-theme-a-os-green-1-2.1.9.noarch.rpm  
e17-theme-a-os-miguel-v3-1-2.1.9.noarch.rpm  
e17-theme-a-os-vision-v3-1-2.1.9.noarch.rpm  
e17-theme-darkness-81195-2.1.9.noarch.rpm
e17-theme-default-0.17.6-2.1.10.x86_64.rpm   
e17-theme-edjy-1-2.1.9.noarch.rpm
e17-theme-openSUSE-20131101.1-2.1.9.noarch.rpm
Ishtar:/suse132 bind 'set print-completions-horizontally' off
Ishtar:/suse132 ls e17complete
e17-0.17.6-2.1.10.x86_64.rpm 
e17-branding-openSUSE-0.1-10.1.2.x86_64.rpm  
e17-branding-upstream-0.17.6-2.1.10.x86_64.rpm   
e17-devel-0.17.6-2.1.10.x86_64.rpm   
e17-doc-html-0.17.6-2.1.10.x86_64.rpm
e17-profiles-openSUSE-20131019-2.1.4.noarch.rpm  
e17-theme-a-os-agust-v3-1-2.1.9.noarch.rpm   
e17-theme-a-os-detour-1-2.1.9.noarch.rpm 
e17-theme-a-os-green-1-2.1.9.noarch.rpm  
e17-theme-a-os-miguel-v3-1-2.1.9.noarch.rpm  
e17-theme-a-os-vision-v3-1-2.1.9.noarch.rpm  
e17-theme-darkness-81195-2.1.9.noarch.rpm
e17-theme-default-0.17.6-2.1.10.x86_64.rpm   
e17-theme-edjy-1-2.1.9.noarch.rpm
e17-theme-openSUSE-20131101.1-2.1.9.noarch.rpm

Is this right? test -v $hash{defined_value} == false?

[Linda Walsh]

 set -A hash
 hash=([defined_value]=22)
 echo ${hash[defined_value]}

22

 [[ -v ${hash[defined_value]} ]]  echo pass
 [[ -v ${hash[defined_value]} ]] || echo fail

fail
 [[ -v {hash[defined_value]} ]] || echo fail  

fail
 [[ -v hash[defined_value] ]] || echo fail 

fail
[[ -v \${hash[defined_value]} ]] || echo fail
fail

 echo $BASH_VERSION

4.2.45(1)-release


How do I test if a value is defined in a hash?

This seems more messy and isn't -v supposed
to help clean up such things?

 [[ ${hash[defined_value]:-} ]] || echo fail  #it passes
 [[ ${hash[defined_value1]:-} ]] || echo fail

fail

So why no '-v'??  :-(

Re: eval a=b\ c

[Linda Walsh]

d...@yost.com wrote:

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu' 
-DCONF_VENDOR='unknown' -DLOCALEDIR='/usr/local/bash/4.3.39/share/locale' 
-DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H   -I.  -I../bash-4.3 
-I../bash-4.3/include -I../bash-4.3/lib   -g -O2
uname output: Linux s6.millcomputing.com 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed 
Apr 22 06:48:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-unknown-linux-gnu

Bash Version: 4.3
Patch Level: 39
Release Status: release

Description:

# Echo the arguments, then execute them as a command.
function echodo() {
  if [[ -n $ZSH_VERSION ]] ; then
echo [ ${(q)@} ]
eval${(q)@}
  else
echo [ $@ ]
eval$@
  fi
}
echodo echo b\ c
echodo a=b\ c
echo $a

The result in zsh is:
  $ echodo echo b\ c
  [ echo b\ c ]
  b c
  $ echodo a=b\ c
  [ a=b\ c ]
  $ echo $a
  b c
  $

Bash gets an error:
  $ echodo echo b\ c


You need to put \\ there:


echodo () {

echo [ $a ]
eval $@
}
Ishtar:law echodo a=b\\ c
[ b\ c ]still quoted here in echo
Ishtar:law echo $a
b c

Your first BackSlash was removed on the calling command line,
that means (as the 'echo' shows, that the BS is no longer there.

so you are telling eval to evaluate both expressions a=b  'c'

But if you use \\, one goes away on call, and the other is till there for eval.




  [ echo b c ]
  b c
  $ echodo a=b\ c
  [ a=b c ]
  bash: c: command not found
  $ echo $a
  
  $


I can't find a way to implement echodo in bash. Apparently this is because bash 
is unable to expand a variable with quoting intact, as zsh can do with its (q) 
expansion flag.

Bash behaves differently in this case::


echodo export a=b\\ c 

[ export a=b\ c ]
Ishtar:law echo $a
b c
---
Same ... You are using 2 levels of dequoting, so you need 2 BS's.
Make sense?


Alternatively, you can pass 1 BS if you quote the initial expression:

echodo 'export a=b\ c'
[ export a=b\ c ]
Ishtar:law typeset -p a  
declare -x a=b c


(I used typeset (or declare) -p to also show the -x (export) flag was set).

Alternatively you could also do it this way:


echodo () {
printf -v v -- %q  $@

echo $v; eval $v
}
Ishtar:law echodo export a=b\ c
export a=b\ c 
Ishtar:law typeset -p a
declare -x a=b c


You can do what you want in bash, you just have to be clear about
what you are telling it (same goes for most any program! ;-))

Re: eval a=b\ c

[Linda Walsh]

Dennis Williamson wrote:
I'm trying to put a command in a variable, but the complex cases always 
fail! : http://mywiki.wooledge.org/BashFAQ/050


Eval command and security issues : http://mywiki.wooledge.org/BashFAQ/048



Dunno, but I see nothing on that page about using 
printf -v %q or using single quotes inside of doubles... in his

first example:
# This code is evil and should never be used! 
fifth() { 
_fifth_array=$1 
eval echo \The fifth element is \${$_fifth_array[4]}\ # DANGER!

}
a=(zero one two three four five) 
fifth a

---
Had been written:
fifth() {
printf -v _fifth_array %q $1
eval echo 'The fifth element is ${ echo ${_fifth_array[4]} }' 
}

---
Then his arbitrary arg function throws  an error:

fifth 'x}'; date;

-bash: 'The fifth element is ${ echo ${_fifth_array[4]} }': bad substitution


However, if someone takes user input... it needs to be way sterilized --
i.e. if expecting a single char -- only accept a single char.
if expecting a number... ${i//[^0-9]/} :
i=948392480
Ishtar:law echo ${i//[^0-9]/}
948392480
i=9'\010'3{92}480  
echo ${i//[^0-9]/}

9010392480

---  
But I think a difference between greg's thinking and mine is that

I tend to write scripts to help me do things on my system.

If someone else wants to use my scripts -- and then use them to try to
break into  their own systemum... *oh well*... ;-)

Re: [Help-bash] make function local

[Linda Walsh]

Greg Wooledge wrote:

AFAIK, _exec_gvim, can only be called from within function gvim, no?


That is not correct.  In bash, all function definitions are global.

imadev:~$ f1() { f2() { echo I'm f2; }; }; f1; f2
I'm f2


You left out a few important parts, like f2 being called inside f1 before
it returns, and unsetting f2:

f1() { f2() { echo I'm f2 $1; unset -f f2; }; f2 internal; }; f1; f2 external
I'm f2 internal
-bash: f2: command not found

Hmmm...
Seems to work for me.

Re: [Help-bash] make function local

[Linda Walsh]

Eduardo A. Bustamante López wrote:

On Thu, Apr 16, 2015 at 10:38:22PM -0700, Linda Walsh wrote:
[...]

AFAIK, _exec_gvim, can only be called from within function gvim, no?


No. Doing this:

outerfunc() {
 func() { ...; }
 func args
 unset -f func
}
outerfunc

Doesn't guarantee that `func' will be only called from outerfunc. If you have
traps defined they can switch to other contexts where that funtion will be
available.

It's sad. Why do people keep pretending bash is something it isn't :-)?


It's sad, when you have to work at breaking people's example.

The program doesn't use traps.

Primarily, I didn't want to leave '_exec_vim' left in the namespace
after it had run.  It accomplishes that.

You can try hard enough to break anything.  Java was designed to
be a secure computing language from the start, yet it's track
record in security doesn't set it apart from any other language.

If you can't start with a 'fresh' language to design something
secure, how can you expect to add duct-tape and bailing wire to
30-40 year-old technology to make it secure.

Most security experts have advanced beyond the stage of the
silver bullet, and have finally come into agreement with where
I was, at least, 15 years ago -- security in layers.  None of the
layers are secure, but by the time a cracker finishes peeling off
the layers of the onion, they'll be in tears.

Re: [Help-bash] make function local

[Linda Walsh]

Greg Wooledge wrote:

The problem is, ALL function definitions (and un-definitions) are global.

---
yeah...

If there was a func defined at the global scope, you've obliterated it.
I don't understand why you don't understand this.

---
I don't understand why you think I don't understand this.

	I showed an example where I wanted to use a local function in 
a stand-alone script -- there were no libraries involved.  There was 
no previous definition.  The function was deleted if it was called

once.  No other interactions were specified.

I don't get why you toss in all sorts of other possible interactions
that I didn't specify nor that existed.  Sure, if I pull the plug from
the wall all of your examples on your website won't work.  So should
I question your examples because they don't work under all possible
circumstances?  That's ridiculous.  




Doing trickery to make it LOOK LIKE you have a local function just leads
to confusion.  The person who inherits this script from you after you're
gone may think the function is local, when it isn't.

---
I'm quite clear about how I use throw-away functions.



So, what is the point of doing this convoluted trickery, when it
accomplishes nothing (it doesn't create a local function, and it doesn't
prevent contamination of the global function namespace)?  Just to
see how clever you can be?

---
No, as an example where a local function would be of use --
and the clever trickery (I said it as a trite example) involved.


If I wanted clever trickery, I'd use a function generator
that generated function names with a GUID ending -- then test if that
exists and repeat if there is a collision.  That would be more my idea
of clever trickery, since Guids are considered to be Globally Unique
(even though they really aren't), it would be as solid as a true local
functions as much as GUIDS are really globally unique -- sorta something
like:

sub uuid_name {
 my uuid=$(uuidgen)
 echo ${uuid//-/_}
}

sub filterarray {
 my arrayname=${1:?}
 my filter=${2:?}
 my t=$(type -t $filter)
 if [[ $t!=function  $t!=alias ]]; then
   my uname=F_$(uuid_name)
   eval sub $uname { $filter }
 fi
}
---
Would be more my idea of clever trickery -- and would be just
as globally unique as GUID's are taken to be (regen GUID's until
you find one that doesn't collide (note the above code doesn't
loop on a test, but was a random scriplet written a year ago
to capture the idea of wrapping some 'filter' code in a sub
if it wasn't already an alias or a sub.

Of course, I'm sure you already thought of creating function
names on the fly using GUID's to ensure they don't collide
globally, tricky guy that you are...  ;-)