Re: Bash-5.1-rc1 available

2020-11-02 Thread Bob Proulx
Chet Ramey wrote:
> Tomas Janousek wrote:
> > Chet Ramey wrote:
> >> Yes, you can disable bracketed paste mode. I'll make sure that turning off
> >> bracketed paste mode disables the active region for incremental searches.
> > 
> > Would you please consider making this configurable separately? I'd love to
> > keep bracketed paste mode enabled, but I find the highlighting annoying 
> > (and a
> > bit buggy, more on that later). It seems I'm not alone:
> >

I would also much prefer if the highlight was separate.

> I'm going to see what the reaction is with bash-5.1 and go from there. The
> entire rationale for including this feature was to highlight the text a
> bracketed paste inserts. Since there was no separate use, and the
> contributed patch didn't include a configuration option, I didn't add one.

Highlighting the region is a completely separate concept from
bracketed paste mode as the region is used in many completely
unrelated places.  It pains me that they are coupled.


Re: Bash-5.1-rc1 available

2020-10-20 Thread Bob Proulx
Chet Ramey wrote:
> This release fixes several outstanding bugs in bash-5.0 and introduces
> several new features.

An unlisted change (I couldn't locate it in the changes) is that
'reverse-search-history (C-r)' now highlights the search pattern.  Is
that because it is the search pattern or because it now sets the region?

'reverse-search-history (C-r)'
 Search backward starting at the current line and moving 'up'
 through the history as necessary.  This is an incremental search.
 This command sets the region to the matched text and activates the

> f. New active mark and face feature: when enabled, it will highlight the text
>inserted by a bracketed paste (the `active region') and the text found by
>incremental and non-incremental history searches.

I find the reverse video highlight to be very difficult to read.  For
me it is extreme eye strain.  Is there a way to disable the reverse
video?  For both the paste text and the search text?

This is so bad for me that for the moment I had to revert to the
previous version of bash to avoid it.  (Note that disabling bracketed
paste mode avoids it for the paste but doesn't address the reverse
video of search.)


Re: GNU Bash v.4.4.23-5 built-in function 'set' produces variable outputs

2018-10-23 Thread Bob Proulx
Robert Elz wrote:
> ps: I did  not suggest that $_ should go away, I know that's not going to
> happen ... just that it would be nice for those who really don't want it to
> be able to turn it off.

I'm having a hard time understanding why one would want to turn off
this feature.  It isn't something that everyone uses but it is a
feature that has a lot of use.  If one goes down that path then the
end result taken to the logical extreme would be that every feature
would have a control to turn them on and off.  That just seems
extreme.  And for the case of $_ if you didn't know it existed then
one probably goes about their life continuing to not be bothered by it
too.  The original report wasn't really about $_ anyway.

Ricky Tigg wrote:
> Built-in function 'set' produces variable outputs.
> $ export SSLKEYLOGFILE=/home/user/test
> $ set | grep SSL
> SSLKEYLOGFILE=/home/user/test
> $ set | grep SSL
> SSLKEYLOGFILE=/home/user/test

The original report was about the output being different in different
invocations.  But I think that is an invalid reason.  Because if so
then 'echo $RANDOM' is also a bug because it produces different output
in different invocations too.  And because set | grep is not a correct
way to look at the environment as such either.  The 'set' command is
designed to set or unset shell options or the positional parameters.
Without any arguments "set shall write the names and values of all
shell variables in the collation sequence of the current locale".
Since $_ is a shell variable it writes it out along with possibly
other data too.  I don't think anyone grep'ing for a string should
complain that the shell also prints out the contents of $_.

As an additional point 'set' writes out the internal data which
includes a lot of *stuff*.  It would be better in this case to use
'env' to write out only the exported variables.  And clearly in the
original report the string being looked for was part of the exported
data.  There is no problem then.

  $ export SSLKEYLOGFILE=/home/user/test
  $ env | grep SSL
  $ env | grep SSL

However even 'env' isn't the appropriate tool either.  There may be
other variables that happen to hit the grep pattern.  And there is the
problem of the variable value including newlines.  The -z,--null helps
here but life isn't simple.

Not that I would do it this way but 'printenv' seems to be the right
matching utility here.

  $ export SSLKEYLOGFILE=/home/user/test
  $ printenv SSLKEYLOGFILE

Using grep is fine.  But then the human must interpret the results
accordingly.  I think here understanding that 'set' is doing what is
expected and reasonable is enough.  However I would use 'env' to avoid
the internal data state.  And programatically neither are sufficient
for a fully robust program and other methods should be used.


Re: bash sockets: printf \x0a does TCP fragmentation

2018-10-11 Thread Bob Proulx
Dirk Wetter wrote:
> PS + @Bob: fd 5 is not a tty in the program -- but interactively in
> this PoC you want to make sure it is not taken yet.

Understood.  I originally mentioned the isatty() thing because that is
how libc decides to line buffer or not.  But Chet has confirmed that
bash's internal printf line buffers regardless, because it has always
done it that way since the beginning.  So for the moment at least
whether it is a tty or not doesn't matter to the bash printf.  Since
this is a lessor traveled corner of bash it hasn't been tripped over
much before.  And probably never discussed here.  However I think it
likely (I didn't check) that it does for the coreutils printf using
libc's default stdio buffering.  So it is good to know about

I think it would make sense if bash's internal commands like printf
performed buffering the same way as the default libc does.  Then it
would behave the same as the rest and be less likely to cause
problems.  And it would have meant it would have worked by default in
your case.

Dirk Wetter wrote:
> Sorry, forget about my mail. In the actual file I was using there
> was a variable which I haven't quoted correctly. The PoC works as
> expected and it looks I have a solution.

The smallest of details can break the largest of programs. :-)

Good to hear you have things resolved and now working for you!


Re: Command not found with builtins and variables

2018-10-10 Thread Bob Proulx
Chet Ramey wrote:
> wrote:
> > The thing I noticed is that when setting a variable and then running the
> > command "time", bash reports "command not found". For example, the
> > command:
> > 
> > TEST=1 time ls
> > 
> > Results in "bash: time: command not found"

That would work if you installed the GNU time package.

And it would produce the portable output format in both cases with the
time -p option, which is available both in the internal time and the
external GNU time command.  Otherwise GNU time has its own unique and
different default output format.

  $ TEST=1 time -p sleep 1
  real 1.00
  user 0.00
  sys 0.00

But regardless of the external standalone time program...  Order when
using the internal time here matters.  Since 'time' is special it
should always be first.  Put any variables to be set on the line
before the command by putting it before the command but after the
'time' word.

  $ time TEST=1 sleep 1
  sys 0m0.000s

> You don't have a command named `time' in a directory in your $PATH.

Well...  *They* don't have a time program in PATH.  But they could
have if they installed it.  And many people do.  :-)

> Bash doesn't have a `time' builtin; it has a `time' reserved word.
> Preceding `time' with a variable assignment removes its special meaning
> and causes it to not be recognized as a reserved word.

I might also mention that the bash 'time' needs this special status
because it times the entire pipeline.

  $ time sleep 1 | sleep 2
  sys 0m0.002s

Therefore conversely if someone is using the external time it might
give them an unexpected result.  Since the external time will not know
anything about a pipeline.  So this might catch people off guard too.

  $ TEST=foo time -p sleep 1 | sleep 2
  real 1.00
  user 0.00
  sys 0.00

When used it is best to always put 'time' first on the command line,
variables second, and commands after that.  Order matters[1].


[1] OT: In English the order also matters and it is always
opinion, size, age, shape, color, origin, material, purpose, noun.
Using any other order and it sounds really odd!  Reference
"The Elements Of Eloquence: How To Turn The Perfect English Phrase."
by Mark Forsyth.

Re: Segfault on recursive trap/kill

2018-10-08 Thread Bob Proulx
Mike Gerwitz wrote:
> Bob Proulx wrote:
> > Let me give the discussion this way and I think you will be
> > convinced. :-)
> Well, thanks for taking the time for such a long reply. :)
> > How is your example any different from a C program?  Or Perl, Python,
> > Ruby, and so forth?  All of those also allow infinite recursion and
> > the kernel will terminate them with a segfault.  Because all of those
> > also allow infinite recursion.  A program that executes an infinite
> > recursion would use infinite stack space.  But real machines have a
> > finite amount of stack available and therefore die when the stack is
> > exceeded.
> I expect this behavior when writing in C, certainly.  But in languages
> where the user does not deal with memory management, I'm used to a more
> graceful abort when the stack gets out of control.  A segfault means
> something to a C hacker.  It means very little to users who are
> unfamiliar with the concepts that you were describing.
> I don't have enough experience with Perl, Python, or Ruby to know how
> they handle stack issues.  But, out of interest, I gave it a try:

Some of those must be maintaining the stack in program data space
instead of in the machine stack space.  (shrug)

>   $ perl -e 'sub foo() { foo(); }; foo()'
>   Out of memory!

Perl apparently will let you use all available memory without

>   $ python <<< 'def foo():
>   >  foo()
>   >foo()' |& tail -n1
>   RuntimeError: maximum recursion depth exceeded

I wonder how they decide to set the depth.

>   $ php -r 'function foo() { foo(); } foo();'
>   Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to
>   allocate 262144 bytes) in Command line code on line 1

134M seems arbitrarily small but at least it does say exactly what it
is doing there.

>   $ guile -e '(let x () (+ (x)))'
>   allocate_stack failed: Cannot allocate memory
>   Warning: Unwind-only `stack-overflow' exception; skipping pre-unwind 
> handler.
>   $ emacs --batch --eval '(message (defun foo () (foo)) (foo))'
>   Lisp nesting exceeds ‘max-lisp-eval-depth’

I think these two are different.  It looks like guile is using
libunwind to set up a stack exception handler whereas emacs appears to
be using a tracked max-lisp-eval-depth variable defaulting to 800 on
my system in emacs v25.

This limit serves to catch infinite recursions for you before they cause
actual stack overflow in C, which would be fatal for Emacs.
You can safely make it considerably larger than its default value,
if that proves inconveniently small.  However, if you increase it too far,
Emacs could overflow the real C stack, and crash.

> I understand that in C you usually don't manage your own stack and,
> consequently, you can't say that it falls under "memory management" in
> the sense of malloc(3) and brk(2) and such.  But C programmers are aware
> of the mechanisms behind the stack (or at least better be) and won't be
> surprised when they get a segfault in this situation.
> But if one of my coworkers who knows some web programming and not much
> about system programming gets a segfault, that's not a friendly
> error.  If Bash instead said something like the above languages, then
> that would be useful.
> When I first saw the error, I didn't know that my trap was
> recursing.  My immediate reaction was "shit, I found a bug".  Once I saw
> it was the trap, I _assumed_ it was just exhausting the stack, but I
> wanted to report it regardless, just in case; I didn't have the time to
> dig deeper, and even so, I wasn't sure if it was intended behavior to
> just let the kernel handle it.

My interpretation of the above is that you would want bash to use
libunwind (or whatever is appropriate) to set up a stack overflow
exception trap in order to handle stack overflow specially and then to
make an improved error reporting to the user when it happens.

Frankly I wasn't even aware of libunwind before this.  And haven't
learned enough about it to even know if that is what I should be
mentioning here yet. :-)

> > Shell script code is program source code.  Infinite loops or infinite
> > recursion are bugs in the shell script source code not the interpreter
> > that is executing the code as written.
> I also agree.  But the context is very different.  Shell is a very,
> very high-level language.

My mind reels at the statement, "Shell is a very, very high-level
language."  What?!  The shell is a very simple low level command and
control language.  It is very good at what it does.  But if one needs
to do high level things then one should switch over to a high level
language. :-)

> > Hope this helps!
> There was useful information, yes.

After sifting out the non-useful information.  :-)

> I hope I was able to further clarify my concerns as well.

You are always eloquent! :-)


Description: PGP signature

Re: Auto-update program cache feature

2018-10-08 Thread Bob Proulx
Chet Ramey wrote:
> Bob Proulx wrote:
> > Put this in your ~/.bashrc file and I believe your use case will be
> > much happier.
> > 
> >   shopt -s checkhash
> How many installers put a new version of an existing package into a
> possibly-different directory with a different name? Even using a symlink
> pointing to a new name would not trigger this behavior.

The use case (reading between the lines) had to do with ping-pong'ing
between /usr/bin and /usr/local/bin though.  And I know that without
checkhash that I used to see that condition too.  But with checkhash
the complaint (from me) went away entirely.  I admit I didn't look at
it in great detail here now because I have had checkhash in use for so
long and not seen this problem since.  I'll need to experiment.


Re: Segfault on recursive trap/kill

2018-10-06 Thread Bob Proulx
Hi Mike,

Mike Gerwitz wrote:
> ... but are you saying that terminating with a segfault is the
> intended behavior for runaway recursion?

Let me give the discussion this way and I think you will be
convinced. :-)

How is your example any different from a C program?  Or Perl, Python,
Ruby, and so forth?  All of those also allow infinite recursion and
the kernel will terminate them with a segfault.  Because all of those
also allow infinite recursion.  A program that executes an infinite
recursion would use infinite stack space.  But real machines have a
finite amount of stack available and therefore die when the stack is

This following complete C program recurses infinitely.  Or at least
until the stack is exhausted.  At which time it triggers a segfault
because it tries to use memory beyond the page mapped stack.

  int main() {
return main();

  $ gcc -o forever forever.c
  $ ./forever
  Segmentation fault
  $ echo $?
  139  # Signal 11 + 128

   The return value of a simple command is its exit status, or 128+n if
   the command is terminated by signal n.

Would you say that is a bug in the C language?  A bug in gcc that
compiled it?  A bug in the Unix/Linux kernel for memory management
that trapped the error?  The parent shell that reported the exit code
of the program?  Or in the program source code?  I am hoping that we
will all agree that it is a bug in the program source code and not
either gcc or the kernel. :-)

Shell script code is program source code.  Infinite loops or infinite
recursion are bugs in the shell script source code not the interpreter
that is executing the code as written.

This feels to me to be related to The Halting Problem.

> As long as there is no exploitable flaw here, then I suppose this isn't
> a problem;

It's not a privilege escalation.  Nor a buffer overflow.  Whether this
is otherwise exploitable depends upon the surrounding environment usage.

> I haven't inspected the code to see if this is an access violation
> or if Bash is intentionally signaling SIGSEGV.

It is the kernel that manages memory, maps pages, detects page faults,
kills the program.  The parent bash shell is only reporting the exit
code that resulted.  The interpreting shell executed the shell script
souce code as written.

Other shells are also fun to check:

  $ dash -c 'trap "kill 0" TERM; kill 0'
  Segmentation fault

  $ ash -c 'trap "kill 0" TERM; kill 0'
  Segmentation fault

  $ mksh -c 'trap "kill 0" TERM; kill 0'
  Segmentation fault

  $ ksh93 -c 'trap "kill 0" TERM; kill 0'
  $ echo $?

  $ posh -c 'trap "kill 0" TERM; kill 0'

Testing zsh is interesting because it seems to keep the interpreter
stack in data space and therefore can consume a large amount of memory
if it is available.  And then can trap the result of being out of data
memory and then kills itself with a SIGTERM.  Note that in my testing
I have Linux memory overcommit disabled.

This finds what look like bugs in posh and ksh93.

> it's just that most users assume that a segfault represents a
> problem with the program

Yes.  And here it indicates a bug too.  It is indicating a bug in the
shell program code which sets up the infinite recursion.  Programs
should avoid doing that. :-)

  bash -c 'trap "kill 0" TERM; kill 0'

The trap handler was not set back to the default before the program
sent the signal to itself.  The way to fix this is:

  $ bash -c 'trap "trap - TERM; kill 0" TERM; kill 0'
  $ echo $?
  143  # killed on SIGTERM as desired, good

If ARG is absent (and a single SIGNAL_SPEC is supplied) or `-',
each specified signal is reset to its original value.

The proper way for a program to terminate itself upon catching a
signal is to set the signal handler back to the default value and then
send the signal to itself so that it will be terminated as a result of
the signal and therefore the exit status will be set correctly.

For example the following is useful boilerplate:

  unset tmpfile
  cleanup() {
test -n "$tmpfile" && rm -f "$tmpfile" && unset tmpfile
  trap "cleanup" EXIT
  trap "cleanup; trap - HUP; kill -HUP $$" HUP
  trap "cleanup; trap - INT; kill -INT $$" INT
  trap "cleanup; trap - QUIT; kill -QUIT $$" QUIT
  trap "cleanup; trap - TERM; kill -TERM $$" TERM
  tmpfile=$(mktemp) || exit 1

If a program traps a signal then it should restore the default signal
handler for that signal and send the signal back to itself.  Otherwise
the exit code will be incorrect.  Otherwise parent programs won't know
that the child was killed with a signal.

For a highly recommended deep dive into this:

Hope this helps!

Description: PGP signature

Re: Auto-update program cache feature

2018-10-06 Thread Bob Proulx
Jeffrey Walton wrote:
> I think a useful feature for Bash would be to automatically update the
> program cache after an install.

Put this in your ~/.bashrc file and I believe your use case will be
much happier.

  shopt -s checkhash

In the bash manual:

  If set, bash checks that a command found in the hash
  table exists before trying to execute it.  If a hashed
  command no longer exists, a normal path search is

I would prefer that to be a default.  But it changed a behavior back
in the day when it was added and deviated from the previous csh
behavior.  Therefore being an optional option makes sense.  But I
don't see a downside to defaulting to it now.  In any case I always
add it to my bashrc file.  (Along with "shopt -s checkwinsize" too.)


Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-23 Thread Bob Proulx
Chet Ramey wrote:
> Bob Proulx wrote:
> > It was also noted in another message that the external standalone
> > printf command line utility did buffer as you desired.  That seems
> > another very good solution too.  Simply use "command printf ..." to
> > force using the external version.
> This won't work the way you want. The `command' builtin only inhibits
> execution of shell functions. It still executes builtins.  You want to
> either get the full pathname of a printf utility using `type -ap printf'
> and use that, or use the env or exec variants I recommended in my last
> message.

Oh drat!  Now I have had to learn *TWO* things today.  :-)

> > Anyway...  Since printf is a text oriented utility it makes sense to
> > me that I would operate in line buffered output mode.
> It's that bash sets stdout and stderr to be line-buffered, not anything
> printf-specific.

I still think 'printf' feels more like a plain text utility and not
something one reaches for when working with binary data blobs.


Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-23 Thread Bob Proulx
Robert Elz wrote:
> Bob Proulx wrote:
>   | Using the same buffer size
>   | for input and output is usually most efficient.
> Yes, but as the objective seemed to be to make big packets, that is probably
> not as important.

The original complaint concerned flushing a data blob content upon
every newline (0x0a) character due to line buffering, write(2)'ing the
buffer up to that point.  As I am sure you already know that will
cause the network stack in the kernel to emit the buffered data up to
that point with whatever has been read up to that point.  Which was
apparently a small'ish amount of data.  And then instead of having
some number of full MTU sized packets there were many more smaller
ones.  It shouldn't have been about big packets, nor fragmentation,
but about streaming efficiency and performance.  Though achieving
correct behavior with more buffer flushes than desired this was
apparently less efficient than they wanted and were therefore
complaining about it.  They wanted the data blob buffered as much as
possible so as to use the fewest number of TCP network packets.  My
choice of a large one meg buffer size was to be larger than any
network MTU size.  My intention was that the network stack would then
split the data blob up into MTU sizes for transmission.  The largest
MTU size that I routinely see is 64k.  I expect that to increase
further in size in the future when 1 meg might not be big enough.  And
I avoid mentioning jumbo frames.

>   |   $ printf -- "%s\n" one two | strace -o /tmp/out -e write,read dd 
> status=none obs=1M ; cat /tmp/out
>   |   one
>   |   two
>   |   ...
>   |   read(0, "one\ntwo\n", 512)  = 8
> What is relevant there is that you're getrting both lines from the printf in 
> one read.  If that had happened, there would ne no need for any rebuffering.
> The point of the original complaint was that  that was not ahppening, and
> the reads were being broken at the \n ... here it might easily make a 
> difference whether the output is a pipe or a socket (I have no idea.)

I dug into this further and see that we were both right. :-)

I was getting misdirected by the Linux kernel's pipeline buffering.
The pipeline buffering was causing me to think that it did not matter.
But digging deeper I see that it was a race condition timing issue and
could go either way.  That's obviously a mistake on my part.

You are right that depending upon timing this must be handled properly
or it might fail.  I am wrong that it would always work regardless of
timing.  However it was working in my test case which is why I had not
noticed.  Thank you for pushing me to see the problem here.

>   | It can then use the same buffer that data was read into for the output
>   | buffer directly.
> No, it can't, that's what bs= does - you're right, that is most effecient,
> but there is no rebuffering, whatever is read, is written, and in that case
> even more effecient is not to interpose dd at all.  The whole point was
> to get the rebuffering.
> Try tests more like
>   { printf %s\\n aaa; sleep 1; printf %s\\n bbb ; } | dd 
> so there will be clearly 2 different writes, and small reads for dd
> (however big the input buffer has) - with obs= (somethingbig enough)
> there will be just 1 write, with bs= (anything big enough for the whole
> output) there will still be two writes.

  $ { command printf "one\n"; command printf "two\n" ;} | strace -v -o 
/tmp/dd.strace.out -e write,read dd status=none bs=1M ; head /tmp/*.strace.out
  read(0, "one\ntwo\n", 1048576)  = 8
  write(1, "one\ntwo\n", 8)   = 8
  read(0, "", 1048576)= 0
  +++ exited with 0 +++

Above the data is definitely written in two different processes but
due to Linux kernel buffering in the pipeline it is read in one read.
The data is written into the pipeline so quickly, before the next
stage of the pipeline could read it out, that by the time the read
eventually happened it was able to read the multiple writes as one
data block.  This is what I had been seeing but you are right that it
is a timing related success and could also be a timing related

  $ { command printf "one\n"; sleep 1; command printf "two\n" ;} | strace -v -o 
/tmp/dd.strace.out -e write,read dd status=none bs=1M ; head /tmp/*.strace.out
  read(0, "one\n", 1048576)   = 4
  write(1, "one\n", 4)= 4
  read(0, "two\n", 1048576)   = 4
  write(1, "two\n", 4)= 4
  read(0, "", 1048576)= 0
  +++ exited with 0 +++

The above illustrates the point you were trying to make.  Thank you
for persevering in educating me as to the issue. :-)

  $ { 

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread Bob Proulx
Robert Elz wrote:
> ps: f there was actually a desire to use dd to do re-buffering, the
> correct usage is not to use "bs=" (which simply does read write with
> a buffer that size) but "obs=" which reads (using ibs if needed, which it
> would not be here), copies to an output buffer and writes only when that
> buffer is full (or on EOF on the input).

If the goal is to minimize writes then it won't matter as long as the
buffer size picked is larger than needed.  Using the same buffer size
for input and output is usually most efficient.

First let's set it to something small to prove that it is buffering as

  $ printf -- "%s\n" one two | strace -o /tmp/out -e read,write dd status=none 
bs=2 ; cat /tmp/out
  read(0, "on", 2)= 2
  write(1, "on", 2)   = 2
  read(0, "e\n", 2)   = 2
  write(1, "e\n", 2)  = 2
  read(0, "tw", 2)= 2
  write(1, "tw", 2)   = 2
  read(0, "o\n", 2)   = 2
  write(1, "o\n", 2)  = 2
  read(0, "", 2)  = 0
  +++ exited with 0 +++

Lots of reads and writes but all as expected.

Or set just the output buffer size large.  Then the input buffer size
defaults to 512 bytes on my system.

  $ printf -- "%s\n" one two | strace -o /tmp/out -e write,read dd status=none 
obs=1M ; cat /tmp/out
  read(0, "one\ntwo\n", 512)  = 8
  read(0, "", 512)= 0
  write(1, "one\ntwo\n", 8)   = 8
  +++ exited with 0 +++

But even if ibs is much too small it still behaves okay with a small
input buffer size and a large output buffer size.

  $ printf -- "%s\n" one two | strace -o /tmp/out -e write,read dd status=none 
ibs=2 obs=1M ; cat /tmp/out
  read(0, "on", 2)= 2
  read(0, "e\n", 2)   = 2
  read(0, "tw", 2)= 2
  read(0, "o\n", 2)   = 2
  read(0, "", 2)  = 0
  write(1, "one\ntwo\n", 8)   = 8
  +++ exited with 0 +++

Then set both ibs and obs to be something quite large using bs= and
let it gather up all of the input and write with that buffer size.

  $ printf -- "%s\n" one two | strace -o /tmp/out -e write,read dd status=none 
bs=1M ; cat /tmp/out
  read(0, "one\ntwo\n", 1048576)  = 8
  write(1, "one\ntwo\n", 8)   = 8
  read(0, "", 1048576)= 0
  +++ exited with 0 +++

It seems to me that using a large buffer size for both read and write
would be the most efficient.  It can then use the same buffer that
data was read into for the output buffer directly.


Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread Bob Proulx
I see that you have subscribed now.  Awesome!  If you and others would
be so kind as to list-reply instead of CC'ing me directly that would
be great.  I read the replies on the mailing list. wrote:
> Bob Proulx wrote:
> > You are doing something that is quite unusual.  You are using a shell
> > script direction on a TCP socket.  That isn't very common.  
> Do you think there should be a paragraph NOT COMMON where bash sockets
> should rather belong to?

You actually had not included enough background information to know if
you were using the bash built in network implementation or not.  You
only showed that you had set up fd 5 connected to a network socket.
That can happen because, for example, a script was used to service an
inetd configuration or similar.  It doesn't actually need to be the
built in network protocol at all.  But now that you have said the
above I guess I can assume that you are using the built in

As to whether the documentation should say this or not that is not
really practical.  There are a godzillian different things that are
not typically addressed by writing a shell script.  As a practical
matter it is impossible to list everything out explicitly.  And if one
tries then the complaint is that the documentation is so long and
detailed that is is unusable due to it.

Primarily a shell script is a command and control program.  It is very
good for that purpose.  It is typically used for that purpose.  That
is the mainstream use and it is very unlikely one will run into
unusual situations there.

But programming tasks that are much different from command and control
tasks, such as your program interacting by TCP with other devices on
the network, are not as common.  I don't have facts to back that up
but I do believe that to be true based upon the way I have seen shell
scripts being programmed and used over a long period of time.  Of
course if you have spent the last 20 years programming network shell
scripts then your observations will bias you the other way. :-)

> > More
> > typically one would use a C program instead.  So it isn't surprising
> > that you are finding interactions that are not well known.
> Bob, my intention was not to discuss program languages and what is typical
> with you or anybody else here.

Hmm...  Put yourself in our shoes.  You stood up on the podium that is
this public mailing list and spoke into the megaphone addressing all
of us complaining that bash's printf was buggy.  But from my
perspective printf is behaving as expected.  It is designed to deal
with line oriented data.  It will also deal with binary data if one is
careful.  But it appears that your application wasn't careful enough
and had tripped over some problems.

Should we (me!) keep silent about those very obvious problems?  It
feels obvious to me but apparently not to the author of the above.  As
has often been said many eyes make all bugs apparent.  I was pointing
this out to you as a public service.  But in response you seem hostile
by the language above and below.  That isn't encouraging any help. :-(

> >> printf -- "$data" >&5 2>/dev/null
> > 
> > Why is stderr discarded?  That is almost always bad because it
> > discards any errors that might occur.  You probably shouldn't do this.>
> > What happens if $data contains % format strings?  What happens if the
> > format contains a sequence such as \c?  This looks problematic.  This
> > is not a safe programming proctice.
> I doubt you can judge on this by just looking at a single line
> of code -- the project has > 18k LoC in bash.

That single line of code was problematic just by itself standing alone
without the rest of the program around it.  That is independent of
anything the rest of the program might contain.

However if you would like to pass sections of the rest of the program
through the help-bash mailing list then I am sure the group there
would help improve the quality of it.

> Github is the place to discuss and do PRs for our project.

No.  Sorry.  You came here to this mailing list.  Therefore this is
the place to discuss it.  Please put yourself in my shoes.  If the
case were reversed and I came over to Github and then stated that
Github was not the place for the discussion but that you needed to set
up email and come over to my mailing list and discuss it there
instead.  How would you feel?  I had come into your house, asked you
for help, then wanted you to go elsewhere?  How would you feel?  I can
tell you that I do not feel very welcome by it.

Also remember that Github is a non-free service.  That is free as in
freedom, not free as in beer.  The free in Free Software.  Or in this
case the opposite of it being non-free.  We try not to use software
that does not respect our freedoms nor ask others to do so either.
It's a philosophy of life thing

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-21 Thread Bob Proulx wrote:
> we discovered a strange phenomenon in the project

You are doing something that is quite unusual.  You are using a shell
script direction on a TCP socket.  That isn't very common.  More
typically one would use a C program instead.  So it isn't surprising
that you are finding interactions that are not well known.

> printf -- "$data" >&5 2>/dev/null

Why is stderr discarded?  That is almost always bad because it
discards any errors that might occur.  You probably shouldn't do this.

What happens if $data contains % format strings?  What happens if the
format contains a sequence such as \c?  This looks problematic.  This
is not a safe programming proctice.

> does not do what it is intended.

"Intent" is in the eye of the beholder.

> "$data" is  a ClientHello like
> '\x16\x03\x01\x2\x00\x01\x00\x1\xfc\x03\x03\x54\x51\x1e\x7a\xde\xad\xbe\xef\x31\x33\x07\x00\x00\x00\x00\x00\xcf\xbd\x39\x04\xcc\x16\x0a\...'
> Each \x0a like the last one causes a new TCP fragment to begin which can be 
> easily
> spotted when using wireshark while running e.g.

As Chet said the libc stdio library is probably doing line oriented
buffering.  The newline is causing a flush at that time.

> One would assume that a bash socket connection cannot influence the TCP
> fragmentation but obviously it does.

One would be in error to assume this.

> If there's a workaround, please let me know. (tried to add "%b" with no
> effect). Otherwise I believe it's a bug.

You can re-block the output stream using other tools such as 'cat' or
'dd'.  Since you are concerned about block size then perhaps dd is the
better of the two.

  | cat

Or probably better:

  | dd status=none bs=1M

Or use whatever block size you wish.  The 'dd' program will read the
input into its buffer and then output that block of data all in one
write(2).  That seems to be what you are wanting.

Good luck! :-)


P.S. You can possibly use the 'stdbuf' command to control the output
buffering depending upon the program.

  info stdbuf

Re: Add sleep builtin

2018-08-25 Thread Bob Proulx
Greg Wooledge wrote:
> L A Walsh wrote:
> > I  remember some implementations only wait till the clock reaches the
> > new time in seconds -- meaning that the actual sleep time could be
> > .001 seconds.
> If true, that would actually violate POSIX.  The sleep time must be
> "at least" as long as the operand.  It may be longer, but never shorter.
> You might be remembering it wrong.

I also remember that at one time the Unix sleep was implemented by
polling a second granularity clock register.  If one was not lucky
then polling just before the second turned, and then just after, would
cause sleep to exit because the second had changed.  A 1 second sleep
was 0-1 seconds.  Therefore if one required at least a 1 second sleep
one would need to ask for at least 2 seconds or it was possible for
there to be very little sleep at all.  A sleep 2 might be 1-2 seconds.
Of course sleeping for 300 would rarely care if it was actually 299 or
300 so longer sleeps were never a practical problem.

I recall this as a historical Unix kernel limitation.  I forget which
versions now and am not in a position to do a search through the
historical record to figure out which kernels did it that way.  I
never worked on any such running system but recall it from comments
written at the time.  I don't think it is true with any modern
kernels.  Or even with older ones such as the venerable HP-UX 10.20
from 1996!  I think it would be difficult to impossible to find any
running Unix-like system outside of a museum environment that still
behaved that old way.

And certainly any using the fractional second extension is past any
need to worry about the problem.  Because any such system with the
problem would only handle to a granularity of integer seconds anyway.


Re: Unexpected delay in using arguments.

2018-08-15 Thread Bob Proulx
Bize Ma wrote:
> > but just the same it is almost always possible
> > to refactor a program into a different algorithm that avoids the need
> > to enumerate so many arguments in memory.
> As you say: "almost".

I still believe that to be true.  Since you haven't shared what the
actual task is there is no way for us to propose any counter example
improvements.  So "almost" is as far as I can go.  Instead should I
say 99.44% of the time?  Since I have never come across a counter
example yet?

> Take a look at the Stéphane Chazelas example to convince yourself.

Nothing Stéphane said changed my statement at all.

It does look like bash can be more efficient with argument handling.
Since, for example, dash does it.


Re: Unexpected delay in using arguments.

2018-08-14 Thread Bob Proulx
I followed along through the script:

> m=2
> test1() { ...

Eventually I got to this line:

>   set -- $(seq $m)

At that line I see that bash must allocate enough memory to hold all
of the numbers from 1 through 2 in memory all at one time.  That
is very inefficient.  That is at least 100K of memory.

  $ seq 2 | wc -c

Of course I assume this is only a proxy simpler reproducer for the
actual problem program but just the same it is almost always possible
to refactor a program into a different algorithm that avoids the need
to enumerate so many arguments in memory.  I suggest refactoring the
program to avoid the need for this memory stress.


Re: Rational Range Interpretation for bash-5.0?

2018-08-06 Thread Bob Proulx
Chet Ramey wrote:
> Hi. I am considering making bash glob expansion implement rational range
> interpretation starting with bash-5.0 -- basically making globasciiranges
> the default. It looks like glibc is going to do this for version 2.28 (at
> least for a-z, A-Z, and 0-9), and other GNU utilities have done it for some
> time. What do folks think?

I think the non-rational ranges in libc were a terrible idea that we
have all been suffering with over the last decade plus.


Re: Empty ""s in ARG in ${x:+ARG} expand to no words instead of the empty word if prepended/appended with space

2018-07-21 Thread Bob Proulx
Chet Ramey wrote:
> Bob Proulx wrote:
> > Denys Vlasenko wrote:
> >> $ f() { for i; do echo "|$i|"; done; }
> >> $ x=x
> >> $ e=
> >> $ f ${x:+ ""}
> >> ^^^ prints nothing, bug?
> >>
> >> $  ${x:+"" }
> >> ^^^ prints nothing, bug?
> > 
> > Insufficient quoting.  That argument should be quoted to avoid the
> > whitespace getting stripped.  (Is that during word splitting phase
> > using the IFS?  I think so.)
> Even if the whitespace gets stripped out, the quoted null string should
> result in an empty argument. Different shells are inconsistent about this,
> but I believe that if word splitting occurs, the empty string (or "$e",
> anything that expands to an empty string) should result in a null word.

Gotcha.  Thanks!

And further testing confirms the inconsistent shells.  bash and ksh
both seem to be the outliers in different ways in terms of behavior.
dash and zsh seem to be consistent and correct in this.


Re: Empty ""s in ARG in ${x:+ARG} expand to no words instead of the empty word if prepended/appended with space

2018-07-20 Thread Bob Proulx
Denys Vlasenko wrote:
> $ f() { for i; do echo "|$i|"; done; }
> $ x=x
> $ e=
> $ f ${x:+ ""}
> ^^^ prints nothing, bug?
> $  ${x:+"" }
> ^^^ prints nothing, bug?

Insufficient quoting.  That argument should be quoted to avoid the
whitespace getting stripped.  (Is that during word splitting phase
using the IFS?  I think so.)

Try this:

  f "${x:+ ""}"
  f "${x:+"" }"

Also in the future please say what version of bash you are using.
There have been a lot of bugs fixed in various versions.


Re: announcing bashj

2018-06-23 Thread Bob Proulx
Chet Ramey wrote:
> On 6/23/18 3:44 PM, L A Walsh wrote:
> > Seems like the original missed being sent toe the bug-bash alias.
> It was not posted to bug-bash.

For bug-bash was in the hold queue for human moderation.  Although it
seems off topic for this mailing list it has already had three replies
in the thread and therefore I have now approved it and passed it on.
For better or worse.  For everyone subscribed
to the package list got a copy from there already.


Re: [annoyance/non-feature] On OS X, every process is "occupying" CWD, making disk ejects impossible after cd'ing into them.

2018-01-20 Thread Bob Proulx
Danyel Bayraktar wrote:
> Bash should idle in (e.g.) /tmp or /var/folders/xyz, i.e. occupy
> that file as ‘CWD’ towards the OS, but obviously not expose this to
> depending code, so still spawn every child process/builtin/keyword
> inside the (memory-stored) cwd which will be updated with `cd` and
> used throughout.

That would create regressions.  Consider the case where a shell
changes directory to some directory.  Then that directory is renamed.
Currently the shell having it as a current working directory will
continue to operate.  In your proposal the shell would suddenly fail
because it would not know the directory was renamed and would be
unable to return to it.


Re: xmalloc crash

2018-01-06 Thread Bob Proulx
Eduardo A. Bustamante López wrote:
> Alfred Baroti wrote:
> > Hi,
> > I found this from long long time ago.
> > Is this a serious bug?
> [...]
> This is not a serious bug at all. It's just a memory allocation failure.

Agreed.  Not a bug at all.

> You can see that the brk() system call actually succeeds.

If you are running under the Linux kernel in the default configuration
then memory overcommit is enabled.

  $ sysctl vm.overcommit_memory

With overcommit enabled brk(), and malloc(), will always succeed even
if there isn't really enough memory.  Neither will fork().  It doesn't
mean there actually is that much memory available.  Later if the
kernel runs out of memory to implement the needs then it will fail and
trigger the OOM Out Of Memory Killer to virtually kill -9 processes
until enough memory is freed up to balance the books.  Processes
killed by the OOM have no opportunity to clean up or log anything.

Or vm.overcommit_memory can be configured the traditional Unix way
where enough virtual memory needs to exist or malloc() and fork() will
fail.  IMNHO overcommit off is the best way for enterprise servers.
However overcommit on is convenient for laptops.


Re: simple function causes BASH to exit when -e in effect

2017-12-24 Thread Bob Proulx
Kevin Layer wrote:
> It took me hours of work to distill this down from a very large and long
> running script.

> set -eu

Hours and hours of debug time could be saved if people stopped using
'set -e' in their programs.  It creates more problems than people
think it solves.  Do not meddle in the affairs of 'set -e' for it is
subtle and quick to anger.


Re: Bash default /etc/bash.bashrc overwrites PS1 even if SUDO_PS1 is set

2017-12-19 Thread Bob Proulx
Eric Engstrom wrote:
> doh - thanks - didn't think of that, and that makes more sense.
> Nevermind then.

In the /usr/share/doc/bash/README.Debian.gz file:

5. What is /etc/bash.bashrc? It doesn't seem to be documented.

   The Debian version of bash is compiled with a special option
   (-DSYS_BASHRC) that makes bash read /etc/bash.bashrc before ~/.bashrc
   for interactive non-login shells. So, on Debian systems,
   /etc/bash.bashrc is to ~/.bashrc as /etc/profile is to


Re: ~/.profile and ~/.bash_profile aren't executed on login

2017-12-10 Thread Bob Proulx
Yuri wrote:
> Bob Proulx wrote:
> > How is the user logging in?  Are they logging in with 'ssh' over the
> > network?  Or are they logging in through an "xdm" X Display Manager
> > login from a graphical login display?
> User logs in locally through the display manager.

Which graphical display manager?  There are many.  You not saying
means that we are not knowing.


Re: ~/.profile and ~/.bash_profile aren't executed on login

2017-12-09 Thread Bob Proulx
Yuri wrote:
> None of these files are executed when bash is a user's default shell on
> FreeBSD.
> No special options were selected. Despite shell.c saying that they should be
> executed they just aren't.

How is the user logging in?  Are they logging in with 'ssh' over the
network?  Or are they logging in through an "xdm" X Display Manager
login from a graphical login display?

If from 'ssh' then that is the standard Unix login path with no surprises.

If from a graphical login display, which I think most likely is the
case, then it all depends upon how the shell is launched from that
point.  The shell looks at its own name, or the -l or --login option,
to determine if it is a login shell or not.  If it is a login shell
then it sources the login files.  If not then it does not and sources
the .bashrc environment file all as documented in the manual.

The most typical problem IMNHO is that graphical login managers should
set things up for sourcing the login files but by default do not.  I
am not familiar with BSD.  I am only familiar with the GNU/Linux
distributions and they often fail to set things up properly to source
the login files correctly.  That is not a bug in bash but is a bug in
the graphical login environment.

This is also a case of "bike shedding" because this is something that
everyone seems to have an opinion on about how this should be done.
And the opinions conflict with each other.  And therefore it will
likely never get resolved where it works automatically without
individual customization to make it work.

If your environment has the capability for $HOME/.xsessionrc then that
is usually the simplest suggestion to place login environment setting
actions there.  However note that it is a #!/bin/sh script and not a
bash script and therefore should use only POSIX portable shell syntax
in that file.  And also note that there are many other configurations
possible too.  I personally do something quite different for example.

Beyond this any other suggestions would require a *long* discussion of
how each graphical login environment works.  Therefore more
information about your environment would be needed.

> Is this a bug?

I feel confident that it is a consequence of how the shell is started.


Re: (OffTopic,) Gnu-Environment restrictions (was Re: Question )

2017-09-28 Thread Bob Proulx
Robert Elz wrote:
> I will send one more reply on this topic - I am replying to this message
> as I agree with the change of Subject - this is off topic for this list,
> which is why I will not respond any further to messages about this here.

You are most enlightened and I will do the same with this being my
last comment here upon it too.

> Bob Proulx said (in a message with the original subject):
>   | You can use the shared kitchen but please leave it in a better state than
>   | you found it.
> That would be fine, and would be free use.  But
>   You can use the shared kitchen provided that you leave it in
>   a better state than you found it.
> is not.   It isn't necessarily unreasonable, but it is not free.

Obviously IANAL and am speaking colloquially here.  If I were writing
a program I could be more exact.  Apparently my use of please softens
the requirement too much such that it still allows people to leave the
kitchen dirty.

I think in this discussion of free software you are conflating "free"
with the concept of "charge" or "cost".  But the "free" is free as in
freedom and not free of cost.  Which is why I often use the letters
free(dom) when talking about it.  Or "libre" which in latin based
languages have a better connotation.

But free(dom) software in the GPL sense is, to my understanding, not
without cost.  There is most definitely a cost to using it.  That cost
is that the source must remain free in the freedom sense of the word.
The idea being that if you want to dip into the pool of software that
it is just like the shared kitchen analogy I posited.  You can use it
but then the cost is that you must keep derivations from it free as in
freedom too.

This seems like a perfectly reasonable exchange to me.  In any case
the author chooses the license for their creations.

> This, and the GPL, are just regular commercial transactions -- you give me
> something, I give you something (that's called consideration in legal
> circles, and just a promise to do, or not do, something, is sufficient).
> While you (and perhaps even I, sometimes) might agree that the price is
> reasonable, there is still a price, and software subject to the GPL is
> not free.

And here I acknowledge that you are probably not that far different
from my thinking on this matter.  It is down to the very small
semantics of the word "free".  Which I emphasize is free as in freedom
and not free of charge.  Because I agree with you that there is a
cost.  There is most definitely a cost.  But where we differ (or
perhaps not if we discussed it out to the very end) is whether the
software is free as in freedom, ahem, free(dom) software or not.
Which is why I will sometimes emphasize this with libre free(dom)
software putting excessive emphasis on it.

I also note that those in the camp that don't like the GPL's free as
in freedom restriction often prefer other software licenses such as
MIT or Apache or other because they are free of charge.  There is
truly no cost to them.  That's great!  I'm happy for them.  But my
observation is that many people like them mostly so that they can keep
derivations proprietary.  There are many who take but they do not give
back.  They like it that others are charitable.  But they do not
reciprocate that charity.  And they don't like that the GPL allows
authors who choose it to require derivations to be just as charitable.
And as to whether which strategy is long term most successful only
time will tell.


Disclaimer: IANAL and use fuzzy language when communicating with
people because people tend to like it better that way.  Ambiguity of
language makes for a lot of unnecessary arguments.

Re: Question

2017-09-19 Thread Bob Proulx
Robert Elz wrote:
> Bob Proulx wrote:
> > However this difference in philosophy does not change the technology
> > that creating a compiler for an interpreted language is very hard.
> > Maybe impossible.
> No.  Not impossible, not even all that difficult, just in this case,
> not all that beneficial either.   (There are lisp compilers around for
> example.)

Lisp is often used as the canonical example.  :-)

> It has been a long time since I looked inside bash, and I remember none
> of it, but I maintain another shell (currently anyway) and it most
> certainly contains a compiler - it just never emits the compiled code
> into any external form, it remains internal, without that it would be
> far too slow.   While bash is slow, it isn't that slow (ie: it is fast
> enough for its purpose) so I assume it does the same (or similar.)

Stephen Bourne himself thought about writing a shell compiler.
  "And the other thing I would like to have done is written a compiler
  for it.  I got halfway through writing a shell script compiler but
  shelved it because nobody was complaining about performance at the

> What is relevant though is that compilers for languages like these
> rarely generate machine code that is directly executed, because of the
> point made earlier that code can be generated on the fly (the canonical
> case is:
>   read var
>   eval "$var"
> ) the resulting binary must contain the complete compiler, so it can

Thank you for mentioning the canonical example above.  I almost
mentioned it.  But decided not to go there.

> handle cases like this - so the usual course of action is to compile into
> an intermediate form, then interpret that (rather than directly interpreting
> the untouched input format).   This also keeps the whole thing portable,
> so no variation (beyond writing portable code in the first place) is needed
> to make the whole system run on different architectures.

Of course that is a description of the JVM too.

However then it has all of the user interaction benefits of working
with a compiler and all of the performance benefits of using an
interpreter.  It is definitely a compromise tradeoff. :-(

> The reasons that no-one has bothered to do something more than what is now
> done for sh, is that the benefits just aren't there - not that it is
> impossible, or even very hard.   As was noted, with any normal shell script
> (there are exceptions) almost all the time is used by the utilities that
> the script runs, not by the script itself, so in many cases, taking the
> extra time to produce a compiled external form, then read it in again every
> time it is run would not usually lead to any net execution speed increase.

Agreed.  It just doesn't seem useful for the shell.

Note that 'go run golangprogram' is extremely fast, all things considered.
I'm impressed by it anyway.

> Now I believe that was not the objective here, which was just to make the
> code harder for others to read/modify/steal

In which case working with a language that has good support for
compilers from the start is probably a better strategy.

> - unfortunately that would not
> really work either.  Along with compilers come disassemblers (they are needed
> if for no other reason than to help debug the compiler - the shell I maintain
> has one built in - it is not usually included in the versions people use,
> but can be, just by recompiling it) and for these kinds of languages, they
> are usually very good (aside from white space, comments, and ordering of
> redirects relative to the rest of the words on the command line, the one I
> maintain produces essentially identical output to the script input - it might
> occasionally leave out redundant useless syntax that was in the original.)

Agreed.  Obscurity is not true security.  But it does keep the casual
observers out.

> Now to the philosophical ... I agree with the aims of free software, but
> I believe that should be interpreted correctly - that is, if software is
> free, it should impose no restrictions at all upon is recipients, which
> includes hidden "you should reciprocate and make your software free too"
> (which is beyond what GNU actually expects I believe, but many people
> seem to - which really means that those people are not really offering free
> software, they're just bartering, I'll give you mine, you give me yours ...
> that's just commerce.)

Libre Free Software under the GPL is not without cost.  The cost, said
way too simply, is to use and share alike.

Many people in the community have worked very hard to contribute to
the pool of free software.  People benefit from having this pool
available.  I don't think it is bad to as

Re: Question

2017-09-14 Thread Bob Proulx
Saint Michael wrote:
> I use Linux. My business to provide services for problems that I solve, as
> you mention, by calling Awk, sed, join, etc., and databases.

Sounds good.  I classify myself the same way. :-)

> I allow my customers to login to a box that I provide in my
> datacenter. I cannot accept that it is impossible to restrict them
> to only call my *,sh scripts.

That sounds more like a restricted shell environment.  There has been
lots of talk and lots of discussion about restricted shells for many
years.  It is a very hard thing to successfully do.  Many try.  If you
search the archives you will find much discussion on this topic.

> I found a company that has a nice obfuscating and encrypting product
> for shell script, and I am trying it. I am no worried only about
> passwords, but the overall logic itself.

At this point we will have to agree to disagree that this is a good
thing to do.  Because philosophically the Free Software (also known as
Libre Software) community deals with software that respects the user's
freedoms.  The "free" is Free Software is not about cost but about
"freedom".  In some languages "libre" meaning liberty gives a better
feeling of it.  Obfuscating your software is your right but the users
of your software do not have the "four essential freedoms".

You can read something of this philosophy here:

> Many thanks for the time you took to respond. But in this case you fall
> close to an error. The market needs a tool so we can generate solutions and
> also protect our intellectual property. I am not smart enough to write it,
> but somebody will.

I think the opposite.  For example you are using a GNU/Linux system
which is Libre Free Software.  You are making use of a lot of hard
work by many people in the community.  They have given this to you and
all of us.  You are not charged money for this.  Yet you do not wish
to reciprocate and give back to the community.  You wish your work to
be proprietary.  At that we are philosophically opposed in values. :-(

Also when you say "intellectual property" it is problematic.  You
might want to read some as to why:

However this difference in philosophy does not change the technology
that creating a compiler for an interpreted language is very hard.
Maybe impossible.  Therefore one hasn't been created.  (I don't like
to say anything is impossible.  Because often clever people find a way
to make the impossible possible.  And in this case by changing the
problem and restricting the language it is possible.  Just not useful
at that point.)

Therefore regardless of our differences what you want is not easy to
do on a technical level.

I don't want to suggest this but...  Instead of obfuscating the shell
script program it would be better if you chose a compiled language
from the start.  Then you could simply compile it and have what you


Re: Bash handling of ENOENT on missing files and directories

2017-09-14 Thread Bob Proulx
Jonny Grant wrote:
> Please keep my email address in any replies
> Bob Proulx wrote:
> > Jonny Grant wrote:
> > > Yes, it's a known limitation of POSIX that it uses a shared error code for
> > > both files and directors, ENOENT. Which without programmers handling and
> > > checking the stat() flags, means the error isn't completely clear in the
> > > case where a file or dir does exist.
> > 
> > I don't see how POSIX is involved in this.  Blaming POSIX is a
> > complete non-sequitur here.
> Why do you feel it isn't?

The "No such file or directory" message I see in the Unix v7 code
circa 1979 and therefore appeared at least that early but probably was
there since the beginning.  That predates the first POSIX standard by
many years.

File errlst.c:
char*sys_errlist[] = {
"Error 0",
"Not owner",
"No such file or directory",

File errno.h:
#define ENOENT  2

> > > I imagine we have spoken already for longer about this, than it would have
> > > been to fix it.
> > 
> > I see no bug to fix here.  However I fear that trying to fix an
> > imaginary one would introduce a bug here.
> How can an easy update to clarify message "No such file or directory"
> introduce a bug?

Because "No such file or directory" is the exact error message that
corresponds to the error returned by the OS kernel as defined by the
system in the /usr/include definition file which is used by all
programs.  It is an important commonality.  All programs should say
the same thing for that error.

If every program created a different error message for that error then
it would be hard to learn what each and every program said for each
and every individual possible error.  OMG!  That would be horrible.
Changing it to make it something different would obscure the error
that is being returned by the OS.  It would make this one program say
something different for that error than other programs.  Different
makes it harder to understand.

Let's look at your cases again.

> $ cd missingdir
> bash: cd: missingdir: No such file or directory

Why do you think that is undesirable?  That is the system error.  The
requested "missingdir" to change directory to failed because there is
no entry by that name, uh, no such file or directory by that name. :-)

> Likewise, software that deals with directories, or in the case of
> "cd" in bash, trying to change directory, can very easily report "No
> such directory"

I think some of the problem might be that you are thinking that
directories are not files.  But in Unix like systems everything is a
file.  (And in a Linux like system everything is a file system.  It's
a joke but also with some truth to it.)

If I were writing that error message I may have said "No such entry"
since that is the literal error.  Meaning directory entry.  But I
could see having written "No such file", stopped there without the
directory hint, and defended it as being correct because everything is
a file.  But I forgive them for helping the user out a little by
hinting that no files, not even special files such as directories,
were found.  Directories are files.  They are special files.  But
files just the same.
  "The most common special file is the directory."

Let's try some other shells to see what error messages they produce.

$ ksh
$ cd nonexistent
ksh: cd: nonexistent: [No such file or directory]

$ mksh
$ cd nonexistent
mksh: cd: /home/rwp/nonexistent: No such file or directory
$ ^D

$ zsh
% cd nonexistent
cd: no such file or directory: nonexistent

$ csh
% cd nonexistent
nonexistent: No such file or directory.
% exit

We can also try a programming solution too just to double check what
error message is provided using a one-liner.

  $ perl -e 'chdir("doesnotexist") or die "$!\n";'
  No such file or directory

It is good that everyone uses the same error messages.

Let's look at your other case.

> $ ./main
> -bash: ./main: No such file or directory

$ ksh
$ ./nonexistent
ksh: ./nonexistent: not found [No such file or directory]

$ mksh
$ ./nonexistent
mksh: ./nonexistent: not found

$ zsh
% ./nonexistent
zsh: no such file or directory: ./nonexistent

$ csh
% ./non-existent
./non-existent: Command not found.
% exit

It looks like mksh and csh are the odd ones out here.


Re: Question

2017-09-11 Thread Bob Proulx
Saint Michael wrote:
> Dear Maintainer

Note that I am not the maintainer.

> Is there a commercial or free software that can take a Bash script and
> transparently turn it into a C executable, provided the machines where it
> runs has any of the external commands like awk, etc?

Not as far as I am aware.  And since no one else has spoken up with a
suggestion then I assume not.

> Something like a Java virtual machine, for Shell.

Yes.  That already exists!  It is the bash shell interpreter itself.
If you install a JVM then it can interpret Java programs.  If you
install bash then it can interpret bash programs.  Same thing.

The underlying issue is that shells are by design mostly there to
invoke external programs.  Therefore most of the activity of the shell
is to run other programs such as awk, grep, sed, and so forth.  It
isn't enough to have only the bash program but also the entire set of
other programs that is invoked.  Java by comparison can also execute
external programs but since that is not the primary design goal of it
most Java programs do not spend 99% of their code doing so.

> I think this language is powerful and I want to distribute some
> tools that contain critical information.
> So far I have been googling and there is only obfuscation tools, not
> compilation tools.

Making a shell script compiler would be a very hard problem because
the shell is an interpretive language with a primary task of invoking
external programs.  It would be very, very hard.  Impossible actually.
The nature of interpretive languages are that they can do things that
compiled languages cannot do.  For example interpreted programs can
generate code on the fly and interpret it.  Such as creating functions
during runtime.  Such as dynamically creating and executing code based
upon dynamic input.  Such tasks are easily done using an interpreted
language but must be done differently when using a compiled language.
One cannot create a compiler that can compile every capability of an
interpreted language without itself embedding an interpreter for that

If you restrict an interpretive language down to only features that
can be compiled then it is always possible to write a compiler for it.
But then it is for a restricted set of language features as compared
to the original language.  It is a different language.  Sometimes this
is an acceptable tradeoff.  But I will hazard a guess that for the
shell and for programs written for it this would be a quite different
language and no longer useful.  Possible.  But not useful.

> I think this language is powerful and I want to distribute some
> tools that contain critical information.

You don't say what critical information you are talking about.  But it
makes me think things like accounts and passwords that you would not
want to be available in clear text.  Note that obfuscating them in a
compiled program does not make them inaccessible.

You also don't say which OS you are using.  Most of us here will
probably be using a GNU/Linux distribution which includes the shell
natively or the Apple Mac which includes a working environment.
Therefore I will assume you are using MS-Windows.  I suggest either
bundling an entire environment such as MinGW such as Bash for Windows
or Cygwin so that all of your external dependencies are satisfied.
Or using a different language such as C which is natively compiled.
Or using a restricted set of Perl, Python, Ruby, or other that already
has a compiler available for that restricted subset.


Re: Bash handling of ENOENT on missing files and directories

2017-09-11 Thread Bob Proulx
Jonny Grant wrote:
> Greg Wooledge wrote:
> > The wording is taken directly from perror() and related library calls,
> > as translated for your locale.
> Yes, it's a known limitation of POSIX that it uses a shared error code for
> both files and directors, ENOENT. Which without programmers handling and
> checking the stat() flags, means the error isn't completely clear in the
> case where a file or dir does exist.

I don't see how POSIX is involved in this.  Blaming POSIX is a
complete non-sequitur here.

> I imagine we have spoken already for longer about this, than it would have
> been to fix it.

I see no bug to fix here.  However I fear that trying to fix an
imaginary one would introduce a bug here.


Re: Execution continuing after SIGINT received

2017-08-05 Thread Bob Proulx
Kevin Brodsky wrote:
> The thing is, "the next level of the program" really is another program,
> i.e. the one that invoked it, and you are communicating via the exit
> status, so it's certainly not as explicit as re-throwing an exception in
> C++, for instance. But sure, once you are aware of this mechanism, it's
> not difficult to understand the rationale.

One shouldn't confuse exception handling with signal handling.  They
are really quite different things. :-)

> Actually, IMHO, what makes it look very counter-intuitive is the fact
> that you need to first reset the signal handler for SIGINT. Of course
> this is necessary to avoid invoking the handler recursively, but it
> feels very much like a workaround.

You probably remember that signal(2) is unsafe because it does exactly
that and resets the signal handler to the default.  This creates a
race condition.  There is a window of time in the signal handler when
a second signal can catch the program with the default state set
before it can reset the signal handler.  That is the main reason why
sigaction(2) should be used instead.

> WIFSIGNALED is true if "the child process [...] terminated due to
> the receipt of a signal that was not caught". That's not really what
> we want to know here; we want to know if the child process received
> a signal that caused it to terminate. Whether it handled SIGINT to
> clean up resources is irrelevant; what's relevant is that it
> eventually terminated as a consequence of SIGINT. Ideally, exit()ing
> from a signal handler should set a bit in the exit status expressing
> exactly this.

Well...  Maybe.  But how are you going to know if the program
"eventually terminated as a consequence of SIGINT" or not?  And of
course then you would need a way to provide a program to do the
opposite when required.

But it has been 40 years already with the current way things are done
and it is a little late to change things now.

> I'll stop digressing, POSIX is what it is and we won't change it anyway
> ;-) For now, there's no other way to communicate with the shell, so
> that's fair enough.

I don't see how POSIX is even involved in this.  Other than stopping
the proliferation of differences on different systems.  Before POSIX
came along every system did this slightly differently.  It was awful.
POSIX.1 simply picked BSD as the behavior to standardize upon.  And
then we could stop putting #ifdef's in our code for every different
system quirk.

What you are seeing here is just the basic behavior of the old Unix
and BSD kernels and now every kernel that has followed since.  Since
that is the common behavior among all systems the standards bodies
will mostly say, freeze on that behavior, do it the same way, don't do
it differently.  Surely that is a good thing, right?

The subtleties of about how
programs should behave were learned not all at once but over the
course of time.


Re: Execution continuing after SIGINT received

2017-08-04 Thread Bob Proulx
Kevin Brodsky wrote:
> $ bash -c '(trap "echo INT; exit 1" INT; sleep 60s); echo after'
> after
> $

This is a good example of a bad example case.  You shouldn't "exit 1"
or you will replace the information that the process is exiting due to
a signal with an error code.  The trap handler should kill itself
instead.  Use this test case instead.

> $ bash -c '(trap "echo INT; trap - INT; kill -s INT $$" INT; sleep 60); echo 
> after'

Of course that doesn't change the end result here.  But at least the
program exit WIFSIGNALED information is now correct.

Signal handlers should always raise the signal on themselves after
handling whatever they need to handle first.

In any case I can't recreate your problem when using real processes in
separate shells not all on one line.


Re: bug in cd

2017-07-19 Thread Bob Proulx
Kae Verens wrote:
> but, if you do this:
> cd // ; pwd
> then the CLI reports
> //

Using two "//" at the start of a path name is special because in some
systems it means the path component that follows is a hostname.
Therefore it must be preserved verbatim.


Re: How to run bash test suite

2017-07-06 Thread Bob Proulx
Phi Debian wrote:
> I tried to subscribe to bug-bash but never got the confirmation mail.

You must have successfully received and replied to the confirmation
email because I looked at the subscriber list and you are subscribed.
Obviously not needed now but if you have trouble with the mailing
lists write to the mailing list owner at to
contact the humans who help with the mailing list.  This is the email
address convention for all of the Mailman managed mailing lists.


Re: null pointer deref, segfault

2017-01-31 Thread Bob Proulx
Chet Ramey wrote:
> On 1/24/17 2:07 AM, Brian 'geeknik' Carpenter wrote:
> > Going through some ancient bug reports and I came across
> > which apparently nobody
> > uses anymore.
> > 
> > <<$(()())|>_[$($(<<0)) crashes bash on Debian, Red Hat, FreeBSD, etc.
> It's marked as `Done'.

Since it is both marked as Done and also reported to be resolved in
bash 4.4 then I have closed the ticket.


Re: Why does bash use xmalloc?

2016-11-06 Thread Bob Proulx
Peng Yu wrote:
> My point is why bash terminates when it runs an external command that
> requires a large memory. Shouldn't bash return an exit code on behalf
> of the failed command and continue to the next command?

Note that I am going to say something without looking at the source
code of bash to say it.  Therefore I might be wrong about bash.  But
it is a general statement about shells.

> ... runs an external command that requires a large memory

If bash is running an external comand then it will be doing so in a
fork()'d off subshell using exec().  It won't be bash anymore but the
other process.  If it exits after the fork but before the exec then it
will be the subshell child process that exits not the main bash
execution thread.

If the bash process itself has grown very large then the fork() might
fail on systems that do not use memory overcommit.  (Traditional Unix
kernels or Linux kernels with overcommit disabled for robustness.)  If
the fork fails then I am not sure what bash should do in that case.
It could potentially loop and retry a few times but if all fail then
exiting is probably the only reasonable action.

If bash itself is running out of memory then it is almost certainly
due to inefficient shell scripting.  For example like one that I
analyzed in this thread.  Of course that is just one of many


Re: forked before bash subshell

2016-10-13 Thread Bob Proulx
XiaoBing Jiang wrote:
> Greg Wooledge wrote:
> > If you want to ENSURE that the child shell process is replaced by the
> > external sleep 20, use an explicit exec.
> yes, I want to know why bash not optimize this. or any strategy ?

Because it wouldn't save anything significant.  Since the parent shell
is bash the extra bash in memory is already in memory.  A forked
process uses almost the same OS memory as the original.  The OS will
share the code pages between the two processes at the OS memory page
level.  Data pages will split when they are individually modified.
Even if you use 'exec' to replace the forked and backgrounded copy
another one is still in memory as the script runs.  Only unique data
pages will go through the copy-on-write process as the two bash
processes diverge.  Therefore even if you *think* you are saving
something by optimizing it out the actual savings is insignificantly

Plus this is really only a problem if the main part of the script
exits leaving the forked subshell running in the background.  However
that is not a good thing to do.  If you want to leave the process
running that is a daemon process.  Launching a daemon process needs
more setup than this to avoid other problems.  Since leaving processes
behind like that is a bad thing to do it again doesn't make much sense
to try to optimize the doing of it.

It is possible to contrive some examples where it makes sense to do
this optimization.  But mostly they are very unusual cases.  Not
typical cases.


Re: Syslog output from bash

2016-08-23 Thread Bob Proulx
Richard Lohman wrote:
> Problem is, I'd like the output to resemble other syslog messages:
>   Mmm dd HH:MM:SS hostname bash[pid]: command
> And ultimately drop the username in as well. Since only bash is logging in

I suggest using the 'logger' command instead of modifying bash.

Normally I am doing things like this:

  logger -t cmdname "my log message here"

But you can get the format you wish with:

  logger -t "cmdname[$$]" "your log message here"


Re: Curious case statement error

2016-08-13 Thread Bob Proulx wrote:
> [a-z]) echo "Character is in Lowercase";;
> [A-Z]) echo "Character is in Uppercase";;

What is the output of 'locale' for you?  It will almost certainly show
that your LC_COLLATE is *NOT* set to the C locale but to some other
locale.  Your statements above are correct only in the C locale.  It
depends upon your locale setting and the program's specific handling
of it.  If it is en_US.UTF-8 then the above does not apply.  Instead
it is more likely this:

  [a-z]) echo "Character is in aAbBcC...z range";;
  [A-Z]) echo "Character is in AbBcC...zZ range";;

This is due to the locale collation ordering in your environment.  It
is not specific to bash and also affects grep, sed, awk, sort, and so
forth.  (However newer versions of most programs are specifically
working around this now.  The problem used to be more common a few
years ago but with recent releases the problem is disappearing.)

Using the human language locales en_US.UTF-8 ranges one must use the
[:lower:] and [:upper:] ranges.

  [[:lower:]]) echo "Character is in Lowercase";;
  [[:upper:]]) echo "Character is in Uppercase";;

The grep man page explains this in detail so let me quote it here:

  Within a bracket expression, a range expression consists of two
  characters separated by a hyphen.  It matches any single character
  that sorts between the two characters, inclusive, using the locale's
  collating sequence and character set.  For example, in the default C
  locale, [a-d] is equivalent to [abcd].  Many locales sort characters
  in dictionary order, and in these locales [a-d] is typically not
  equivalent to [abcd]; it might be equivalent to [aBbCcDd], for
  example.  To obtain the traditional interpretation of bracket
  expressions, you can use the C locale by setting the LC_ALL
  environment variable to the value C.

For reasons that most of us common people disagree with the powers
that be decided that locale specific collation sequences would ignore
punctuation and would fold case using "dictionary" collation ordering.

Note also that bash's collation sequence is set when it is started.
In other words changing the LC_ALL or LC_COLLATE variables only
affects newly launched programs.  It will have no effect on the
currently running bash shell.  In other words to change it for bash
you would need something like this:

  $ with LC_COLLATE set to en_US.UTF-8 with bad collation ...
  $ env LC_COLLATE=C bash
  $ ... works now ...

I mention this because otherwise people try changing the variable and
then don't see a change in the already running bash shell.


Re: [Documentation] -d returns true for symlink to directory

2016-07-21 Thread Bob Proulx
Reuti wrote:
> While we are on this: wondering about the difference about -h and -L I found 
> that `man test` outputs on OS X:
> " -h file   True if file exists and is a symbolic link.  This 
> operator is retained for compatibility with pre-
>vious versions of this program. Do not rely on its 
> existence; use -L instead."
> while on Linux `man test` just states that it's the same. Is there any 
> preference which one should be used for the bash builtin?

Originally Unix did not provide symlinks.  Then when symlinks were
added they were not completely added in a uniform regular way through
every command.  Bummer!

The original option letter used by test to check for the presence of a
symlink was -h.  I don't know why.  But in those days the only test
option to test for a symlink was -h.  The -L came later.  This legacy
is still visible in the HP-UX test man page for example.  No -L
documented there.

However note that because POSIX requires it the /bin/sh posix shell
must implement -L as identical to -h.  And the reverse.  Therefore you
are safe to use either on any POSIX system.  Remember too that POSIX
wasn't originally meant as a design document but as an OS feature
non-proliferation treaty to make it possible to write portable
programs.  POSIX features must be present on any POSIX system and
therefore you can use the feature and expect it to work.

If you need to interact with legacy systems which predate POSIX and
have never been updated to comply then you must use -h because that is
all there is there.  But if you never interact with anything non-POSIX
then using -L is okay too.  Either is the same as the other now.

I think the statement in the Apple man page is there because someone
liked -L over -h and wanted to scare everyone into using one over the
other.  But there isn't any reason to do so.


Re: Cannot clone from anonymous git

2016-05-15 Thread Bob Proulx
Eduardo A. Bustamante López wrote:
> When I try to clone:
> |  dualbus@hp ~ % git clone git://
> |  Cloning into 'bash'...
> |  fatal: Could not read from remote repository.

Connection was closed immediately.

> |  Please make sure you have the correct access rights
> |  and the repository exists.

Misleading error message.  It couldn't connect.

> The links here appear to work:

Since Friday the 6th there was a large move with various assorted
related and unrelated changes.  Which has makes it difficult to tell
what broke what.  I am deep in the middle of *.savannah and can give
the most information about it.  Immediately after the move there was a
systematic breakage of uids and ACLs which were fixed last weekend.

But the problem is only partially Savannah.  Most of the problem is
global network connectivity.  It seems to be worse from Europe than
from the US.  For example me in Colorado I rarely have any problem.
But at the same time people overseas report complete failures.  There
are reports of problems across all of the servers from fencepost to to the web server systems.  The common change to all of
them was a core router change.

On Savannah vcs the network connectivity tickles a problem with the
git daemon.  It causes them to get stuck.  This causes them to hang
around endlessly.  They are launched from xinetd with 100 process
limit.  Especially during busy activity they fill up the slots.  Then
xinetd has a bug IMNHO that instead off allowing connections to queue
it simply closes any additional connection.  This is what you see and
describe in the above.  git:// failed because all 100 slots were in
use and xinetd closed your additional connection.  Meanwhile http://
worked because that used apache.  Maybe the problem is actually the
xinetd which is common to the cvs, hg, bzr, svn daemons which are also

Now before everyone decided to switch to apache note that apache also
has process limits.  It just so happens that fewer people are using it
and so it wasn't limited.  Yet.  If everyone switched to apache
http:// then it would simply switch the problem from one daemon to the
other.  We see all of the source daemons stack up.  This includes cvs,
hg, bzr, and svn processes too.  Something network related is causing
them to hang around after they should have exited.  I have been
tweaking the tcp keepalives to try to makedead the stale processes
faster.  Assaf has been doing the majority of the process triage and
manually pruning processes in order to keep things functional.

On the client side it is impossible to distinguish between git daemons
consuming the limit of process slots versus router connectivity
problems.  Both have the same effect.  However if you can get through
with ssh:// or http:// then that indicates xinetd process slots are
full.  But over the last week (mostly Europe and other overseas
clients) reports endless problems trying to post bugs using the web
server.  Endless problems trying to SMTP mail in and out.  Reports of
rsync failing.  Sometimes IPv4 works when IPv6 is failing.  All of
those are separate except for the global network connectivity through
the routers.

Obviously the most urgent problem to fix is the global network
connectivity problem.  The FSF admins are aware of the problem.  They
post (terse) status updates to so do
please keep an eye on it for out-of-band status updates.  Especially
when there are Boston local network problems.

If you are suffering problems I encourage a trouble report being made.
Please include details.  Say where from and where to.  Say whether it
was IPv4 or IPv6.  Time the problem occurred.  Then if later it works
update and say that.  Because one of the problems is that this seems
worse from Europe than from the US.  There isn't a reproducible test
case to show if a change fixes things or not.  I in Colorado have a
hard time triggering any problems but have been able to on occasion.
But people in Europe have been most of the problem reports.  Most of
them have been using IPv6.

And that is all I know.


Re: Avoid asterisk expansion when it selects "everything"

2016-04-13 Thread Bob Proulx
Anis ELLEUCH wrote:
> I would like to ask if it is possible to disable expanding asterisk when it
> selects all entries ?

You ask if it is possible and the answer is no it is not possible.
Because the shell expands the "*" before it passes the result as
arguments to the rm command.  The rm command has no idea it was the
result of an expansion from a "*" wildcard.  The rm command simply
receives the list of files.

> Another idea: `*` and `/*` should not be interpreted and the user has to
> enter another sequence "more powerful" to emphasize selecting all entries (
> `^*` would it work just fine ?)

But then the same thing exists there.  If the user accidentally enters
that sequence then it is the same thing all over again.

If I am ever uncertain about a file glob expansion then I use echo
first to see what the expansion will do.  If dealing with dangerous
commands such as rm then it is wise to check twice before invoking.

  echo rm * .jpg

If you saw the result then you would fix it before invoking the
command for effect.  If the result is bad then correct it.  If the
result is good then remove the echo and run the command for effect.


Re: Fwd: read builtin return non-zero status reason

2016-02-24 Thread Bob Proulx
Cuong Manh Le wrote:
> I'm sorry for that. It's not the time between post but after sending the
> email, I look at the help-bash archive and see it have no active email
> since January.

There has been lots and lots of email in the archive since January.

Maybe the archive you were looking at is dead?

Note that the archive only updates every half hour.  Therefore
messages don't show up there immediately even when they have been sent
to the list immediately.

> Anyway, still my fault.

Don't worry about it.  It is all in the past now.


Re: Fwd: read builtin return non-zero status reason

2016-02-24 Thread Bob Proulx
Greg Wooledge wrote:
> On Wed, Feb 24, 2016 at 11:31:40AM +0700, Cuong Manh Le wrote:
> > I send this email to help-bash but it seems to be inactive. Please help me
> > with this problem. Thanks.
> help-bash is active.  You probably just didn't wait long enough.
> If this is your first message to help-bash, your message is probably
> waiting in some moderation/antispam queue.

Impatience is the problem.  Here are the times of the three messages.

  help-bash: Date: Wed, 24 Feb 2016 11:14:31 +0700
  help-bash: Date: Wed, 24 Feb 2016 11:20:46 +0700
   bug-bash: Date: Wed, 24 Feb 2016 11:31:40 +0700

You waited six minutes for a response to help-bash and then sent it
there again.  You then waited eleven minutes before forwarding it to
bug-bash.  Neither six nor eleven minutes is a reasonable amount of
time to wait for a response.

Plus your timezone is +0700 while most of the moderation folks and
bash folks operate out of -0500 through -0800.  It may be the middle
of the day for you but that is the middle of the night for others.
Your first message was held for moderation and released at a very
early morning time of 05:44:56 -0800.  Your bug-bash message had no
moderation delay probably due to previous contact there.

Please have some patience.


Re: How to lock a terminal

2016-02-17 Thread Bob Proulx
Benno Schulenberg wrote:
> Bob Proulx wrote:
> > I do not have a nanorc file at all.  The defaults are okay for
> > suspension without any specific configuration.
> ??  How does that work?  What version of nano are you using?

Since this has nothing to do bash I am going to take this to the nano
list only.


Re: How to lock a terminal

2016-02-16 Thread Bob Proulx
Benno Schulenberg wrote:
> Bob Proulx wrote:
> > [...] this is the perfect case for job control.  No need for a
> > second terminal.  Here is an example.  Use Control-Z to stop the
> > foreground job.
> For that to work, it requires having 'set suspend' in your
> nanorc.  (Which I don't have, because it annoys me when nano
> drops into the background when I accidentally hit ^Z.)

I do not have a nanorc file at all.  The defaults are okay for
suspension without any specific configuration.

Since you are specifically configuring suspend disabled one must
presume that you will already know about job control and have
specifically disabled it and will work around it by other means.  Such
as in your case M-Z or another terminal or other method.


Re: How to lock a terminal

2016-02-16 Thread Bob Proulx
Nick Warne wrote:
> I was in a SSH session, and checking something inadvertently issued:
> > nano /var/log/messages | grep a
> (I was searching for something else than an 'a', but the above example shows
> the issue - about to use 'nano', but then forgot to change it to 'cat').
> The terminal just sits there doing nothing - CTRL+C doesn't do anything; in
> a SSH session, the only option is to kill the terminal.  On a local machine,
> you can use kill -9 from another terminal to get out of it.

On a remote machine you can do the same.  There really is no
difference between local and remote here.  You just use a second
terminal for it.

However this is the perfect case for job control.  No need for a
second terminal.  Here is an example.  Use Control-Z to stop the
foreground job.

  rwp@havoc:~$ nano /var/log/messages | grep a
  [1]+  Stopped nano /var/log/messages | grep a
  rwp@havoc:~$ jobs
  [1]+  Stopped nano /var/log/messages | grep a
  rwp@havoc:~$ kill %1
  Received SIGHUP or SIGTERM
  rwp@havoc:~$ jobs
  [1]+  Terminated  nano /var/log/messages | grep a
  rwp@havoc:~$ jobs

Simply stop the process and then kill it using the same terminal.


P.S. The other suggestions to use Control-X to exit nano are also good
too but job control is general for the entire class type of commands
like this and I think good to know too.

Re: [PATCH/RFC] do not source/exec scripts on noexec mount points

2015-12-12 Thread Bob Proulx
Mike Frysinger wrote:
> But bash itself has no problem running this file:
>   $ bash /dev/shm/
>   hi
> This detracts from the security of the overall system.  People
> writing scripts sometimes want to save/restore state (like
> variables) and will restore the content from a noexec point using
> the aforementioned source command without realizing that it executes
> code too.  Of course their code is wrong, but it would be nice if
> the system would catch & reject it explicitly to stave of
> inadvertent usage.

I don't think it makes sense for a userland program to be an enforcer
of this type of check.  It gives a false impression of a security that
does not exist.  Which I think is more dangerous.  It will almost
certainly get in the way of a reasonable use case.  And nothing
prevents one from running a private copy of a shell without such a
check.  Or any of the many compatible /bin/sh variants such as ksh,
zsh, ash, dash, and so forth.


Re: Design question(s), re: why use of tmp-files or named-pipes(/dev/fd/N) instead of plain pipes?

2015-10-23 Thread Bob Proulx
Linda Walsh wrote:
> Bob Proulx wrote:
> >Where does the OS claim it is a file descriptor?
> 1) in the kernel documentation:
> file 'devices.txt:
> /dev/fd   /proc/self/fd   symbolicFile descriptors

That shows the file descriptor map for the process.  It isn't the
program's actual internal file descriptor.  Obviously it can't be.
They have abbreviated it trying to make it be concise and fit there.
If you disagree with the description then take that up with the
devices.txt authors.

> 2) ls -Ll shows the type of what the symlink points to:
> >ls -Ll /dev/fd/
> total 0
> crw--w 1 law tty  136, 0 Oct 21 19:49 0
> crw--w 1 law tty  136, 0 Oct 21 19:49 1
> crw--w 1 law tty  136, 0 Oct 21 19:49 2

The -L hides the symlink.  Try it without the -L.

  $ ls -log /dev/fd/
  total 0
  lrwx-- 1 64 Oct 23 15:45 0 -> /dev/pts/13
  lrwx-- 1 64 Oct 23 15:45 1 -> /dev/pts/13
  lrwx-- 1 64 Oct 23 15:45 2 -> /dev/pts/13
  lr-x-- 1 64 Oct 23 15:45 3 -> /proc/31883/fd/

It is showing where the program's file descriptors are redirected to
and from.  But those are still file path names.  Paths.  Names.  This
is an email from me to you.  I am not defining this exactly for a
standard.  It is a string of characters.  It isn't the integer value
index offset into the data structure holding the file data inside the
program.  It's not.  Don't confuse the two.

> and 3)
> YOU claim it is a descriptor further down in this note:
> >  tmpname=<(sort /dev/null); ls -l "$tmpname"
> >ls: cannot access /dev/fd/63: No such file or directory
> >
> >But as soon as the first commend is finished at the ';' then the
> >descriptor will be closed and the path will evaporate.  Therefore the
>  ^
> >path won't exist for the second command.

At that point I am talking about the program's internal behavior.  I
also mispelled command too.  I am talking conversationally here.

   if (close(fd) < 0) { ...; exit(1); }

Don't confuse the integer value of the program's file descriptor,
which is an offset into the program's data structure array, with a
path to a file.  In this case they are confusingly similar because the
/proc one is actively trying to represent the other!  But life is like
an analogy and being similar does not make it the actual thing.
Actively trying to represent the other here means one is really a
mapping show how one is mapped to the other.  That doesn't make it
actually the other.

>   If you close a file-descriptor to a normal file
> it doesn't disappear.  If it was deleted via 'rm' the name disappears
> immediately, and the only access to it is through the descriptor.
> In this case, closing the descriptor deletes the temporary
> name in the process's 'file descriptor' "/fd/".  Normal filenames
> are not "auto-deleted" when you close an I/O handle to them.

Confusion such as yours is precisely why I cringe whenever I see
someone using /dev/stdin, /dev/stdout, /dev/stderr or any of the
/dev/fd/* symlinks in a program instead of using the normal
redirections.  It causes people to think they are something that they
absolutely are not.  That and the unportant nature of them.

The /proc/self/fd/ directory is a kernel construct.  It does not have
normal file system semantics.  I am sorry to be the bearer of bad news
to you.  But it doesn't.  If you don't like this then that would be a
discussion with the kernel folks responsible for it.

And really what would be a sane purpose in being able to remove files
from it?  What would that do?  Would that make any sense at all?  I
think it wouldn't make any sense.

> >But if you use it for the ls command itself then it exists for that
> >command.
> >
> >  ls -l <(sort /dev/null)
> >lr-x-- 1 rwp rwp 64 Oct 19 15:56 /dev/fd/63 -> pipe:[102059434]
>   But only as a pointer to something one can do I/O on.
> You can't set any file attributes or metadata on "pipe:[]" It's not a
> real file somewhere.

Exactly!  Stick with that thought.  You are on the right track with
it.  It is only a symlink in a /proc kernel space representation.  You
can do I/O on it but you can't actually change permissions or remove
it or other things.


Re: Design question(s), re: why use of tmp-files or named-pipes(/dev/fd/N) instead of plain pipes?

2015-10-19 Thread Bob Proulx
Linda Walsh wrote:
> Greg Wooledge wrote:
> >A simple example:
> >diff -u <(sort file1) <(sort file2)
>   You claim <(sort file1) is a filename?

Me too.

> >touch a
> >mv <(sort a) <(sort a)
> mv: cannot move ‘/dev/fd/63’ to ‘/dev/fd/62’: Operation not permitted
> The OS claims that <() generates a file descriptor -- not a filename.

Where does the OS claim it is a file descriptor?  "/dev/fd/63" is a
file name not a file descriptor.  And /dev/fd is a symlink into /proc

  $ ls -ld /dev/fd
  lrwxrwxrwx 1 root root 13 Sep 21 18:19 /dev/fd -> /proc/self/fd

Since it is in /proc it is simply a kernel space construct.  It won't
allow things like rename(2) to work upon it.

I think you were confused by the "fd" part of "/dev/fd" but it is
simply a textual name of the file path.

> It's the same with 'FILENAME'a used with "read < FILENAME >FILENAME"
> >read < FILE > FILE

Unless FILE is a pipe or a tty or other such device then reading and
writing the same file at the same time is a bad idea.  It will get
opened for truncate and empty the file before it can be read.

> >a=<(sort a); chmod +x "$a"

The above is confusing a file named "a" and a variable named "a" which
is always a bad idea.  Better to use different names in experients
such as that so that they don't get confused with each other.

  tmpname=<(sort /dev/null); ls -l "$tmpname"
ls: cannot access /dev/fd/63: No such file or directory

But as soon as the first commend is finished at the ';' then the
descriptor will be closed and the path will evaporate.  Therefore the
path won't exist for the second command.

But if you use it for the ls command itself then it exists for that

  ls -l <(sort /dev/null)
lr-x-- 1 rwp rwp 64 Oct 19 15:56 /dev/fd/63 -> pipe:[102059434]

> chmod: cannot access ‘/dev/fd/63’: No such file or directory
> >echo aaa > a
> >read var <(sort a) && echo "var=$var" >a
> ## the above hangs: if <(sort a), was a filename, read would
> ## have read 'aaa' ## from it, and returned immediately.  ## Then the 'echo'
> command would have put 'var=aaa' ## in 'a': instead:

Syntax error in the above.  You probably meant to have 'read' read
from the pipe and not hand it to read as a variable name argument.

  echo read var <(sort a)
read var /dev/fd/63

You probably meant to use a '<' input redirection there.

  read var < <(sort a) && echo "var=$var" >a
# equates to: read var < /dev/fd/63

That works as you expected.


Re: why must bash zap last search string just because we hit ^C?

2015-10-18 Thread Bob Proulx
Eduardo A. Bustamante López wrote:
> $ some long command I've typed 
> And then boss calls: URGENT! Run uptime and tell me the output, I'll:
> $ some long command I've typed 
> Press Ctrl-u
> $ _
> Write uptime RET
> $ uptime
>  11:23:45 up 4 days, 22:39,  4 users,  load average: 0.51, 0.86, 0.75
> $ _
> Press Ctrl-y
> $ some long command I've typed 
> :-)

Additionally let's say that along the way I killed some other text so
that C-y yanks back that more recently killed text instead of the one
I saved first.  In that case ESC C-y will rotate the kill ring to the
previous killed text.  Repeating ESC C-y again rotates to the one
before it.  So can walk back through until finding the text I want to
yank back.  :-)


Re: ! in PS1 in posix mode

2015-10-15 Thread Bob Proulx
Linda Walsh wrote:
> Haven't used ksh for some timeI thought the '!' stuff came
> from csh?  It seemed so pointless, since having to look up
> things by command number I thought, was way too much work... searching
> via a string in the line seemed so much faster...

You are not thinking about the environment in which csh was designed.
If you were not working on a CRT but were on a paper terminal then it
wasn't possible to implement a C-r reverse-search through the command
history and WYSIWYG editor.  That is why we have 'ed' and 'ex'
although at the time I used 'qed'.

During that era CRTs were new fangled things.  At the time csh was
popular most of us used paper terminals!  I missed the teletypes by a
few years but my introduction to computing at university was on
printing paper terminals.  Students were only allowed on the low end
paper terminals.  The CRTs were reserved for faculty and grad
students.  (Almost all of my CS classes were on paper terminals
dialing a rotary phone with an acoustic coupler at 300 bps to a
Honeywell.  "Brrappph" was the sound the terminal made as it printed
out lines of paper.  It would shake the table as the print head on the
carriage moved back and forth.  Working in the computer lab was very
LOUD back then.  Especially in a room full of other people all working
with the same type of terminals.)

In that environment the !67 type of commands make a lot of sense.  One
always put the command number in the prompt so that it was printed out
with the command.  Once run that number would always be unchanged
forever on the paper and easy to see and know exactly what was going
to happen.  Just look over the printed out history on the paper and
select some program arguments from one command and other arguments
from other commands and stitch them together.  Since they were all in
the growing roll of trash piling up out of the back to the terminal it
was fairly easy to do.  Take a pen and underline the numbers you used
often in that session to make them easier to see.


Re: SIGINT handling

2015-09-22 Thread Bob Proulx
Greg Wooledge wrote:
> Just for the record, ping is the *classic* example of an incorrectly
> written application that traps SIGINT but doesn't kill itself with
> SIGINT afterward.  (This seems to be true on multiple systems -- at
> the very least, HP-UX and Linux pings both suffer from it.)

The command I run into the problem most with is 'rsync' in a loop.

   0  Success
   20 Received SIGUSR1 or SIGINT

Which forces me to write such things this way.

  rsync ...
  if [ $rc -eq 20 ]; then
kill -INT $$
  if [ $rc -ne 0 ]; then
echo "Error: failed: ..." 1>&2
exit 1


Re: 4-byte script triggers null ptr deref and segfault

2015-09-17 Thread Bob Proulx
Greg Wooledge wrote:
> Brian Carpenter wrote:
> > While fuzzing GNU bash version 4.3.42(1)-release
> > (x86_64-unknown-linux-gnu) with AFL(, I
> > stumbled upon a 4-byte 'script' that triggers a null ptr deref and causes a
> > segfault.
> > 
> >
> Well, that's an annoying web-to-mail interface.  It didn't include the
> full bug report?

There is no web-to-mail interface.  The above was simply an email from
Brian citing the bug report he filed above.  He filed a ticket on the
web site and then mailed a reference to it here.


Re: Why does a Bash shell script write prompts followed by reads and do it right?

2015-09-02 Thread Bob Proulx
Robert Parker wrote:
> fputs(shortprompt, stdout);
> fflush(stdin);

Uhm...  fflush'ing stdin?  That doesn't make sense.  Obviously you
meant to flush stdout.

fputs(shortprompt, stdout);

That is likely the specific problem.  Unfortunately I have no time for
comments about the rest of the code.


Re: Why does a Bash shell script write prompts followed by reads and do it right?

2015-09-01 Thread Bob Proulx
John McKown wrote:
> Not a bug, so likely the wrong forum.


> Have you tried doing a fflush() after the fputs()? I.e. something to tell
> the Kernel to "write this out immediately!".

Almost.  It is the C library libc that buffers output with fputs() not
the kernel.  The libc normally checks to see if the output is a "slow"
device such as a tty device.  If so then there is no buffering.  If
the output is a "fast" device such as a file then libc buffers the
output into larger writes for efficiency.  But that only applies when
the output is redirected.  If it is a prompt then the output will be
to the tty and there won't be any buffering.  The fflush() in that
case won't do anything.

> In the case of write(), I think you need to use the O_SYNC flag in
> the open() call.

No.  That isn't necessary at all.  It doesn't matter to programs in
userland if the write is synchronous or not.  Userland programs can't
tell the difference.  Using synchronous I/O is all about what happens
at the instant of a power failure or hard system crash.

Robert Parker wrote:
> Yet when I attempt the same in a C program, the system always writes 2
> prompts, then waits for a read.

Basically that doesn't make sense.  I do not believe that is what is
actually happening.  Please go back and look more deeply.

Since you are talking about prompts I assume you are writing to a tty
device.  There is no buffering and the output will appear on the tty
immediately without need for opening it synchronously or flushing or
anything extra.


Re: Why does a Bash shell script write prompts followed by reads and do it right?

2015-09-01 Thread Bob Proulx
Robert Parker wrote:
> Yet when I attempt the same in a C program, the system always writes 2
> prompts, then waits for a read.

This isn't a C list.  But if you don't show us the code then no one
can give good help.

If you write, read, write and then it *must* try the read before the
second write.  If you are seeing the second write then it means that
the read returned after the first write.  What you say, writes two
prompts and then read doesn't make sense.

> Does not matter if I write(1, "..."); read(0, number, buffer); or use
> fputs("...", stdout); fgets(.
> The result is the same.
> And I have tried using readline without any better result.

This sounds like you simply have some basic problem issue.  You will
need to figure that out first.  If you write(), if you fputs(), and
you are not seeing the output then start there.  Don't move on to
anything else until you understand that part of the problem.

Your problem is really one for discussion on a C programming list.

> I have downloaded the source code for Bash.
> Would someone please let me know where to look to learn to make my code
> work like yours?

That is the wrong direction to go.  It is like saying, I have this
problem building a small bike shed.  I couldn't figure it out.
Therefore I downloaded the plans to the Eiffel Tower in order to
compare the two.  They are not really comparable.  Trying to do so is
just a distraction from solving your real problem.


Re: In the bash man page the entry for the 'read' command has two spaces after it instead of one.

2015-08-21 Thread Bob Proulx
Andreas Schwab wrote:
 Ángel González writes:
  entering  «/^ *read⤶» may be easier to type (and remember)
 It won't match, though.

It matches fine for me.  Although I suggest adding the square bracket
so as to avoid the other false positives as long as one knows that
read has options documented.  This works.

  /^  *read  *\[


Re: Worth mentioning in documentation

2015-08-10 Thread Bob Proulx
Greg Wooledge wrote:
 Juanma wrote:
   [ is an ordinary command (a shell builtin)
  Here is another point I find confusing: I thought a shell builtin didn't
  have a separate binary executable file, like 'cd' (which cd = fail), but
  some of them do have such form (which [ = /usr/bin/[ ; which pwd =
  /bin/pwd). I also fail to see how 'test' modifies the state of the shell
  itself (like 'cd' does), or why it is impossible or inconvenient to obtain
  [its functionality] with separate utilities.
 Don't use which(1).  Which is an external program, so it has no knowledge
 of the shell's builtins, aliases, functions and keywords.  Instead, use

Another problem with 'which' is that it was originally designed and
written for csh users specifically.  It was a csh script and would
source the users ~/.cshrc file so as to acquire their aliases from
there so as to be able to report user aliases.  I think it is still
that way on HP-UX (and probably others) for example.

However that obviously won't work well for bash, ksh, zsh, and other
shell users.  And IMNHO csh is a terrible shell regardless of the tcsh
users using it who think otherwise.  Therefore some distributions such
as Debian have rewritten 'which' as a /bin/sh script meaning that
'which' behaves differently on different systems.  It is non-portable.

 imadev:~$ type cd
 cd is a shell builtin
 imadev:~$ type [[
 [[ is a shell keyword
 imadev:~$ type -a test
 test is a shell builtin
 test is /usr/bin/test
 test is /bin/test
 Bash implements test as a builtin not because it's necessary, but simply
 for efficiency.  Forking a whole process to test whether two strings are
 equal would be horribly wasteful.

An important mental concept is that test and [ must *behave* the same
as if they were an external program.  They are internal builtins for
efficiency now but everything behaves the same if they are external
programs.  This is why quoting as if they were external programs is
required.  On the other hand [[ has always been a builtin and
therefore the shell can avoid one layer of quoting and does.


Re: bash displays strange characters after base64 decoding

2015-08-07 Thread Bob Proulx
Valentin Schmidt wrote:
 Subject: bash displays strange characters after base64 decoding
 Bash displays strange characters (including command prompt) (see attached
 png file) after the execution of the base64 (GNU coreutils 8.21) command
 like described as follows:
 base64 -d base64.txt

This is not related to your command shell bash.  It is only your
terminal that is involved.  Your terminal interprets escape sequences.
Move the cursor.  Change the color.  Use reverse video.  Use a
different character set.  Things like that.

You have sent random binary characters to your terminal.  This is
similar to if you decided to eat random non-food items from random
shelves in a store.  Newspapers.  Shampoo.  Sunscreen.  Things like
that.  It won't be good for you!  Eventually you will eat something
that will upset your tummy and send your body into a bad state.  Not

Sending random binary characters to your terminal is similar to this.
It is not good for your terminal.  With enough random characters it is
inevitable that you will eventually send your terminal something that
will instruct it to go into an undesirable state such as you have

 The aim was actually to direct the output of the base64 command into a file
 (would have been a .jpg file) but the decoded.jpg was forgotten.

Oops. :-(

 Accordingly the output was directed to stdout instead. This resulted in the
 command prompt consisting of strange characters after decoding.

That is unfortunate.  However terminals are called smart terminals
for the reason that they interpret escape sequences.  Because this is
useful for them to do this.  There are dumb terminals available too.
Dumb terminals do not interpret escape sequences and would be safe
from this problem.  Except people generally do not like dumb terminals
because they do not have any of the useful capabilities we have come
to like.  Other than switching to a dumb terminal there is little that
can be done to prevent this.  Other than avoiding feeding your
terminal random binary characters.

 Also any command entered via the keyboard resulted in strange characters
 being displayed instead of the typed characters.
 Also the output of the ls (which seems to be executed) command (as an
 example) is displayed as strange characters.

I don't recognize the terminal emulator program you are using.  One of
the menu options may offer a way to reset it.  Otherwise you can try
the reset command to reset the terminal back to a sane state.

  man reset

   When invoked as reset, tset sets cooked and echo modes, turns
   off cbreak and raw modes, turns on newline translation and
   resets any unset special characters to their default values
   before doing the terminal initialization described above.  This
   is useful after a program dies leaving a terminal in an
   abnormal state.  Note, you may have to type


   (the line-feed character is normally control-J) to get the
   terminal to work, as carriage-return may no longer work in the
   abnormal state.  Also, the terminal will often not echo the

Otherwise the best you can do is to close the terminal and start a new


Re: ShellShock where my browser window randomly re-encodes with what looks like Chinese

2015-07-06 Thread Bob Proulx
S. Irelan wrote:
 I have vested so very many hours and attempts and well, can't get
 out from under what I understand is the Shellshock /bash hacking
 browser attacking NIGHTMARE !!!

Your report does not indicate any similarities to the well known
Shellshock bug.

 Subject: Re: ShellShock where my browser window randomly re-encodes
 with what looks like Chinese

Nor does it seem related to anything that would have any result in a
web browser.

 My name is shaun, I have a knoppix that might very well be knoppix64
 and I have tried the suggested knoppix/adriane fuse thingy (pardon
 my lack of prowess) and I need so help to understand how I can fix
 without having the ability to enter alot of code.
 I have tried within terminal:
 [ 7397.465665] usb 2-1: Manufacturer: Kingston[ 7397.465670] usb 2-1: 
 SerialNumber: 5B6C10933DEF
 [ 7397.466183] usb-storage 2-1:1.0: USB Mass Storage device detected
 [ 7397.466740] scsi9 : usb-storage 2-1:1.0
 [ 7398.470159] scsi 9:0:0:0: Direct-Access Kingston DataTraveler 2.0 PMAP 
 PQ: 0 ANSI: 0 CCS
 [ 7398.470441] sd 9:0:0:0: Attached scsi generic sg2 type 0
 [ 7398.471394] sd 9:0:0:0: [sdb] 4030464 512-byte logical blocks: (2.06 
 GB/1.92 GiB)
 [ 7398.474041] sd 9:0:0:0: [sdb] Write Protect is off
 [ 7398.474046] sd 9:0:0:0: [sdb] Mode Sense: 23 00 00 00
 [ 7398.474632] sd 9:0:0:0: [sdb] No Caching mode page found
 [ 7398.474637] sd 9:0:0:0: [sdb] Assuming drive cache: write through
 [ 7398.478157]  sdb: sdb1
 [ 7398.481770] sd 9:0:0:0: [sdb] Attached SCSI removable disk
 [ 7399.305618] FAT-fs (sdb1): utf8 is not a recommended IO charset for FAT 
 filesystems, filesystem will be case sensitive!
 [ 7399.312661] FAT-fs (sdb1): Volume was not properly unmounted. Some data 
 may be corrupt. Please run fsck.

I see you have mounted a USB storage device containing a FAT
filesystem.  That is all that the above shows.

 knoppix@Microknoppix:~$ sudo dd if=gnome-3.16.x86_64.iso.crdownload 
 of=/dev/home/knoppix bs=8M conv=fsyncdd: opening 
 `gnome-3.16.x86_64.iso.crdownload': No such file or directory

You are trying to dd copy a file but that file does not exist and dd
reports that problem to you.

You apparently did not notice that the file name ended with the
temporary suffix .crdownload and (appears to me) to be a temporary
file name for the file.  It was likely renamed to be the actual name
after the download was completed.

None of that appears to be related to bash in any way.  Neither does
it seem to be related to any web browser problem either.

I don't know where I would direct you for further assistance.  I think
if you could locate a local user group community they would provide
useful assistance to you face-to-face.


Please pardon the incorrectly handled anti-spam

2015-07-06 Thread Bob Proulx
Please pardon the incorrectly handled anti-spam message re-send seen
today.  One of the listhelper volunteers had a problem vivifying an
incorrectly classified false-positive-as-spam message.  Accidents
happen.  Your patience and tolerance is appreciated.


Re: Feature : walkthrough lines of stdout

2015-06-17 Thread Bob Proulx
Hrazel wrote:
 It would be so cool just to [shift] + [arrow up]  to go to the line of
 interest. [arrow right]  to copy it to the clipboard and return to the
 normal console again. then I can decide what to do with it next. 

This would need some type of wrapper around everything such as GNU
'screen' or as Andreas Schwab hints at emacs interactive shell mode.

Have you looked at screen's copy and paste mechanism?

There are also programs such as xclip and xsel that interface with X
and allow you to cut and paste text from the command line.  If you
know you want the output of a program into the cut buffer you can pipe
it to xclip.  Or pull text out of the buffer with xclip.


Re: Should nested case statements within command substitutions work on in bash 3.2.x?

2015-03-21 Thread Bob Proulx
Jon Seymour wrote:
 I was surprised that this didn't work with the OSX version of bash 3.2:
  /bin/bash -c 'echo $(case yes in yes) echo yes; ;; no) echo no; ;; 
 /bin/bash: -c: line 0: syntax error near unexpected token `;;'
 /bin/bash: -c: line 0: `echo $(case yes in yes) echo yes; ;; no)
 echo no; ;; esac)'

 It does work with bash 4.x.

It was improved later.  But I see that zsh as of 5.0.7 still has the
same problem with that construct.

 Is this a known issue with 3.2 or is it particular to the OSX
 implementation (which in my case is 3.2.53(1))?

It is that way in 3.2 and not specific to your OS X implementation.

Of course you know that if you match the parens then it will parse
correctly.  Matching parens has been the standard way to handle this
case for all of the shells.

  $ bash -c 'echo $(case yes in (yes) echo yes ;; (no) echo no ;; esac)'


Re: Feature: disallow |

2015-03-18 Thread Bob Proulx
Ed Avis wrote:
 Thanks.  So 'whoami | out' working is a natural consequence of the
 fact that 'out' is a command by itself.

It is a natural consequence of parsing redirections before parsing
commands.  The redirection happens before the command execution and an
empty command is valid.  That redirections happen in an earlier pass
across the command line is why they can occur anywhere in the command.
By convention we put rediections at the end.  But they can also
validly occur first (as the previous example showed) or in the middle.

 IMHO it would have been better for the Unix shell to forbid that,
 and require ': out' if you really do want to run the null command
 and redirect its output, but it's too late now.

If it did then it would be incompatible with all of the rest that
allow it.  As I recall one of the obscure fringe Unix lookalike
systems did exactly that in their shell.  I forget details now.  But
because of that some of us do always use : instead of an empty
redirection simply due to it.


Re: Command line prefixed with space is not saved

2015-02-10 Thread Bob Proulx
Vladimir Kanazir wrote:
 uname output: Linux canny 3.13.0-34-generic #60-Ubuntu SMP Wed Aug 13 
 15:45:27 $

   When you type space before the command, the command is not
   saved in the history.
   You can't see it when arrow up / CTRL+P is pressed. It is
   like the command is never executed.

The bash default is not to set that variable.  But it looks like you
are using Ubuntu and the Ubuntu package default is to set it in the
default .bashrc file as a user level customization.

If you don't like that configuration then edit your ~/.bashrc file and
change this:

  # don't put duplicate lines or lines starting with space in the history.
  # See bash(1) for more options

Change that to something different.  Perhaps simply ignoredups?


Or whatever you prefer there.  It is your .bashrc file.  Customize it
as you desire.


Re: Command line prefixed with space is not saved

2015-02-10 Thread Bob Proulx
Vladimir Kanazir wrote:
 Bob Proulx wrote:
 The bash default is not to set that variable.  But it looks like you
 are using Ubuntu and the Ubuntu package default is to set it in the
 default .bashrc file as a user level customization.
 Yes, they broke many things in 14.04 LTS...

Well...  It has been packaged that way for years and years and years.
I didn't look to see how long but this really isn't anything new.

Note if you are not aware...  The default files are packaged into the
/etc/skel directory and then used whenever a new user is created.  The
/etc files are conffiles and if you wish to customize the behavior on
your system then changing them there would be the place to do it.
Then any new users created on your system get your new customizations.


Re: Does [ -f FILE ] have a bug on testing a symlink ?

2015-02-09 Thread Bob Proulx
Cheng Rk wrote:
 According this documentation `help test`, I am expecting it should
 return false on anything other than a regular file,
 -f FILETrue if file exists and is a regular file.
 but why it returned true on a symlink to a regular file?
 $ [ -f tmp/sym-link ]  echo true

Symlinks originated in BSD.  GNU is implementing previously existing
behavior.  It is a BSD feature that migrated to System V and then to
GNU and elsewhere.  The behavior is now standardized by POSIX so that
it will always be the same everywhere and it is possible to write
portable scripts that behavior the same everywhere.

The idea is that symlinks should be completely transparent and
invisible to most applications.  The idea is that when normal things
are run they don't realize they are using a symlink.  Behavior would
be exactly the same as if it were a regular file system link.  That is
what is happening in your test case.  The symlink references an
existing file, is a regular file and exists, therefore the status is

If you have an application that needs to know if something is a
symlink then then you will need to test for that case explicitly.

Since this is a question and not a bug in bash it would be better
discussed in the mailing list instead.  Please send
future use discussion there.  Thanks.


Re: bash complete.c

2014-12-27 Thread Bob Proulx
Vincent Leduc wrote:
 The null redirect was used for the /etc/issue to not be shown
 here. I mean although we dont have any tty attached, i assume the
 builtin should not do this. I only tested it with an older release,
 as you said, it seems to be corrected.

If I don't include -t then I get the following back from bash.

  bash: cannot set terminal process group (-1): Inappropriate ioctl for device
  bash: no job control in this shell

I assumed that was what you were trying to redirect.  Adding the -t
sets up a tty and that particular part of the problem is avoided.

As far as /etc/issue goes another useful thing is to touch a file
named $HOME/.hushlogin.  If .hushlogin exists then the /etc/issue
and /etc/motd are not emitted at login time.  That is much nicer for
me.  I am well aware of the uname -a and license status of the systems
I am using.  I don't need to see that same message again and again.
This is documented in the login(1) man page.

  man login

   Suppress printing of system messages.

And so a normal thing for me when getting an account set up is to
always create that file in order to make hopping around systems less

  touch ~/.hushlogin


Re: bash complete.c

2014-12-26 Thread Bob Proulx
Vincent Leduc wrote:
 sh-3.2# ssh localhost bash -i 2/dev/null
 root@localhost's password:
 Display all 1797 possibilities? (y or n)[enter]

Although I can reproduce this in bash 4.2 it is not reproducible for
me in bash 4.3.  I assume that it was improved there.

One problem in your example is that your bash does not have a tty
attached to it.  Add -t to ssh and you will then get a tty and you
won't need to redirect 2/dev/null anymore.

  ssh -t localhost bash


Re: Documentation Update

2014-12-12 Thread Bob Proulx
Greg Wooledge wrote:
 David J. Haines wrote:
  When started interactively, bash sets the extglob shopt; however, that
  fact seems to have been overlooked in the manpage.
 This is a compile-time setting.  Your vendor probably turned this on.
 imadev:~$ bash-4.3
 imadev:~$ shopt | grep extglob
 extglob off
 imadev:~$ exit

Also some vendors place a lot of customization in /etc/profile,
/etc/profile.d/*, /etc/bash_otherplaces sourced in the above.  It
might be turned on by one of those.

I see a lot of hits in the bash_completion add-on files.  Should we be
looking at:

  $ grep -r shopt -s extglob /etc/bash*
  /etc/bash_completion.d/nmh:shopt -s extglob
  /etc/bash_completion.d/ $g -eq 0  shopt -s extglob
  /etc/bash_completion.d/mercurial:shopt -s extglob
  /etc/bash_completion.d/subversion:# pattern matching enabled (use 'shopt -s 
extglob progcomp' to enable
  /etc/bash_completion.d/subversion:shopt -s extglob


Re: Documentation Update

2014-12-12 Thread Bob Proulx
Dennis Williamson wrote:
 Bob Proulx wrote:
  I see a lot of hits in the bash_completion add-on files.  Should we be
  looking at:
$ grep -r shopt -s extglob /etc/bash*
/etc/bash_completion.d/nmh:shopt -s extglob
/etc/bash_completion.d/ $g -eq 0  shopt -s
/etc/bash_completion.d/mercurial:shopt -s extglob
/etc/bash_completion.d/subversion:# pattern matching enabled (use 'shopt
  -s extglob progcomp' to enable
/etc/bash_completion.d/subversion:shopt -s extglob

 Perhaps you should check for shopt -u extglob reversing those.

Good point.

  $ grep -r shopt -. extglob /etc/bash*
  /etc/bash_completion.d/nmh:orig_opts=$(shopt -p extglob) 
  /etc/bash_completion.d/nmh:shopt -s extglob
  /etc/bash_completion.d/ -q extglob  g=1
  /etc/bash_completion.d/ $g -eq 0  shopt -s extglob
  /etc/bash_completion.d/  test $g -eq 0  
shopt -u extglob
  /etc/bash_completion.d/ $g -eq 0  shopt -u extglob
  /etc/bash_completion.d/mercurial:shopt -s extglob
  /etc/bash_completion.d/subversion:shopt -s extglob


Re: bash 4.3: Clear Scrollback and Reset (Ctrl-Shift-X) in konsole stopped to work as expected when upgrading from upstream bash 4.2 to upstream bash 4.3

2014-11-29 Thread Bob Proulx
Askar Safin wrote:
 Okey, so, Chet, what will you say about resizing bug? Is this a bug?
 At this moment I doesn't ask where (readline or konsole) this bug
 resides. I'm just asking: is this a bug? Or long line doesn't move
 on resize is intended behavior?

In my opinion the bug is in your expectations.  You are asking the
terminal emulator to clear the screen history.  It does.  This
includes the prompt too.  It does exactly what you asked.  There isn't
any surprise.  You see the screen and know its state before the action
and know whether it is displaying a prompt or whether it is currently
running a program.  You clear the screen.  The screen is cleared.
There is no prompt but you should know that in order to get a prompt
all you need to do is hit Enter or Control-l and bash will print
another prompt which will then be displayed.  Or you could type in a
command right then regardless.  And of course it might be running a
program and not at the bash prompt so whatever happens must work
reasonably with any program.

However you are expecting that after instructing the terminal to clear
screen history that bash should emit a new prompt.  I think that is an
unreasonable expectation.  There hasn't ever been a communication
interface for doing such a thing.  One could be created.  But if so
then that would be a *new* interface.


Description: Digital signature

Re: Sorting processes by NI

2014-11-14 Thread Bob Proulx
Artem Yegorov wrote:
 Hi, why does not the sort processes by priority? This is a bug or not?
 I attached a screenshot with a problem.
 I use VMware Workstation and Ubuntu 14.04 LTS

You seem to have confused the 'ps' command with 'bash' and reported
something that you think should be different about 'ps' to the 'bash'
bug list.  Wrong mailing list.  This is the bug reporting list for
bash and not for ps.

Also please do not send graphical screenshots to mailing lists.  Your
109K image was only a few bytes of actual information.  In the future
if you want to show text output then simply cut and paste the text
output.  Do not include an image capture of the text.

As to your 'ps' issue, why would 'ps' sort by priority?  The man page
for 'ps' clearly says Output is unsorted by default.  If you want it
sorted then you will have to read the 'ps' man page and add your
preferred sort order.

If you wish to followup on the ps issue please do so on the mailing list and not here on the bash bug list.
The help-gnu-utils list is a relatively generic mailing list for
random help discussion about the utilities and the discussion would be
fine there.


Re: [PATCH] bracketed paste support

2014-11-04 Thread Bob Proulx
Daniel Colascione wrote:
 Bob Proulx wrote:
  I use paste into the shell with an embedded newline in order to
  immediately execute a command *a lot*.  If that were removed I would
  be very unhappy.
 I strongly doubt that your use case is typical. 

It doesn't matter if it is typical or not.  A feature need not have a
51% majority in order to be retained.  Everyone does things
differently.  If we were to examine your operating modes carefully I
am sure that we would find some practice that you use often that few
other people are using and would be annoyed greatly if removed.
Furthermore it is impossible to know how many people use it.  This is
a long standing feature that if removed would disenfranchise the users
of it.  That is all that we can say about it.

I very often triple-click to select by line and then paste it into the
shell for immediate execution.  I less often triple-click to select
multiple lines of a command set and paste them into the shell for
immediate execution.  I am sure there are many others not in this
discussion who do the same.

 I've asked several of my colleagues; all have complained about
 accidentally pasting a large amount of text into the shell at one
 time or another. Nobody has complained about losing automatic
 execution of code after paste.

Unfortunately that is not a useful survey.  It is self-fullfilling!
If I ask ten of my peers of whom I have each taught a feature if they
use that feature then the answer will of course be yes.  If on the
other hand I have vilified a feature with my peers then it is likely
that the reverse would be true.  A survey asking colleagues around you
is simply not a worthwhile survey.  Sorry.

Knowledge and use tends to group and cluster.  Birds of a feather
flock together.  This is one of the reasons why things like
conferences, newsgroups, and mailing lists are so useful.  It allows
people to meet others and learn about practices outside of their
normal routine.  People not only learn other things for themselves but
they also learn that there is diversity in others too.  In this case
you learn that a feature that you despise is one that is liked by

 Speaking from personal experience, I've been using terminal emulators of
 various sorts for almost 20 years, and in that time, I've accidentally
 pasted documents into my terminal orders of magnitude more often than
 I've deliberately pasted multi-command sequences.

I have used terminals for a very long time as well.  I am using one
now.  I won't say that I haven't sometimes pasted in paragraphs of
text into a terminal by mistake.  I have.  I also use kitchen knives
to chop vegetables.  I have sometimes cut myself doing so.  That
doesn't prevent me from avoiding cutting up vegetables with a kitchen
knife.  I have also accidentally removed a file with rm that I didn't
intend to remove.  In none of those cases do I want to remove rm or
kitchen knives from the world to prevent anyone from doing either of
those things again.

 As far as I'm concerned, automatic execution of code on paste isn't a
 feature: it's a bug and security hole. Users should have to opt into
 security holes. 

It is not a security hole.  Simply declaring it to be so does not
make it so.  I get that it is a feature you hate.  But labeling it
with incorrect negative labels is simply a normal day of US political
advertisements.  It has no business on a technical list.

 We've only lived with the existing behavior so long because it's
 only recently become possible to distinguish pastes from other

And I think it is great that you can now have it operate differently.
Isn't it a wonderful world that we live in?


Re: [PATCH] bracketed paste support

2014-10-31 Thread Bob Proulx
Chet Ramey wrote:
 * paste into the text editor invoked by the `edit-and-execute-command' key
   binding (C-xC-e in emacs mode), edit the command if desired, and have the
   shell automatically execute the contents of the editing buffer when the
   editor exits

Cool!  This was an idea that I hadn't thought about before.  (I often
would paste into my editor and clean it up.  But doing it as part of
the edit-and-execute-command is very nice!)  

Thanks for this hint!


Re: [PATCH] bracketed paste support

2014-10-30 Thread Bob Proulx
Daniel Colascione wrote:
 Well, I don't know whether Chet left the feature enabled by
 default. I hope he did, though, since preventing execution of pasted
 commands is one of the feature's key benefits. In bash, you should
 be able to execute a pasted command sequence by typing RET after the
 paste, but a paste by itself should never begin execution.

I use paste into the shell with an embedded newline in order to
immediately execute a command *a lot*.  If that were removed I would
be very unhappy.

However I read from Chet's replies that this is not the case.  So I am
not worried.  Just voicing a counter opinion that this shouldn't be
removed.  If you want it then you should enable it.  This is like many
other options available in the shell.  They are optional.

 For better or worse, people copy and paste commands from websites
 all the time. Even if a bit of shell looks innocuous, a malicious
 bit of JavaScript could change the pasted text at the last second
 without the user being aware of the switch. 

Then the browser shouldn't copy it.  The place to fix the problem is
in the browser.  A problem with a browser should not break cutting and
pasting in general.

 (Tynt uses this technique to slightly less malicious ends.) If
 pasting in a terminal immediately begins execution, there's no
 opportunity to review pasted code. With bracketed paste support, the
 shell requires additional user interaction after a paste to begin
 execution, making this attack much less effective.

I am very happy this is a feature you like.  However I would hate that
feature.  It should definitely not default to on or it will make
people like me very unhappy.


Re: Random loss of bash history

2014-10-09 Thread Bob Proulx
Darshit Shah wrote:
 I'm running Bash 4.3.30(1)-release on Arch Linux. Recently after I rebooted
 by system I realized that all my Bash History had been erased. The
 ~/.bash_history file existed, but it was completely empty.
 The situation occurred just now once again. I boot up my system and realize
 that I have no bash history at all. The issue does not occur at every boot,
 it's random and I am unable to correlate the two instances in which this has
 happened to me.
 I haven't any idea as to how I should debug this problem to provide more
 information. Could you please start out by helping me there?

What type of file system is your ~/.bash_history located upon?  This
sounds like a deja vu from the ext4 file system.  Do you observe any
other zero sized files after a reboot?


Re: Detecting invocation as /bin/sh ?

2014-09-30 Thread Bob Proulx
Alexandre Ferrieux wrote:
 Greg Wooledge wrote:
  Unfortunately, most people don't actually express an intent when they
  use #!/bin/sh.  They just copy some code from a random script they found
  somewhere, without understanding what it means.
 That song reminds me of PHP. I was under the optimistic assumption
 that bash, given its noble ancestry, was not exactly targeting the
 same audience.

The shell is part of the landscape.  It has been around since the
beginning.  Although some try to avoid it others embrace it.  It is
not a matter of targeting an audience.  It is simply the way things

  The number of scripts that use #!/bin/sh but then go on to use Bash
  syntax is higher than you might believe. 

Agreed.  But those are not portable.  And they are one of the reasons
behind dash and the other targeted shells.

 OK. Sh*t happens. Bozos write scripts. So what ?
 A more interesting question IMO is:
  How many packaged init scripts / mission-critical daemons are
  written by people that cannot be bothered to write #! /bin/bash
  when they require bash ?

Lots and lots and lots.  When Debian converted from /bin/sh being bash
over to it being dash there were many bugs found.  The process of
getting those instances cleaned up took a long time.  Years.

  It's not a stretch of the imagination to suppose that someone has used
  exported functions in a #!/bin/sh script on a Linux box where /bin/sh
  is a symlink to bash.
 Except that script would instantly break if *dash* was installed,
 instead of bash, as target of the sh symlink. So that beast would
 already be extremely fragile.

Agreed to both.  Definitely fragile.  But people do it just the same.
Personally I would vote to have bash not support exported functions in
/bin/sh mode.  But as Chet has indicated the task of maintaining a
stripped down shell is the target of dash and posh and others and not
the target for bash.  It would be duplicated effort if bash also
targeted that too.  So while bash might or might not do some of that
it isn't really the project for it.


Re: Not so useless use of cat

2014-09-16 Thread Bob Proulx
Ralf Goertz wrote:
 Actually things are more complicated. I do need the /dev/stdout part. I
 obiously don't have the problem with `cat' but with some other program
 that doesn't write to stdout per se and expects a -o parameter for the
 output file. And this program just accepts one input file. I merely used
 the first `cat' in my example to make my point. So what I wanted was

I suspected that was the case.  That is generally the way it goes.

But does the program really have no way to write to stdout?  I despise
programs such as those.  They should be fixed so that there is a way
to use them as a filter.  Because working around those programs by
using Linux specific features such as /dev/stdout makes the entire
bundle non-portable to systems that don't have /dev/stdout.

 $ for i in file[12] ; do program -i $i -o /dev/stdout ; done  outfile
 which I assumed to be elegant and would do as I expected except it
 didn't and I really thought it could have been a bug. That's why I
 reported it here.

I guess you could create as many temporary files as needed and then
concatenate them with cat afterward.  Or if the program can't be fixed
to append or write to stdout directly then use the not so useless cat
after all.  Probably needs a comment in that case though! :-)

There is another possibility.  I don't actually like this one as
much.  It seems more confusing.  It uses more processes.  But try this:

  for i in file[12] ; do program -i $i -o (cat) ; done outfile

Good luck!

Re: Not so useless use of cat

2014-09-16 Thread Bob Proulx
Greg Wooledge wrote:
 It's important to note that the following two cases are *not*
cat $i /dev/stdout
program -i $i -o /dev/stdout
 In the first case, the /dev/stdout is part of a redirection.  On
 platforms that do not have a native /dev/stdout in the file system,
 Bash handles this *internally* (and it would actually work the way
 you expected).
 In the second case, /dev/stdout is just a string as far as Bash is
 concerned.  It is passed verbatim to program as an argument.  There is
 no internal Bash magic happening.  You get the underlying operating
 system implementation, or you get no such file or directory.

That is one of the reasons I don't like the /dev/std{err,in,out}
things.  They are not portable.  They do different things on different
systems.  I avoid them.


Re: Not so useless use of cat

2014-09-16 Thread Bob Proulx
Chet Ramey wrote:
 Bob Proulx wrote:
  That is one of the reasons I don't like the /dev/std{err,in,out}
  things.  They are not portable.  They do different things on different
  systems.  I avoid them.
 I've considered emulating it everywhere, regardless of what the OS
 provides, but I'd get just as many complaints if I did that.

Agreed.  There isn't any way to win.

Tonight I came up with yet another reason I don't like use of the
/dev/stdout and others.  And also the use of /proc too.  That reason
is use of programs in chroots.  I ran into a program installer that
required that both /proc and /dev be mounted in the chroot I was
using.  I can't see any reason those are actually needed.  But it
required them both in order to function.  Things like that get in the
way of using lightweight chroots which would otherwise work fine as a
light weight container.


Re: Not so useless use of cat

2014-09-13 Thread Bob Proulx
Dennis Williamson wrote:
 Bob Proulx wrote:
  And the subshell isn't needed either.  Use a list.
{ for i in file[12] ; do cat $i ; done ;}  both
 There's no need for the curly braces and the last semicolon.

Of course you are totally right.  I was distracted by the subshell as
a concept.  For re-stitching file with redirections a subshell isn't
needed and a list is convenient.

 Note that the loop in this case can be replaced by
 cat file[12]  both
 I failed to spot that in my earlier reply.

Me too.  (But usually these are from more complex examples that can't
be simplified as much.  It is just that all of the details aren't shown.)


Re: Not so useless use of cat

2014-09-12 Thread Bob Proulx
Ralf Goertz me@myprovider.invalid wrote:

Since you have used an invalid address I assume you are reading the
mailing list via a web archive or other means and did not CC you.

 Why do I need cat (the second on) here?

You don't.

 $ echo first file1
 $ echo second file2
 $ (for i in file[12] ; do cat $i  /dev/stdout ; done) | cat  both

Why do you want /dev/stdout there?

 $ cat both


 If I omit the | cat after the loop I get

  (for i in file[12] ; do cat $i  /dev/stdout ; done)  both

 $ cat both

Because the /dev/stdout truncates the output.  It writes the first.
Then the second one truncates the file and then writes the second.
Remove that from the line and it will print both lines.

  (for i in file[12] ; do cat $i ; done)  both

 Even when using  both instead of  both only second makes it
 into both. Why?

All that the , , 21, types of redirections do is to attach the
associated files with different files.  Once associated the further
actions such as truncation happen to those files.  Therefore the
truncation associated with /dev/null truncates the file that on the
far right was both.

Try this:

  (for i in file[12] ; do cat $i  /dev/stdout ; done)  both

But really I think you are better off forgetting about /dev/stdout.
That is almost like the useless use of cat.

And the subshell isn't needed either.  Use a list.

  { for i in file[12] ; do cat $i ; done ;}  both

 I would have thought that using a subshell should be
 enough to protect the both from being overwritten. 

The subshell isn't really part of the issue.  The issue is that the
file descriptor is attached and truncated and that is orthogonal to
process boundaries.

Hope that helps!

In the future please consider asking questions such as these on for general discussion of bash shell scripting.  If
you expose a bug then report the bug to the bug reporting address.
The help-bash list is for help with bash scripting and this would have
been perfect there.


Re: SEGFAULT if bash script make source for itself

2014-08-28 Thread Bob Proulx wrote:
 Eric Blake wrote: wrote:
   IMHO any user action should not lead to SIGSEGV! I am not objecting 
   recursive sourse itself. But when I got SIGSEGV from bash, I have no
   idea why this is happened. I have made recursive sourse by mistake and
   spend a lot of time looking up what exactly lead to SIGSEGV.

But you wrote the program that caused the SIGSEGV.  At that point you
are no longer just a user but are now a programmer.  Technically
speaking the SIGSEGV problem would be a bug in your bash script
program.  You as the programmer of that script have the responsibility
for it.

  SIGSEGV is what happens on stack overflow, unless you integrate a stack
  overflow detector like GNU libsigsegv with your sources to catch the
  segv and replace it with a nice error message.

 I know when and why program can get SIGSEGV.

Then you already know that it is a recursion problem that has run out
of stack space.  Any program that allows recursion might be programmed
in error.  If this is not suitable then using a programming language
that does not allow recursion might be the choice for you.  For
example the old FORTRAN did not allow recursion and yet it enjoyed a
lot of popularity at one time.

 So why I should got SIGSEGV instead of nice, detailed error message in
 recursion? We can detect it?

Because in general it is a hard problem to solve.  And it isn't always
about making it bigger.  It may simply be a very small environment
without enough stack to complete.  The program may completely fine if
there were enough stack.  It is hard to tell if the program is in an
infinite recursion or if it simply didn't have enough stack space to
complete and would complete if there were more.  All that can be said
is that it didn't have enough stack.

  The GNU Coding Standards state that GNU software cannot have arbitrary
  limits by default.  Any limit we pick, other than unlimited (your

 IBM produce new CPU - it solve infinite loop in 8 seconds.
 How bigger amount of memory save from infinite recursion? It lead to bigger
 delay before SIGSEGV and nothing else.

Haha.  Of course it is a hardware problem.  If we only had a machine
with an infinitely large stack then we would never run out of stack
space and could never SIGSEGV due to stack overflow.

Of course then the program would simply run forever since it would
continue to do exactly what it had been programmed to do.  Which is
one of the things that makes this so hard.  In order for an automated
detection the program must say that the program should not do what the
programmer told it to do.  That is where it runs into problems.
Similar to why auto-correction spell checkers force so many spelling
errors on humans.

  $ ulimit -a
 data seg size   (kbytes, -d) unlimited
 so... in real life we have a limits. Some of them turned off, but they
 exists and can be ajusted.

Those are explicit process limits in addition to the actual limits.
It is always easier to make them smaller.  But you can't make the
actual limits larger.

For example feel free to try to make use of that unlimited data set
size.  Let's assume you have a X sized memory machine.  Try to use a
thousand times X that amount of memory regardless of it being set to
unlimited and see how well things work.

 And if I have an option, which I can change to some suitable value for me
 and this can save me/show me good error message in case of infinite
 recursion - I will use it. Other can leave it in ifinite position. We can
 have 2 options - one set recursion level limit, other set action when this
 limit is reached - deny deeper recursion / print warning.

There is the sharp kitchen knife thought-problem.  Have you ever cut
yourself on a kitchen knife?  Of course most of us have at one time or
another.  Can you envision a way to make kitchen knives safer?  A way
to make it impossible for you to cut yourself on one?  Think about how
you would make a kitchen knife safer.  Give it a serious think.  Then
ask yourself this question.  Would you use such a knife that you
designed yourself?  The answer is invariable no.  There has always
been some inherent danger when using a kitchen knife.  We accept this
because the alternatives are all worse.


Re: Interactive Expect from bash Script

2014-03-29 Thread Bob Proulx
Esben Stien wrote:
 If I run this, it works: 

Sounds good.  Is there anything particular bad about running it as an
expect script?  (Or in native python using pexepect?  Or in native perl
using Perl's expect library?  Or Ruby's?)

 Andreas Schwab writes:
  Since you are passing the expect script on stdin the interact command
  will immediately read EOF, and expect will run into its implicit
 Hmm, I see, so how am I supposed to run it?;)

Personally I would use #!/use/bin/env expect and be done with it.
But if you really want to call it then the first easy way is to create
a script that does the expect portion just by itself.  This is very
common such as often seen with 'sed' scripts called like:

  sed -f sedscript infile  outfile

But if you want the expect script to be contained within the shell
script then you need to write it out from the shell script first.
Then clean up the script.  Which means things get more complicated due
to the need for trap handling and temporary file cleanup.  Which is
why the use of expect directly seems best to me.

But if you really want to know then the following should do it.  Note
that I didn't test it.  There may be some detail still broken.

Note that with #!/bin/bash using trap on EXIT is sufficient.  To use
this with a #!/bin/sh script one must also trap properly on every
other signal that might happen, usually HUP, INT, QUIT, TERM, and
perhaps PIPE though PIPE is definitely subtle and special.  It is
rather of a mess.  Bash is definitely better in this regard.



unset tmpfile
cleanup() {
  test -n $tmpfile  rm -f $tmpfile
trap cleanup EXIT

tmpfile=$(mktemp) || exit 1

cat $tmpfile 'EOF'
set passwd $env(_passwd)
set usr $env(_usr)
set host $env(_host)

spawn /usr/bin/ssh $usr@$host
expect {
-re .*Are.*.*yes.*no.* {
send yes\n
#look for the password prompt

*?assword:* {
send $passwd
send \n
#The expect command will now return

env _passwd=foo _usr=foo _host=foo expect $tmpfile

Re: Browsing the history of commands. Inconsistency between Bash and Emacs

2014-02-13 Thread Bob Proulx
Dani Moncayo wrote:
 Emacs uses M-p/M-n to browse the minibuffer history (and C-p/C-n to
 move to the previous/next line in a multi-line buffer), whereas Bash
 uses C-n/C-p for browsing the command history (and doesn't use M-p/M-n
 for anything, AFAIK).
 It would be nice to remove this inconsistency (this is the obvious
 part), and IMO TRT would be to make Bash behave like Emacs, that is:
 1. Use M-p/M-n to browse the command history (instead of the current C-p/C-n).
 2. Use C-p/C-n to move to the previous/next line, in the current
 command being edited.

This is completely putting the cart before the horse.  And going down
that road creates a circular line of reasoning which has no end to the
loop cycle.  Plus it is a radical change in fundamental behavior.
Please don't.

The entire reason that bash's readline emacs mode uses C-n and C-p is
that those are the emacs keys for next-line and previous-line.  If
emacs were using M-p and M-n for previous and next then bash would
have done so too.  Bash didn't make this up.  It is using the keys
from emacs for emacs mode and those keys are C-n and C-p.  Editing the
history is just like editing a file.  C-p takes you to the previous
line.  C-n takes you to the next line.

Now enter emacs.  I mean literally.  (I have been using emacs for a
very long time by the way.  I am using it now.)  Emacs has the feature
including of being able to edit the minibuffer and also being able to
run an inferior shell process in a buffer.  Both are very similar
cases and should be discussed together.  In those buffers if you want
to go to the previous line or the next line then what keys do you use?
You use C-p and C-n to go to the previous line or the next line.

But that is not editing the history.  That is editing the buffer.
There the previous and next lines are parts of the visible screen.  It
isn't a history of the screen.  For using bash in an inferior shell if
you want to recall shell history you can't use C-p and C-n because
those are shadowed by the emacs layer of keybindings that move
previous and next lines.  Therefore M-p and M-n were the natural
second choice to navigate by adding another mental layer of
navigation.  (Although emacs itself is keeping the input history.)
Same thing for the minibuffer.

So now people say, I am now used to using M-p and M-n in emacs to
avoid the C-p/C-n.  Let's set that up for bash.  (Of course you can
easily do this if you desire.  Just do it.  Several people suggested
it.)  Well, let's say for discussion that you get used to that
setting.  The entire argument so far is that people are used to it.

Now people become used to navigating the previous line with M-p and
the next line with M-n in bash.  Now they go to emacs.  What do they
do there?  They find that in emacs M-p and M-n *do not navigate* the
previous line and next line.  They find that bash and emacs are once
again inconsistent!

In that future time let me file a bug with emacs asking for them to
change their previous-line and next-line key bindings to be compatible
with the new bash bindings of M-p and M-n for previous and next line.
Why?  Because they are used to it.  That request is just as valid as
this request to do so with bash.  So why not?  Why wouldn't emacs
change M-p and M-n to be previous line and next line?  Let's say they
do for continued discussion.

Now in the future, future time where emacs has now adopted M-p and M-n
for previous and next line.  Now they need a way to escape the emacs
layer again but with the new set of keys.  In emacs I really don't
want to need to use C-c as a prefix key for the minibuffer and the
inferior shell buffer like in term mode.  But let's say they do.
Would there then be a request for bash to set up C-c M-n and C-c M-p
for next and previous line?  I think there would be and it would just
be an endless cycle.

As Chet said the mental model of what is happening at those two points
of action in each program is different.


P.S. This is going to be a rant and run because I am offline for
several days after this.

Re: Please accept M-p as well as C-p

2014-02-13 Thread Bob Proulx
Andreas Schwab wrote:
 Chet Ramey writes:
  Andreas Schwab wrote:
  AIUI there is no readline command the moves
  vertically within the commandline (in readline a line always includes
  any embedded newlines).
  That's true.  How should such a command work?  Move point forward and
  backward by a multiple of the screenwidth?
 It should work like previous-line/next-line in Emacs: move over the
 previous/next newline and then forward to the same column as before.

In emacs 24 this is the new feature controlled by the line-move-visual
variable.  Call me a Luddite if you want but in emacs 24 I turn that
feature off.  I am very much used to using the traditional behavior
and like it.

Thinking about it I think that the recent addition of line-move-visual
to emacs and not to libreadline is probably the reason for the request
for this feature.


help-bash pointer added to web page

2014-02-02 Thread Bob Proulx
One of the recent postings mentioned that
directed questions to bug-bash.  We never updated the web page to
mention the new help-bash mailing list!  I updated the web page to say
to send questions for help with bash to the help-bash list and bugs or
development issues to bug-bash.


Re: unsigned int for loop in bash

2014-02-01 Thread Bob Proulx
Mathieu Malaterre wrote:
 I am getting a weird behavior in bash. Would it be possible for the
 next release of bash to not get a SIGSEV ?
 for i in {0..4294967295}; do
   echo $i

That is one of those expressions that I see and my eyes go *WIDE* with
shock!  The {X..Y} expression is generating arguments.
{0..4294967295} is generating 4 GIG of arguments.  And since those
arguements are strings they will take up a lot of memory.

Let's try some experiments to make it more obvious what it is doing.

  $ echo {0..10}
  0 1 2 3 4 5 6 7 8 9 10
  $ echo {0..50}
  0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
  $ echo {0..100}
  0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
  $ echo {0..250}
  ...much too long for an email...redacted...

That is getting large very fast!  Faster than just the number of
numbers because each number is taking up a larger amount of characters
as it gets largert in magnitude.  Let's stop printing them and start
counting them.

  $ echo {0..100} | wc -c
  $ echo {0..1000} | wc -c
  $ echo {0..1} | wc -c
  $ echo {0..10} | wc -c
  $ echo {0..100} | wc -c
  $ time echo {0..1000} | wc -c
  sys 0m1.860s

To print to 1M consumes 6,888,898 bytes and we are only a fraction of
the way to 4 gig!  To print out 10M consumes 78,888,899 bytes and
starts to take a long time to complete.  On my machine the bash
process gets up to around 1.8G of ram on that one and force other
memory to swap out.  There isn't much memory left for other processes
to use.  Any larger values and the machine starts to thrash swap and
slow down to disk I/O speeds.  And we are only up to 10M and not yet
anywhere close to your 4,000+M request.

The reason you are seeing a segfault is because you are trying to use
more memory than exists in your system.  You are trying to use more
memory than exists in most super computers.  You are trying to use an
astronomical amount of memory!

If you are adventuresome you might try to actually produce that many
arguments.  I expect that it will crash due to being out of memory.
If you are lucky it will only produce a bash malloc error.  If you are
running Linux with the OOM enabled then it might kill something in
appropriate and damage your system.

  echo {0..4294967295} /dev/null

This has been discussed previously.  I suggest reading through this
thread.  It includes good suggestions for alternative ways of looping
without consuming endless memory.



2014-01-29 Thread Bob Proulx
Eduardo A. Bustamante López wrote:
 Chris Down wrote:
  Well, you can do this with PROMPT_COMMAND='sleep 1'. I don't see the
  need for another internal variable to do this.

 How would that work to avoid filling the prompt with the paste buffer
 when I fat finger and paste a whole email to the prompt? It will just
 slow down the process, but not cancel it.

It would give you plenty of time to hit Control-C to interrupt it.
Try it and you will see.


Re: Builtins should canonicalize path arguments

2014-01-09 Thread Bob Proulx
Chet Ramey wrote:
 Ondrej Oprala wrote:
  Hi, I investigated this bug report:
  and found out that some of bash's builtins (source at the very
  least) do not canonicalize pathnames given as arguments (builtin
  open is instead fed with the path - failing in the BZ case).

The bug report referenced was a missing a simple reproducer.  It is
also missing the information entirely on the location of the
rpmbuild directory.

The condensed bug report:

  $ set +o physical | grep physical
  [aledvink@dhcp-25-145 rpmbuild]$ ls -ld ~/repo
  lrwxrwxrwx. 1 aledvink aledvink 8 Jan 26 12:08 /home/aledvink/repo - /al/repo
  [aledvink@dhcp-25-145 rpmbuild]$ cd ~/repo/ipmitool.f
  [aledvink@dhcp-25-145 ipmitool.f]$ . ../../rpmbuild/runme
  bash: ../../rpmbuild/runme: No such file or directory
  [aledvink@dhcp-25-145 ipmitool.f]$ cd ../../rpmbuild
  [aledvink@dhcp-25-145 rpmbuild]$

Where is rpmbuild located?  Because of the confusion I can guess that
it must only be located where there will be maximum confusion but a
bug report really should say these things explicitly.

Hmm...  Perhaps rpmbuild is in /home/aledvink/rpmbuild maybe?

  /home/aledvink/repo - /al/repo

I reproduced the confusion with this sequence.

  rwp@havoc:~$ set +o physical
  rwp@havoc:~$ rm -rf /tmp/testdir
  rwp@havoc:~$ mkdir /tmp/testdir
  rwp@havoc:~$ mkdir /tmp/testdir/al
  rwp@havoc:~$ mkdir /tmp/testdir/al/repo
  rwp@havoc:~$ mkdir /tmp/testdir/home
  rwp@havoc:~$ mkdir /tmp/testdir/home/aledvink
  rwp@havoc:~$ mkdir /tmp/testdir/home/aledvink/rpmbuild
  rwp@havoc:~$ ln -s /tmp/testdir/al/repo /tmp/testdir/home/aledvink/
  rwp@havoc:~$ mkdir /tmp/testdir/al/repo/ipmitool.f
  rwp@havoc:~$ echo echo inside runme file  
  rwp@havoc:~$ cd /tmp/testdir/home/aledvink/repo/ipmitool.f
  rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ /bin/pwd
  rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ echo $PWD

That sets up the test case.

  rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ . ../../rpmbuild/runme
  bash: ../../rpmbuild/runme: No such file or directory

Can't source the script using a logical path because it isn't
physically there.  The canonical path for the file is elsewhere.  And
because of this no utility can access it by that path.

  rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ ls -ld 
  ls: cannot access ../../rpmbuild/runme: No such file or directory
  rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ stat 
  stat: cannot stat ‘../../rpmbuild/runme’: No such file or directory
  rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ cat ../../rpmbuild/runme
  cat: ../../rpmbuild/runme: No such file or directory

That is not a correct path to access the runme file.

Only the shell maintaining $PWD and a fake model of how the user
arrived there is able to do this.

  rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ cd ../../rpmbuild

Why isn't the file there for anything other than the shell?  Because
. and .. are actual directory entries.  Let's track them down by
inode number.

  rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ ls -ldogi . .. ../.. 
  23842833 drwxrwxr-x 2 40 Jan  9 13:46 .
  23840406 drwxrwxr-x 3 60 Jan  9 13:46 ..
  23840405 drwxrwxr-x 3 60 Jan  9 13:46 ../..
  23842828 drwxrwxr-x 4 80 Jan  9 13:46 ../../..

  rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ find /tmp/testdir -type 
d -exec ls -ldogi {} +
  23842828 drwxrwxr-x 4 80 Jan  9 13:46 /tmp/testdir
  23840405 drwxrwxr-x 3 60 Jan  9 13:46 /tmp/testdir/al
  23840406 drwxrwxr-x 3 60 Jan  9 13:46 /tmp/testdir/al/repo
  23842833 drwxrwxr-x 2 40 Jan  9 13:46 /tmp/testdir/al/repo/ipmitool.f
  23840407 drwxrwxr-x 3 60 Jan  9 13:46 /tmp/testdir/home
  23842831 drwxrwxr-x 3 80 Jan  9 13:46 /tmp/testdir/home/aledvink
  23840410 drwxrwxr-x 2 60 Jan  9 13:46 /tmp/testdir/home/aledvink/rpmbuild

When you try to access ../ the .. directory is
/tmp/testdir/al/repo.  When accessing ../../ that directory is

  The builtin cd seems to handle relative paths correctly.

Actually cd in logical mode works in conjuction with a logical path
as displayed in $PWD to handle a fake model of the file system in the
presence of symbolic links.  Whether this fake model is correct or not
is a matter of perspective.  From my perspective it is most definitely
NOT correct.

I acknowledge that it is a convenient lie that many like.  Which is
exactly why there is the physical bash shell option to set how to
handle it.  Some people like it one way and others like it the other
way.  I do not like it and so I always set the 

Re: print float number

2014-01-07 Thread Bob Proulx
Hi lina,

lina wrote:
 How can I print the 0.05, 0.10 ... 1.0 out, with the step of 0.05
 Thanks ahead for your suggestions,

First please do not hijack threads.  You replied to Chris Down's
message responding to DanielB about For loop prints two $values
instead of one and then changed the subject but your message isn't
about anything to do with that thread.  If you want to start a new
thread like you did here please compose a new message.  Please don't
reply to an existing thread to start a new conversation.

Second is that this is simply a question and not a bug report.
Questions should go to the mailing list.

 How can I print the 0.05, 0.10 ... 1.0 out, with the step of 0.05

As to your question there are many ways to solve the problem.  I would
be inclined to use the GNU seq command.

  $ seq 0.05 0.05 1.0

However seq is somewhat of a troubled command.  It does math with
floating point and sometimes suffers from the same problems as
anything that does compter based floating point math with various
approximation and rounding errors.  You must be careful concerning it
or odd errors will occur.

A related question about for loops in bash is discussed in this FAQ
entry.  It isn't exactly your case but you might be interested in
reading about the examples it provides.

I doubt just printing out these numbers were really your question.  If
you wrote a message to and described your task in
more detail there are many people who would be happy to help you use
bash effectively.


Re: Replacing addresses

2013-11-28 Thread Bob Proulx
Allodoxaphobia wrote:
 Chris Down wrote:
  Peter Cordes wrote:
   Oh, also, the online bug-bash archive has a bad habbit of replacing
   code with address@hidden.  There was a whole thread about setting
   PS1=whatever that is now a complete mystery to non-subscribers!
  Yes, let's please stop this useless obfuscation. If you *must*
  obfuscate, then it should at least be retrievable somehow that doesn't
  require clicking through tons of stuff. Besides, the list is archived in
  other places that don't do this, so it doesn't really help anyone.
  rot13   :-)

Doesn't help.  The '@' isn't rotated and so the result still looks
like an email address to the redaction code.

  $ echo | rot13


Re: Replacing addresses

2013-11-28 Thread Bob Proulx
Chet Ramey wrote:
 Chris Down wrote:
  Peter Cordes wrote:
  Oh, also, the online bug-bash archive has a bad habbit of replacing
  code with address@hidden.  There was a whole thread about setting
  PS1=whatever that is now a complete mystery to non-subscribers!
  Yes, let's please stop this useless obfuscation. If you *must*
  obfuscate, then it should at least be retrievable somehow that doesn't
  require clicking through tons of stuff. Besides, the list is archived in
  other places that don't do this, so it doesn't really help anyone.
 I think this is a global setting

Yes.  It is a global setting.  And directly due to complaints from
people when it wasn't used.  People complain if it isn't there.
People complain if it is there.

 and probably does more good than the annoyance it causes.

I don't think it can do much good.  The idea is to hide your email
address so that spammers don't know it and can't find it.  And yet
they use it for sending email.  Hiding email addresses simply does not
work.  The genie of information is easier to escape from the lamp than
to force back into it.


P.S. I looked over at the archive to see how they handled
things and there even my quoted '@' which obviously is not an email
address got obfuscated to  at .  So it appears there that no use
of an at sign anywhere is allowed.

Re: RFC: turn off word splitting for vars but keep for read

2013-11-25 Thread Bob Proulx
Greg Wooledge wrote:
 Bob Proulx wrote:
  Aleksey Midenkov wrote:
   rmcmd=$rmprog -f
   Because it will fail for files with spaces.
  I am sure we disagree but I don't consider making rm program names
  have spaces to be reasonable.  :-/
  What would it be?  remove command?  I definitely would not consider
  an executable named with spaces that way to be reasonable.
 How about /mnt/c/Program Files/something/DEL.EXE?  The directory might
 contain spaces even if the command name itself does not.

That would be using a full hard coded path.  I think using full hard
coded paths is bad for yet different reasons.

Here in this context it might be argued that hard coding the path is
part of a specific host's customization.  But IMNHO it is always
better to set PATH in that case.  Because it is never just one
command.  It will almost always be several commands in an open ended
list of possibilities.  In which case you will need PATH set
appropriately.  Having a PATH with spaces in it won't cause a problem
in this context.  But in any case such host customizations are better
to occur in a sourced configuration file and not in a production
script itself.

As a matter of style I very rarly would set options with the program
name in a program name variable.  (RSYNC_RSH comes to mind.)  I would
more typically set a companion $program_args variable instead.  But
then expect word splitting to occur in that args variable.  So it is
still the same issue just moved to a slightly different location.


Re: Feature request for Bash

2013-11-25 Thread Bob Proulx
Ryan Campbell Cunningham wrote:
 I would like to request that Bash delete the character
 immediately preceding a '#', provided the '#'
 (The request was inspired by an article in the seventh
 edition of the UNIX Programmer's Manual. This feature
 is not required by POSIX, but came from the traditional
 Bourne shell.)

This isn't a feature of the shell but of the tty driver.  It still
exists.  It is still possible to use it today exactly as it was used
in Unix V7.

  $ stty -a | grep --color erase
  intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = undef;

  $ stty erase '#'

  $ stty -a | grep --color erase
  intr = ^C; quit = ^\; erase = #; kill = ^U; eof = ^D; eol = undef;

Feel free to try it.

You can also customize the behavior of your bash libreadline by
setting the backward-delete-char key binding.  Place the following in
your $HOME/.inputrc file for example.

  # backward-delete-char

 I would like to request that Bash delete the character
 immediately preceding a '#', provided the '#'
* does not begin a new word,
* is not included in any quoted string or variable,
* is not preceded by a '\', and
* is only found in an interactive command line (not
  in a script* or Bash initialization argument).
 An exception: If the character immediately preceding
 is also a '#', Bash should skip backward to the previous
 non-'#' character and delete as much characters as the
 number of consecutive '#' characters after them in the
 same word.

Oh my, isn't that a complicated set of rules!  That is bound to cause
trouble.  Not good.  Plus that isn't how it worked in V7 days.


  1   2   3   4   5   >