Re: bug with edit-and-execute-command and multiline commands
Emanuele Torre wrote: > Also here is a paste with some examples by rwp: > https://paste.debian.net/plain/1206849 That paste is going to exire so here is the contents of it. I wanted to show exactly what characters were in the file being edited and therefore set VISUAL to od in order to dump it. $ env VISUAL='od -tx1 -c' bash $ echo $BASH_VERSION 5.1.8(1)-release $ if true > then echo yes > else echo no > fiC-xC-e 000 69 66 20 74 72 75 65 3b 20 74 68 65 6e 20 65 63 i f t r u e ; t h e n e c 020 68 6f 20 79 65 73 3b 20 65 6c 73 65 20 65 63 68 h o y e s ; e l s e e c h 040 6f 20 6e 6f 3b 20 66 69 0a o n o ; f i \n 051 if true; then echo yes; else echo no; fi yes > C-xC-e 000 69 66 20 74 72 75 65 3b 20 74 68 65 6e 20 65 63 i f t r u e ; t h e n e c 020 68 6f 20 79 65 73 3b 20 65 6c 73 65 20 65 63 68 h o y e s ; e l s e e c h 040 6f 20 6e 6f 3b 20 66 69 0a o n o ; f i \n 051 if true; then echo yes; else echo no; fi yes > C-xC-e 000 69 66 20 74 72 75 65 3b 20 74 68 65 6e 20 65 63 i f t r u e ; t h e n e c 020 68 6f 20 79 65 73 3b 20 65 6c 73 65 20 65 63 68 h o y e s ; e l s e e c h 040 6f 20 6e 6f 3b 20 66 69 0a o n o ; f i \n 051 if true; then echo yes; else echo no; fi yes > C-d bash: syntax error: unexpected end of file exit I edited in the C-xC-e and C-d so that it would be visible where I sent those commands. $ env VISUAL="ed -p 'ed> '" bash $ if true > then echo yes > else echo no > fiC-xC-e 41 ed> %p if true; then echo yes; else echo no; fi ed> q if true; then echo yes; else echo no; fi yes > C-xC-e 41 ed> %p if true; then echo yes; else echo no; fi ed> q if true; then echo yes; else echo no; fi yes > C-d bash: syntax error: unexpected end of file exit Just another rendition of the same thing. Bob
Re: Bash-5.1-rc1 available
Chet Ramey wrote: > Tomas Janousek wrote: > > Chet Ramey wrote: > >> Yes, you can disable bracketed paste mode. I'll make sure that turning off > >> bracketed paste mode disables the active region for incremental searches. > > > > Would you please consider making this configurable separately? I'd love to > > keep bracketed paste mode enabled, but I find the highlighting annoying > > (and a > > bit buggy, more on that later). It seems I'm not alone: > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=972344 I would also much prefer if the highlight was separate. > I'm going to see what the reaction is with bash-5.1 and go from there. The > entire rationale for including this feature was to highlight the text a > bracketed paste inserts. Since there was no separate use, and the > contributed patch didn't include a configuration option, I didn't add one. Highlighting the region is a completely separate concept from bracketed paste mode as the region is used in many completely unrelated places. It pains me that they are coupled. Bob
Re: Bash-5.1-rc1 available
Chet Ramey wrote: > This release fixes several outstanding bugs in bash-5.0 and introduces > several new features. An unlisted change (I couldn't locate it in the changes) is that 'reverse-search-history (C-r)' now highlights the search pattern. Is that because it is the search pattern or because it now sets the region? 'reverse-search-history (C-r)' Search backward starting at the current line and moving 'up' through the history as necessary. This is an incremental search. This command sets the region to the matched text and activates the mark. > f. New active mark and face feature: when enabled, it will highlight the text >inserted by a bracketed paste (the `active region') and the text found by >incremental and non-incremental history searches. I find the reverse video highlight to be very difficult to read. For me it is extreme eye strain. Is there a way to disable the reverse video? For both the paste text and the search text? This is so bad for me that for the moment I had to revert to the previous version of bash to avoid it. (Note that disabling bracketed paste mode avoids it for the paste but doesn't address the reverse video of search.) Bob
Re: GNU Bash v.4.4.23-5 built-in function 'set' produces variable outputs
Robert Elz wrote: > ps: I did not suggest that $_ should go away, I know that's not going to > happen ... just that it would be nice for those who really don't want it to > be able to turn it off. I'm having a hard time understanding why one would want to turn off this feature. It isn't something that everyone uses but it is a feature that has a lot of use. If one goes down that path then the end result taken to the logical extreme would be that every feature would have a control to turn them on and off. That just seems extreme. And for the case of $_ if you didn't know it existed then one probably goes about their life continuing to not be bothered by it too. The original report wasn't really about $_ anyway. Ricky Tigg wrote: > Built-in function 'set' produces variable outputs. > $ export SSLKEYLOGFILE=/home/user/test > $ set | grep SSL > SSLKEYLOGFILE=/home/user/test > _=SSLKEYLOGFILE > $ set | grep SSL > SSLKEYLOGFILE=/home/user/test The original report was about the output being different in different invocations. But I think that is an invalid reason. Because if so then 'echo $RANDOM' is also a bug because it produces different output in different invocations too. And because set | grep is not a correct way to look at the environment as such either. The 'set' command is designed to set or unset shell options or the positional parameters. Without any arguments "set shall write the names and values of all shell variables in the collation sequence of the current locale". Since $_ is a shell variable it writes it out along with possibly other data too. I don't think anyone grep'ing for a string should complain that the shell also prints out the contents of $_. As an additional point 'set' writes out the internal data which includes a lot of *stuff*. It would be better in this case to use 'env' to write out only the exported variables. And clearly in the original report the string being looked for was part of the exported data. There is no problem then. $ export SSLKEYLOGFILE=/home/user/test $ env | grep SSL SSLKEYLOGFILE=/home/user/test $ env | grep SSL SSLKEYLOGFILE=/home/user/test However even 'env' isn't the appropriate tool either. There may be other variables that happen to hit the grep pattern. And there is the problem of the variable value including newlines. The -z,--null helps here but life isn't simple. Not that I would do it this way but 'printenv' seems to be the right matching utility here. $ export SSLKEYLOGFILE=/home/user/test $ printenv SSLKEYLOGFILE /home/user/test Using grep is fine. But then the human must interpret the results accordingly. I think here understanding that 'set' is doing what is expected and reasonable is enough. However I would use 'env' to avoid the internal data state. And programatically neither are sufficient for a fully robust program and other methods should be used. Bob
Re: bash sockets: printf \x0a does TCP fragmentation
Dirk Wetter wrote: > PS + @Bob: fd 5 is not a tty in the program -- but interactively in > this PoC you want to make sure it is not taken yet. Understood. I originally mentioned the isatty() thing because that is how libc decides to line buffer or not. But Chet has confirmed that bash's internal printf line buffers regardless, because it has always done it that way since the beginning. So for the moment at least whether it is a tty or not doesn't matter to the bash printf. Since this is a lessor traveled corner of bash it hasn't been tripped over much before. And probably never discussed here. However I think it likely (I didn't check) that it does for the coreutils printf using libc's default stdio buffering. So it is good to know about regardless. I think it would make sense if bash's internal commands like printf performed buffering the same way as the default libc does. Then it would behave the same as the rest and be less likely to cause problems. And it would have meant it would have worked by default in your case. Dirk Wetter wrote: > Sorry, forget about my mail. In the actual file I was using there > was a variable which I haven't quoted correctly. The PoC works as > expected and it looks I have a solution. The smallest of details can break the largest of programs. :-) Good to hear you have things resolved and now working for you! Bob
Re: Command not found with builtins and variables
Chet Ramey wrote: > b...@feusi.co wrote: > > The thing I noticed is that when setting a variable and then running the > > command "time", bash reports "command not found". For example, the > > command: > > > > TEST=1 time ls > > > > Results in "bash: time: command not found" That would work if you installed the GNU time package. https://www.gnu.org/software/time/ And it would produce the portable output format in both cases with the time -p option, which is available both in the internal time and the external GNU time command. Otherwise GNU time has its own unique and different default output format. $ TEST=1 time -p sleep 1 real 1.00 user 0.00 sys 0.00 But regardless of the external standalone time program... Order when using the internal time here matters. Since 'time' is special it should always be first. Put any variables to be set on the line before the command by putting it before the command but after the 'time' word. $ time TEST=1 sleep 1 real0m1.003s user0m0.000s sys 0m0.000s > You don't have a command named `time' in a directory in your $PATH. Well... *They* don't have a time program in PATH. But they could have if they installed it. And many people do. :-) > Bash doesn't have a `time' builtin; it has a `time' reserved word. > Preceding `time' with a variable assignment removes its special meaning > and causes it to not be recognized as a reserved word. I might also mention that the bash 'time' needs this special status because it times the entire pipeline. $ time sleep 1 | sleep 2 real0m2.004s user0m0.003s sys 0m0.002s Therefore conversely if someone is using the external time it might give them an unexpected result. Since the external time will not know anything about a pipeline. So this might catch people off guard too. $ TEST=foo time -p sleep 1 | sleep 2 real 1.00 user 0.00 sys 0.00 When used it is best to always put 'time' first on the command line, variables second, and commands after that. Order matters[1]. Bob [1] OT: In English the order also matters and it is always opinion, size, age, shape, color, origin, material, purpose, noun. Using any other order and it sounds really odd! Reference "The Elements Of Eloquence: How To Turn The Perfect English Phrase." by Mark Forsyth. http://www.npr.org/2016/09/08/493157931/bbc-editor-highlights-often-overlooked-english-language-rule
Re: Segfault on recursive trap/kill
Mike Gerwitz wrote:
> Bob Proulx wrote:
> > Let me give the discussion this way and I think you will be
> > convinced. :-)
>
> Well, thanks for taking the time for such a long reply. :)
>
> > How is your example any different from a C program? Or Perl, Python,
> > Ruby, and so forth? All of those also allow infinite recursion and
> > the kernel will terminate them with a segfault. Because all of those
> > also allow infinite recursion. A program that executes an infinite
> > recursion would use infinite stack space. But real machines have a
> > finite amount of stack available and therefore die when the stack is
> > exceeded.
>
> I expect this behavior when writing in C, certainly. But in languages
> where the user does not deal with memory management, I'm used to a more
> graceful abort when the stack gets out of control. A segfault means
> something to a C hacker. It means very little to users who are
> unfamiliar with the concepts that you were describing.
>
> I don't have enough experience with Perl, Python, or Ruby to know how
> they handle stack issues. But, out of interest, I gave it a try:
Some of those must be maintaining the stack in program data space
instead of in the machine stack space. (shrug)
> $ perl -e 'sub foo() { foo(); }; foo()'
> Out of memory!
Perl apparently will let you use all available memory without
limitation.
> $ python <<< 'def foo():
> > foo()
> >foo()' |& tail -n1
> RuntimeError: maximum recursion depth exceeded
>...
I wonder how they decide to set the depth.
> $ php -r 'function foo() { foo(); } foo();'
>
> Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to
> allocate 262144 bytes) in Command line code on line 1
134M seems arbitrarily small but at least it does say exactly what it
is doing there.
> $ guile -e '(let x () (+ (x)))'
> allocate_stack failed: Cannot allocate memory
> Warning: Unwind-only `stack-overflow' exception; skipping pre-unwind
> handler.
>
> $ emacs --batch --eval '(message (defun foo () (foo)) (foo))'
> Lisp nesting exceeds ‘max-lisp-eval-depth’
I think these two are different. It looks like guile is using
libunwind to set up a stack exception handler whereas emacs appears to
be using a tracked max-lisp-eval-depth variable defaulting to 800 on
my system in emacs v25.
This limit serves to catch infinite recursions for you before they cause
actual stack overflow in C, which would be fatal for Emacs.
You can safely make it considerably larger than its default value,
if that proves inconveniently small. However, if you increase it too far,
Emacs could overflow the real C stack, and crash.
> I understand that in C you usually don't manage your own stack and,
> consequently, you can't say that it falls under "memory management" in
> the sense of malloc(3) and brk(2) and such. But C programmers are aware
> of the mechanisms behind the stack (or at least better be) and won't be
> surprised when they get a segfault in this situation.
>
> But if one of my coworkers who knows some web programming and not much
> about system programming gets a segfault, that's not a friendly
> error. If Bash instead said something like the above languages, then
> that would be useful.
>
> When I first saw the error, I didn't know that my trap was
> recursing. My immediate reaction was "shit, I found a bug". Once I saw
> it was the trap, I _assumed_ it was just exhausting the stack, but I
> wanted to report it regardless, just in case; I didn't have the time to
> dig deeper, and even so, I wasn't sure if it was intended behavior to
> just let the kernel handle it.
My interpretation of the above is that you would want bash to use
libunwind (or whatever is appropriate) to set up a stack overflow
exception trap in order to handle stack overflow specially and then to
make an improved error reporting to the user when it happens.
Frankly I wasn't even aware of libunwind before this. And haven't
learned enough about it to even know if that is what I should be
mentioning here yet. :-)
> > Shell script code is program source code. Infinite loops or infinite
> > recursion are bugs in the shell script source code not the interpreter
> > that is executing the code as written.
>
> I also agree. But the context is very different. Shell is a very,
> very high-level language.
My mind reels at the statement, "Shell is a very, very high-level
language." What?! The shell is a very simple low level command and
control language. It is very good at what it does. But if one needs
to do high level things then one should switch over to a high level
language. :-)
> > Hope this helps!
>
> There was useful information, yes.
After sifting out the non-useful information. :-)
> I hope I was able to further clarify my concerns as well.
You are always eloquent! :-)
Bob
signature.asc
Description: PGP signature
Re: Auto-update program cache feature
Chet Ramey wrote: > Bob Proulx wrote: > > Put this in your ~/.bashrc file and I believe your use case will be > > much happier. > > > > shopt -s checkhash > > How many installers put a new version of an existing package into a > possibly-different directory with a different name? Even using a symlink > pointing to a new name would not trigger this behavior. The use case (reading between the lines) had to do with ping-pong'ing between /usr/bin and /usr/local/bin though. And I know that without checkhash that I used to see that condition too. But with checkhash the complaint (from me) went away entirely. I admit I didn't look at it in great detail here now because I have had checkhash in use for so long and not seen this problem since. I'll need to experiment. Bob
Re: Segfault on recursive trap/kill
Hi Mike,
Mike Gerwitz wrote:
> ... but are you saying that terminating with a segfault is the
> intended behavior for runaway recursion?
Let me give the discussion this way and I think you will be
convinced. :-)
How is your example any different from a C program? Or Perl, Python,
Ruby, and so forth? All of those also allow infinite recursion and
the kernel will terminate them with a segfault. Because all of those
also allow infinite recursion. A program that executes an infinite
recursion would use infinite stack space. But real machines have a
finite amount of stack available and therefore die when the stack is
exceeded.
This following complete C program recurses infinitely. Or at least
until the stack is exhausted. At which time it triggers a segfault
because it tries to use memory beyond the page mapped stack.
int main() {
return main();
}
$ gcc -o forever forever.c
$ ./forever
Segmentation fault
$ echo $?
139 # Signal 11 + 128
The return value of a simple command is its exit status, or 128+n if
the command is terminated by signal n.
Would you say that is a bug in the C language? A bug in gcc that
compiled it? A bug in the Unix/Linux kernel for memory management
that trapped the error? The parent shell that reported the exit code
of the program? Or in the program source code? I am hoping that we
will all agree that it is a bug in the program source code and not
either gcc or the kernel. :-)
Shell script code is program source code. Infinite loops or infinite
recursion are bugs in the shell script source code not the interpreter
that is executing the code as written.
This feels to me to be related to The Halting Problem.
> As long as there is no exploitable flaw here, then I suppose this isn't
> a problem;
It's not a privilege escalation. Nor a buffer overflow. Whether this
is otherwise exploitable depends upon the surrounding environment usage.
> I haven't inspected the code to see if this is an access violation
> or if Bash is intentionally signaling SIGSEGV.
It is the kernel that manages memory, maps pages, detects page faults,
kills the program. The parent bash shell is only reporting the exit
code that resulted. The interpreting shell executed the shell script
souce code as written.
Other shells are also fun to check:
$ dash -c 'trap "kill 0" TERM; kill 0'
Segmentation fault
$ ash -c 'trap "kill 0" TERM; kill 0'
Segmentation fault
$ mksh -c 'trap "kill 0" TERM; kill 0'
Segmentation fault
$ ksh93 -c 'trap "kill 0" TERM; kill 0'
$ echo $?
0
$ posh -c 'trap "kill 0" TERM; kill 0'
Terminated
Terminated
Terminated
...
Terminated
^C
Testing zsh is interesting because it seems to keep the interpreter
stack in data space and therefore can consume a large amount of memory
if it is available. And then can trap the result of being out of data
memory and then kills itself with a SIGTERM. Note that in my testing
I have Linux memory overcommit disabled.
This finds what look like bugs in posh and ksh93.
> it's just that most users assume that a segfault represents a
> problem with the program
Yes. And here it indicates a bug too. It is indicating a bug in the
shell program code which sets up the infinite recursion. Programs
should avoid doing that. :-)
bash -c 'trap "kill 0" TERM; kill 0'
The trap handler was not set back to the default before the program
sent the signal to itself. The way to fix this is:
$ bash -c 'trap "trap - TERM; kill 0" TERM; kill 0'
Terminated
$ echo $?
143 # killed on SIGTERM as desired, good
If ARG is absent (and a single SIGNAL_SPEC is supplied) or `-',
each specified signal is reset to its original value.
The proper way for a program to terminate itself upon catching a
signal is to set the signal handler back to the default value and then
send the signal to itself so that it will be terminated as a result of
the signal and therefore the exit status will be set correctly.
For example the following is useful boilerplate:
unset tmpfile
cleanup() {
test -n "$tmpfile" && rm -f "$tmpfile" && unset tmpfile
}
trap "cleanup" EXIT
trap "cleanup; trap - HUP; kill -HUP $$" HUP
trap "cleanup; trap - INT; kill -INT $$" INT
trap "cleanup; trap - QUIT; kill -QUIT $$" QUIT
trap "cleanup; trap - TERM; kill -TERM $$" TERM
tmpfile=$(mktemp) || exit 1
If a program traps a signal then it should restore the default signal
handler for that signal and send the signal back to itself. Otherwise
the exit code will be incorrect. Otherwise parent programs won't know
that the child was killed with a signal.
For a highly recommended deep dive into this:
https://www.cons.org/cracauer/sigint.html
Hope this helps!
Bob
signature.asc
Description: PGP signature
Re: Auto-update program cache feature
Jeffrey Walton wrote: > I think a useful feature for Bash would be to automatically update the > program cache after an install. Put this in your ~/.bashrc file and I believe your use case will be much happier. shopt -s checkhash In the bash manual: checkhash If set, bash checks that a command found in the hash table exists before trying to execute it. If a hashed command no longer exists, a normal path search is performed. I would prefer that to be a default. But it changed a behavior back in the day when it was added and deviated from the previous csh behavior. Therefore being an optional option makes sense. But I don't see a downside to defaulting to it now. In any case I always add it to my bashrc file. (Along with "shopt -s checkwinsize" too.) Bob
Re: bash sockets: printf \x0a does TCP fragmentation
Chet Ramey wrote: > Bob Proulx wrote: > > It was also noted in another message that the external standalone > > printf command line utility did buffer as you desired. That seems > > another very good solution too. Simply use "command printf ..." to > > force using the external version. > > This won't work the way you want. The `command' builtin only inhibits > execution of shell functions. It still executes builtins. You want to > either get the full pathname of a printf utility using `type -ap printf' > and use that, or use the env or exec variants I recommended in my last > message. Oh drat! Now I have had to learn *TWO* things today. :-) > > Anyway... Since printf is a text oriented utility it makes sense to > > me that I would operate in line buffered output mode. > > It's that bash sets stdout and stderr to be line-buffered, not anything > printf-specific. I still think 'printf' feels more like a plain text utility and not something one reaches for when working with binary data blobs. Bob
Re: bash sockets: printf \x0a does TCP fragmentation
Robert Elz wrote:
> Bob Proulx wrote:
> | Using the same buffer size
> | for input and output is usually most efficient.
>
> Yes, but as the objective seemed to be to make big packets, that is probably
> not as important.
The original complaint concerned flushing a data blob content upon
every newline (0x0a) character due to line buffering, write(2)'ing the
buffer up to that point. As I am sure you already know that will
cause the network stack in the kernel to emit the buffered data up to
that point with whatever has been read up to that point. Which was
apparently a small'ish amount of data. And then instead of having
some number of full MTU sized packets there were many more smaller
ones. It shouldn't have been about big packets, nor fragmentation,
but about streaming efficiency and performance. Though achieving
correct behavior with more buffer flushes than desired this was
apparently less efficient than they wanted and were therefore
complaining about it. They wanted the data blob buffered as much as
possible so as to use the fewest number of TCP network packets. My
choice of a large one meg buffer size was to be larger than any
network MTU size. My intention was that the network stack would then
split the data blob up into MTU sizes for transmission. The largest
MTU size that I routinely see is 64k. I expect that to increase
further in size in the future when 1 meg might not be big enough. And
I avoid mentioning jumbo frames.
> | $ printf -- "%s\n" one two | strace -o /tmp/out -e write,read dd
> status=none obs=1M ; cat /tmp/out
> | one
> | two
> | ...
> | read(0, "one\ntwo\n", 512) = 8
>
> What is relevant there is that you're getrting both lines from the printf in
> one read. If that had happened, there would ne no need for any rebuffering.
> The point of the original complaint was that that was not ahppening, and
> the reads were being broken at the \n ... here it might easily make a
> difference whether the output is a pipe or a socket (I have no idea.)
I dug into this further and see that we were both right. :-)
I was getting misdirected by the Linux kernel's pipeline buffering.
The pipeline buffering was causing me to think that it did not matter.
But digging deeper I see that it was a race condition timing issue and
could go either way. That's obviously a mistake on my part.
You are right that depending upon timing this must be handled properly
or it might fail. I am wrong that it would always work regardless of
timing. However it was working in my test case which is why I had not
noticed. Thank you for pushing me to see the problem here.
> | It can then use the same buffer that data was read into for the output
> | buffer directly.
>
> No, it can't, that's what bs= does - you're right, that is most effecient,
> but there is no rebuffering, whatever is read, is written, and in that case
> even more effecient is not to interpose dd at all. The whole point was
> to get the rebuffering.
>
> Try tests more like
>
> { printf %s\\n aaa; sleep 1; printf %s\\n bbb ; } | dd
>
> so there will be clearly 2 different writes, and small reads for dd
> (however big the input buffer has) - with obs= (somethingbig enough)
> there will be just 1 write, with bs= (anything big enough for the whole
> output) there will still be two writes.
$ { command printf "one\n"; command printf "two\n" ;} | strace -v -o
/tmp/dd.strace.out -e write,read dd status=none bs=1M ; head /tmp/*.strace.out
one
two
...
read(0, "one\ntwo\n", 1048576) = 8
write(1, "one\ntwo\n", 8) = 8
read(0, "", 1048576)= 0
+++ exited with 0 +++
Above the data is definitely written in two different processes but
due to Linux kernel buffering in the pipeline it is read in one read.
The data is written into the pipeline so quickly, before the next
stage of the pipeline could read it out, that by the time the read
eventually happened it was able to read the multiple writes as one
data block. This is what I had been seeing but you are right that it
is a timing related success and could also be a timing related
failure.
$ { command printf "one\n"; sleep 1; command printf "two\n" ;} | strace -v -o
/tmp/dd.strace.out -e write,read dd status=none bs=1M ; head /tmp/*.strace.out
one
two
...
read(0, "one\n", 1048576) = 4
write(1, "one\n", 4)= 4
read(0, "two\n", 1048576) = 4
write(1, "two\n", 4)= 4
read(0, "", 1048576)= 0
+++ exited with 0 +++
The above illustrates the point you were trying to make. Thank you
for persevering in educating me as to the issue. :-)
$ {
Re: bash sockets: printf \x0a does TCP fragmentation
Robert Elz wrote: > ps: f there was actually a desire to use dd to do re-buffering, the > correct usage is not to use "bs=" (which simply does read write with > a buffer that size) but "obs=" which reads (using ibs if needed, which it > would not be here), copies to an output buffer and writes only when that > buffer is full (or on EOF on the input). If the goal is to minimize writes then it won't matter as long as the buffer size picked is larger than needed. Using the same buffer size for input and output is usually most efficient. First let's set it to something small to prove that it is buffering as expected. $ printf -- "%s\n" one two | strace -o /tmp/out -e read,write dd status=none bs=2 ; cat /tmp/out one two ... read(0, "on", 2)= 2 write(1, "on", 2) = 2 read(0, "e\n", 2) = 2 write(1, "e\n", 2) = 2 read(0, "tw", 2)= 2 write(1, "tw", 2) = 2 read(0, "o\n", 2) = 2 write(1, "o\n", 2) = 2 read(0, "", 2) = 0 +++ exited with 0 +++ Lots of reads and writes but all as expected. Or set just the output buffer size large. Then the input buffer size defaults to 512 bytes on my system. $ printf -- "%s\n" one two | strace -o /tmp/out -e write,read dd status=none obs=1M ; cat /tmp/out one two ... read(0, "one\ntwo\n", 512) = 8 read(0, "", 512)= 0 write(1, "one\ntwo\n", 8) = 8 +++ exited with 0 +++ But even if ibs is much too small it still behaves okay with a small input buffer size and a large output buffer size. $ printf -- "%s\n" one two | strace -o /tmp/out -e write,read dd status=none ibs=2 obs=1M ; cat /tmp/out one two ... read(0, "on", 2)= 2 read(0, "e\n", 2) = 2 read(0, "tw", 2)= 2 read(0, "o\n", 2) = 2 read(0, "", 2) = 0 write(1, "one\ntwo\n", 8) = 8 +++ exited with 0 +++ Then set both ibs and obs to be something quite large using bs= and let it gather up all of the input and write with that buffer size. $ printf -- "%s\n" one two | strace -o /tmp/out -e write,read dd status=none bs=1M ; cat /tmp/out one two ... read(0, "one\ntwo\n", 1048576) = 8 write(1, "one\ntwo\n", 8) = 8 read(0, "", 1048576)= 0 +++ exited with 0 +++ It seems to me that using a large buffer size for both read and write would be the most efficient. It can then use the same buffer that data was read into for the output buffer directly. Bob
Re: bash sockets: printf \x0a does TCP fragmentation
I see that you have subscribed now. Awesome! If you and others would be so kind as to list-reply instead of CC'ing me directly that would be great. I read the replies on the mailing list. dirk+b...@testssl.sh wrote: > Bob Proulx wrote: > > You are doing something that is quite unusual. You are using a shell > > script direction on a TCP socket. That isn't very common. > > Do you think there should be a paragraph NOT COMMON where bash sockets > should rather belong to? You actually had not included enough background information to know if you were using the bash built in network implementation or not. You only showed that you had set up fd 5 connected to a network socket. That can happen because, for example, a script was used to service an inetd configuration or similar. It doesn't actually need to be the built in network protocol at all. But now that you have said the above I guess I can assume that you are using the built in implementation. As to whether the documentation should say this or not that is not really practical. There are a godzillian different things that are not typically addressed by writing a shell script. As a practical matter it is impossible to list everything out explicitly. And if one tries then the complaint is that the documentation is so long and detailed that is is unusable due to it. Primarily a shell script is a command and control program. It is very good for that purpose. It is typically used for that purpose. That is the mainstream use and it is very unlikely one will run into unusual situations there. But programming tasks that are much different from command and control tasks, such as your program interacting by TCP with other devices on the network, are not as common. I don't have facts to back that up but I do believe that to be true based upon the way I have seen shell scripts being programmed and used over a long period of time. Of course if you have spent the last 20 years programming network shell scripts then your observations will bias you the other way. :-) > > More > > typically one would use a C program instead. So it isn't surprising > > that you are finding interactions that are not well known. > > Bob, my intention was not to discuss program languages and what is typical > with you or anybody else here. Hmm... Put yourself in our shoes. You stood up on the podium that is this public mailing list and spoke into the megaphone addressing all of us complaining that bash's printf was buggy. But from my perspective printf is behaving as expected. It is designed to deal with line oriented data. It will also deal with binary data if one is careful. But it appears that your application wasn't careful enough and had tripped over some problems. Should we (me!) keep silent about those very obvious problems? It feels obvious to me but apparently not to the author of the above. As has often been said many eyes make all bugs apparent. I was pointing this out to you as a public service. But in response you seem hostile by the language above and below. That isn't encouraging any help. :-( > >> printf -- "$data" >&5 2>/dev/null > > > > Why is stderr discarded? That is almost always bad because it > > discards any errors that might occur. You probably shouldn't do this.> > > What happens if $data contains % format strings? What happens if the > > format contains a sequence such as \c? This looks problematic. This > > is not a safe programming proctice. > > I doubt you can judge on this by just looking at a single line > of code -- the project has > 18k LoC in bash. That single line of code was problematic just by itself standing alone without the rest of the program around it. That is independent of anything the rest of the program might contain. However if you would like to pass sections of the rest of the program through the help-bash mailing list then I am sure the group there would help improve the quality of it. > Github is the place to discuss and do PRs for our project. No. Sorry. You came here to this mailing list. Therefore this is the place to discuss it. Please put yourself in my shoes. If the case were reversed and I came over to Github and then stated that Github was not the place for the discussion but that you needed to set up email and come over to my mailing list and discuss it there instead. How would you feel? I had come into your house, asked you for help, then wanted you to go elsewhere? How would you feel? I can tell you that I do not feel very welcome by it. Also remember that Github is a non-free service. That is free as in freedom, not free as in beer. The free in Free Software. Or in this case the opposite of it being non-free. We try not to use software that does not respect our freedoms nor ask others to do so either. It's a philosophy of life thing
Re: bash sockets: printf \x0a does TCP fragmentation
dirk+b...@testssl.sh wrote: > we discovered a strange phenomenon in the project testssl.sh: You are doing something that is quite unusual. You are using a shell script direction on a TCP socket. That isn't very common. More typically one would use a C program instead. So it isn't surprising that you are finding interactions that are not well known. > printf -- "$data" >&5 2>/dev/null Why is stderr discarded? That is almost always bad because it discards any errors that might occur. You probably shouldn't do this. What happens if $data contains % format strings? What happens if the format contains a sequence such as \c? This looks problematic. This is not a safe programming proctice. > does not do what it is intended. "Intent" is in the eye of the beholder. > "$data" is a ClientHello like > > '\x16\x03\x01\x2\x00\x01\x00\x1\xfc\x03\x03\x54\x51\x1e\x7a\xde\xad\xbe\xef\x31\x33\x07\x00\x00\x00\x00\x00\xcf\xbd\x39\x04\xcc\x16\x0a\...' > > Each \x0a like the last one causes a new TCP fragment to begin which can be > easily > spotted when using wireshark while running e.g. As Chet said the libc stdio library is probably doing line oriented buffering. The newline is causing a flush at that time. > One would assume that a bash socket connection cannot influence the TCP > fragmentation but obviously it does. One would be in error to assume this. > If there's a workaround, please let me know. (tried to add "%b" with no > effect). Otherwise I believe it's a bug. You can re-block the output stream using other tools such as 'cat' or 'dd'. Since you are concerned about block size then perhaps dd is the better of the two. | cat Or probably better: | dd status=none bs=1M Or use whatever block size you wish. The 'dd' program will read the input into its buffer and then output that block of data all in one write(2). That seems to be what you are wanting. Good luck! :-) Bob P.S. You can possibly use the 'stdbuf' command to control the output buffering depending upon the program. info stdbuf
Re: Add sleep builtin
Greg Wooledge wrote: > L A Walsh wrote: > > I remember some implementations only wait till the clock reaches the > > new time in seconds -- meaning that the actual sleep time could be > > .001 seconds. > > If true, that would actually violate POSIX. The sleep time must be > "at least" as long as the operand. It may be longer, but never shorter. > > You might be remembering it wrong. I also remember that at one time the Unix sleep was implemented by polling a second granularity clock register. If one was not lucky then polling just before the second turned, and then just after, would cause sleep to exit because the second had changed. A 1 second sleep was 0-1 seconds. Therefore if one required at least a 1 second sleep one would need to ask for at least 2 seconds or it was possible for there to be very little sleep at all. A sleep 2 might be 1-2 seconds. Of course sleeping for 300 would rarely care if it was actually 299 or 300 so longer sleeps were never a practical problem. I recall this as a historical Unix kernel limitation. I forget which versions now and am not in a position to do a search through the historical record to figure out which kernels did it that way. I never worked on any such running system but recall it from comments written at the time. I don't think it is true with any modern kernels. Or even with older ones such as the venerable HP-UX 10.20 from 1996! I think it would be difficult to impossible to find any running Unix-like system outside of a museum environment that still behaved that old way. And certainly any using the fractional second extension is past any need to worry about the problem. Because any such system with the problem would only handle to a granularity of integer seconds anyway. Bob
Re: Unexpected delay in using arguments.
Bize Ma wrote: > > but just the same it is almost always possible > > to refactor a program into a different algorithm that avoids the need > > to enumerate so many arguments in memory. > > As you say: "almost". I still believe that to be true. Since you haven't shared what the actual task is there is no way for us to propose any counter example improvements. So "almost" is as far as I can go. Instead should I say 99.44% of the time? Since I have never come across a counter example yet? > Take a look at the Stéphane Chazelas example to convince yourself. Nothing Stéphane said changed my statement at all. It does look like bash can be more efficient with argument handling. Since, for example, dash does it. Bob
Re: Unexpected delay in using arguments.
I followed along through the script:
> m=2
...
> test1() { ...
Eventually I got to this line:
> set -- $(seq $m)
At that line I see that bash must allocate enough memory to hold all
of the numbers from 1 through 2 in memory all at one time. That
is very inefficient. That is at least 100K of memory.
$ seq 2 | wc -c
108894
Of course I assume this is only a proxy simpler reproducer for the
actual problem program but just the same it is almost always possible
to refactor a program into a different algorithm that avoids the need
to enumerate so many arguments in memory. I suggest refactoring the
program to avoid the need for this memory stress.
Bob
Re: Rational Range Interpretation for bash-5.0?
Chet Ramey wrote: > Hi. I am considering making bash glob expansion implement rational range > interpretation starting with bash-5.0 -- basically making globasciiranges > the default. It looks like glibc is going to do this for version 2.28 (at > least for a-z, A-Z, and 0-9), and other GNU utilities have done it for some > time. What do folks think? I think the non-rational ranges in libc were a terrible idea that we have all been suffering with over the last decade plus. Bob
Re: Empty ""s in ARG in ${x:+ARG} expand to no words instead of the empty word if prepended/appended with space
Chet Ramey wrote:
> Bob Proulx wrote:
> > Denys Vlasenko wrote:
> >> $ f() { for i; do echo "|$i|"; done; }
> >> $ x=x
> >> $ e=
> >> $ f ${x:+ ""}
> >> ^^^ prints nothing, bug?
> >>
> >> $ ${x:+"" }
> >> ^^^ prints nothing, bug?
> >
> > Insufficient quoting. That argument should be quoted to avoid the
> > whitespace getting stripped. (Is that during word splitting phase
> > using the IFS? I think so.)
>
> Even if the whitespace gets stripped out, the quoted null string should
> result in an empty argument. Different shells are inconsistent about this,
> but I believe that if word splitting occurs, the empty string (or "$e",
> anything that expands to an empty string) should result in a null word.
Gotcha. Thanks!
And further testing confirms the inconsistent shells. bash and ksh
both seem to be the outliers in different ways in terms of behavior.
dash and zsh seem to be consistent and correct in this.
Bob
Re: Empty ""s in ARG in ${x:+ARG} expand to no words instead of the empty word if prepended/appended with space
Denys Vlasenko wrote:
> $ f() { for i; do echo "|$i|"; done; }
> $ x=x
> $ e=
> $ f ${x:+ ""}
> ^^^ prints nothing, bug?
>
> $ ${x:+"" }
> ^^^ prints nothing, bug?
Insufficient quoting. That argument should be quoted to avoid the
whitespace getting stripped. (Is that during word splitting phase
using the IFS? I think so.)
Try this:
f "${x:+ ""}"
f "${x:+"" }"
Also in the future please say what version of bash you are using.
There have been a lot of bugs fixed in various versions.
Bob
Re: announcing bashj
Chet Ramey wrote: > On 6/23/18 3:44 PM, L A Walsh wrote: > > Seems like the original missed being sent toe the bug-bash alias. > > It was not posted to bug-bash. For bug-bash was in the hold queue for human moderation. Although it seems off topic for this mailing list it has already had three replies in the thread and therefore I have now approved it and passed it on. For better or worse. For b...@packages.debian.org everyone subscribed to the package list got a copy from there already. Bob
Re: [annoyance/non-feature] On OS X, every process is "occupying" CWD, making disk ejects impossible after cd'ing into them.
Danyel Bayraktar wrote: > Bash should idle in (e.g.) /tmp or /var/folders/xyz, i.e. occupy > that file as ‘CWD’ towards the OS, but obviously not expose this to > depending code, so still spawn every child process/builtin/keyword > inside the (memory-stored) cwd which will be updated with `cd` and > used throughout. That would create regressions. Consider the case where a shell changes directory to some directory. Then that directory is renamed. Currently the shell having it as a current working directory will continue to operate. In your proposal the shell would suddenly fail because it would not know the directory was renamed and would be unable to return to it. Bob
Re: xmalloc crash
Eduardo A. Bustamante López wrote: > Alfred Baroti wrote: > > Hi, > > I found this from long long time ago. > > Is this a serious bug? > [...] > > This is not a serious bug at all. It's just a memory allocation failure. Agreed. Not a bug at all. > You can see that the brk() system call actually succeeds. If you are running under the Linux kernel in the default configuration then memory overcommit is enabled. $ sysctl vm.overcommit_memory With overcommit enabled brk(), and malloc(), will always succeed even if there isn't really enough memory. Neither will fork(). It doesn't mean there actually is that much memory available. Later if the kernel runs out of memory to implement the needs then it will fail and trigger the OOM Out Of Memory Killer to virtually kill -9 processes until enough memory is freed up to balance the books. Processes killed by the OOM have no opportunity to clean up or log anything. Or vm.overcommit_memory can be configured the traditional Unix way where enough virtual memory needs to exist or malloc() and fork() will fail. IMNHO overcommit off is the best way for enterprise servers. However overcommit on is convenient for laptops. Bob
Re: simple function causes BASH to exit when -e in effect
Kevin Layer wrote: > It took me hours of work to distill this down from a very large and long > running script. > set -eu Hours and hours of debug time could be saved if people stopped using 'set -e' in their programs. It creates more problems than people think it solves. Do not meddle in the affairs of 'set -e' for it is subtle and quick to anger. Bob
Re: Bash default /etc/bash.bashrc overwrites PS1 even if SUDO_PS1 is set
Eric Engstrom wrote: > doh - thanks - didn't think of that, and that makes more sense. > > Nevermind then. In the /usr/share/doc/bash/README.Debian.gz file: 5. What is /etc/bash.bashrc? It doesn't seem to be documented. The Debian version of bash is compiled with a special option (-DSYS_BASHRC) that makes bash read /etc/bash.bashrc before ~/.bashrc for interactive non-login shells. So, on Debian systems, /etc/bash.bashrc is to ~/.bashrc as /etc/profile is to ~/.bash_profile. Bob
Re: ~/.profile and ~/.bash_profile aren't executed on login
Yuri wrote: > Bob Proulx wrote: > > How is the user logging in? Are they logging in with 'ssh' over the > > network? Or are they logging in through an "xdm" X Display Manager > > login from a graphical login display? > > User logs in locally through the display manager. Which graphical display manager? There are many. You not saying means that we are not knowing. Bob
Re: ~/.profile and ~/.bash_profile aren't executed on login
Yuri wrote: > None of these files are executed when bash is a user's default shell on > FreeBSD. > No special options were selected. Despite shell.c saying that they should be > executed they just aren't. How is the user logging in? Are they logging in with 'ssh' over the network? Or are they logging in through an "xdm" X Display Manager login from a graphical login display? If from 'ssh' then that is the standard Unix login path with no surprises. If from a graphical login display, which I think most likely is the case, then it all depends upon how the shell is launched from that point. The shell looks at its own name, or the -l or --login option, to determine if it is a login shell or not. If it is a login shell then it sources the login files. If not then it does not and sources the .bashrc environment file all as documented in the manual. The most typical problem IMNHO is that graphical login managers should set things up for sourcing the login files but by default do not. I am not familiar with BSD. I am only familiar with the GNU/Linux distributions and they often fail to set things up properly to source the login files correctly. That is not a bug in bash but is a bug in the graphical login environment. This is also a case of "bike shedding" because this is something that everyone seems to have an opinion on about how this should be done. And the opinions conflict with each other. And therefore it will likely never get resolved where it works automatically without individual customization to make it work. If your environment has the capability for $HOME/.xsessionrc then that is usually the simplest suggestion to place login environment setting actions there. However note that it is a #!/bin/sh script and not a bash script and therefore should use only POSIX portable shell syntax in that file. And also note that there are many other configurations possible too. I personally do something quite different for example. Beyond this any other suggestions would require a *long* discussion of how each graphical login environment works. Therefore more information about your environment would be needed. > Is this a bug? I feel confident that it is a consequence of how the shell is started. Bob
Re: (OffTopic,) Gnu-Environment restrictions (was Re: Question )
Robert Elz wrote: > I will send one more reply on this topic - I am replying to this message > as I agree with the change of Subject - this is off topic for this list, > which is why I will not respond any further to messages about this here. You are most enlightened and I will do the same with this being my last comment here upon it too. > Bob Proulx said (in a message with the original subject): > | You can use the shared kitchen but please leave it in a better state than > | you found it. > > That would be fine, and would be free use. But > > You can use the shared kitchen provided that you leave it in > a better state than you found it. > > is not. It isn't necessarily unreasonable, but it is not free. Obviously IANAL and am speaking colloquially here. If I were writing a program I could be more exact. Apparently my use of please softens the requirement too much such that it still allows people to leave the kitchen dirty. I think in this discussion of free software you are conflating "free" with the concept of "charge" or "cost". But the "free" is free as in freedom and not free of cost. Which is why I often use the letters free(dom) when talking about it. Or "libre" which in latin based languages have a better connotation. But free(dom) software in the GPL sense is, to my understanding, not without cost. There is most definitely a cost to using it. That cost is that the source must remain free in the freedom sense of the word. The idea being that if you want to dip into the pool of software that it is just like the shared kitchen analogy I posited. You can use it but then the cost is that you must keep derivations from it free as in freedom too. This seems like a perfectly reasonable exchange to me. In any case the author chooses the license for their creations. > This, and the GPL, are just regular commercial transactions -- you give me > something, I give you something (that's called consideration in legal > circles, and just a promise to do, or not do, something, is sufficient). > > While you (and perhaps even I, sometimes) might agree that the price is > reasonable, there is still a price, and software subject to the GPL is > not free. And here I acknowledge that you are probably not that far different from my thinking on this matter. It is down to the very small semantics of the word "free". Which I emphasize is free as in freedom and not free of charge. Because I agree with you that there is a cost. There is most definitely a cost. But where we differ (or perhaps not if we discussed it out to the very end) is whether the software is free as in freedom, ahem, free(dom) software or not. Which is why I will sometimes emphasize this with libre free(dom) software putting excessive emphasis on it. I also note that those in the camp that don't like the GPL's free as in freedom restriction often prefer other software licenses such as MIT or Apache or other because they are free of charge. There is truly no cost to them. That's great! I'm happy for them. But my observation is that many people like them mostly so that they can keep derivations proprietary. There are many who take but they do not give back. They like it that others are charitable. But they do not reciprocate that charity. And they don't like that the GPL allows authors who choose it to require derivations to be just as charitable. And as to whether which strategy is long term most successful only time will tell. Bob Disclaimer: IANAL and use fuzzy language when communicating with people because people tend to like it better that way. Ambiguity of language makes for a lot of unnecessary arguments.
Re: Question
Robert Elz wrote: > Bob Proulx wrote: > > However this difference in philosophy does not change the technology > > that creating a compiler for an interpreted language is very hard. > > Maybe impossible. > > No. Not impossible, not even all that difficult, just in this case, > not all that beneficial either. (There are lisp compilers around for > example.) Lisp is often used as the canonical example. :-) > It has been a long time since I looked inside bash, and I remember none > of it, but I maintain another shell (currently anyway) and it most > certainly contains a compiler - it just never emits the compiled code > into any external form, it remains internal, without that it would be > far too slow. While bash is slow, it isn't that slow (ie: it is fast > enough for its purpose) so I assume it does the same (or similar.) Stephen Bourne himself thought about writing a shell compiler. https://www.computerworld.com.au/article/279011/a-z_programming_languages_bourne_shell_sh/?pp=3 "And the other thing I would like to have done is written a compiler for it. I got halfway through writing a shell script compiler but shelved it because nobody was complaining about performance at the time." > What is relevant though is that compilers for languages like these > rarely generate machine code that is directly executed, because of the > point made earlier that code can be generated on the fly (the canonical > case is: > read var > eval "$var" > ) the resulting binary must contain the complete compiler, so it can Thank you for mentioning the canonical example above. I almost mentioned it. But decided not to go there. > handle cases like this - so the usual course of action is to compile into > an intermediate form, then interpret that (rather than directly interpreting > the untouched input format). This also keeps the whole thing portable, > so no variation (beyond writing portable code in the first place) is needed > to make the whole system run on different architectures. Of course that is a description of the JVM too. However then it has all of the user interaction benefits of working with a compiler and all of the performance benefits of using an interpreter. It is definitely a compromise tradeoff. :-( > The reasons that no-one has bothered to do something more than what is now > done for sh, is that the benefits just aren't there - not that it is > impossible, or even very hard. As was noted, with any normal shell script > (there are exceptions) almost all the time is used by the utilities that > the script runs, not by the script itself, so in many cases, taking the > extra time to produce a compiled external form, then read it in again every > time it is run would not usually lead to any net execution speed increase. Agreed. It just doesn't seem useful for the shell. Note that 'go run golangprogram' is extremely fast, all things considered. I'm impressed by it anyway. > Now I believe that was not the objective here, which was just to make the > code harder for others to read/modify/steal In which case working with a language that has good support for compilers from the start is probably a better strategy. > - unfortunately that would not > really work either. Along with compilers come disassemblers (they are needed > if for no other reason than to help debug the compiler - the shell I maintain > has one built in - it is not usually included in the versions people use, > but can be, just by recompiling it) and for these kinds of languages, they > are usually very good (aside from white space, comments, and ordering of > redirects relative to the rest of the words on the command line, the one I > maintain produces essentially identical output to the script input - it might > occasionally leave out redundant useless syntax that was in the original.) Agreed. Obscurity is not true security. But it does keep the casual observers out. > Now to the philosophical ... I agree with the aims of free software, but > I believe that should be interpreted correctly - that is, if software is > free, it should impose no restrictions at all upon is recipients, which > includes hidden "you should reciprocate and make your software free too" > (which is beyond what GNU actually expects I believe, but many people > seem to - which really means that those people are not really offering free > software, they're just bartering, I'll give you mine, you give me yours ... > that's just commerce.) Libre Free Software under the GPL is not without cost. The cost, said way too simply, is to use and share alike. Many people in the community have worked very hard to contribute to the pool of free software. People benefit from having this pool available. I don't think it is bad to as
Re: Question
Saint Michael wrote: > I use Linux. My business to provide services for problems that I solve, as > you mention, by calling Awk, sed, join, etc., and databases. Sounds good. I classify myself the same way. :-) > I allow my customers to login to a box that I provide in my > datacenter. I cannot accept that it is impossible to restrict them > to only call my *,sh scripts. That sounds more like a restricted shell environment. There has been lots of talk and lots of discussion about restricted shells for many years. It is a very hard thing to successfully do. Many try. If you search the archives you will find much discussion on this topic. > I found a company that has a nice obfuscating and encrypting product > for shell script, and I am trying it. I am no worried only about > passwords, but the overall logic itself. At this point we will have to agree to disagree that this is a good thing to do. Because philosophically the Free Software (also known as Libre Software) community deals with software that respects the user's freedoms. The "free" is Free Software is not about cost but about "freedom". In some languages "libre" meaning liberty gives a better feeling of it. Obfuscating your software is your right but the users of your software do not have the "four essential freedoms". You can read something of this philosophy here: https://www.gnu.org/philosophy/free-software-even-more-important.html > Many thanks for the time you took to respond. But in this case you fall > close to an error. The market needs a tool so we can generate solutions and > also protect our intellectual property. I am not smart enough to write it, > but somebody will. I think the opposite. For example you are using a GNU/Linux system which is Libre Free Software. You are making use of a lot of hard work by many people in the community. They have given this to you and all of us. You are not charged money for this. Yet you do not wish to reciprocate and give back to the community. You wish your work to be proprietary. At that we are philosophically opposed in values. :-( Also when you say "intellectual property" it is problematic. You might want to read some as to why: https://www.gnu.org/philosophy/not-ipr.en.html However this difference in philosophy does not change the technology that creating a compiler for an interpreted language is very hard. Maybe impossible. Therefore one hasn't been created. (I don't like to say anything is impossible. Because often clever people find a way to make the impossible possible. And in this case by changing the problem and restricting the language it is possible. Just not useful at that point.) Therefore regardless of our differences what you want is not easy to do on a technical level. I don't want to suggest this but... Instead of obfuscating the shell script program it would be better if you chose a compiled language from the start. Then you could simply compile it and have what you want. Bob
Re: Bash handling of ENOENT on missing files and directories
Jonny Grant wrote:
> Please keep my email address in any replies
>
> Bob Proulx wrote:
> > Jonny Grant wrote:
> > > Yes, it's a known limitation of POSIX that it uses a shared error code for
> > > both files and directors, ENOENT. Which without programmers handling and
> > > checking the stat() flags, means the error isn't completely clear in the
> > > case where a file or dir does exist.
> >
> > I don't see how POSIX is involved in this. Blaming POSIX is a
> > complete non-sequitur here.
>
> Why do you feel it isn't?
The "No such file or directory" message I see in the Unix v7 code
circa 1979 and therefore appeared at least that early but probably was
there since the beginning. That predates the first POSIX standard by
many years.
File errlst.c:
char*sys_errlist[] = {
"Error 0",
"Not owner",
"No such file or directory",
...
File errno.h:
#define ENOENT 2
> > > I imagine we have spoken already for longer about this, than it would have
> > > been to fix it.
> >
> > I see no bug to fix here. However I fear that trying to fix an
> > imaginary one would introduce a bug here.
>
> How can an easy update to clarify message "No such file or directory"
> introduce a bug?
Because "No such file or directory" is the exact error message that
corresponds to the error returned by the OS kernel as defined by the
system in the /usr/include definition file which is used by all
programs. It is an important commonality. All programs should say
the same thing for that error.
If every program created a different error message for that error then
it would be hard to learn what each and every program said for each
and every individual possible error. OMG! That would be horrible.
Changing it to make it something different would obscure the error
that is being returned by the OS. It would make this one program say
something different for that error than other programs. Different
makes it harder to understand.
Let's look at your cases again.
> $ cd missingdir
> bash: cd: missingdir: No such file or directory
Why do you think that is undesirable? That is the system error. The
requested "missingdir" to change directory to failed because there is
no entry by that name, uh, no such file or directory by that name. :-)
> Likewise, software that deals with directories, or in the case of
> "cd" in bash, trying to change directory, can very easily report "No
> such directory"
I think some of the problem might be that you are thinking that
directories are not files. But in Unix like systems everything is a
file. (And in a Linux like system everything is a file system. It's
a joke but also with some truth to it.)
If I were writing that error message I may have said "No such entry"
since that is the literal error. Meaning directory entry. But I
could see having written "No such file", stopped there without the
directory hint, and defended it as being correct because everything is
a file. But I forgive them for helping the user out a little by
hinting that no files, not even special files such as directories,
were found. Directories are files. They are special files. But
files just the same.
https://en.wikipedia.org/wiki/Unix_file_types
"The most common special file is the directory."
Let's try some other shells to see what error messages they produce.
$ ksh
$ cd nonexistent
ksh: cd: nonexistent: [No such file or directory]
$ mksh
$ cd nonexistent
mksh: cd: /home/rwp/nonexistent: No such file or directory
$ ^D
$ zsh
% cd nonexistent
cd: no such file or directory: nonexistent
$ csh
% cd nonexistent
nonexistent: No such file or directory.
% exit
We can also try a programming solution too just to double check what
error message is provided using a one-liner.
$ perl -e 'chdir("doesnotexist") or die "$!\n";'
No such file or directory
It is good that everyone uses the same error messages.
Let's look at your other case.
> $ ./main
> -bash: ./main: No such file or directory
$ ksh
$ ./nonexistent
ksh: ./nonexistent: not found [No such file or directory]
$ mksh
$ ./nonexistent
mksh: ./nonexistent: not found
$
$ zsh
% ./nonexistent
zsh: no such file or directory: ./nonexistent
%
$ csh
% ./non-existent
./non-existent: Command not found.
% exit
It looks like mksh and csh are the odd ones out here.
Bob
Re: Question
Saint Michael wrote: > Dear Maintainer Note that I am not the maintainer. > Is there a commercial or free software that can take a Bash script and > transparently turn it into a C executable, provided the machines where it > runs has any of the external commands like awk, etc? Not as far as I am aware. And since no one else has spoken up with a suggestion then I assume not. > Something like a Java virtual machine, for Shell. Yes. That already exists! It is the bash shell interpreter itself. If you install a JVM then it can interpret Java programs. If you install bash then it can interpret bash programs. Same thing. The underlying issue is that shells are by design mostly there to invoke external programs. Therefore most of the activity of the shell is to run other programs such as awk, grep, sed, and so forth. It isn't enough to have only the bash program but also the entire set of other programs that is invoked. Java by comparison can also execute external programs but since that is not the primary design goal of it most Java programs do not spend 99% of their code doing so. > I think this language is powerful and I want to distribute some > tools that contain critical information. > > So far I have been googling and there is only obfuscation tools, not > compilation tools. Making a shell script compiler would be a very hard problem because the shell is an interpretive language with a primary task of invoking external programs. It would be very, very hard. Impossible actually. The nature of interpretive languages are that they can do things that compiled languages cannot do. For example interpreted programs can generate code on the fly and interpret it. Such as creating functions during runtime. Such as dynamically creating and executing code based upon dynamic input. Such tasks are easily done using an interpreted language but must be done differently when using a compiled language. One cannot create a compiler that can compile every capability of an interpreted language without itself embedding an interpreter for that language. If you restrict an interpretive language down to only features that can be compiled then it is always possible to write a compiler for it. But then it is for a restricted set of language features as compared to the original language. It is a different language. Sometimes this is an acceptable tradeoff. But I will hazard a guess that for the shell and for programs written for it this would be a quite different language and no longer useful. Possible. But not useful. > I think this language is powerful and I want to distribute some > tools that contain critical information. You don't say what critical information you are talking about. But it makes me think things like accounts and passwords that you would not want to be available in clear text. Note that obfuscating them in a compiled program does not make them inaccessible. You also don't say which OS you are using. Most of us here will probably be using a GNU/Linux distribution which includes the shell natively or the Apple Mac which includes a working environment. Therefore I will assume you are using MS-Windows. I suggest either bundling an entire environment such as MinGW such as Bash for Windows or Cygwin so that all of your external dependencies are satisfied. Or using a different language such as C which is natively compiled. Or using a restricted set of Perl, Python, Ruby, or other that already has a compiler available for that restricted subset. Bob
Re: Bash handling of ENOENT on missing files and directories
Jonny Grant wrote: > Greg Wooledge wrote: > > The wording is taken directly from perror() and related library calls, > > as translated for your locale. > > Yes, it's a known limitation of POSIX that it uses a shared error code for > both files and directors, ENOENT. Which without programmers handling and > checking the stat() flags, means the error isn't completely clear in the > case where a file or dir does exist. I don't see how POSIX is involved in this. Blaming POSIX is a complete non-sequitur here. > I imagine we have spoken already for longer about this, than it would have > been to fix it. I see no bug to fix here. However I fear that trying to fix an imaginary one would introduce a bug here. Bob
Re: Execution continuing after SIGINT received
Kevin Brodsky wrote: > The thing is, "the next level of the program" really is another program, > i.e. the one that invoked it, and you are communicating via the exit > status, so it's certainly not as explicit as re-throwing an exception in > C++, for instance. But sure, once you are aware of this mechanism, it's > not difficult to understand the rationale. One shouldn't confuse exception handling with signal handling. They are really quite different things. :-) > Actually, IMHO, what makes it look very counter-intuitive is the fact > that you need to first reset the signal handler for SIGINT. Of course > this is necessary to avoid invoking the handler recursively, but it > feels very much like a workaround. You probably remember that signal(2) is unsafe because it does exactly that and resets the signal handler to the default. This creates a race condition. There is a window of time in the signal handler when a second signal can catch the program with the default state set before it can reset the signal handler. That is the main reason why sigaction(2) should be used instead. > WIFSIGNALED is true if "the child process [...] terminated due to > the receipt of a signal that was not caught". That's not really what > we want to know here; we want to know if the child process received > a signal that caused it to terminate. Whether it handled SIGINT to > clean up resources is irrelevant; what's relevant is that it > eventually terminated as a consequence of SIGINT. Ideally, exit()ing > from a signal handler should set a bit in the exit status expressing > exactly this. Well... Maybe. But how are you going to know if the program "eventually terminated as a consequence of SIGINT" or not? And of course then you would need a way to provide a program to do the opposite when required. But it has been 40 years already with the current way things are done and it is a little late to change things now. > I'll stop digressing, POSIX is what it is and we won't change it anyway > ;-) For now, there's no other way to communicate with the shell, so > that's fair enough. I don't see how POSIX is even involved in this. Other than stopping the proliferation of differences on different systems. Before POSIX came along every system did this slightly differently. It was awful. POSIX.1 simply picked BSD as the behavior to standardize upon. And then we could stop putting #ifdef's in our code for every different system quirk. What you are seeing here is just the basic behavior of the old Unix and BSD kernels and now every kernel that has followed since. Since that is the common behavior among all systems the standards bodies will mostly say, freeze on that behavior, do it the same way, don't do it differently. Surely that is a good thing, right? The subtleties of https://www.cons.org/cracauer/sigint.html about how programs should behave were learned not all at once but over the course of time. Bob
Re: Execution continuing after SIGINT received
Kevin Brodsky wrote: > $ bash -c '(trap "echo INT; exit 1" INT; sleep 60s); echo after' > ^CINT > after > $ This is a good example of a bad example case. You shouldn't "exit 1" or you will replace the information that the process is exiting due to a signal with an error code. The trap handler should kill itself instead. Use this test case instead. > $ bash -c '(trap "echo INT; trap - INT; kill -s INT $$" INT; sleep 60); echo > after' Of course that doesn't change the end result here. But at least the program exit WIFSIGNALED information is now correct. Signal handlers should always raise the signal on themselves after handling whatever they need to handle first. In any case I can't recreate your problem when using real processes in separate shells not all on one line. Bob
Re: bug in cd
Kae Verens wrote: > but, if you do this: > cd // ; pwd > > then the CLI reports > // Using two "//" at the start of a path name is special because in some systems it means the path component that follows is a hostname. Therefore it must be preserved verbatim. Bob
Re: How to run bash test suite
Phi Debian wrote: > I tried to subscribe to bug-bash but never got the confirmation mail. You must have successfully received and replied to the confirmation email because I looked at the subscriber list and you are subscribed. Obviously not needed now but if you have trouble with the mailing lists write to the mailing list owner at bug-bash-ow...@gnu.org to contact the humans who help with the mailing list. This is the email address convention for all of the Mailman managed mailing lists. Bob
Re: null pointer deref, segfault
Chet Ramey wrote: > On 1/24/17 2:07 AM, Brian 'geeknik' Carpenter wrote: > > Going through some ancient bug reports and I came across > > https://savannah.gnu.org/support/index.php?108884 which apparently nobody > > uses anymore. > > > > <<$(()())|>_[$($(<<0)) crashes bash on Debian, Red Hat, FreeBSD, etc. > > It's marked as `Done'. Since it is both marked as Done and also reported to be resolved in bash 4.4 then I have closed the ticket. Bob
Re: Why does bash use xmalloc?
Peng Yu wrote: > My point is why bash terminates when it runs an external command that > requires a large memory. Shouldn't bash return an exit code on behalf > of the failed command and continue to the next command? Note that I am going to say something without looking at the source code of bash to say it. Therefore I might be wrong about bash. But it is a general statement about shells. > ... runs an external command that requires a large memory If bash is running an external comand then it will be doing so in a fork()'d off subshell using exec(). It won't be bash anymore but the other process. If it exits after the fork but before the exec then it will be the subshell child process that exits not the main bash execution thread. If the bash process itself has grown very large then the fork() might fail on systems that do not use memory overcommit. (Traditional Unix kernels or Linux kernels with overcommit disabled for robustness.) If the fork fails then I am not sure what bash should do in that case. It could potentially loop and retry a few times but if all fail then exiting is probably the only reasonable action. If bash itself is running out of memory then it is almost certainly due to inefficient shell scripting. For example like one that I analyzed in this thread. Of course that is just one of many possibilities. http://lists.gnu.org/archive/html/bug-bash/2011-11/msg00189.html Bob
Re: forked before bash subshell
XiaoBing Jiang wrote: > Greg Wooledge wrote: > > If you want to ENSURE that the child shell process is replaced by the > > external sleep 20, use an explicit exec. > > yes, I want to know why bash not optimize this. or any strategy ? Because it wouldn't save anything significant. Since the parent shell is bash the extra bash in memory is already in memory. A forked process uses almost the same OS memory as the original. The OS will share the code pages between the two processes at the OS memory page level. Data pages will split when they are individually modified. Even if you use 'exec' to replace the forked and backgrounded copy another one is still in memory as the script runs. Only unique data pages will go through the copy-on-write process as the two bash processes diverge. Therefore even if you *think* you are saving something by optimizing it out the actual savings is insignificantly small. Plus this is really only a problem if the main part of the script exits leaving the forked subshell running in the background. However that is not a good thing to do. If you want to leave the process running that is a daemon process. Launching a daemon process needs more setup than this to avoid other problems. Since leaving processes behind like that is a bad thing to do it again doesn't make much sense to try to optimize the doing of it. It is possible to contrive some examples where it makes sense to do this optimization. But mostly they are very unusual cases. Not typical cases. Bob
Re: Syslog output from bash
Richard Lohman wrote: > Problem is, I'd like the output to resemble other syslog messages: > Mmm dd HH:MM:SS hostname bash[pid]: command > And ultimately drop the username in as well. Since only bash is logging in I suggest using the 'logger' command instead of modifying bash. Normally I am doing things like this: logger -t cmdname "my log message here" But you can get the format you wish with: logger -t "cmdname[$$]" "your log message here" Bob
Re: Curious case statement error
psko...@gmail.com wrote: > [a-z]) echo "Character is in Lowercase";; > [A-Z]) echo "Character is in Uppercase";; What is the output of 'locale' for you? It will almost certainly show that your LC_COLLATE is *NOT* set to the C locale but to some other locale. Your statements above are correct only in the C locale. It depends upon your locale setting and the program's specific handling of it. If it is en_US.UTF-8 then the above does not apply. Instead it is more likely this: [a-z]) echo "Character is in aAbBcC...z range";; [A-Z]) echo "Character is in AbBcC...zZ range";; This is due to the locale collation ordering in your environment. It is not specific to bash and also affects grep, sed, awk, sort, and so forth. (However newer versions of most programs are specifically working around this now. The problem used to be more common a few years ago but with recent releases the problem is disappearing.) Using the human language locales en_US.UTF-8 ranges one must use the [:lower:] and [:upper:] ranges. [[:lower:]]) echo "Character is in Lowercase";; [[:upper:]]) echo "Character is in Uppercase";; The grep man page explains this in detail so let me quote it here: Within a bracket expression, a range expression consists of two characters separated by a hyphen. It matches any single character that sorts between the two characters, inclusive, using the locale's collating sequence and character set. For example, in the default C locale, [a-d] is equivalent to [abcd]. Many locales sort characters in dictionary order, and in these locales [a-d] is typically not equivalent to [abcd]; it might be equivalent to [aBbCcDd], for example. To obtain the traditional interpretation of bracket expressions, you can use the C locale by setting the LC_ALL environment variable to the value C. For reasons that most of us common people disagree with the powers that be decided that locale specific collation sequences would ignore punctuation and would fold case using "dictionary" collation ordering. Note also that bash's collation sequence is set when it is started. In other words changing the LC_ALL or LC_COLLATE variables only affects newly launched programs. It will have no effect on the currently running bash shell. In other words to change it for bash you would need something like this: $ ...shell with LC_COLLATE set to en_US.UTF-8 with bad collation ... $ env LC_COLLATE=C bash $ ... works now ... I mention this because otherwise people try changing the variable and then don't see a change in the already running bash shell. Bob
Re: [Documentation] -d returns true for symlink to directory
Reuti wrote: > While we are on this: wondering about the difference about -h and -L I found > that `man test` outputs on OS X: > > " -h file True if file exists and is a symbolic link. This > operator is retained for compatibility with pre- >vious versions of this program. Do not rely on its > existence; use -L instead." > > while on Linux `man test` just states that it's the same. Is there any > preference which one should be used for the bash builtin? Originally Unix did not provide symlinks. Then when symlinks were added they were not completely added in a uniform regular way through every command. Bummer! The original option letter used by test to check for the presence of a symlink was -h. I don't know why. But in those days the only test option to test for a symlink was -h. The -L came later. This legacy is still visible in the HP-UX test man page for example. No -L documented there. http://nixdoc.net/man-pages/HP-UX/man1/test.1.html However note that because POSIX requires it the /bin/sh posix shell must implement -L as identical to -h. And the reverse. Therefore you are safe to use either on any POSIX system. Remember too that POSIX wasn't originally meant as a design document but as an OS feature non-proliferation treaty to make it possible to write portable programs. POSIX features must be present on any POSIX system and therefore you can use the feature and expect it to work. If you need to interact with legacy systems which predate POSIX and have never been updated to comply then you must use -h because that is all there is there. But if you never interact with anything non-POSIX then using -L is okay too. Either is the same as the other now. I think the statement in the Apple man page is there because someone liked -L over -h and wanted to scare everyone into using one over the other. But there isn't any reason to do so. Bob
Re: Cannot clone from anonymous git
Eduardo A. Bustamante López wrote: > When I try to clone: > | dualbus@hp ~ % git clone git://git.savannah.gnu.org/bash.git > | Cloning into 'bash'... > | fatal: Could not read from remote repository. Connection was closed immediately. > | Please make sure you have the correct access rights > | and the repository exists. Misleading error message. It couldn't connect. > The links here appear to work: http://git.savannah.gnu.org/cgit/bash.git Since Friday the 6th there was a large move with various assorted related and unrelated changes. Which has makes it difficult to tell what broke what. I am deep in the middle of *.savannah and can give the most information about it. Immediately after the move there was a systematic breakage of uids and ACLs which were fixed last weekend. But the problem is only partially Savannah. Most of the problem is global network connectivity. It seems to be worse from Europe than from the US. For example me in Colorado I rarely have any problem. But at the same time people overseas report complete failures. There are reports of problems across all of the servers from fencepost to lists.gnu.org to the web server systems. The common change to all of them was a core router change. On Savannah vcs the network connectivity tickles a problem with the git daemon. It causes them to get stuck. This causes them to hang around endlessly. They are launched from xinetd with 100 process limit. Especially during busy activity they fill up the slots. Then xinetd has a bug IMNHO that instead off allowing connections to queue it simply closes any additional connection. This is what you see and describe in the above. git:// failed because all 100 slots were in use and xinetd closed your additional connection. Meanwhile http:// worked because that used apache. Maybe the problem is actually the xinetd which is common to the cvs, hg, bzr, svn daemons which are also suffering. Now before everyone decided to switch to apache note that apache also has process limits. It just so happens that fewer people are using it and so it wasn't limited. Yet. If everyone switched to apache http:// then it would simply switch the problem from one daemon to the other. We see all of the source daemons stack up. This includes cvs, hg, bzr, and svn processes too. Something network related is causing them to hang around after they should have exited. I have been tweaking the tcp keepalives to try to makedead the stale processes faster. Assaf has been doing the majority of the process triage and manually pruning processes in order to keep things functional. On the client side it is impossible to distinguish between git daemons consuming the limit of process slots versus router connectivity problems. Both have the same effect. However if you can get through with ssh:// or http:// then that indicates xinetd process slots are full. But over the last week (mostly Europe and other overseas clients) reports endless problems trying to post bugs using the web server. Endless problems trying to SMTP mail in and out. Reports of rsync failing. Sometimes IPv4 works when IPv6 is failing. All of those are separate except for the global network connectivity through the routers. Obviously the most urgent problem to fix is the global network connectivity problem. The FSF admins are aware of the problem. They post (terse) status updates to https://pumprock.net/fsfstatus so do please keep an eye on it for out-of-band status updates. Especially when there are Boston local network problems. If you are suffering problems I encourage a trouble report being made. Please include details. Say where from and where to. Say whether it was IPv4 or IPv6. Time the problem occurred. Then if later it works update and say that. Because one of the problems is that this seems worse from Europe than from the US. There isn't a reproducible test case to show if a change fixes things or not. I in Colorado have a hard time triggering any problems but have been able to on occasion. But people in Europe have been most of the problem reports. Most of them have been using IPv6. And that is all I know. Bob
Re: Avoid asterisk expansion when it selects "everything"
Anis ELLEUCH wrote: > I would like to ask if it is possible to disable expanding asterisk when it > selects all entries ? You ask if it is possible and the answer is no it is not possible. Because the shell expands the "*" before it passes the result as arguments to the rm command. The rm command has no idea it was the result of an expansion from a "*" wildcard. The rm command simply receives the list of files. > Another idea: `*` and `/*` should not be interpreted and the user has to > enter another sequence "more powerful" to emphasize selecting all entries ( > `^*` would it work just fine ?) But then the same thing exists there. If the user accidentally enters that sequence then it is the same thing all over again. If I am ever uncertain about a file glob expansion then I use echo first to see what the expansion will do. If dealing with dangerous commands such as rm then it is wise to check twice before invoking. echo rm * .jpg If you saw the result then you would fix it before invoking the command for effect. If the result is bad then correct it. If the result is good then remove the echo and run the command for effect. Bob
Re: Fwd: read builtin return non-zero status reason
Cuong Manh Le wrote: > I'm sorry for that. It's not the time between post but after sending the > email, I look at the help-bash archive and see it have no active email > since January. There has been lots and lots of email in the archive since January. http://lists.gnu.org/archive/html/help-bash/2016-02/threads.html Maybe the archive you were looking at is dead? Note that the gnu.org archive only updates every half hour. Therefore messages don't show up there immediately even when they have been sent to the list immediately. > Anyway, still my fault. Don't worry about it. It is all in the past now. Bob
Re: Fwd: read builtin return non-zero status reason
Greg Wooledge wrote: > On Wed, Feb 24, 2016 at 11:31:40AM +0700, Cuong Manh Le wrote: > > I send this email to help-bash but it seems to be inactive. Please help me > > with this problem. Thanks. > > help-bash is active. You probably just didn't wait long enough. > If this is your first message to help-bash, your message is probably > waiting in some moderation/antispam queue. Impatience is the problem. Here are the times of the three messages. help-bash: Date: Wed, 24 Feb 2016 11:14:31 +0700 help-bash: Date: Wed, 24 Feb 2016 11:20:46 +0700 bug-bash: Date: Wed, 24 Feb 2016 11:31:40 +0700 You waited six minutes for a response to help-bash and then sent it there again. You then waited eleven minutes before forwarding it to bug-bash. Neither six nor eleven minutes is a reasonable amount of time to wait for a response. Plus your timezone is +0700 while most of the moderation folks and bash folks operate out of -0500 through -0800. It may be the middle of the day for you but that is the middle of the night for others. Your first message was held for moderation and released at a very early morning time of 05:44:56 -0800. Your bug-bash message had no moderation delay probably due to previous contact there. Please have some patience. Bob
Re: How to lock a terminal
Benno Schulenberg wrote: > Bob Proulx wrote: > > I do not have a nanorc file at all. The defaults are okay for > > suspension without any specific configuration. > > ?? How does that work? What version of nano are you using? Since this has nothing to do bash I am going to take this to the nano list only. Bob
Re: How to lock a terminal
Benno Schulenberg wrote: > Bob Proulx wrote: > > [...] this is the perfect case for job control. No need for a > > second terminal. Here is an example. Use Control-Z to stop the > > foreground job. > > For that to work, it requires having 'set suspend' in your > nanorc. (Which I don't have, because it annoys me when nano > drops into the background when I accidentally hit ^Z.) I do not have a nanorc file at all. The defaults are okay for suspension without any specific configuration. Since you are specifically configuring suspend disabled one must presume that you will already know about job control and have specifically disabled it and will work around it by other means. Such as in your case M-Z or another terminal or other method. Bob
Re: How to lock a terminal
Nick Warne wrote: > I was in a SSH session, and checking something inadvertently issued: > > > nano /var/log/messages | grep a > > (I was searching for something else than an 'a', but the above example shows > the issue - about to use 'nano', but then forgot to change it to 'cat'). > > The terminal just sits there doing nothing - CTRL+C doesn't do anything; in > a SSH session, the only option is to kill the terminal. On a local machine, > you can use kill -9 from another terminal to get out of it. On a remote machine you can do the same. There really is no difference between local and remote here. You just use a second terminal for it. However this is the perfect case for job control. No need for a second terminal. Here is an example. Use Control-Z to stop the foreground job. rwp@havoc:~$ nano /var/log/messages | grep a ^Z [1]+ Stopped nano /var/log/messages | grep a rwp@havoc:~$ jobs [1]+ Stopped nano /var/log/messages | grep a rwp@havoc:~$ kill %1 Received SIGHUP or SIGTERM rwp@havoc:~$ jobs [1]+ Terminated nano /var/log/messages | grep a rwp@havoc:~$ jobs rwp@havoc:~$ Simply stop the process and then kill it using the same terminal. Bob P.S. The other suggestions to use Control-X to exit nano are also good too but job control is general for the entire class type of commands like this and I think good to know too.
Re: [PATCH/RFC] do not source/exec scripts on noexec mount points
Mike Frysinger wrote: > But bash itself has no problem running this file: > $ bash /dev/shm/test.sh > hi >... > This detracts from the security of the overall system. People > writing scripts sometimes want to save/restore state (like > variables) and will restore the content from a noexec point using > the aforementioned source command without realizing that it executes > code too. Of course their code is wrong, but it would be nice if > the system would catch & reject it explicitly to stave of > inadvertent usage. I don't think it makes sense for a userland program to be an enforcer of this type of check. It gives a false impression of a security that does not exist. Which I think is more dangerous. It will almost certainly get in the way of a reasonable use case. And nothing prevents one from running a private copy of a shell without such a check. Or any of the many compatible /bin/sh variants such as ksh, zsh, ash, dash, and so forth. Bob
Re: Design question(s), re: why use of tmp-files or named-pipes(/dev/fd/N) instead of plain pipes?
Linda Walsh wrote:
> Bob Proulx wrote:
> >Where does the OS claim it is a file descriptor?
> 1) in the kernel documentation:
> file 'devices.txt:
> /dev/fd /proc/self/fd symbolicFile descriptors
That shows the file descriptor map for the process. It isn't the
program's actual internal file descriptor. Obviously it can't be.
They have abbreviated it trying to make it be concise and fit there.
If you disagree with the description then take that up with the
devices.txt authors.
> 2) ls -Ll shows the type of what the symlink points to:
> >ls -Ll /dev/fd/
> total 0
> crw--w 1 law tty 136, 0 Oct 21 19:49 0
> crw--w 1 law tty 136, 0 Oct 21 19:49 1
> crw--w 1 law tty 136, 0 Oct 21 19:49 2
The -L hides the symlink. Try it without the -L.
$ ls -log /dev/fd/
total 0
lrwx-- 1 64 Oct 23 15:45 0 -> /dev/pts/13
lrwx-- 1 64 Oct 23 15:45 1 -> /dev/pts/13
lrwx-- 1 64 Oct 23 15:45 2 -> /dev/pts/13
lr-x-- 1 64 Oct 23 15:45 3 -> /proc/31883/fd/
It is showing where the program's file descriptors are redirected to
and from. But those are still file path names. Paths. Names. This
is an email from me to you. I am not defining this exactly for a
standard. It is a string of characters. It isn't the integer value
index offset into the data structure holding the file data inside the
program. It's not. Don't confuse the two.
> and 3)
>
> YOU claim it is a descriptor further down in this note:
> > tmpname=<(sort /dev/null); ls -l "$tmpname"
> >ls: cannot access /dev/fd/63: No such file or directory
> >
> >But as soon as the first commend is finished at the ';' then the
> >descriptor will be closed and the path will evaporate. Therefore the
> ^
> >path won't exist for the second command.
At that point I am talking about the program's internal behavior. I
also mispelled command too. I am talking conversationally here.
if (close(fd) < 0) { ...; exit(1); }
exit(0);
Don't confuse the integer value of the program's file descriptor,
which is an offset into the program's data structure array, with a
path to a file. In this case they are confusingly similar because the
/proc one is actively trying to represent the other! But life is like
an analogy and being similar does not make it the actual thing.
Actively trying to represent the other here means one is really a
mapping show how one is mapped to the other. That doesn't make it
actually the other.
> If you close a file-descriptor to a normal file
> it doesn't disappear. If it was deleted via 'rm' the name disappears
> immediately, and the only access to it is through the descriptor.
>
> In this case, closing the descriptor deletes the temporary
> name in the process's 'file descriptor' "/fd/". Normal filenames
> are not "auto-deleted" when you close an I/O handle to them.
Confusion such as yours is precisely why I cringe whenever I see
someone using /dev/stdin, /dev/stdout, /dev/stderr or any of the
/dev/fd/* symlinks in a program instead of using the normal
redirections. It causes people to think they are something that they
absolutely are not. That and the unportant nature of them.
The /proc/self/fd/ directory is a kernel construct. It does not have
normal file system semantics. I am sorry to be the bearer of bad news
to you. But it doesn't. If you don't like this then that would be a
discussion with the kernel folks responsible for it.
And really what would be a sane purpose in being able to remove files
from it? What would that do? Would that make any sense at all? I
think it wouldn't make any sense.
> >But if you use it for the ls command itself then it exists for that
> >command.
> >
> > ls -l <(sort /dev/null)
> >lr-x-- 1 rwp rwp 64 Oct 19 15:56 /dev/fd/63 -> pipe:[102059434]
>
> But only as a pointer to something one can do I/O on.
> You can't set any file attributes or metadata on "pipe:[]" It's not a
> real file somewhere.
Exactly! Stick with that thought. You are on the right track with
it. It is only a symlink in a /proc kernel space representation. You
can do I/O on it but you can't actually change permissions or remove
it or other things.
Bob
Re: Design question(s), re: why use of tmp-files or named-pipes(/dev/fd/N) instead of plain pipes?
Linda Walsh wrote: > Greg Wooledge wrote: > >A simple example: > >diff -u <(sort file1) <(sort file2) > You claim <(sort file1) is a filename? Me too. > >touch a > >mv <(sort a) <(sort a) > mv: cannot move ‘/dev/fd/63’ to ‘/dev/fd/62’: Operation not permitted > > The OS claims that <() generates a file descriptor -- not a filename. Where does the OS claim it is a file descriptor? "/dev/fd/63" is a file name not a file descriptor. And /dev/fd is a symlink into /proc space. $ ls -ld /dev/fd lrwxrwxrwx 1 root root 13 Sep 21 18:19 /dev/fd -> /proc/self/fd Since it is in /proc it is simply a kernel space construct. It won't allow things like rename(2) to work upon it. I think you were confused by the "fd" part of "/dev/fd" but it is simply a textual name of the file path. > It's the same with 'FILENAME'a used with "read < FILENAME >FILENAME" > > >read < FILE > FILE Unless FILE is a pipe or a tty or other such device then reading and writing the same file at the same time is a bad idea. It will get opened for truncate and empty the file before it can be read. > >a=<(sort a); chmod +x "$a" The above is confusing a file named "a" and a variable named "a" which is always a bad idea. Better to use different names in experients such as that so that they don't get confused with each other. tmpname=<(sort /dev/null); ls -l "$tmpname" ls: cannot access /dev/fd/63: No such file or directory But as soon as the first commend is finished at the ';' then the descriptor will be closed and the path will evaporate. Therefore the path won't exist for the second command. But if you use it for the ls command itself then it exists for that command. ls -l <(sort /dev/null) lr-x-- 1 rwp rwp 64 Oct 19 15:56 /dev/fd/63 -> pipe:[102059434] > chmod: cannot access ‘/dev/fd/63’: No such file or directory > >echo aaa > a > >read var <(sort a) && echo "var=$var" >a > ## the above hangs: if <(sort a), was a filename, read would > ## have read 'aaa' ## from it, and returned immediately. ## Then the 'echo' > command would have put 'var=aaa' ## in 'a': instead: Syntax error in the above. You probably meant to have 'read' read from the pipe and not hand it to read as a variable name argument. echo read var <(sort a) read var /dev/fd/63 You probably meant to use a '<' input redirection there. read var < <(sort a) && echo "var=$var" >a # equates to: read var < /dev/fd/63 That works as you expected. Bob
Re: why must bash zap last search string just because we hit ^C?
Eduardo A. Bustamante López wrote: > $ some long command I've typed > And then boss calls: URGENT! Run uptime and tell me the output, I'll: > $ some long command I've typed > Press Ctrl-u > $ _ > Write uptime RET > $ uptime > 11:23:45 up 4 days, 22:39, 4 users, load average: 0.51, 0.86, 0.75 > $ _ > Press Ctrl-y > $ some long command I've typed > :-) Additionally let's say that along the way I killed some other text so that C-y yanks back that more recently killed text instead of the one I saved first. In that case ESC C-y will rotate the kill ring to the previous killed text. Repeating ESC C-y again rotates to the one before it. So can walk back through until finding the text I want to yank back. :-) Bob
Re: SIGINT handling
Greg Wooledge wrote: > Just for the record, ping is the *classic* example of an incorrectly > written application that traps SIGINT but doesn't kill itself with > SIGINT afterward. (This seems to be true on multiple systems -- at > the very least, HP-UX and Linux pings both suffer from it.) The command I run into the problem most with is 'rsync' in a loop. EXIT VALUES 0 Success ... 20 Received SIGUSR1 or SIGINT Which forces me to write such things this way. rsync ... rc=$? if [ $rc -eq 20 ]; then kill -INT $$ fi if [ $rc -ne 0 ]; then echo "Error: failed: ..." 1>&2 exit 1 fi Bob
Re: 4-byte script triggers null ptr deref and segfault
Greg Wooledge wrote: > Brian Carpenter wrote: > > While fuzzing GNU bash version 4.3.42(1)-release > > (x86_64-unknown-linux-gnu) with AFL(http://lcamtuf.coredump.cx/afl), I > > stumbled upon a 4-byte 'script' that triggers a null ptr deref and causes a > > segfault. > > > > https://savannah.gnu.org/support/index.php?108885 > > Well, that's an annoying web-to-mail interface. It didn't include the > full bug report? There is no web-to-mail interface. The above was simply an email from Brian citing the bug report he filed above. He filed a ticket on the web site and then mailed a reference to it here. Bob
Re: Why does a Bash shell script write prompts followed by reads and do it right?
Robert Parker wrote: > fputs(shortprompt, stdout); > fflush(stdin); Uhm... fflush'ing stdin? That doesn't make sense. Obviously you meant to flush stdout. fputs(shortprompt, stdout); fflush(stdout); That is likely the specific problem. Unfortunately I have no time for comments about the rest of the code. Bob
Re: Why does a Bash shell script write prompts followed by reads and do it right?
John McKown wrote: > Not a bug, so likely the wrong forum. Agreed. > Have you tried doing a fflush() after the fputs()? I.e. something to tell > the Kernel to "write this out immediately!". Almost. It is the C library libc that buffers output with fputs() not the kernel. The libc normally checks to see if the output is a "slow" device such as a tty device. If so then there is no buffering. If the output is a "fast" device such as a file then libc buffers the output into larger writes for efficiency. But that only applies when the output is redirected. If it is a prompt then the output will be to the tty and there won't be any buffering. The fflush() in that case won't do anything. > In the case of write(), I think you need to use the O_SYNC flag in > the open() call. No. That isn't necessary at all. It doesn't matter to programs in userland if the write is synchronous or not. Userland programs can't tell the difference. Using synchronous I/O is all about what happens at the instant of a power failure or hard system crash. Robert Parker wrote: > Yet when I attempt the same in a C program, the system always writes 2 > prompts, then waits for a read. Basically that doesn't make sense. I do not believe that is what is actually happening. Please go back and look more deeply. Since you are talking about prompts I assume you are writing to a tty device. There is no buffering and the output will appear on the tty immediately without need for opening it synchronously or flushing or anything extra. Bob
Re: Why does a Bash shell script write prompts followed by reads and do it right?
Robert Parker wrote:
> Yet when I attempt the same in a C program, the system always writes 2
> prompts, then waits for a read.
This isn't a C list. But if you don't show us the code then no one
can give good help.
If you write, read, write and then it *must* try the read before the
second write. If you are seeing the second write then it means that
the read returned after the first write. What you say, writes two
prompts and then read doesn't make sense.
> Does not matter if I write(1, "..."); read(0, number, buffer); or use
> fputs("...", stdout); fgets(.
> The result is the same.
> And I have tried using readline without any better result.
This sounds like you simply have some basic problem issue. You will
need to figure that out first. If you write(), if you fputs(), and
you are not seeing the output then start there. Don't move on to
anything else until you understand that part of the problem.
Your problem is really one for discussion on a C programming list.
> I have downloaded the source code for Bash.
> Would someone please let me know where to look to learn to make my code
> work like yours?
That is the wrong direction to go. It is like saying, I have this
problem building a small bike shed. I couldn't figure it out.
Therefore I downloaded the plans to the Eiffel Tower in order to
compare the two. They are not really comparable. Trying to do so is
just a distraction from solving your real problem.
Bob
Re: In the bash man page the entry for the 'read' command has two spaces after it instead of one.
Andreas Schwab wrote: Ángel González writes: entering «/^ *read⤶» may be easier to type (and remember) It won't match, though. It matches fine for me. Although I suggest adding the square bracket so as to avoid the other false positives as long as one knows that read has options documented. This works. /^ *read *\[ Bob
Re: Worth mentioning in documentation
Greg Wooledge wrote: Juanma wrote: [ is an ordinary command (a shell builtin) Here is another point I find confusing: I thought a shell builtin didn't have a separate binary executable file, like 'cd' (which cd = fail), but some of them do have such form (which [ = /usr/bin/[ ; which pwd = /bin/pwd). I also fail to see how 'test' modifies the state of the shell itself (like 'cd' does), or why it is impossible or inconvenient to obtain [its functionality] with separate utilities. Don't use which(1). Which is an external program, so it has no knowledge of the shell's builtins, aliases, functions and keywords. Instead, use type. Another problem with 'which' is that it was originally designed and written for csh users specifically. It was a csh script and would source the users ~/.cshrc file so as to acquire their aliases from there so as to be able to report user aliases. I think it is still that way on HP-UX (and probably others) for example. However that obviously won't work well for bash, ksh, zsh, and other shell users. And IMNHO csh is a terrible shell regardless of the tcsh users using it who think otherwise. Therefore some distributions such as Debian have rewritten 'which' as a /bin/sh script meaning that 'which' behaves differently on different systems. It is non-portable. imadev:~$ type cd cd is a shell builtin imadev:~$ type [[ [[ is a shell keyword imadev:~$ type -a test test is a shell builtin test is /usr/bin/test test is /bin/test Bash implements test as a builtin not because it's necessary, but simply for efficiency. Forking a whole process to test whether two strings are equal would be horribly wasteful. An important mental concept is that test and [ must *behave* the same as if they were an external program. They are internal builtins for efficiency now but everything behaves the same if they are external programs. This is why quoting as if they were external programs is required. On the other hand [[ has always been a builtin and therefore the shell can avoid one layer of quoting and does. Bob
Re: bash displays strange characters after base64 decoding
Valentin Schmidt wrote: Subject: bash displays strange characters after base64 decoding Bash displays strange characters (including command prompt) (see attached png file) after the execution of the base64 (GNU coreutils 8.21) command like described as follows: base64 -d base64.txt This is not related to your command shell bash. It is only your terminal that is involved. Your terminal interprets escape sequences. Move the cursor. Change the color. Use reverse video. Use a different character set. Things like that. You have sent random binary characters to your terminal. This is similar to if you decided to eat random non-food items from random shelves in a store. Newspapers. Shampoo. Sunscreen. Things like that. It won't be good for you! Eventually you will eat something that will upset your tummy and send your body into a bad state. Not good! Sending random binary characters to your terminal is similar to this. It is not good for your terminal. With enough random characters it is inevitable that you will eventually send your terminal something that will instruct it to go into an undesirable state such as you have shown. The aim was actually to direct the output of the base64 command into a file (would have been a .jpg file) but the decoded.jpg was forgotten. Oops. :-( Accordingly the output was directed to stdout instead. This resulted in the command prompt consisting of strange characters after decoding. That is unfortunate. However terminals are called smart terminals for the reason that they interpret escape sequences. Because this is useful for them to do this. There are dumb terminals available too. Dumb terminals do not interpret escape sequences and would be safe from this problem. Except people generally do not like dumb terminals because they do not have any of the useful capabilities we have come to like. Other than switching to a dumb terminal there is little that can be done to prevent this. Other than avoiding feeding your terminal random binary characters. Also any command entered via the keyboard resulted in strange characters being displayed instead of the typed characters. Also the output of the ls (which seems to be executed) command (as an example) is displayed as strange characters. I don't recognize the terminal emulator program you are using. One of the menu options may offer a way to reset it. Otherwise you can try the reset command to reset the terminal back to a sane state. man reset When invoked as reset, tset sets cooked and echo modes, turns off cbreak and raw modes, turns on newline translation and resets any unset special characters to their default values before doing the terminal initialization described above. This is useful after a program dies leaving a terminal in an abnormal state. Note, you may have to type LFresetLF (the line-feed character is normally control-J) to get the terminal to work, as carriage-return may no longer work in the abnormal state. Also, the terminal will often not echo the command. Otherwise the best you can do is to close the terminal and start a new one. Bob
Re: ShellShock where my browser window randomly re-encodes with what looks like Chinese
S. Irelan wrote: I have vested so very many hours and attempts and well, can't get out from under what I understand is the Shellshock /bash hacking browser attacking NIGHTMARE !!! Your report does not indicate any similarities to the well known Shellshock bug. https://en.wikipedia.org/wiki/Shellshock_(software_bug) Subject: Re: ShellShock where my browser window randomly re-encodes with what looks like Chinese Nor does it seem related to anything that would have any result in a web browser. My name is shaun, I have a knoppix that might very well be knoppix64 and I have tried the suggested knoppix/adriane fuse thingy (pardon my lack of prowess) and I need so help to understand how I can fix without having the ability to enter alot of code. I have tried within terminal: [ 7397.465665] usb 2-1: Manufacturer: Kingston[ 7397.465670] usb 2-1: SerialNumber: 5B6C10933DEF [ 7397.466183] usb-storage 2-1:1.0: USB Mass Storage device detected [ 7397.466740] scsi9 : usb-storage 2-1:1.0 [ 7398.470159] scsi 9:0:0:0: Direct-Access Kingston DataTraveler 2.0 PMAP PQ: 0 ANSI: 0 CCS [ 7398.470441] sd 9:0:0:0: Attached scsi generic sg2 type 0 [ 7398.471394] sd 9:0:0:0: [sdb] 4030464 512-byte logical blocks: (2.06 GB/1.92 GiB) [ 7398.474041] sd 9:0:0:0: [sdb] Write Protect is off [ 7398.474046] sd 9:0:0:0: [sdb] Mode Sense: 23 00 00 00 [ 7398.474632] sd 9:0:0:0: [sdb] No Caching mode page found [ 7398.474637] sd 9:0:0:0: [sdb] Assuming drive cache: write through [ 7398.478157] sdb: sdb1 [ 7398.481770] sd 9:0:0:0: [sdb] Attached SCSI removable disk [ 7399.305618] FAT-fs (sdb1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive! [ 7399.312661] FAT-fs (sdb1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck. I see you have mounted a USB storage device containing a FAT filesystem. That is all that the above shows. knoppix@Microknoppix:~$ sudo dd if=gnome-3.16.x86_64.iso.crdownload of=/dev/home/knoppix bs=8M conv=fsyncdd: opening `gnome-3.16.x86_64.iso.crdownload': No such file or directory You are trying to dd copy a file but that file does not exist and dd reports that problem to you. You apparently did not notice that the file name ended with the temporary suffix .crdownload and (appears to me) to be a temporary file name for the file. It was likely renamed to be the actual name after the download was completed. None of that appears to be related to bash in any way. Neither does it seem to be related to any web browser problem either. I don't know where I would direct you for further assistance. I think if you could locate a local user group community they would provide useful assistance to you face-to-face. Bob
Please pardon the incorrectly handled anti-spam
Please pardon the incorrectly handled anti-spam message re-send seen today. One of the listhelper volunteers had a problem vivifying an incorrectly classified false-positive-as-spam message. Accidents happen. Your patience and tolerance is appreciated. Bob
Re: Feature : walkthrough lines of stdout
Hrazel wrote: It would be so cool just to [shift] + [arrow up] to go to the line of interest. [arrow right] to copy it to the clipboard and return to the normal console again. then I can decide what to do with it next. This would need some type of wrapper around everything such as GNU 'screen' or as Andreas Schwab hints at emacs interactive shell mode. Have you looked at screen's copy and paste mechanism? There are also programs such as xclip and xsel that interface with X and allow you to cut and paste text from the command line. If you know you want the output of a program into the cut buffer you can pipe it to xclip. Or pull text out of the buffer with xclip. Bob
Re: Should nested case statements within command substitutions work on in bash 3.2.x?
Jon Seymour wrote: I was surprised that this didn't work with the OSX version of bash 3.2: /bin/bash -c 'echo $(case yes in yes) echo yes; ;; no) echo no; ;; esac)' /bin/bash: -c: line 0: syntax error near unexpected token `;;' /bin/bash: -c: line 0: `echo $(case yes in yes) echo yes; ;; no) echo no; ;; esac)' It does work with bash 4.x. It was improved later. But I see that zsh as of 5.0.7 still has the same problem with that construct. Is this a known issue with 3.2 or is it particular to the OSX implementation (which in my case is 3.2.53(1))? It is that way in 3.2 and not specific to your OS X implementation. Of course you know that if you match the parens then it will parse correctly. Matching parens has been the standard way to handle this case for all of the shells. $ bash -c 'echo $(case yes in (yes) echo yes ;; (no) echo no ;; esac)' yes Bob
Re: Feature: disallow |
Ed Avis wrote: Thanks. So 'whoami | out' working is a natural consequence of the fact that 'out' is a command by itself. It is a natural consequence of parsing redirections before parsing commands. The redirection happens before the command execution and an empty command is valid. That redirections happen in an earlier pass across the command line is why they can occur anywhere in the command. By convention we put rediections at the end. But they can also validly occur first (as the previous example showed) or in the middle. IMHO it would have been better for the Unix shell to forbid that, and require ': out' if you really do want to run the null command and redirect its output, but it's too late now. If it did then it would be incompatible with all of the rest that allow it. As I recall one of the obscure fringe Unix lookalike systems did exactly that in their shell. I forget details now. But because of that some of us do always use : instead of an empty redirection simply due to it. Bob
Re: Command line prefixed with space is not saved
Vladimir Kanazir wrote: uname output: Linux canny 3.13.0-34-generic #60-Ubuntu SMP Wed Aug 13 15:45:27 $ When you type space before the command, the command is not saved in the history. You can't see it when arrow up / CTRL+P is pressed. It is like the command is never executed. The bash default is not to set that variable. But it looks like you are using Ubuntu and the Ubuntu package default is to set it in the default .bashrc file as a user level customization. If you don't like that configuration then edit your ~/.bashrc file and change this: # don't put duplicate lines or lines starting with space in the history. # See bash(1) for more options HISTCONTROL=ignoreboth Change that to something different. Perhaps simply ignoredups? HISTCONTROL=ignoredups Or whatever you prefer there. It is your .bashrc file. Customize it as you desire. Bob
Re: Command line prefixed with space is not saved
Vladimir Kanazir wrote: Bob Proulx wrote: The bash default is not to set that variable. But it looks like you are using Ubuntu and the Ubuntu package default is to set it in the default .bashrc file as a user level customization. Yes, they broke many things in 14.04 LTS... Well... It has been packaged that way for years and years and years. I didn't look to see how long but this really isn't anything new. Note if you are not aware... The default files are packaged into the /etc/skel directory and then used whenever a new user is created. The /etc files are conffiles and if you wish to customize the behavior on your system then changing them there would be the place to do it. Then any new users created on your system get your new customizations. Bob
Re: Does [ -f FILE ] have a bug on testing a symlink ?
Cheng Rk wrote: According this documentation `help test`, I am expecting it should return false on anything other than a regular file, -f FILETrue if file exists and is a regular file. but why it returned true on a symlink to a regular file? $ [ -f tmp/sym-link ] echo true true Symlinks originated in BSD. GNU is implementing previously existing behavior. It is a BSD feature that migrated to System V and then to GNU and elsewhere. The behavior is now standardized by POSIX so that it will always be the same everywhere and it is possible to write portable scripts that behavior the same everywhere. The idea is that symlinks should be completely transparent and invisible to most applications. The idea is that when normal things are run they don't realize they are using a symlink. Behavior would be exactly the same as if it were a regular file system link. That is what is happening in your test case. The symlink references an existing file, is a regular file and exists, therefore the status is true. If you have an application that needs to know if something is a symlink then then you will need to test for that case explicitly. Since this is a question and not a bug in bash it would be better discussed in the help-b...@gnu.org mailing list instead. Please send future use discussion there. Thanks. Bob
Re: bash complete.c
Vincent Leduc wrote: The null redirect was used for the /etc/issue to not be shown here. I mean although we dont have any tty attached, i assume the builtin should not do this. I only tested it with an older release, as you said, it seems to be corrected. If I don't include -t then I get the following back from bash. bash: cannot set terminal process group (-1): Inappropriate ioctl for device bash: no job control in this shell I assumed that was what you were trying to redirect. Adding the -t sets up a tty and that particular part of the problem is avoided. As far as /etc/issue goes another useful thing is to touch a file named $HOME/.hushlogin. If .hushlogin exists then the /etc/issue and /etc/motd are not emitted at login time. That is much nicer for me. I am well aware of the uname -a and license status of the systems I am using. I don't need to see that same message again and again. This is documented in the login(1) man page. man login $HOME/.hushlogin Suppress printing of system messages. And so a normal thing for me when getting an account set up is to always create that file in order to make hopping around systems less verbose. touch ~/.hushlogin Bob
Re: Documentation Update
Greg Wooledge wrote: David J. Haines wrote: When started interactively, bash sets the extglob shopt; however, that fact seems to have been overlooked in the manpage. This is a compile-time setting. Your vendor probably turned this on. imadev:~$ bash-4.3 imadev:~$ shopt | grep extglob extglob off imadev:~$ exit Also some vendors place a lot of customization in /etc/profile, /etc/profile.d/*, /etc/bash_otherplaces sourced in the above. It might be turned on by one of those. I see a lot of hits in the bash_completion add-on files. Should we be looking at: $ grep -r shopt -s extglob /etc/bash* /etc/bash_completion.d/nmh:shopt -s extglob /etc/bash_completion.d/libreoffice.sh:test $g -eq 0 shopt -s extglob /etc/bash_completion.d/mercurial:shopt -s extglob /etc/bash_completion.d/subversion:# pattern matching enabled (use 'shopt -s extglob progcomp' to enable /etc/bash_completion.d/subversion:shopt -s extglob Bob
Re: Documentation Update
Dennis Williamson wrote: Bob Proulx wrote: I see a lot of hits in the bash_completion add-on files. Should we be looking at: $ grep -r shopt -s extglob /etc/bash* /etc/bash_completion.d/nmh:shopt -s extglob /etc/bash_completion.d/libreoffice.sh:test $g -eq 0 shopt -s extglob /etc/bash_completion.d/mercurial:shopt -s extglob /etc/bash_completion.d/subversion:# pattern matching enabled (use 'shopt -s extglob progcomp' to enable /etc/bash_completion.d/subversion:shopt -s extglob Perhaps you should check for shopt -u extglob reversing those. Good point. $ grep -r shopt -. extglob /etc/bash* /etc/bash_completion.d/nmh:orig_opts=$(shopt -p extglob) /etc/bash_completion.d/nmh:shopt -s extglob /etc/bash_completion.d/libreoffice.sh:shopt -q extglob g=1 /etc/bash_completion.d/libreoffice.sh:test $g -eq 0 shopt -s extglob /etc/bash_completion.d/libreoffice.sh: test $g -eq 0 shopt -u extglob /etc/bash_completion.d/libreoffice.sh:test $g -eq 0 shopt -u extglob /etc/bash_completion.d/mercurial:shopt -s extglob /etc/bash_completion.d/subversion:shopt -s extglob Bob
Re: bash 4.3: Clear Scrollback and Reset (Ctrl-Shift-X) in konsole stopped to work as expected when upgrading from upstream bash 4.2 to upstream bash 4.3
Askar Safin wrote: Okey, so, Chet, what will you say about resizing bug? Is this a bug? At this moment I doesn't ask where (readline or konsole) this bug resides. I'm just asking: is this a bug? Or long line doesn't move on resize is intended behavior? In my opinion the bug is in your expectations. You are asking the terminal emulator to clear the screen history. It does. This includes the prompt too. It does exactly what you asked. There isn't any surprise. You see the screen and know its state before the action and know whether it is displaying a prompt or whether it is currently running a program. You clear the screen. The screen is cleared. There is no prompt but you should know that in order to get a prompt all you need to do is hit Enter or Control-l and bash will print another prompt which will then be displayed. Or you could type in a command right then regardless. And of course it might be running a program and not at the bash prompt so whatever happens must work reasonably with any program. However you are expecting that after instructing the terminal to clear screen history that bash should emit a new prompt. I think that is an unreasonable expectation. There hasn't ever been a communication interface for doing such a thing. One could be created. But if so then that would be a *new* interface. Bob signature.asc Description: Digital signature
Re: [PATCH] bracketed paste support
Daniel Colascione wrote: Bob Proulx wrote: I use paste into the shell with an embedded newline in order to immediately execute a command *a lot*. If that were removed I would be very unhappy. I strongly doubt that your use case is typical. It doesn't matter if it is typical or not. A feature need not have a 51% majority in order to be retained. Everyone does things differently. If we were to examine your operating modes carefully I am sure that we would find some practice that you use often that few other people are using and would be annoyed greatly if removed. Furthermore it is impossible to know how many people use it. This is a long standing feature that if removed would disenfranchise the users of it. That is all that we can say about it. I very often triple-click to select by line and then paste it into the shell for immediate execution. I less often triple-click to select multiple lines of a command set and paste them into the shell for immediate execution. I am sure there are many others not in this discussion who do the same. I've asked several of my colleagues; all have complained about accidentally pasting a large amount of text into the shell at one time or another. Nobody has complained about losing automatic execution of code after paste. Unfortunately that is not a useful survey. It is self-fullfilling! If I ask ten of my peers of whom I have each taught a feature if they use that feature then the answer will of course be yes. If on the other hand I have vilified a feature with my peers then it is likely that the reverse would be true. A survey asking colleagues around you is simply not a worthwhile survey. Sorry. Knowledge and use tends to group and cluster. Birds of a feather flock together. This is one of the reasons why things like conferences, newsgroups, and mailing lists are so useful. It allows people to meet others and learn about practices outside of their normal routine. People not only learn other things for themselves but they also learn that there is diversity in others too. In this case you learn that a feature that you despise is one that is liked by another. Speaking from personal experience, I've been using terminal emulators of various sorts for almost 20 years, and in that time, I've accidentally pasted documents into my terminal orders of magnitude more often than I've deliberately pasted multi-command sequences. I have used terminals for a very long time as well. I am using one now. I won't say that I haven't sometimes pasted in paragraphs of text into a terminal by mistake. I have. I also use kitchen knives to chop vegetables. I have sometimes cut myself doing so. That doesn't prevent me from avoiding cutting up vegetables with a kitchen knife. I have also accidentally removed a file with rm that I didn't intend to remove. In none of those cases do I want to remove rm or kitchen knives from the world to prevent anyone from doing either of those things again. As far as I'm concerned, automatic execution of code on paste isn't a feature: it's a bug and security hole. Users should have to opt into security holes. It is not a security hole. Simply declaring it to be so does not make it so. I get that it is a feature you hate. But labeling it with incorrect negative labels is simply a normal day of US political advertisements. It has no business on a technical list. We've only lived with the existing behavior so long because it's only recently become possible to distinguish pastes from other input. And I think it is great that you can now have it operate differently. Isn't it a wonderful world that we live in? Bob
Re: [PATCH] bracketed paste support
Chet Ramey wrote: * paste into the text editor invoked by the `edit-and-execute-command' key binding (C-xC-e in emacs mode), edit the command if desired, and have the shell automatically execute the contents of the editing buffer when the editor exits Cool! This was an idea that I hadn't thought about before. (I often would paste into my editor and clean it up. But doing it as part of the edit-and-execute-command is very nice!) Thanks for this hint! Bob
Re: [PATCH] bracketed paste support
Daniel Colascione wrote: Well, I don't know whether Chet left the feature enabled by default. I hope he did, though, since preventing execution of pasted commands is one of the feature's key benefits. In bash, you should be able to execute a pasted command sequence by typing RET after the paste, but a paste by itself should never begin execution. I use paste into the shell with an embedded newline in order to immediately execute a command *a lot*. If that were removed I would be very unhappy. However I read from Chet's replies that this is not the case. So I am not worried. Just voicing a counter opinion that this shouldn't be removed. If you want it then you should enable it. This is like many other options available in the shell. They are optional. For better or worse, people copy and paste commands from websites all the time. Even if a bit of shell looks innocuous, a malicious bit of JavaScript could change the pasted text at the last second without the user being aware of the switch. Then the browser shouldn't copy it. The place to fix the problem is in the browser. A problem with a browser should not break cutting and pasting in general. (Tynt uses this technique to slightly less malicious ends.) If pasting in a terminal immediately begins execution, there's no opportunity to review pasted code. With bracketed paste support, the shell requires additional user interaction after a paste to begin execution, making this attack much less effective. I am very happy this is a feature you like. However I would hate that feature. It should definitely not default to on or it will make people like me very unhappy. Bob
Re: Random loss of bash history
Darshit Shah wrote: I'm running Bash 4.3.30(1)-release on Arch Linux. Recently after I rebooted by system I realized that all my Bash History had been erased. The ~/.bash_history file existed, but it was completely empty. The situation occurred just now once again. I boot up my system and realize that I have no bash history at all. The issue does not occur at every boot, it's random and I am unable to correlate the two instances in which this has happened to me. I haven't any idea as to how I should debug this problem to provide more information. Could you please start out by helping me there? What type of file system is your ~/.bash_history located upon? This sounds like a deja vu from the ext4 file system. Do you observe any other zero sized files after a reboot? Bob
Re: Detecting invocation as /bin/sh ?
Alexandre Ferrieux wrote: Greg Wooledge wrote: Unfortunately, most people don't actually express an intent when they use #!/bin/sh. They just copy some code from a random script they found somewhere, without understanding what it means. That song reminds me of PHP. I was under the optimistic assumption that bash, given its noble ancestry, was not exactly targeting the same audience. The shell is part of the landscape. It has been around since the beginning. Although some try to avoid it others embrace it. It is not a matter of targeting an audience. It is simply the way things are. The number of scripts that use #!/bin/sh but then go on to use Bash syntax is higher than you might believe. Agreed. But those are not portable. And they are one of the reasons behind dash and the other targeted shells. OK. Sh*t happens. Bozos write scripts. So what ? A more interesting question IMO is: How many packaged init scripts / mission-critical daemons are written by people that cannot be bothered to write #! /bin/bash when they require bash ? Lots and lots and lots. When Debian converted from /bin/sh being bash over to it being dash there were many bugs found. The process of getting those instances cleaned up took a long time. Years. It's not a stretch of the imagination to suppose that someone has used exported functions in a #!/bin/sh script on a Linux box where /bin/sh is a symlink to bash. Except that script would instantly break if *dash* was installed, instead of bash, as target of the sh symlink. So that beast would already be extremely fragile. Agreed to both. Definitely fragile. But people do it just the same. Personally I would vote to have bash not support exported functions in /bin/sh mode. But as Chet has indicated the task of maintaining a stripped down shell is the target of dash and posh and others and not the target for bash. It would be duplicated effort if bash also targeted that too. So while bash might or might not do some of that it isn't really the project for it. Bob
Re: Not so useless use of cat
Ralf Goertz wrote: Actually things are more complicated. I do need the /dev/stdout part. I obiously don't have the problem with `cat' but with some other program that doesn't write to stdout per se and expects a -o parameter for the output file. And this program just accepts one input file. I merely used the first `cat' in my example to make my point. So what I wanted was I suspected that was the case. That is generally the way it goes. But does the program really have no way to write to stdout? I despise programs such as those. They should be fixed so that there is a way to use them as a filter. Because working around those programs by using Linux specific features such as /dev/stdout makes the entire bundle non-portable to systems that don't have /dev/stdout. $ for i in file[12] ; do program -i $i -o /dev/stdout ; done outfile which I assumed to be elegant and would do as I expected except it didn't and I really thought it could have been a bug. That's why I reported it here. I guess you could create as many temporary files as needed and then concatenate them with cat afterward. Or if the program can't be fixed to append or write to stdout directly then use the not so useless cat after all. Probably needs a comment in that case though! :-) There is another possibility. I don't actually like this one as much. It seems more confusing. It uses more processes. But try this: for i in file[12] ; do program -i $i -o (cat) ; done outfile Good luck! Bob
Re: Not so useless use of cat
Greg Wooledge wrote:
It's important to note that the following two cases are *not*
equivalent:
cat $i /dev/stdout
program -i $i -o /dev/stdout
In the first case, the /dev/stdout is part of a redirection. On
platforms that do not have a native /dev/stdout in the file system,
Bash handles this *internally* (and it would actually work the way
you expected).
In the second case, /dev/stdout is just a string as far as Bash is
concerned. It is passed verbatim to program as an argument. There is
no internal Bash magic happening. You get the underlying operating
system implementation, or you get no such file or directory.
That is one of the reasons I don't like the /dev/std{err,in,out}
things. They are not portable. They do different things on different
systems. I avoid them.
Bob
Re: Not so useless use of cat
Chet Ramey wrote:
Bob Proulx wrote:
That is one of the reasons I don't like the /dev/std{err,in,out}
things. They are not portable. They do different things on different
systems. I avoid them.
I've considered emulating it everywhere, regardless of what the OS
provides, but I'd get just as many complaints if I did that.
Agreed. There isn't any way to win.
Tonight I came up with yet another reason I don't like use of the
/dev/stdout and others. And also the use of /proc too. That reason
is use of programs in chroots. I ran into a program installer that
required that both /proc and /dev be mounted in the chroot I was
using. I can't see any reason those are actually needed. But it
required them both in order to function. Things like that get in the
way of using lightweight chroots which would otherwise work fine as a
light weight container.
Bob
Re: Not so useless use of cat
Dennis Williamson wrote:
Bob Proulx wrote:
And the subshell isn't needed either. Use a list.
{ for i in file[12] ; do cat $i ; done ;} both
There's no need for the curly braces and the last semicolon.
Of course you are totally right. I was distracted by the subshell as
a concept. For re-stitching file with redirections a subshell isn't
needed and a list is convenient.
Note that the loop in this case can be replaced by
cat file[12] both
I failed to spot that in my earlier reply.
Me too. (But usually these are from more complex examples that can't
be simplified as much. It is just that all of the details aren't shown.)
Bob
Re: Not so useless use of cat
Ralf Goertz me@myprovider.invalid wrote:
Since you have used an invalid address I assume you are reading the
mailing list via a web archive or other means and did not CC you.
Why do I need cat (the second on) here?
You don't.
$ echo first file1
$ echo second file2
$ (for i in file[12] ; do cat $i /dev/stdout ; done) | cat both
Why do you want /dev/stdout there?
$ cat both
first
second
Okay.
If I omit the | cat after the loop I get
(for i in file[12] ; do cat $i /dev/stdout ; done) both
$ cat both
second
Because the /dev/stdout truncates the output. It writes the first.
Then the second one truncates the file and then writes the second.
Remove that from the line and it will print both lines.
(for i in file[12] ; do cat $i ; done) both
Even when using both instead of both only second makes it
into both. Why?
All that the , , 21, types of redirections do is to attach the
associated files with different files. Once associated the further
actions such as truncation happen to those files. Therefore the
truncation associated with /dev/null truncates the file that on the
far right was both.
Try this:
(for i in file[12] ; do cat $i /dev/stdout ; done) both
But really I think you are better off forgetting about /dev/stdout.
That is almost like the useless use of cat.
And the subshell isn't needed either. Use a list.
{ for i in file[12] ; do cat $i ; done ;} both
I would have thought that using a subshell should be
enough to protect the both from being overwritten.
The subshell isn't really part of the issue. The issue is that the
file descriptor is attached and truncated and that is orthogonal to
process boundaries.
Hope that helps!
In the future please consider asking questions such as these on
help-b...@gnu.org for general discussion of bash shell scripting. If
you expose a bug then report the bug to the bug reporting address.
The help-bash list is for help with bash scripting and this would have
been perfect there.
Bob
Re: SEGFAULT if bash script make source for itself
bogun.dmit...@gmail.com wrote: Eric Blake wrote: bogun.dmit...@gmail.com wrote: IMHO any user action should not lead to SIGSEGV! I am not objecting against recursive sourse itself. But when I got SIGSEGV from bash, I have no idea why this is happened. I have made recursive sourse by mistake and spend a lot of time looking up what exactly lead to SIGSEGV. But you wrote the program that caused the SIGSEGV. At that point you are no longer just a user but are now a programmer. Technically speaking the SIGSEGV problem would be a bug in your bash script program. You as the programmer of that script have the responsibility for it. SIGSEGV is what happens on stack overflow, unless you integrate a stack overflow detector like GNU libsigsegv with your sources to catch the segv and replace it with a nice error message. I know when and why program can get SIGSEGV. Then you already know that it is a recursion problem that has run out of stack space. Any program that allows recursion might be programmed in error. If this is not suitable then using a programming language that does not allow recursion might be the choice for you. For example the old FORTRAN did not allow recursion and yet it enjoyed a lot of popularity at one time. So why I should got SIGSEGV instead of nice, detailed error message in recursion? We can detect it? Because in general it is a hard problem to solve. And it isn't always about making it bigger. It may simply be a very small environment without enough stack to complete. The program may completely fine if there were enough stack. It is hard to tell if the program is in an infinite recursion or if it simply didn't have enough stack space to complete and would complete if there were more. All that can be said is that it didn't have enough stack. The GNU Coding Standards state that GNU software cannot have arbitrary limits by default. Any limit we pick, other than unlimited (your ... IBM produce new CPU - it solve infinite loop in 8 seconds. How bigger amount of memory save from infinite recursion? It lead to bigger delay before SIGSEGV and nothing else. Haha. Of course it is a hardware problem. If we only had a machine with an infinitely large stack then we would never run out of stack space and could never SIGSEGV due to stack overflow. Of course then the program would simply run forever since it would continue to do exactly what it had been programmed to do. Which is one of the things that makes this so hard. In order for an automated detection the program must say that the program should not do what the programmer told it to do. That is where it runs into problems. Similar to why auto-correction spell checkers force so many spelling errors on humans. $ ulimit -a data seg size (kbytes, -d) unlimited ... so... in real life we have a limits. Some of them turned off, but they exists and can be ajusted. Those are explicit process limits in addition to the actual limits. It is always easier to make them smaller. But you can't make the actual limits larger. For example feel free to try to make use of that unlimited data set size. Let's assume you have a X sized memory machine. Try to use a thousand times X that amount of memory regardless of it being set to unlimited and see how well things work. And if I have an option, which I can change to some suitable value for me and this can save me/show me good error message in case of infinite recursion - I will use it. Other can leave it in ifinite position. We can have 2 options - one set recursion level limit, other set action when this limit is reached - deny deeper recursion / print warning. There is the sharp kitchen knife thought-problem. Have you ever cut yourself on a kitchen knife? Of course most of us have at one time or another. Can you envision a way to make kitchen knives safer? A way to make it impossible for you to cut yourself on one? Think about how you would make a kitchen knife safer. Give it a serious think. Then ask yourself this question. Would you use such a knife that you designed yourself? The answer is invariable no. There has always been some inherent danger when using a kitchen knife. We accept this because the alternatives are all worse. Bob
Re: Interactive Expect from bash Script
Esben Stien wrote:
If I run this, it works:
#!/usr/bin/expect
Sounds good. Is there anything particular bad about running it as an
expect script? (Or in native python using pexepect? Or in native perl
using Perl's expect library? Or Ruby's?)
Andreas Schwab writes:
Since you are passing the expect script on stdin the interact command
will immediately read EOF, and expect will run into its implicit
timeout.
Hmm, I see, so how am I supposed to run it?;)
Personally I would use #!/use/bin/env expect and be done with it.
But if you really want to call it then the first easy way is to create
a script that does the expect portion just by itself. This is very
common such as often seen with 'sed' scripts called like:
#!/bin/sh
sed -f sedscript infile outfile
But if you want the expect script to be contained within the shell
script then you need to write it out from the shell script first.
Then clean up the script. Which means things get more complicated due
to the need for trap handling and temporary file cleanup. Which is
why the use of expect directly seems best to me.
But if you really want to know then the following should do it. Note
that I didn't test it. There may be some detail still broken.
Note that with #!/bin/bash using trap on EXIT is sufficient. To use
this with a #!/bin/sh script one must also trap properly on every
other signal that might happen, usually HUP, INT, QUIT, TERM, and
perhaps PIPE though PIPE is definitely subtle and special. It is
rather of a mess. Bash is definitely better in this regard.
Bob
#!/bin/bash
unset tmpfile
cleanup() {
test -n $tmpfile rm -f $tmpfile
}
trap cleanup EXIT
tmpfile=$(mktemp) || exit 1
cat $tmpfile 'EOF'
set passwd $env(_passwd)
set usr $env(_usr)
set host $env(_host)
spawn /usr/bin/ssh $usr@$host
expect {
-re .*Are.*.*yes.*no.* {
send yes\n
exp_continue
#look for the password prompt
}
*?assword:* {
send $passwd
send \n
interact
#The expect command will now return
}
}
EOF
env _passwd=foo _usr=foo _host=foo expect $tmpfile
Re: Browsing the history of commands. Inconsistency between Bash and Emacs
Dani Moncayo wrote: Emacs uses M-p/M-n to browse the minibuffer history (and C-p/C-n to move to the previous/next line in a multi-line buffer), whereas Bash uses C-n/C-p for browsing the command history (and doesn't use M-p/M-n for anything, AFAIK). It would be nice to remove this inconsistency (this is the obvious part), and IMO TRT would be to make Bash behave like Emacs, that is: 1. Use M-p/M-n to browse the command history (instead of the current C-p/C-n). 2. Use C-p/C-n to move to the previous/next line, in the current command being edited. WDYT? This is completely putting the cart before the horse. And going down that road creates a circular line of reasoning which has no end to the loop cycle. Plus it is a radical change in fundamental behavior. Please don't. The entire reason that bash's readline emacs mode uses C-n and C-p is that those are the emacs keys for next-line and previous-line. If emacs were using M-p and M-n for previous and next then bash would have done so too. Bash didn't make this up. It is using the keys from emacs for emacs mode and those keys are C-n and C-p. Editing the history is just like editing a file. C-p takes you to the previous line. C-n takes you to the next line. Now enter emacs. I mean literally. (I have been using emacs for a very long time by the way. I am using it now.) Emacs has the feature including of being able to edit the minibuffer and also being able to run an inferior shell process in a buffer. Both are very similar cases and should be discussed together. In those buffers if you want to go to the previous line or the next line then what keys do you use? You use C-p and C-n to go to the previous line or the next line. But that is not editing the history. That is editing the buffer. There the previous and next lines are parts of the visible screen. It isn't a history of the screen. For using bash in an inferior shell if you want to recall shell history you can't use C-p and C-n because those are shadowed by the emacs layer of keybindings that move previous and next lines. Therefore M-p and M-n were the natural second choice to navigate by adding another mental layer of navigation. (Although emacs itself is keeping the input history.) Same thing for the minibuffer. So now people say, I am now used to using M-p and M-n in emacs to avoid the C-p/C-n. Let's set that up for bash. (Of course you can easily do this if you desire. Just do it. Several people suggested it.) Well, let's say for discussion that you get used to that setting. The entire argument so far is that people are used to it. Now people become used to navigating the previous line with M-p and the next line with M-n in bash. Now they go to emacs. What do they do there? They find that in emacs M-p and M-n *do not navigate* the previous line and next line. They find that bash and emacs are once again inconsistent! In that future time let me file a bug with emacs asking for them to change their previous-line and next-line key bindings to be compatible with the new bash bindings of M-p and M-n for previous and next line. Why? Because they are used to it. That request is just as valid as this request to do so with bash. So why not? Why wouldn't emacs change M-p and M-n to be previous line and next line? Let's say they do for continued discussion. Now in the future, future time where emacs has now adopted M-p and M-n for previous and next line. Now they need a way to escape the emacs layer again but with the new set of keys. In emacs I really don't want to need to use C-c as a prefix key for the minibuffer and the inferior shell buffer like in term mode. But let's say they do. Would there then be a request for bash to set up C-c M-n and C-c M-p for next and previous line? I think there would be and it would just be an endless cycle. As Chet said the mental model of what is happening at those two points of action in each program is different. Bob P.S. This is going to be a rant and run because I am offline for several days after this.
Re: Please accept M-p as well as C-p
Andreas Schwab wrote: Chet Ramey writes: Andreas Schwab wrote: AIUI there is no readline command the moves vertically within the commandline (in readline a line always includes any embedded newlines). That's true. How should such a command work? Move point forward and backward by a multiple of the screenwidth? It should work like previous-line/next-line in Emacs: move over the previous/next newline and then forward to the same column as before. In emacs 24 this is the new feature controlled by the line-move-visual variable. Call me a Luddite if you want but in emacs 24 I turn that feature off. I am very much used to using the traditional behavior and like it. Thinking about it I think that the recent addition of line-move-visual to emacs and not to libreadline is probably the reason for the request for this feature. Bob
help-bash pointer added to web page
One of the recent postings mentioned that http://www.gnu.org/software/bash/ directed questions to bug-bash. We never updated the web page to mention the new help-bash mailing list! I updated the web page to say to send questions for help with bash to the help-bash list and bugs or development issues to bug-bash. Bob
Re: unsigned int for loop in bash
Mathieu Malaterre wrote:
I am getting a weird behavior in bash. Would it be possible for the
next release of bash to not get a SIGSEV ?
for i in {0..4294967295}; do
echo $i
done
That is one of those expressions that I see and my eyes go *WIDE* with
shock! The {X..Y} expression is generating arguments.
{0..4294967295} is generating 4 GIG of arguments. And since those
arguements are strings they will take up a lot of memory.
Let's try some experiments to make it more obvious what it is doing.
$ echo {0..10}
0 1 2 3 4 5 6 7 8 9 10
$ echo {0..50}
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
$ echo {0..100}
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
$ echo {0..250}
...much too long for an email...redacted...
That is getting large very fast! Faster than just the number of
numbers because each number is taking up a larger amount of characters
as it gets largert in magnitude. Let's stop printing them and start
counting them.
$ echo {0..100} | wc -c
294
$ echo {0..1000} | wc -c
3895
$ echo {0..1} | wc -c
48896
$ echo {0..10} | wc -c
588897
$ echo {0..100} | wc -c
698
$ time echo {0..1000} | wc -c
7899
real0m20.800s
user0m15.280s
sys 0m1.860s
To print to 1M consumes 6,888,898 bytes and we are only a fraction of
the way to 4 gig! To print out 10M consumes 78,888,899 bytes and
starts to take a long time to complete. On my machine the bash
process gets up to around 1.8G of ram on that one and force other
memory to swap out. There isn't much memory left for other processes
to use. Any larger values and the machine starts to thrash swap and
slow down to disk I/O speeds. And we are only up to 10M and not yet
anywhere close to your 4,000+M request.
The reason you are seeing a segfault is because you are trying to use
more memory than exists in your system. You are trying to use more
memory than exists in most super computers. You are trying to use an
astronomical amount of memory!
If you are adventuresome you might try to actually produce that many
arguments. I expect that it will crash due to being out of memory.
If you are lucky it will only produce a bash malloc error. If you are
running Linux with the OOM enabled then it might kill something in
appropriate and damage your system.
echo {0..4294967295} /dev/null
This has been discussed previously. I suggest reading through this
thread. It includes good suggestions for alternative ways of looping
without consuming endless memory.
http://lists.gnu.org/archive/html/bug-bash/2011-11/msg00181.html
Bob
Re: let's establish BASH_MINIMUM_TIME_BETWEEN_INTERACTIVE_COMMAND
Eduardo A. Bustamante López wrote: Chris Down wrote: Well, you can do this with PROMPT_COMMAND='sleep 1'. I don't see the need for another internal variable to do this. How would that work to avoid filling the prompt with the paste buffer when I fat finger and paste a whole email to the prompt? It will just slow down the process, but not cancel it. It would give you plenty of time to hit Control-C to interrupt it. Try it and you will see. Bob
Re: Builtins should canonicalize path arguments
Chet Ramey wrote:
Ondrej Oprala wrote:
Hi, I investigated this bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=987975
and found out that some of bash's builtins (source at the very
least) do not canonicalize pathnames given as arguments (builtin
open is instead fed with the path - failing in the BZ case).
The bug report referenced was a missing a simple reproducer. It is
also missing the information entirely on the location of the
rpmbuild directory.
The condensed bug report:
$ set +o physical | grep physical
physicaloff
[aledvink@dhcp-25-145 rpmbuild]$ ls -ld ~/repo
lrwxrwxrwx. 1 aledvink aledvink 8 Jan 26 12:08 /home/aledvink/repo - /al/repo
[aledvink@dhcp-25-145 rpmbuild]$ cd ~/repo/ipmitool.f
[aledvink@dhcp-25-145 ipmitool.f]$ . ../../rpmbuild/runme
bash: ../../rpmbuild/runme: No such file or directory
[aledvink@dhcp-25-145 ipmitool.f]$ cd ../../rpmbuild
[aledvink@dhcp-25-145 rpmbuild]$
Where is rpmbuild located? Because of the confusion I can guess that
it must only be located where there will be maximum confusion but a
bug report really should say these things explicitly.
Hmm... Perhaps rpmbuild is in /home/aledvink/rpmbuild maybe?
/
/al
/al/repo
/al/repo/ipmitool.f
/home
/home/aledvink
/home/aledvink/repo - /al/repo
/home/aledvink/rpmbuild
/home/aledvink/rpmbuild/runme
I reproduced the confusion with this sequence.
rwp@havoc:~$ set +o physical
rwp@havoc:~$ rm -rf /tmp/testdir
rwp@havoc:~$ mkdir /tmp/testdir
rwp@havoc:~$ mkdir /tmp/testdir/al
rwp@havoc:~$ mkdir /tmp/testdir/al/repo
rwp@havoc:~$ mkdir /tmp/testdir/home
rwp@havoc:~$ mkdir /tmp/testdir/home/aledvink
rwp@havoc:~$ mkdir /tmp/testdir/home/aledvink/rpmbuild
rwp@havoc:~$ ln -s /tmp/testdir/al/repo /tmp/testdir/home/aledvink/
rwp@havoc:~$ mkdir /tmp/testdir/al/repo/ipmitool.f
rwp@havoc:~$ echo echo inside runme file
/tmp/testdir/home/aledvink/rpmbuild/runme
rwp@havoc:~$ cd /tmp/testdir/home/aledvink/repo/ipmitool.f
rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ /bin/pwd
/tmp/testdir/al/repo/ipmitool.f
rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ echo $PWD
/tmp/testdir/home/aledvink/repo/ipmitool.f
That sets up the test case.
rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ . ../../rpmbuild/runme
bash: ../../rpmbuild/runme: No such file or directory
Can't source the script using a logical path because it isn't
physically there. The canonical path for the file is elsewhere. And
because of this no utility can access it by that path.
rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ ls -ld
../../rpmbuild/runme
ls: cannot access ../../rpmbuild/runme: No such file or directory
rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ stat
../../rpmbuild/runme
stat: cannot stat ‘../../rpmbuild/runme’: No such file or directory
rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ cat ../../rpmbuild/runme
cat: ../../rpmbuild/runme: No such file or directory
That is not a correct path to access the runme file.
Only the shell maintaining $PWD and a fake model of how the user
arrived there is able to do this.
rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ cd ../../rpmbuild
rwp@havoc:/tmp/testdir/home/aledvink/rpmbuild$
Why isn't the file there for anything other than the shell? Because
. and .. are actual directory entries. Let's track them down by
inode number.
rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ ls -ldogi . .. ../..
../../..
23842833 drwxrwxr-x 2 40 Jan 9 13:46 .
23840406 drwxrwxr-x 3 60 Jan 9 13:46 ..
23840405 drwxrwxr-x 3 60 Jan 9 13:46 ../..
23842828 drwxrwxr-x 4 80 Jan 9 13:46 ../../..
rwp@havoc:/tmp/testdir/home/aledvink/repo/ipmitool.f$ find /tmp/testdir -type
d -exec ls -ldogi {} +
23842828 drwxrwxr-x 4 80 Jan 9 13:46 /tmp/testdir
23840405 drwxrwxr-x 3 60 Jan 9 13:46 /tmp/testdir/al
23840406 drwxrwxr-x 3 60 Jan 9 13:46 /tmp/testdir/al/repo
23842833 drwxrwxr-x 2 40 Jan 9 13:46 /tmp/testdir/al/repo/ipmitool.f
23840407 drwxrwxr-x 3 60 Jan 9 13:46 /tmp/testdir/home
23842831 drwxrwxr-x 3 80 Jan 9 13:46 /tmp/testdir/home/aledvink
23840410 drwxrwxr-x 2 60 Jan 9 13:46 /tmp/testdir/home/aledvink/rpmbuild
When you try to access ../ the .. directory is
/tmp/testdir/al/repo. When accessing ../../ that directory is
/tmp/testdir/al.
The builtin cd seems to handle relative paths correctly.
Actually cd in logical mode works in conjuction with a logical path
as displayed in $PWD to handle a fake model of the file system in the
presence of symbolic links. Whether this fake model is correct or not
is a matter of perspective. From my perspective it is most definitely
NOT correct.
I acknowledge that it is a convenient lie that many like. Which is
exactly why there is the physical bash shell option to set how to
handle it. Some people like it one way and others like it the other
way. I do not like it and so I always set the
Re: print float number
Hi lina, lina wrote: How can I print the 0.05, 0.10 ... 1.0 out, with the step of 0.05 Thanks ahead for your suggestions, First please do not hijack threads. You replied to Chris Down's message responding to DanielB about For loop prints two $values instead of one and then changed the subject but your message isn't about anything to do with that thread. If you want to start a new thread like you did here please compose a new message. Please don't reply to an existing thread to start a new conversation. Second is that this is simply a question and not a bug report. Questions should go to the help-b...@gnu.org mailing list. How can I print the 0.05, 0.10 ... 1.0 out, with the step of 0.05 As to your question there are many ways to solve the problem. I would be inclined to use the GNU seq command. $ seq 0.05 0.05 1.0 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 However seq is somewhat of a troubled command. It does math with floating point and sometimes suffers from the same problems as anything that does compter based floating point math with various approximation and rounding errors. You must be careful concerning it or odd errors will occur. A related question about for loops in bash is discussed in this FAQ entry. It isn't exactly your case but you might be interested in reading about the examples it provides. http://mywiki.wooledge.org/BashFAQ/018 I doubt just printing out these numbers were really your question. If you wrote a message to help-b...@gnu.org and described your task in more detail there are many people who would be happy to help you use bash effectively. Bob
Re: Replacing addresses
Allodoxaphobia wrote: Chris Down wrote: Peter Cordes wrote: Oh, also, the online bug-bash archive has a bad habbit of replacing code with address@hidden. There was a whole thread about setting PS1=whatever that is now a complete mystery to non-subscribers! Yes, let's please stop this useless obfuscation. If you *must* obfuscate, then it should at least be retrievable somehow that doesn't require clicking through tons of stuff. Besides, the list is archived in other places that don't do this, so it doesn't really help anyone. rot13 :-) Doesn't help. The '@' isn't rotated and so the result still looks like an email address to the redaction code. $ echo u...@example.com | rot13 h...@rknzcyr.pbz Bob
Re: Replacing addresses
Chet Ramey wrote: Chris Down wrote: Peter Cordes wrote: Oh, also, the online bug-bash archive has a bad habbit of replacing code with address@hidden. There was a whole thread about setting PS1=whatever that is now a complete mystery to non-subscribers! Yes, let's please stop this useless obfuscation. If you *must* obfuscate, then it should at least be retrievable somehow that doesn't require clicking through tons of stuff. Besides, the list is archived in other places that don't do this, so it doesn't really help anyone. I think this is a global setting Yes. It is a global setting. And directly due to complaints from people when it wasn't used. People complain if it isn't there. People complain if it is there. and probably does more good than the annoyance it causes. I don't think it can do much good. The idea is to hide your email address so that spammers don't know it and can't find it. And yet they use it for sending email. Hiding email addresses simply does not work. The genie of information is easier to escape from the lamp than to force back into it. Bob P.S. I looked over at the gmane.org archive to see how they handled things and there even my quoted '@' which obviously is not an email address got obfuscated to at . So it appears there that no use of an at sign anywhere is allowed.
Re: RFC: turn off word splitting for vars but keep for read
Greg Wooledge wrote: Bob Proulx wrote: Aleksey Midenkov wrote: rmcmd=$rmprog -f Because it will fail for files with spaces. I am sure we disagree but I don't consider making rm program names have spaces to be reasonable. :-/ What would it be? remove command? I definitely would not consider an executable named with spaces that way to be reasonable. How about /mnt/c/Program Files/something/DEL.EXE? The directory might contain spaces even if the command name itself does not. That would be using a full hard coded path. I think using full hard coded paths is bad for yet different reasons. Here in this context it might be argued that hard coding the path is part of a specific host's customization. But IMNHO it is always better to set PATH in that case. Because it is never just one command. It will almost always be several commands in an open ended list of possibilities. In which case you will need PATH set appropriately. Having a PATH with spaces in it won't cause a problem in this context. But in any case such host customizations are better to occur in a sourced configuration file and not in a production script itself. As a matter of style I very rarly would set options with the program name in a program name variable. (RSYNC_RSH comes to mind.) I would more typically set a companion $program_args variable instead. But then expect word splitting to occur in that args variable. So it is still the same issue just moved to a slightly different location. Bob
Re: Feature request for Bash
Ryan Campbell Cunningham wrote: I would like to request that Bash delete the character immediately preceding a '#', provided the '#' ... (The request was inspired by an article in the seventh edition of the UNIX Programmer's Manual. This feature is not required by POSIX, but came from the traditional Bourne shell.) This isn't a feature of the shell but of the tty driver. It still exists. It is still possible to use it today exactly as it was used in Unix V7. $ stty -a | grep --color erase intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = undef; $ stty erase '#' $ stty -a | grep --color erase intr = ^C; quit = ^\; erase = #; kill = ^U; eof = ^D; eol = undef; Feel free to try it. You can also customize the behavior of your bash libreadline by setting the backward-delete-char key binding. Place the following in your $HOME/.inputrc file for example. # backward-delete-char I would like to request that Bash delete the character immediately preceding a '#', provided the '#' * does not begin a new word, * is not included in any quoted string or variable, * is not preceded by a '\', and * is only found in an interactive command line (not in a script* or Bash initialization argument). An exception: If the character immediately preceding is also a '#', Bash should skip backward to the previous non-'#' character and delete as much characters as the number of consecutive '#' characters after them in the same word. Oh my, isn't that a complicated set of rules! That is bound to cause trouble. Not good. Plus that isn't how it worked in V7 days. Bob
Re: RFC: turn off word splitting for vars but keep for read
Aleksey Midenkov wrote:
I would say, that this style generally is wrong:
rmprog=${RMPROG-rm}
rmcmd=$rmprog -f
Because it will fail for files with spaces.
I am sure we disagree but I don't consider making rm program names
have spaces to be reasonable. :-/
What would it be? remove command? I definitely would not consider
an executable named with spaces that way to be reasonable.
Bob
Re: How to create bash-pattern to exclude a matching param in param expansionmatching?
Eric Blake wrote: for $var in *; do case $var in *-IGN-*) ;; *) # whatever you wanted on the remaining files ;; esac done A couple of minor syntax errors in the above. But we know it is simply that perl influence that dragged you that way. :-) This is a typical pattern for me. But I optimize it a little bit when possible by doing it this way. for var in *; do case $var in *-IGN-*) continue ;; esac whatever you wanted on the remaining files done Bob
