Re: I've found a vulnerability in bash

[Alex fxmbsw7 Ratchev]

this is due to parsing of args by specific app, -- arg indicates
end-of-switches

On Wed, Nov 17, 2021, 13:41 Marshall Whittaker 
wrote:

> Software: bash
> Version: 5.0.17(1)-release (x86_64-pc-linux-gnu)
>
> --- SNIP ---
> [marshall@jerkon]{04:09 AM}: [~/bashful] $ touch -- '--version'
> [marshall@jerkon]{04:09 AM}: [~/bashful] $ touch a && mkdir b
> [marshall@jerkon]{04:09 AM}: [~/bashful] $ ls -l
> total 4
> -rw-rw-r-- 1 marshall marshall0 Nov 17 04:09 a
> drwxrwxr-x 2 marshall marshall 4096 Nov 17 04:09 b
> -rw-rw-r-- 1 marshall marshall0 Nov 17 04:09 --version
> [marshall@jerkon]{04:09 AM}: [~/bashful] $ mv * b
> mv (GNU coreutils) 8.30
> Copyright (C) 2018 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> https://gnu.org/licenses/gpl.html>;.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Written by Mike Parker, David MacKenzie, and Jim Meyering.
> [marshall@jerkon]{04:09 AM}: [~/bashful] $ rm *
> rm (GNU coreutils) 8.30
> Copyright (C) 2018 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> https://gnu.org/licenses/gpl.html>;.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Written by Paul Rubin, David MacKenzie, Richard M. Stallman,
> and Jim Meyering.
> [marshall@jerkon]{04:09 AM}: [~/bashful] $
> --- SNIP ---
>
> This shouldn't happen beacuse you can drop a file and then redirect
> other code for example calling a script if you only have access to drop
> a file.  Say a cronjob was running every hour, and it did rm * on some
> folder, by expansion, you could expand it to -riv or whatever you
> wanted and redirect program flow from there.
>
> Thanks,
> Marshall Whittaker / oxagast
>
>
>

Re: bash conditional expressions

[Michael J. Baars]

On Mon, 2021-11-15 at 09:23 -0500, Chet Ramey wrote:
> On 11/12/21 4:36 AM, Mischa Baars wrote:
> 
> > Could you please restore the Fedora 32 behaviour? Someone must have read
> > the bash manual a little too precise, because now the statement only
> > returns true when a 'touch -a test' is given and not when a 'touch -am
> > test' is given.
> > 
> > As I understand it, -N stands for NEW and therefore should return a true
> > when either a 'touch -a test' or a 'touch -am test' is given.
> 
> Why do you think `touch -am', which sets the atime and mtime to the same
> value, should make -N true?

When -N stands for NEW, and touch (-am) gives you a new file, then -N should 
return true on a newly created file and the documentation is incomplete.

> 
> If test -N is a strict test that mtime > atime, it is working correctly
> and you have managed to defeat it by setting atime == mtimne.
> 

Re: bash conditional expressions

[Michael J. Baars]

On Fri, 2021-11-12 at 19:48 +0100, Andreas Schwab wrote:
>   FILE1 -nt FILE2  True if file1 is newer than file2 (according to
>modification date).
> 
> Andreas.
> 

This would indeed also solve the problem at hand :)

I've found a vulnerability in bash

[Marshall Whittaker]

Software: bash
Version: 5.0.17(1)-release (x86_64-pc-linux-gnu)

--- SNIP ---
[marshall@jerkon]{04:09 AM}: [~/bashful] $ touch -- '--version'
[marshall@jerkon]{04:09 AM}: [~/bashful] $ touch a && mkdir b
[marshall@jerkon]{04:09 AM}: [~/bashful] $ ls -l
total 4
-rw-rw-r-- 1 marshall marshall0 Nov 17 04:09 a
drwxrwxr-x 2 marshall marshall 4096 Nov 17 04:09 b
-rw-rw-r-- 1 marshall marshall0 Nov 17 04:09 --version
[marshall@jerkon]{04:09 AM}: [~/bashful] $ mv * b
mv (GNU coreutils) 8.30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
https://gnu.org/licenses/gpl.html>;.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Parker, David MacKenzie, and Jim Meyering.
[marshall@jerkon]{04:09 AM}: [~/bashful] $ rm *
rm (GNU coreutils) 8.30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
https://gnu.org/licenses/gpl.html>;.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Paul Rubin, David MacKenzie, Richard M. Stallman,
and Jim Meyering.
[marshall@jerkon]{04:09 AM}: [~/bashful] $
--- SNIP ---

This shouldn't happen beacuse you can drop a file and then redirect
other code for example calling a script if you only have access to drop
a file.  Say a cronjob was running every hour, and it did rm * on some
folder, by expansion, you could expand it to -riv or whatever you
wanted and redirect program flow from there.

Thanks,
Marshall Whittaker / oxagast

I've found a vulnerability in bash

[Marshall Whittaker]

Software: bash
Version: 5.0.17(1)-release
--- SNIP ---
[marshall@jerkon]{04:09 AM}: [~/bashful] $ touch -- '--version'
[marshall@jerkon]{04:09 AM}: [~/bashful] $ touch a && mkdir b
[marshall@jerkon]{04:09 AM}: [~/bashful] $ ls -l
total 4
-rw-rw-r-- 1 marshall marshall0 Nov 17 04:09 a
drwxrwxr-x 2 marshall marshall 4096 Nov 17 04:09 b
-rw-rw-r-- 1 marshall marshall0 Nov 17 04:09 --version
[marshall@jerkon]{04:09 AM}: [~/bashful] $ mv * b
mv (GNU coreutils) 8.30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
https://gnu.org/licenses/gpl.html>;;.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Parker, David MacKenzie, and Jim Meyering.
[marshall@jerkon]{04:09 AM}: [~/bashful] $ rm *
rm (GNU coreutils) 8.30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
https://gnu.org/licenses/gpl.html>;;.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Paul Rubin, David MacKenzie, Richard M. Stallman,
and Jim Meyering.
[marshall@jerkon]{04:09 AM}: [~/bashful] $
--- SNIP ---

This shouldn't happen beacuse you can drop a file and then redirect
other code for example calling a script if you only have access to drop
a file.  Say a cronjob was running every hour, and it did rm * on some
folder, by expansion, you could expand it to -riv or whatever you
wanted and redirect program flow from there, say blocking the removal
of sensitive temporary files.

Re: I've found a vulnerability in bash

[Greg Wooledge]

On Wed, Nov 17, 2021 at 03:47:15PM +0200, Ilkka Virta wrote:
> I don't see this in BashFAQ, though. Is it because it's not strictly about
> Bash? Greg?

https://mywiki.wooledge.org/BashPitfalls#pf3

Re: bash conditional expressions

[Ilkka Virta]

On Wed, Nov 17, 2021 at 1:33 PM Andreas Schwab 
wrote:

> On Nov 17 2021, Michael J. Baars wrote:
>
> > When -N stands for NEW, and touch (-am) gives you a new file
>
> It doesn't.  The file hasn't been modified after it was last read.
>

touch creates the given file if it doesn't previously exist. Immediately
afterwards,
it could be called "new" in the usual English meaning, and would be new in
the
sense that nothing was done to it after it was created. But:

$ echo $BASH_VERSION
5.1.8(3)-maint
$ rm foo.txt
$ ls -l foo.txt
ls: cannot access 'foo.txt': No such file or directory
$ touch -am foo.txt
$ if test -N foo.txt; then echo is new; else echo is NOT new; fi
is NOT new

Of course "new" is not an exact concept, it could be defined e.g. to
compare the
file timestamps with the current time.

Anyway, the documentation doesn't seem to say 'test -N' tests if the file
is "new".

Re: I've found a vulnerability in bash

[Greg Wooledge]

On Wed, Nov 17, 2021 at 04:16:36AM -0500, Marshall Whittaker wrote:
> --- SNIP ---
> [marshall@jerkon]{04:09 AM}: [~/bashful] $ touch -- '--version'

> [marshall@jerkon]{04:09 AM}: [~/bashful] $ mv * b

This isn't a vulnerability in bash.  It's a bug in your script.

Use this instead: mv -- * b

Re: I've found a vulnerability in bash

[Chet Ramey]

On 11/17/21 4:16 AM, Marshall Whittaker wrote:

> This shouldn't happen beacuse you can drop a file and then redirect
> other code for example calling a script if you only have access to drop
> a file.  Say a cronjob was running every hour, and it did rm * on some
> folder, by expansion, you could expand it to -riv or whatever you
> wanted and redirect program flow from there.

That's just bad scripting.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: bash conditional expressions

[Andreas Schwab]

On Nov 17 2021, Michael J. Baars wrote:

> When -N stands for NEW, and touch (-am) gives you a new file

It doesn't.  The file hasn't been modified after it was last read.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: bash conditional expressions

[Chet Ramey]

On 11/17/21 5:16 AM, Michael J. Baars wrote:

>> Why do you think `touch -am', which sets the atime and mtime to the same
>> value, should make -N true?
> 
> When -N stands for NEW

It doesn't, though. It could just as easily be a mnemonic for "new activity
in the file." You're using it to mean `new' because it's convenient for
your application, or the mental model you have for it, but that's just an
invention.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: I've found a vulnerability in bash

[Ilkka Virta]

On Wed, Nov 17, 2021 at 2:42 PM Marshall Whittaker <
marshallwhitta...@gmail.com> wrote:

> [marshall@jerkon]{04:09 AM}: [~/bashful] $ touch -- '--version'
> [marshall@jerkon]{04:09 AM}: [~/bashful] $ rm *
> rm (GNU coreutils) 8.30
> Copyright (C) 2018 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> https://gnu.org/licenses/gpl.html>;.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Written by Paul Rubin, David MacKenzie, Richard M. Stallman,
> and Jim Meyering.
> [marshall@jerkon]{04:09 AM}: [~/bashful] $
>

A common pitfall, due to how the utility can't tell what strings come from
globs and what
were given literally. See e.g.
https://unix.stackexchange.com/questions/1519/how-do-i-delete-a-file-whose-name-begins-with-hyphen-a-k-a-dash-or-minus
and https://dwheeler.com/essays/filenames-in-shell.html (though the latter
is rather long and depressing.)

I don't see this in BashFAQ, though. Is it because it's not strictly about
Bash? Greg?

Also, GNU rm has a helpful helptext about it:

$ rm --help
Usage: rm [OPTION]... [FILE]...
Remove (unlink) the FILE(s).

[...]

To remove a file whose name starts with a '-', for example '-foo',
use one of these commands:
  rm -- -foo

  rm ./-foo

Note that if you use rm to remove a file, it might be possible to recover
some of its contents, given sufficient expertise and/or time.  For greater
assurance that the contents are truly unrecoverable, consider using shred.

Re: Unclosed quotes on heredoc mode

[Greg Wooledge]

On Wed, Nov 17, 2021 at 06:30:08PM +, João Almeida Santos wrote:
> Thank you for your reply Robert and Lawrence!
> 
> I understand the description alone is hard to follow, so I think the image 
> below should make it clearer. Otherwise let me know!
> 
> 
> Kind regards,
> João Almeida Santos

No attachment made it through.  Which is probably a good thing.

We don't want to see an image of text.  Simply paste the text from
your terminal into the body of your email.  Or if you have a script
which is in a file, you can attach the script.  It should be small,
no larger than necessary to demonstrate the problem you're having -- say,
10 to 20 lines tops.

If you aren't programming on a terminal (because you're running Microsoft
Windows or something), then this may be a culture shock to you, but
it'll be worth figuring these things out in the long run.  Text beats
images every time when it comes to programming questions.

Re: Unclosed quotes on heredoc mode

[Robert Elz]

Date:Wed, 17 Nov 2021 18:45:05 +
From:=?utf-8?Q?Jo=C3=A3o_Almeida_Santos?= 

Message-ID:  


  | No, it's on the email...

It wasn't, but some lists filter attachments (remove them) - this might be one.

  | bash-5.1$ echo $PATH
  | 
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/usr/local/bin/:/usr/local/bin/

That's fine, but having /usr/local/bin there 3 times isn't useful for
anything but wasting time.

  | bash-5.1$ cat << $PATH


  | it should have terminated with the upper delimiter!

What do you consider the "upper delimiter" ?

This is one of the weirder aspects of shell syntax, and perhaps one
of bash's oddities.

  | but, bash does not seem to expand PATH.

Which one are you concerned with?   It really should expand neither
reference to $PATH in that example - the word after << is just a string,
it isn't used anywhere except locating the end of the here doc data.
However, that word can sometimes be expanded (by some shells), whereas
all that should happen to it is quote removal.   Avoiding '$' in that
word is a very good idea.

The same is true of the '$PATH' that ends the here document, that's also
just a string - which is also never expanded (no-one should be expanding
that one, ever).

The only point of this string (where you're using $PATH) is as a marker
to show where the here document data ends, it does nothing else at all.
In some instances (where the data might contain just about anything) a
complicated end word (so it is unlikely to be a line in the here document
data) is needed, in other cases something simple like EOF or DONE or my
favourite: !  is all that is needed.

So
cat < I,
  | > ola""

That first " in that last string closes the opening " in the end delimiter
word.   The second " in that line starts a new quoted string, which is
still part of the same word (no separators have occurred)

  | bash-5.1$ cat << ola"
  | > I,
  | > ola""
  | > ola"

Now we have a closing " on for the opening (2nd) one on the previous
line, and the newline that follows that ends that word, and also becomes
the newline after which the here document data starts.  So now the end
delimiter word is complete, it is, as a C string:

"old\nI,\nola\nola"

The internal quotes have been removed, they do not form part of it,
the ones added here are just for clarity, and don't form part of the word.
The \n sequences represent actual newline characters, not the 2 character
sequence '\' 'n'.

That string is what you would need to use, on a line, to end the
here document.   Being able to do that (include embedded newline characters
in a "line") isn't required by the shell specification, and (it has been
a while since I checked) I do not believe that those work in bash (they
do in some other shells).

That means that (in standard shell required syntax, and probably in
bash) there is nothing you can possibly enter which will terminate that
here document correctly.   So an end of file, or interrupt (^C probably)
is your only choice.


  | In the above example, I don't unterstand how to provide the wanted
  | delimiter!

You cannot in bash, I believe, it is simply impossible.

  | I try to close the quote, in case it is needing it, but even with both
  | quotes, just one or none, it doesn't close...not even with a '\n' char

In shell, newlines do not end quoted strings, they just become a character
in the string.   The word that comes after the << cannot (in standard syntax)
contain a newline, so you cannot use an unpaired quote character in it
(unless it is itself quoted - and if you do that, nothing in the here doc
will be able to be expanded).

Since bash doesn't allow end delimiter words that contain newlines to
work, it should probably generate an error when you try to use one, that
would have made things clear.

If you wanted a quote character in the end delimiter word for some
reason, you could do:

cat << ola\"
hello
old"

and that should say "hello" on standard output, as the end delimiter there
is the 4 char string:
'o' 'l' 'a' '"'
but no expansions in the "hello" part can happen, because of the quoting
used in the end delimiter word.

kre

Re: Unclosed quotes on heredoc mode

[João Almeida Santos]

No, it’s on the email...Anyway, here’s the text!

bash-5.1$ echo $PATH
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/usr/local/bin/:/usr/local/bin/

bash-5.1$ cat << $PATH
> /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/usr/local/bin/:/usr/local/bin/
> it should have terminated with the upper delimiter! but, bash does not seem 
> to expand PATH.
> $PATH
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/usr/local/bin/:/usr/local/bin/
it should have terminated with the upper delimiter! but, bash does not seem to 
expand PATH.
bash-5.1$

ok, but now addressing the actual question. If I use unclosed quotes on 
heredoc, I can't use 
the given delimiter to end the heredoc, I end up having to use an EOF. Example:

bash-5.1$
bash-5.1$ cat « ola"
> I,
> ola""
> ola"
> ola
>
bash: warning: here-document at line 31 delimited by end-of-file (wanted 'ola
')
ola""
ola"
ola

In the above example, I don't unterstand how to provide the wanted delimiter! I 
try to close the quote, in case it is needing it, but even with both quotes, 
just one or none, it doesn't close...not even with a '\n' char



Kind regards,
João Almeida Santos

> On 17 Nov 2021, at 18:34, Alex fxmbsw7 Ratchev  wrote:
> 
> u forgot to attach the picture .. ?
> 
> On Wed, Nov 17, 2021, 19:31 João Almeida Santos  > wrote:
> Thank you for your reply Robert and Lawrence!
> 
> I understand the description alone is hard to follow, so I think the image 
> below should make it clearer. Otherwise let me know!
> 
> 
> Kind regards,
> João Almeida Santos

Re: Unclosed quotes on heredoc mode

[Greg Wooledge]

On Wed, Nov 17, 2021 at 06:45:05PM +, João Almeida Santos wrote:
> bash-5.1$ cat << $PATH

That's not how a here-document is intended to be used.  A here-document
lets you drop a blob of text directly into your script and use that as
standard input for some command, without needing to store the text in a
separate file.

Here's an example of how it's often used:

usage() {
cat << 'EOF'
usage: myprogram [-abcxyz] [-f inputfile]

Description of options:
  -a   all the things
  -b   make it bad
  ...
EOF
}


Using the contents of a variable as the sentinel to mark the end of
the here-document is not a good idea, and using $PATH specifically is
a VERY bad idea.

It kinds looks like you actually wanted a here-string, not a here-document,

IFS=: read -ra paths <<< "$PATH"

Was that what you were trying to do?  To use the content of PATH as your
input?  That's a here-string and uses the <<< operator, not <<.

Re: Unclosed quotes on heredoc mode

[Chet Ramey]

On 11/17/21 10:33 AM, Robert Elz wrote:

> There are several (IMO)
> bugs in the way bash processes here documents, 

Such as?


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Unclosed quotes on heredoc mode

[João Almeida Santos]

Thank you for your reply Robert and Lawrence!

I understand the description alone is hard to follow, so I think the image 
below should make it clearer. Otherwise let me know!


Kind regards,
João Almeida Santos

Re: Unclosed quotes on heredoc mode

[João Almeida Santos]

Ok, got it. It makes sense now!
Thank you very much for your detailed explanation guys; now that I understand 
it, I’ll try to implement that on my mini shell.
It’s a bit too soon, but merry Christmas to you all!

Kind regards,
João Almeida Santos

Re: Unclosed quotes on heredoc mode

[Chet Ramey]

On 11/17/21 3:02 PM, Robert Elz wrote:

>   | bash-5.1$ cat << $PATH
> 
> 
>   | it should have terminated with the upper delimiter!
> 
> What do you consider the "upper delimiter" ?
> 
> This is one of the weirder aspects of shell syntax, and perhaps one
> of bash's oddities.

It's not. Every modern shell uses `$PATH' as the here-document delimiter
and checks for the delimiter before any part of the process that expands
the lines in the here-document body.

> First, the EOF should not work, that's a bash bug (IMO) - that should
> generate an error, not just a warning.

It's not. The historical shells used for the basis of the POSIX standard
(ksh88 and the SVR4 sh) silently allow EOF to terminate a here-document;
ksh93 preserves this behavior. Some of the common shells allow this as
well (e.g., dash, zsh and the version of the FreeBSD from a couple of years
ago when I last built it), some do not (e.g., mksh and the netbsd sh). Bash
at least warns you about it.

> 
>  Example:
>   |
>   | bash-5.1$
>   | bash-5.1$ cat << ola"
> 
> OK, here we have another of the oddities of shell syntax.   The spec
> says that a here document starts at the next newline after the << operator,
> but that's not what it really means. 

I think the intent there is that the here document starts at the next
newline after the delimiter.


> Being able to do that (include embedded newline characters
> in a "line") isn't required by the shell specification, and (it has been
> a while since I checked) I do not believe that those work in bash (they
> do in some other shells).

I couldn't fine one where it does.


> Since bash doesn't allow end delimiter words that contain newlines to
> work, it should probably generate an error when you try to use one, that
> would have made things clear.

See above.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Unclosed quotes on heredoc mode

[Alex fxmbsw7 Ratchev]

u forgot to attach the picture .. ?

On Wed, Nov 17, 2021, 19:31 João Almeida Santos 
wrote:

> Thank you for your reply Robert and Lawrence!
>
> I understand the description alone is hard to follow, so I think the image
> below should make it clearer. Otherwise let me know!
>
>
> Kind regards,
> João Almeida Santos

Re: Unclosed quotes on heredoc mode

[Chet Ramey]

On 11/17/21 1:45 PM, João Almeida Santos wrote:
> No, it’s on the email...Anyway, here’s the text!
> 
> bash-5.1$ echo $PATH
> /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/usr/local/bin/:/usr/local/bin/
> 
> bash-5.1$ cat << $PATH
>> /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/usr/local/bin/:/usr/local/bin/
>> it should have terminated with the upper delimiter! but, bash does not seem 
>> to expand PATH.
>> $PATH
> /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/usr/local/bin/:/usr/local/bin/
> it should have terminated with the upper delimiter! but, bash does not seem 
> to expand PATH.

The here document delimiter does not undergo any expansions except quote
removal, so the delimiter is the literal string `$PATH'. The lines of the
here-document undergo a different set of expansions, which happen after the
check for the delimiter is performed, which means that you need to have a
line that consists solely of `$PATH' to terminate the here-document (as you
discovered). I cannot see how you're going to be able to do anything useful
with this construct; it just seems too clever by (more than) half.

This is all in the bash documentation.


> ok, but now addressing the actual question. If I use unclosed quotes on 
> heredoc, I can't use 
> the given delimiter to end the heredoc, I end up having to use an EOF. 
> Example:
> 
> bash-5.1$
> bash-5.1$ cat « ola"
>> I,
>> ola""
>> ola"
>> ola

The delimiter is not what you think it is. The delimiter for a here-
document is a shell word (which can include quoted substrings), and after
it undergoes the appropriate quote removal, your delimiter is
"ola\nI,\nola\nola" (using C string notation).

Now, you're never going to be able to match this; it contains a newline.
When the shell constructs the here-document body, it reads individual lines
from the input source and, after removing the trailing newline, tries to
match them against the delimiter (and backslash doesn't work to quote the
newline). This will obviously never match a delimiter containing a newline.

Some shells (e.g., yash) choose to make this a syntax error. Bash does not.

> In the above example, I don't unterstand how to provide the wanted delimiter!

You simply cannot, not the way you specify it. If you really want to have
the double quote as part of the here-document delimiter, write it as

cat << ola\"

I can't imagine this being useful, either.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Unclosed quotes on heredoc mode

[João Almeida Santos]

Hello,

First of all thank you for doing great (and free) software!
I’m a programming student currently on 42 School in Lisbon, and one of our 
projects is to create a minishell, and to mimic the behavior of bash. 
While testing the heredoc mode, I realized that the $ is not interpreted as 
variable expansion. That’s interesting.
But the reason why I’m emailing you is that I whenever I tried to use an 
unclosed quote on heredoc, it doesn’t seem to handle well…it never finishes the 
heredoc.
I tried \n for paragraph, verbatim inserting enter, closing quotes on the next 
line, …don’t know what else to test.
I’m on version 5.1 btw.
Is this an expected behavior or a bug?

Thank you!

Kind regards,
João Almeida Santos

Re: Unclosed quotes on heredoc mode

[Lawrence Velázquez]

On Wed, Nov 17, 2021, at 7:35 AM, João Almeida Santos wrote:
> I’m a programming student currently on 42 School in Lisbon, and one of 
> our projects is to create a minishell, and to mimic the behavior of 
> bash.

Nice!

> While testing the heredoc mode, I realized that the $ is not 
> interpreted as variable expansion. That’s interesting.

Are you talking about your project or bash?  In the latter (and
related shells), expansion is only suppressed in a here-document
if, when the delimiter is specified, it is at least partially quoted.

$ cat < But the reason why I’m emailing you is that I whenever I tried to use 
> an unclosed quote on heredoc, it doesn’t seem to handle well…it never 
> finishes the heredoc.
> I tried \n for paragraph, verbatim inserting enter, closing quotes on 
> the next line, …don’t know what else to test.

This description is a bit hard to follow.  Could you provide a small
bit of code that demonstrates the issue?

-- 
vq

Re: Unclosed quotes on heredoc mode

[Robert Elz]

Date:Wed, 17 Nov 2021 12:35:42 +
From:=?utf-8?Q?Jo=C3=A3o_Almeida_Santos?= 

Message-ID:  


  | While testing the heredoc mode, I realized that the $ is not
  | interpreted as variable expansion.

It depends how you set up the heredoc, please give an example of
what you're testing (one which is not doing what you expect, and
indicate exactly what you do expect to happen).

  | But the reason why I'm emailing you is that I whenever I tried to
  | use an unclosed quote on heredoc, it doesn't seem to handle well
  | it never finishes the heredoc.

Again, an example is needed to understand what you're doing.

  | I tried \n for paragraph, verbatim inserting enter,
  | closing quotes on the next line, ... don't know what else to test.

Without seeing what you are actually doing, where the missing quote is
for example (in the heredoc data, or in the word that is after the << ?)
it is hard to suggest anything.

  | I'm on version 5.1 btw.

Show the result of "echo $BASH_VERSION" to provide this info.

  | Is this an expected behavior or a bug?

Impossible to say without more information.   There are several (IMO)
bugs in the way bash processes here documents, whether you're encountering
one of those, something new, or just not understanding the way it is
intended to work is impossible to say with the information you've given.

kre

Re: Unclosed quotes on heredoc mode

[Robert Elz]

Date:Wed, 17 Nov 2021 15:47:37 -0500
From:Chet Ramey 
Message-ID:  <420281e7-f3c4-8054-d390-9378080c2...@case.edu>

  | Every modern shell uses `$PATH' as the here-document delimiter

Depends what you call modern shells - some ash derived shells (at least)
don't, because they parse the $PATH into an internal form (in all words
where that makes sense, before knowing what the word is to be used for)
and then cannot match that properly.   While that isn't actually expanding
the word, it still makes things fail badly.

But:

[D] sh-current $ cat foo <<$PATH
sh: 80: Syntax error: Illegal eof marker for << redirection

at least we error out when the user tries, not just fail to ever
find the end of the here doc.


  | and checks for the delimiter before any part of the process that expands
  | the lines in the here-document body.

That yes, I agree, everyone does that.

  | > First, the EOF should not work, that's a bash bug (IMO) - that should
  | > generate an error, not just a warning.
  |
  | It's not. The historical shells used for the basis of the POSIX standard

I didn't say it was a standards violation, I said it was a bug.
That the same bug exists in some other ancient shells isn't a justification.

Blindly taking the whole remainder of the script as a here document, and
processing it as if that were the author's intent, just because they made
a typo somewhere, is simply irrational.   Further, no-one (not anyone I
have ever seen) deliberately relies upon the here doc ending at EOF, not
even if a here doc is in a -c command string or similar).

  | Bash at least warns you about it.

Yes, better than some, but not as good as it should be.

  | > OK, here we have another of the oddities of shell syntax.   The spec
  | > says that a here document starts at the next newline after the << 
operator,
  | > but that's not what it really means. 
  |
  | I think the intent there is that the here document starts at the next
  | newline after the delimiter.

You mean at the newline after the ola" in the example given?   Really?
Surely it must mean newline token, not newline character, mustn't it?
(Even then, there are more, messier, issues, which I know you're aware of;
if we could make it as simple as "after the lexically next newline token"
it would make everything much simpler - that's what it should be.)

  | > Being able to do that (include embedded newline characters
  | > do in some other shells).
  |
  | I couldn't fine one where it does.

They work in (at least) the NetBSD shell, FreeBSD too I expect, since the
two use essentially the same mechanism for recognising the end of the
here doc -- (effectively) after a newline, read chars (from a buffer) one
at a time, comparing them with the end delimiter, until either there is a
match failure, or until the end of the end delimiter (after which one more
char from the buffer is compared to \n).   (Add tab stripping as required).

On no match, reset the buffer pointer back to where all this started,
and continue reading lines into the here doc.  When the end delim is
recognised, the here doc is complete after the last \n that was added to
it, and regular shell input continues after the \n from the buffer which
matched after the end delimiter.   What the chars are that match (including
more newlines, etc) is irrelevant, anything works (but no tab stripping
occurs after any intermediate newlines).

  | > Since bash doesn't allow end delimiter words that contain newlines to
  | > work, it should probably generate an error when you try to use one, that
  | > would have made things clear.
  |
  | See above.

Again, behaving irrationally when it would be trivial to detect the error
(even if a rare one) is poor design, and should be fixed.

kre

Bash-5.1 Official Patch 9

[Chet Ramey]

 BASH PATCH REPORT
 =

Bash-Release:   5.1
Patch-ID:   bash51-009

Bug-Reported-by:Julien Moutinho 
Bug-Reference-ID:   <20211004035906.5kiobuzkpeckm...@sourcephile.fr>
Bug-Reference-URL:  
https://lists.gnu.org/archive/html/bug-bash/2021-10/msg00022.html

Bug-Description:

The bash malloc implementation of malloc_usable_size() does not follow the
specification. This can cause library functions that use it to overwrite
memory bounds checking.

Patch (apply with `patch -p0'):

*** ../bash-5.1-patched/lib/malloc/malloc.c 2020-07-08 10:19:30.0 
-0400
--- lib/malloc/malloc.c 2021-10-05 16:10:55.0 -0400
***
*** 1287,1297 
  }
  
!   /* XXX - should we return 0 if ISFREE? */
!   maxbytes = binsize(p->mh_index);
! 
!   /* So the usable size is the maximum number of bytes in the bin less the
!  malloc overhead */
!   maxbytes -= MOVERHEAD + MSLOP;
!   return (maxbytes);
  }
  
--- 1358,1367 
  }
  
!   /* return 0 if ISFREE */
!   if (p->mh_alloc == ISFREE)
! return 0;
!   
!   /* Since we use bounds checking, the usable size is the last requested 
size. */
!   return (p->mh_nbytes);
  }
  
*** ../bash-5.1/patchlevel.h2020-06-22 14:51:03.0 -0400
--- patchlevel.h2020-10-01 11:01:28.0 -0400
***
*** 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 8
  
  #endif /* _PATCHLEVEL_H_ */
--- 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 9
  
  #endif /* _PATCHLEVEL_H_ */

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Bash-5.1 Official Patch 10

[Chet Ramey]

 BASH PATCH REPORT
 =

Bash-Release:   5.1
Patch-ID:   bash51-010

Bug-Reported-by:Jonas Alfredsson 
Bug-Reference-ID:   

Bug-Reference-URL:  
https://lists.gnu.org/archive/html/bug-bash/2021-05/msg00059.html

Bug-Description:

If `wait -n' is interrupted by a trapped signal other than SIGINT, it does
not completely clean up state, and that can prevent subsequent calls to
`wait -n' from working correctly.

Patch (apply with `patch -p0'):

*** ../bash-5.1-patched/builtins/wait.def   2020-12-16 17:13:12.0 
-0500
--- builtins/wait.def   2021-11-17 10:25:15.0 -0500
***
*** 112,116 
   WORD_LIST *list;
  {
!   int status, code, opt, nflag, wflags;
char *vname;
SHELL_VAR *pidvar;
--- 112,117 
   WORD_LIST *list;
  {
!   int status, code, opt, nflag;
!   volatile int wflags;
char *vname;
SHELL_VAR *pidvar;
***
*** 181,184 
--- 188,193 
status = 128 + wait_signal_received;
wait_sigint_cleanup ();
+   if (wflags & JWAIT_WAITING)
+   unset_waitlist ();
WAIT_RETURN (status);
  }

*** ../bash-5.1/patchlevel.h2020-06-22 14:51:03.0 -0400
--- patchlevel.h2020-10-01 11:01:28.0 -0400
***
*** 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 9
  
  #endif /* _PATCHLEVEL_H_ */
--- 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 10
  
  #endif /* _PATCHLEVEL_H_ */

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Bash-5.1 Official Patch 12

[Chet Ramey]

 BASH PATCH REPORT
 =

Bash-Release:   5.1
Patch-ID:   bash51-012

Bug-Reported-by:Nikolay Borisov 
Bug-Reference-ID:   <1a715205-06ce-413b-c1c0-2f5639ce0...@suse.com>
Bug-Reference-URL:  
https://lists.gnu.org/archive/html/bug-bash/2020-11/msg00091.html

Bug-Description:

There is a possible race condition that arises when a child process receives
a signal trapped by the parent before it can reset the signal dispositions.
The child process is not supposed to trap the signal in this circumstance.

Patch (apply with `patch -p0'):

*** ../bash-20201118/command.h  2020-08-14 15:04:39.0 -0400
--- command.h   2020-11-27 15:18:02.0 -0500
***
*** 125,128 
--- 125,129 
  #define SUBSHELL_COPROC   0x40/* subshell from a coproc pipeline */
  #define SUBSHELL_RESETTRAP 0x80   /* subshell needs to reset trap strings 
on first call to trap */
+ #define SUBSHELL_IGNTRAP 0x100  /* subshell should reset trapped signals from 
trap_handler */
  
  /* A structure which represents a word. */

*** ../bash-20201118/execute_cmd.c  2020-11-23 14:16:48.0 -0500
--- execute_cmd.c   2020-11-27 16:43:25.0 -0500
***
*** 1548,1551 
--- 1548,1554 
reset_signal_handlers ();
subshell_environment |= SUBSHELL_RESETTRAP;
+   /* Note that signal handlers have been reset, so we should no longer
+ reset the handler and resend trapped signals to ourselves. */
+   subshell_environment &= ~SUBSHELL_IGNTRAP;
  
/* We are in a subshell, so forget that we are running a trap handler or
***
*** 4321,4325 
  cmdflags |= CMD_NO_FORK;
  
! subshell_environment = SUBSHELL_FORK; /* XXX */
  if (pipe_in != NO_PIPE || pipe_out != NO_PIPE)
subshell_environment |= SUBSHELL_PIPE;
--- 4324,4329 
  cmdflags |= CMD_NO_FORK;
  
! /* We redo some of what make_child() does with SUBSHELL_IGNTRAP */
! subshell_environment = SUBSHELL_FORK|SUBSHELL_IGNTRAP;/* XXX 
*/
  if (pipe_in != NO_PIPE || pipe_out != NO_PIPE)
subshell_environment |= SUBSHELL_PIPE;
***
*** 4575,4578 
--- 4580,4584 
  reset_signal_handlers ();
  subshell_environment |= SUBSHELL_RESETTRAP;
+ subshell_environment &= ~SUBSHELL_IGNTRAP;
  
  if (async)
***
*** 5515,5518 
--- 5521,5525 
/* Cancel traps, in trap.c. */
restore_original_signals ();
+   subshell_environment &= ~SUBSHELL_IGNTRAP;
  
  #if defined (JOB_CONTROL)

*** ../bash-20201118/jobs.c 2020-08-04 10:17:39.0 -0400
--- jobs.c  2020-11-27 16:39:56.0 -0500
***
*** 2218,2221 
--- 2218,2223 
pid_t mypid;
  
+   subshell_environment |= SUBSHELL_IGNTRAP;
+ 
/* If this ends up being changed to modify or use `command' in the
 child process, go back and change callers who free `command' in
diff -rC 2 ../bash-20201118/nojobs.c nojobs.c
*** ../bash-20201118/nojobs.c   2020-07-08 10:11:25.0 -0400
--- nojobs.c2020-11-27 16:38:36.0 -0500
***
*** 576,579 
--- 576,581 
  #endif
  
+   subshell_environment |= SUBSHELL_IGNTRAP;
+ 
default_tty_job_signals ();
  }

*** ../bash-20201118/sig.c  2020-11-23 13:22:17.0 -0500
--- sig.c   2020-11-28 10:21:43.0 -0500
***
*** 56,60 
  #endif
  
! extern void initialize_siglist ();
  
  #if !defined (JOB_CONTROL)
--- 56,61 
  #endif
  
! extern void initialize_siglist PARAMS((void));
! extern void set_original_signal PARAMS((int, SigHandler *));
  
  #if !defined (JOB_CONTROL)
***
*** 256,259 
--- 257,267 
XHANDLER(i) = oact.sa_handler;
XSAFLAGS(i) = oact.sa_flags;
+ 
+ #if 0
+   set_original_signal (XSIG(i), XHANDLER(i)); /* optimization */
+ #else
+   set_original_signal (XSIG(i), act.sa_handler);  /* optimization */
+ #endif
+ 
/* Don't do anything with signals that are ignored at shell entry
 if the shell is not interactive. */

*** ../bash-20201118/subst.c2020-11-16 10:33:15.0 -0500
--- subst.c 2020-11-27 16:07:00.0 -0500
***
*** 5952,5955 
--- 5952,5956 
/* Cancel traps, in trap.c. */
restore_original_signals ();/* XXX - what about special builtins? 
bash-4.2 */
+   subshell_environment &= ~SUBSHELL_IGNTRAP;
QUIT;   /* catch any interrupts we got post-fork */
setup_async_signals ();
***
*** 6383,6386 
--- 6384,6388 
QUIT;   /* catch any interrupts we got post-fork */
subshell_environment |= SUBSHELL_RESETTRAP;
+   subshell_environment &= ~SUBSHELL_IGNTRAP;
  }
  

diff -rC 2 ../bash-20201118/trap.c trap.c
*** ../bash-20201118/trap.c 2020-11-28 

Bash-5.1 Official Patch 11

[Chet Ramey]

 BASH PATCH REPORT
 =

Bash-Release:   5.1
Patch-ID:   bash51-011

Bug-Reported-by:Alex fxmbsw7 Ratchev 
Bug-Reference-ID:   

Bug-Reference-URL:  
https://lists.gnu.org/archive/html/bug-bash/2020-11/msg00064.html

Bug-Description:

When reading a compound assignment, and running it through the parser to
split it into words, we need to save and restore any alias we're currently
expanding.

Patch (apply with `patch -p0'):

*** ../bash-5.1-patched/parse.y 2020-11-28 12:10:06.0 -0500
--- parse.y 2021-10-13 11:04:27.0 -0400
***
*** 6494,6501 
  
push_stream (1);
- #if 0 /* TAG: bash-5.2 Alex fxmbsw7 Ratchev  11/17/2020 */
if (ea = expanding_alias ())
  parser_save_alias ();
- #endif
last_read_token = WORD; /* WORD to allow reserved words here */
current_command_line_count = 0;
--- 6494,6499 
***
*** 6532,6539 
pop_stream ();
  
- #if 0 /* TAG: bash-5.2 */
if (ea)
  parser_restore_alias ();
- #endif
  
  #if defined (HISTORY)
--- 6530,6535 
*** ../bash-5.1-patched/y.tab.c 2020-11-28 12:17:19.0 -0500
--- y.tab.c 2021-11-17 10:47:35.0 -0500
***
*** 8788,8795 
  
push_stream (1);
- #if 0 /* TAG: bash-5.2 Alex fxmbsw7 Ratchev  11/17/2020 */
if (ea = expanding_alias ())
  parser_save_alias ();
- #endif
last_read_token = WORD; /* WORD to allow reserved words here */
current_command_line_count = 0;
--- 8777,8782 
***
*** 8826,8833 
pop_stream ();
  
- #if 0 /* TAG: bash-5.2 */
if (ea)
  parser_restore_alias ();
- #endif
  
  #if defined (HISTORY)
--- 8813,8818 
*** ../bash-5.1/patchlevel.h2020-06-22 14:51:03.0 -0400
--- patchlevel.h2020-10-01 11:01:28.0 -0400
***
*** 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 10
  
  #endif /* _PATCHLEVEL_H_ */
--- 26,30 
 looks for to find the patch level (for the sccs version string). */
  
! #define PATCHLEVEL 11
  
  #endif /* _PATCHLEVEL_H_ */

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: bash conditional expressions

[Michael J. Baars]

On Fri, 2021-11-12 at 19:48 +0100, Andreas Schwab wrote:
>   FILE1 -nt FILE2  True if file1 is newer than file2 (according to
>modification date).
> 
> Andreas.
> 

So now we have a relation for 'older than' and for 'newer than', but how about 
'oldest' (executable), and 'newest' (executable)?

I could only come up with this:

unset y; for x in $(find bin -mindepth 1 -name "*"); do if [[ ${x} -nt ${y} ]]; 
then y=${x}; fi; done; echo newest: ${y};
y="bin"; for x in $(find bin -mindepth 1 -name "*"); do if [[ ${x} -ot ${y} ]]; 
then y=${x}; fi; done; echo oldest: ${y};

As you can see, the way the commands are initialized is not identical, because:

'-nt' returns a true when 'if file1 exists and file2 does not'  (y in 
initialized by the first condition evaluated)
'-ot' returns a true when 'if file2 exists and file1 does not'  (y is not 
initialized by the first condition evaluated)

When you try to selectively link new executables, I think it is important that 
you do not only have relations for 'older than' and 'newer than', but also 
consistent (identically initializated)
relations for 'oldest' and 'newest'.

Mischa.

Re: bash conditional expressions

[Michael J. Baars]

On Wed, 2021-11-17 at 14:06 +0200, Ilkka Virta wrote:
> On Wed, Nov 17, 2021 at 1:33 PM Andreas Schwab  wrote:
> > On Nov 17 2021, Michael J. Baars wrote:
> > 
> > 
> > 
> > > When -N stands for NEW, and touch (-am) gives you a new file
> > 
> > 
> > 
> > It doesn't.  The file hasn't been modified after it was last read.
> 
> touch creates the given file if it doesn't previously exist. Immediately 
> afterwards,it could be called "new" in the usual English meaning, and would 
> be new in thesense that nothing was done to it
> after it was created. But:
> $ echo $BASH_VERSION5.1.8(3)-maint
> $ rm foo.txt
> $ ls -l foo.txt
> ls: cannot access 'foo.txt': No such file or directory
> $ touch -am foo.txt
> $ if test -N foo.txt; then echo is new; else echo is NOT new; fi
> is NOT new
> 
> Of course "new" is not an exact concept, it could be defined e.g. to compare 
> the
> file timestamps with the current time.
> 
> 
> Anyway, the documentation doesn't seem to say 'test -N' tests if the file is 
> "new".

It seemed logical to assume that '-N' stands for 'new' in some way. The rest of 
the line does indeed not imply '-N' to be equivalent to 'new'.