Re: pipefail with SIGPIPE/EPIPE
> Errexit (a.k.a. set -e) is horrible, > and you should not be using it in any new shell scripts you write. > It exists solely for support of legacy scripts. Wow. For those of us who don't know this, what should we be using instead? Is using a trap on ERR any better? Is your suggestion that || exit 1 be added to the end of every command? (Potentially of mild interest is this thread on Hacker News from earlier today, where multiple people are suggesting the usage of "set -e" along with "set -u" and "set -o pipefile".) https://news.ycombinator.com/item?id=13940322
q// / qq// syntax or built-ins please ?
Please , in some future versions of bash, could it provide
support / help for avoiding "quoting hell", in such situations
as :
$ echo "
Error: Cannot find file '/missing/file_1'.
Error: Cannot find file '/missing/file_2'.
" | while read line; do
cmd='if [[ '$'"'"$line"$'"'' =~
^[^'"\\'"']*['"\\'"']([^'"\\'"']+)['"\\'"'][^'"\\'"']*$ ]]; then echo
${BASH_REMATCH[1]}; fi;';
echo "$cmd" | bash -;
done
See what I have to do to match lines containing a non-empty
single-quoted string ?
ie. I just want to cut-and-paste such lines from the output of some
application, and
weed out the empty lines and print the single-quoted string in lines
containing them (only!), with a simple bash command. If you replace
"echo $cmd | bash -" with
'eval "$cmd"' , it does not work, because the double-quotes which I
had painstakingly inserted with '$'"'' get removed somehow by eval -
why is this?
ie, only if "$line" is empty, does bash evaluate the text:
'if [[ "" = ... ]]; then ...'
else, for the lines I want to match, it would evaluate eg. :
+ eval 'if [[ Error: Cannot find file '\''/missing_file_1'\''.\" =~
^[^\\\'\'']*[\\\'\'']([\\\'\'']+)[\\\'\''][^\\\'\'']*$ ]]; then echo
${BASH_REMATCH[1]}; fi;'
+ set +x
( nothing printed - the single quotes are stripped)
+ eval 'if [[ \"\" =~
^[^\\\'\'']*[\\\'\'']([\\\'\'']+)[\\\'\''][^\\\'\'']*$ ]]; then echo
${BASH_REMATCH[1]}; fi;'
++ [[ "" =~ ^[^\']*[\']([\']+)[\'][^\']*$ ]]
+ set +x
+
I think bash needs some kind of "q/.../'" and 'qq/../' syntax / built-ins, or
whatever syntax its author likes, like PERL has, whereby the single quote
("'") or double quote ('"') respectively are totally ignored within
'/.../' parameter
strings, which should use a different quoting character than '"' or
"'" to delineate
them.
If the author won't develop these, I will & will send a patch.
Regards,
Jason
Re: pipefail with SIGPIPE/EPIPE
On Thu, Mar 23, 2017 at 07:27:19PM +, Jay Freeman (saurik) wrote: > > Errexit (a.k.a. set -e) is horrible, > > and you should not be using it in any new shell scripts you write. > > It exists solely for support of legacy scripts. > > Wow. For those of us who don't know this, what should we be using instead? Is > using a trap on ERR any better? Is your suggestion that || exit 1 be added to > the end of every command? That will work, as long as you know which commands *not* to put it on. In practice, you should know which commands are critical to the script, and just put it on those. > (Potentially of mild interest is this thread on Hacker News from earlier > today, where multiple people are suggesting the usage of "set -e" along with > "set -u" and "set -o pipefile".) > > https://news.ycombinator.com/item?id=13940322 I understand why people want set -e. I really, truly do. If it did what people thought it did, I'd use it too! I use "use strict" in Perl, and in Tcl, you get behavior similar to that of "set -u" (except not broken) by default. The problem is, set -e does NOT do what people think it does. http://mywiki.wooledge.org/BashFAQ/105 has more details. set -u is not quite as bad, but it can also break scripts, just like set -e can. http://mywiki.wooledge.org/BashFAQ/112 has details on that. Let me try a story. Maybe that will help convey how things stand. Once upon a time, a man with a dirty lab coat and long, uncombed hair showed up at the town police station, demanding to see the chief of police. "I've done it!" he exclaimed. "I've built the perfect criminal-catching robot!" The police chief was skeptical, but decided that it might be worth the time to see what the man had invented. Also, he secretly thought, it might be a somewhat unwise move to completely alienate the mad scientist and his army of hunter robots. So, the man explained to the police chief how his invention could tell the difference between a criminal and law-abiding citizen using a series of heuristics. "It's especially good at spotting recently escaped prisoners!" he said. "Guaranteed non-lethal restraints!" Frowning and increasingly skeptical, the police chief nevertheless allowed the man to demonstrate one robot for a week. They decided that the robot should patrol around the jail. Sure enough, there was a jailbreak a few days later, and an inmate digging up through the ground outside of the prison facility was grabbed by the robot and carried back inside the prison. The surprised police chief allowed the robot to patrol a wider area. The next day, the chief received an angry call from the zookeeper. It seems the robot had cut through the bars of one of the animal cages, grabbed the animal, and delivered it to the prison. The chief confronted the robot's inventor, who asked what animal it was. "A zebra," replied the police chief. The man slapped his head and exclaimed, "Curses! It was fooled by the black and white stripes! I shall have to recalibrate!" And so the man set about rewriting the robot's code. Black and white stripes would indicate an escaped inmate UNLESS the inmate had more than two legs. Then it should be left alone. The robot was redeployed with the updated code, and seemed to be operating well enough for a few days. Then on Saturday, a mob of children in soccer clothing, followed by their parents, descended on the police station. After the chaos subsided, the chief was told that the robot had absconded with the referee right in the middle of a soccer game. Scowling, the chief reported this to the scientist, who performed a second calibration. Black and white stripes would indicate an escaped inmate UNLESS the inmate had more than two legs OR had a whistle on a necklace. Despite the second calibration, the police chief declared that the robot would no longer be allowed to operate in his town. However, the news of the robot had spread, and requests from many larger cities were pouring in. The inventor made dozens more robots, and shipped them off to eager police stations around the nation. Every time a robot grabbed something that wasn't an escaped inmate, the scientist was consulted, and the robot was recalibrated. Unfortunately, the inventor was just one man, and he didn't have the time or the resources to recalibrate EVERY robot whenever one of them went awry. The robot in Shangri-La was recalibrated not to grab a grave-digger working on a cold winter night while wearing a ski mask, and the robot in Xanadu was recalibrated not to capture a black and white television set that showed a movie about a prison break, and so on. But the robot in Xanadu would still grab grave-diggers with ski masks (which it turns out was not common due to Xanadu's warmer climate), and the robot in Shangri-La was still a menace to old televisions (of which there were very few, the people of Shangri-La being on the average more wealthy than those of Xanadu). So, after a few years,
Re: pipefail with SIGPIPE/EPIPE
On 23/03/17 09:34, Greg Wooledge wrote: > On Thu, Mar 23, 2017 at 08:50:45AM -0700, Pádraig Brady wrote: >> I was bitten by this again when combined with set -e. >> I.E. this script doesn't finish: >> >> #!/bin/bash >> set -o errexit >> set -o pipefail >> yes | head -n1 >> echo finished >> >> That makes the errexit and pipefail options decidedly less useful. > > No. It makes errexit less useful. Errexit (a.k.a. set -e) is horrible, > and you should not be using it in any new shell scripts you write. > It exists solely for support of legacy scripts. OK let's not derail this into a discussion specific to errexit. Can we please improve things? You say to not use errexit, and instead use `|| exit 1` where appropriate. In that case can we fix this case? set -o pipefail yes | head -n1 || exit 1 echo this is skipped cheers, Pádraig
Re: pipefail with SIGPIPE/EPIPE
On 15/02/15 14:14, Pádraig Brady wrote: > On 15/02/15 21:59, Daniel Colascione wrote: >> On 02/15/2015 01:48 PM, Chet Ramey wrote: >>> On 2/13/15 12:19 PM, Pádraig Brady wrote: I was expecting bash to handle SIGPIPE specially here, as in this context it's informational rather than an indication of error. >>> >>> I don't agree. It's a fatal signal whose default disposition is to >>> terminate a process, which is exactly what happens in your example. >> >> The purpose of pipefail is to make the shell indicate when something has >> gone wrong anywhere in a pipeline. For most programs, SIGPIPE does not >> indicate that something went wrong. Instead, SIGPIPE is expected >> behavior. When pipefail spuriously reports expected behavior as an >> error, Bash comes less useful. > > Exactly. SIGPIPE is special. It indicates the pipe is closed. > That may be due to something having gone wrong down the pipe, > but if that's the case the status code will be that of the > failing process down the pipe. > If it's only SIGPIPE that's significant to the status, > then we know it's only informational, in which case the status > should be 0 to indicate things have gone as expected. > > There are many cases of the pipe being legitimately closed early. > ... | head > ... | grep -m1 ... > etc. > >>> You might consider trapping or ignoring SIGPIPE in situations where it >>> might be an issue. >> >> If I were emperor of the world, I would make SIGPIPE's SIG_DFL action >> terminate the process with exit status 0. But POSIX says we can't do >> that. Even locally, I make my system do that without kernel surgery. >> It's also not reasonable to modify every program that might be part of a >> pipeline so that it exits successfully on EPIPE. >> >> Making Bash treat SIGPIPE death as success is the next best option. > > Only SIG_IGN isn't reset on exec, in which case each process > would be getting EPIPE on write(), which most don't (and don't need to) > handle explicitly. bash handling the SIGPIPE specially seems > like the best option to me too. I was bitten by this again when combined with set -e. I.E. this script doesn't finish: #!/bin/bash set -o errexit set -o pipefail yes | head -n1 echo finished That makes the errexit and pipefail options decidedly less useful. Looking at this the other way, does the current behavior help, or would any existing code be depending on the current behavior of treating SIGPIPE as an error, when it's really only a shortcut informational mechanism. thanks, Pádraig
Re: pipefail with SIGPIPE/EPIPE
On Thu, Mar 23, 2017 at 08:50:45AM -0700, Pádraig Brady wrote: > I was bitten by this again when combined with set -e. > I.E. this script doesn't finish: > > #!/bin/bash > set -o errexit > set -o pipefail > yes | head -n1 > echo finished > > That makes the errexit and pipefail options decidedly less useful. No. It makes errexit less useful. Errexit (a.k.a. set -e) is horrible, and you should not be using it in any new shell scripts you write. It exists solely for support of legacy scripts.
