Re: pipefail with SIGPIPE/EPIPE
On 3/24/17 7:57 AM, Greg Wooledge wrote: > On Thu, Mar 23, 2017 at 10:14:01PM -0700, Pádraig Brady wrote: >> OK let's not derail this into a discussion specific to errexit. >> Can we please improve things? >> You say to not use errexit, and instead use `|| exit 1` where appropriate. >> In that case can we fix this case? >> >> set -o pipefail >> yes | head -n1 || exit 1 >> echo this is skipped > > What do you think is broken, here? > > imadev:~$ yes | head -n1 > y > imadev:~$ declare -p PIPESTATUS > declare -a PIPESTATUS=([0]="141" [1]="0") > > I don't see any problem in bash's behavior. It's exiting because you > asked it to exit if the pipe failed, and the pipe failed. The pipe > failed because yes(1) returned a nonzero exit code, and you turned on > the pipefail option. More precisely, `yes' died due to SIGPIPE, whose default action is to terminate the process, so its exit status is 128+SIGPIPE. > What exactly are you asking to change? That's what he's asking to change. He wants bash to mask the occurrence of SIGPIPE, so that a command that dies due to SIGPIPE doesn't `fail'. He is asking that SIGPIPE be made special, so that a process dying due to it appears to exit with a status of 0. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: pipefail with SIGPIPE/EPIPE
On 3/23/17 3:27 PM, Jay Freeman (saurik) wrote: > (Potentially of mild interest is this thread on Hacker News from earlier > today, where multiple people are suggesting the usage of "set -e" along with > "set -u" and "set -o pipefile".) > > https://news.ycombinator.com/item?id=13940322 It's useful as long as you understand what it does, and when it does not have any effect. I'd say that that set of people is smaller than it should be. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: pipefail with SIGPIPE/EPIPE
On 24/03/17 04:57, Greg Wooledge wrote: > On Thu, Mar 23, 2017 at 10:14:01PM -0700, Pádraig Brady wrote: >> OK let's not derail this into a discussion specific to errexit. >> Can we please improve things? >> You say to not use errexit, and instead use `|| exit 1` where appropriate. >> In that case can we fix this case? >> >> set -o pipefail >> yes | head -n1 || exit 1 >> echo this is skipped > > What do you think is broken, here? > > imadev:~$ yes | head -n1 > y > imadev:~$ declare -p PIPESTATUS > declare -a PIPESTATUS=([0]="141" [1]="0") > > I don't see any problem in bash's behavior. It's exiting because you > asked it to exit if the pipe failed, and the pipe failed. The pipe > failed because yes(1) returned a nonzero exit code, and you turned on > the pipefail option. > > What exactly are you asking to change? I would like bash to treat SIGPIPE specially in this case, as it's not an error, rather the standard "back pressure" mechanism used in pipes. The fact that SIGPIPE kills by default is only a shortcut mechanism allowing processes to exit automatically without changing their code. The shell should not treat this as a real error though. I.E. so I can write code like: set -o pipefail vv=$(yes | head) || exit 1 echo finished which will exit if yes(1) or head(1) segfault or otherwise exit(1) etc., but proceed normally. thanks, Pádraig
Re: pipefail with SIGPIPE/EPIPE
On Thu, Mar 23, 2017 at 10:14:01PM -0700, Pádraig Brady wrote: > OK let's not derail this into a discussion specific to errexit. > Can we please improve things? > You say to not use errexit, and instead use `|| exit 1` where appropriate. > In that case can we fix this case? > > set -o pipefail > yes | head -n1 || exit 1 > echo this is skipped What do you think is broken, here? imadev:~$ yes | head -n1 y imadev:~$ declare -p PIPESTATUS declare -a PIPESTATUS=([0]="141" [1]="0") I don't see any problem in bash's behavior. It's exiting because you asked it to exit if the pipe failed, and the pipe failed. The pipe failed because yes(1) returned a nonzero exit code, and you turned on the pipefail option. What exactly are you asking to change?
Re: pipefail with SIGPIPE/EPIPE
On 23/03/17 09:34, Greg Wooledge wrote: > On Thu, Mar 23, 2017 at 08:50:45AM -0700, Pádraig Brady wrote: >> I was bitten by this again when combined with set -e. >> I.E. this script doesn't finish: >> >> #!/bin/bash >> set -o errexit >> set -o pipefail >> yes | head -n1 >> echo finished >> >> That makes the errexit and pipefail options decidedly less useful. > > No. It makes errexit less useful. Errexit (a.k.a. set -e) is horrible, > and you should not be using it in any new shell scripts you write. > It exists solely for support of legacy scripts. OK let's not derail this into a discussion specific to errexit. Can we please improve things? You say to not use errexit, and instead use `|| exit 1` where appropriate. In that case can we fix this case? set -o pipefail yes | head -n1 || exit 1 echo this is skipped cheers, Pádraig
Re: pipefail with SIGPIPE/EPIPE
> Errexit (a.k.a. set -e) is horrible, > and you should not be using it in any new shell scripts you write. > It exists solely for support of legacy scripts. Wow. For those of us who don't know this, what should we be using instead? Is using a trap on ERR any better? Is your suggestion that || exit 1 be added to the end of every command? (Potentially of mild interest is this thread on Hacker News from earlier today, where multiple people are suggesting the usage of "set -e" along with "set -u" and "set -o pipefile".) https://news.ycombinator.com/item?id=13940322
Re: pipefail with SIGPIPE/EPIPE
On Thu, Mar 23, 2017 at 07:27:19PM +, Jay Freeman (saurik) wrote: > > Errexit (a.k.a. set -e) is horrible, > > and you should not be using it in any new shell scripts you write. > > It exists solely for support of legacy scripts. > > Wow. For those of us who don't know this, what should we be using instead? Is > using a trap on ERR any better? Is your suggestion that || exit 1 be added to > the end of every command? That will work, as long as you know which commands *not* to put it on. In practice, you should know which commands are critical to the script, and just put it on those. > (Potentially of mild interest is this thread on Hacker News from earlier > today, where multiple people are suggesting the usage of "set -e" along with > "set -u" and "set -o pipefile".) > > https://news.ycombinator.com/item?id=13940322 I understand why people want set -e. I really, truly do. If it did what people thought it did, I'd use it too! I use "use strict" in Perl, and in Tcl, you get behavior similar to that of "set -u" (except not broken) by default. The problem is, set -e does NOT do what people think it does. http://mywiki.wooledge.org/BashFAQ/105 has more details. set -u is not quite as bad, but it can also break scripts, just like set -e can. http://mywiki.wooledge.org/BashFAQ/112 has details on that. Let me try a story. Maybe that will help convey how things stand. Once upon a time, a man with a dirty lab coat and long, uncombed hair showed up at the town police station, demanding to see the chief of police. "I've done it!" he exclaimed. "I've built the perfect criminal-catching robot!" The police chief was skeptical, but decided that it might be worth the time to see what the man had invented. Also, he secretly thought, it might be a somewhat unwise move to completely alienate the mad scientist and his army of hunter robots. So, the man explained to the police chief how his invention could tell the difference between a criminal and law-abiding citizen using a series of heuristics. "It's especially good at spotting recently escaped prisoners!" he said. "Guaranteed non-lethal restraints!" Frowning and increasingly skeptical, the police chief nevertheless allowed the man to demonstrate one robot for a week. They decided that the robot should patrol around the jail. Sure enough, there was a jailbreak a few days later, and an inmate digging up through the ground outside of the prison facility was grabbed by the robot and carried back inside the prison. The surprised police chief allowed the robot to patrol a wider area. The next day, the chief received an angry call from the zookeeper. It seems the robot had cut through the bars of one of the animal cages, grabbed the animal, and delivered it to the prison. The chief confronted the robot's inventor, who asked what animal it was. "A zebra," replied the police chief. The man slapped his head and exclaimed, "Curses! It was fooled by the black and white stripes! I shall have to recalibrate!" And so the man set about rewriting the robot's code. Black and white stripes would indicate an escaped inmate UNLESS the inmate had more than two legs. Then it should be left alone. The robot was redeployed with the updated code, and seemed to be operating well enough for a few days. Then on Saturday, a mob of children in soccer clothing, followed by their parents, descended on the police station. After the chaos subsided, the chief was told that the robot had absconded with the referee right in the middle of a soccer game. Scowling, the chief reported this to the scientist, who performed a second calibration. Black and white stripes would indicate an escaped inmate UNLESS the inmate had more than two legs OR had a whistle on a necklace. Despite the second calibration, the police chief declared that the robot would no longer be allowed to operate in his town. However, the news of the robot had spread, and requests from many larger cities were pouring in. The inventor made dozens more robots, and shipped them off to eager police stations around the nation. Every time a robot grabbed something that wasn't an escaped inmate, the scientist was consulted, and the robot was recalibrated. Unfortunately, the inventor was just one man, and he didn't have the time or the resources to recalibrate EVERY robot whenever one of them went awry. The robot in Shangri-La was recalibrated not to grab a grave-digger working on a cold winter night while wearing a ski mask, and the robot in Xanadu was recalibrated not to capture a black and white television set that showed a movie about a prison break, and so on. But the robot in Xanadu would still grab grave-diggers with ski masks (which it turns out was not common due to Xanadu's warmer climate), and the robot in Shangri-La was still a menace to old televisions (of which there were very few, the people of Shangri-La being on the average more wealthy than those of Xanadu). So, after a few years, there
Re: pipefail with SIGPIPE/EPIPE
On Thu, Mar 23, 2017 at 08:50:45AM -0700, Pádraig Brady wrote: > I was bitten by this again when combined with set -e. > I.E. this script doesn't finish: > > #!/bin/bash > set -o errexit > set -o pipefail > yes | head -n1 > echo finished > > That makes the errexit and pipefail options decidedly less useful. No. It makes errexit less useful. Errexit (a.k.a. set -e) is horrible, and you should not be using it in any new shell scripts you write. It exists solely for support of legacy scripts.
Re: pipefail with SIGPIPE/EPIPE
On 15/02/15 14:14, Pádraig Brady wrote: > On 15/02/15 21:59, Daniel Colascione wrote: >> On 02/15/2015 01:48 PM, Chet Ramey wrote: >>> On 2/13/15 12:19 PM, Pádraig Brady wrote: I was expecting bash to handle SIGPIPE specially here, as in this context it's informational rather than an indication of error. >>> >>> I don't agree. It's a fatal signal whose default disposition is to >>> terminate a process, which is exactly what happens in your example. >> >> The purpose of pipefail is to make the shell indicate when something has >> gone wrong anywhere in a pipeline. For most programs, SIGPIPE does not >> indicate that something went wrong. Instead, SIGPIPE is expected >> behavior. When pipefail spuriously reports expected behavior as an >> error, Bash comes less useful. > > Exactly. SIGPIPE is special. It indicates the pipe is closed. > That may be due to something having gone wrong down the pipe, > but if that's the case the status code will be that of the > failing process down the pipe. > If it's only SIGPIPE that's significant to the status, > then we know it's only informational, in which case the status > should be 0 to indicate things have gone as expected. > > There are many cases of the pipe being legitimately closed early. > ... | head > ... | grep -m1 ... > etc. > >>> You might consider trapping or ignoring SIGPIPE in situations where it >>> might be an issue. >> >> If I were emperor of the world, I would make SIGPIPE's SIG_DFL action >> terminate the process with exit status 0. But POSIX says we can't do >> that. Even locally, I make my system do that without kernel surgery. >> It's also not reasonable to modify every program that might be part of a >> pipeline so that it exits successfully on EPIPE. >> >> Making Bash treat SIGPIPE death as success is the next best option. > > Only SIG_IGN isn't reset on exec, in which case each process > would be getting EPIPE on write(), which most don't (and don't need to) > handle explicitly. bash handling the SIGPIPE specially seems > like the best option to me too. I was bitten by this again when combined with set -e. I.E. this script doesn't finish: #!/bin/bash set -o errexit set -o pipefail yes | head -n1 echo finished That makes the errexit and pipefail options decidedly less useful. Looking at this the other way, does the current behavior help, or would any existing code be depending on the current behavior of treating SIGPIPE as an error, when it's really only a shortcut informational mechanism. thanks, Pádraig
Re: pipefail with SIGPIPE/EPIPE
On 15/02/15 21:59, Daniel Colascione wrote: > On 02/15/2015 01:48 PM, Chet Ramey wrote: >> On 2/13/15 12:19 PM, Pádraig Brady wrote: >>> I was expecting bash to handle SIGPIPE specially here, >>> as in this context it's informational rather than an indication of error. >> >> I don't agree. It's a fatal signal whose default disposition is to >> terminate a process, which is exactly what happens in your example. > > The purpose of pipefail is to make the shell indicate when something has > gone wrong anywhere in a pipeline. For most programs, SIGPIPE does not > indicate that something went wrong. Instead, SIGPIPE is expected > behavior. When pipefail spuriously reports expected behavior as an > error, Bash comes less useful. Exactly. SIGPIPE is special. It indicates the pipe is closed. That may be due to something having gone wrong down the pipe, but if that's the case the status code will be that of the failing process down the pipe. If it's only SIGPIPE that's significant to the status, then we know it's only informational, in which case the status should be 0 to indicate things have gone as expected. There are many cases of the pipe being legitimately closed early. ... | head ... | grep -m1 ... etc. >> You might consider trapping or ignoring SIGPIPE in situations where it >> might be an issue. > > If I were emperor of the world, I would make SIGPIPE's SIG_DFL action > terminate the process with exit status 0. But POSIX says we can't do > that. Even locally, I make my system do that without kernel surgery. > It's also not reasonable to modify every program that might be part of a > pipeline so that it exits successfully on EPIPE. > > Making Bash treat SIGPIPE death as success is the next best option. Only SIG_IGN isn't reset on exec, in which case each process would be getting EPIPE on write(), which most don't (and don't need to) handle explicitly. bash handling the SIGPIPE specially seems like the best option to be too. thanks, Pádraig.
Re: pipefail with SIGPIPE/EPIPE
On 02/15/2015 01:48 PM, Chet Ramey wrote: > On 2/13/15 12:19 PM, Pádraig Brady wrote: >> I was expecting bash to handle SIGPIPE specially here, >> as in this context it's informational rather than an indication of error. > > I don't agree. It's a fatal signal whose default disposition is to > terminate a process, which is exactly what happens in your example. The purpose of pipefail is to make the shell indicate when something has gone wrong anywhere in a pipeline. For most programs, SIGPIPE does not indicate that something went wrong. Instead, SIGPIPE is expected behavior. When pipefail spuriously reports expected behavior as an error, Bash comes less useful. > You might consider trapping or ignoring SIGPIPE in situations where it > might be an issue. If I were emperor of the world, I would make SIGPIPE's SIG_DFL action terminate the process with exit status 0. But POSIX says we can't do that. Even locally, I make my system do that without kernel surgery. It's also not reasonable to modify every program that might be part of a pipeline so that it exits successfully on EPIPE. Making Bash treat SIGPIPE death as success is the next best option. signature.asc Description: OpenPGP digital signature
Re: pipefail with SIGPIPE/EPIPE
On 2/13/15 12:19 PM, Pádraig Brady wrote: > I was expecting bash to handle SIGPIPE specially here, > as in this context it's informational rather than an indication of error. I don't agree. It's a fatal signal whose default disposition is to terminate a process, which is exactly what happens in your example. You might consider trapping or ignoring SIGPIPE in situations where it might be an issue. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
