On Mon, Nov 23, 2015 at 11:25 PM, David Ventimiglia <dvent...@gmail.com> wrote:
> How do I use Bash pipefail with GNU Parallel?
:
> I want any failed pipeline command to cause GNU parallel to fail (i.e.,
> return non-zero exit status).
>
> Here's an example of a command stored in the database table.
>
>> curl -sf <some URL> | awk <some program>:
:
> The problem is that
> the individual commands never fail (i.e., they don't return a non-zero exit
> status) because the last command in their pipelines--the call to awk--never
> fails.
:
> Evidently, Bash has a convenient facility for this. One can set the
> 'pipefail' option so that the exit status of pipelines is right-most
> non-zero exit status of its constituent commands, or zero if none of its
> commands fails. That's exactly what I want. The question is this.
>
> How do I pass 'set -o pipefail' to the commands that are run by GNU
> parallel?
>
> I think the answer lies somewhere in the labyrinth of combinations of using
> some GNU Parallel options (-q --shell-quote -I{}) and some Bash options
> (-c).
>
> Here's what I tried so far, and which didn't work.
>
>> sql -n <some database URL> 'select command from mycommands' | parallel
>> --halt 2 bash -c 'set -o pipefail; {}'
You are being hit by quoting and your thinking of -q is correct. -I{}
will not help you, and --shell-quote is used for quite a different
problem.
sql -n <some database URL> 'select command from mycommands' |
parallel -q --halt 2 bash -c 'set -o pipefail; {}'
Alternatives:
myfunc() {
bash -c "set -o pipefail; $1"
}
export -f myfunc
... | parallel --halt 2 myfunc
or:
... | parallel -q --halt 2 bash -c "'set -o pipefail; {}'"
or make a script:
#!/bin/bash
set -o pipefail
"$@"
and call that: ... | parallel myscript {}
Personally I have grown to like functions: They are easy to test
before giving them to GNU Parallel, they are easy to read, they can be
put in the same file/script, they can be exported to remote hosts
(using --env), and (most importantly) they do not require a forest of
mixed '"\ which can be a hell to debug.
/Ole
PS: Just for fun:
fun() {
echo "$1" is fun;
}
export -f fun
parallel -S localhost --env fun -vv fun ::: this
