>On Thu, Mar 29, 2012 at 6:02 PM, Ole Tange <[email protected]> wrote: > On Thu, Mar 29, 2012 at 9:23 PM, David Erickson <[email protected]> wrote: >> Hi there- >> I just discovered parallel and it looks great. > > Good to hear. How did you discover it? What could I have done to make > you discover it earlier?
I found it through one of the comments in this thread: http://stackoverflow.com/questions/463963/parallel-processing-from-a-command-queue-on-linux-bash-python-ruby-whateve I'm not sure what the key words you use on your website are, I believe I tried searching for bash job queue to begin with and it led me to the above. I'm surprised it doesn't come up in any conversation relating to a tool like this because this project has about a zillion more features than any of the others I ran in to in the same category. > >> One existing use case >> I have for commands I would like to run is something like: >> >> parallel 'cmd1 {} 2>&1 | tee somewhere.log' >> >> I have a couple questions regarding this use case: >> >> 1) Will parallel correctly honor my request to redirect stderr to stdout? > > This is easy for you to test, so I will leave that for you as an exercise. > >> 2) Is there some way to use --halt with the error code returned from >> cmd1? IE in bash I can do this with ${PIPESTATUS[0]}, because tee >> will always return 0. > > GNU Parallel looks at the exit code, so you can do: > > parallel 'cmd1 {} 2>&1 | tee somewhere.log; exit ${PIPESTATUS[0]}' > > You can combine this with --halt if you need GNU Parallel not to spawn > more jobs. If you just need to know how many jobs failed, the default > (--halt 0) will work for you. Perfect, with --halt 2 is what I'm looking for. > >> Alternatively, can parallel log both stdout and >> stderr for each command to a unique file with naming of my choice? > > You probably want one of these: > > ... | parallel cmd1 {} > all_stdout 2> all_stderr > ... | parallel --tag cmd1 {} > all_stdout 2> all_stderr > ... | parallel cmd1 {} \> {}_stdout 2\> {}_stderr > ... | parallel cmd1 {} \> {#}_stdout 2\> {#}_stderr > > But more creative ways exists. If every 3rd argument is the > stdout-name and the next is the stderr-name: > > ... | parallel -N3 cmd1 {1} \> {2} 2\> {3} > > If you have a file with the names of the stdout-names and another file > with the stderr-filenames: > > ... | parallel --xapply cmd1 {1} \> {2} 2\> {3} :::: - > stdoutfilenames stderrfilenames > > Also you may find --joblog useful. I'm a little confused by these, what I'd like is the stdout and stderr to be combined into a single file for each job, where I can supply the name of that file per job (something like {}.log or similar). Doable? Thanks! David
