Hello, I was hoping to get some guidance on a mistake that I am making in rewriting a for loop to perform the iterations of the loop in parallel using GNU Parallel. I have changed the for loop to the set-up that I found on many examples online, but I'm afraid that the syntax is not correct still, and I am still getting errors.
When running my current edited script, I am getting errors saying "no
file/directory *Name of file* exists" meaning that the commands that are
supposed to create these files and directories are not being run properly. This
leads me to believe that lines 15 - 19 of my edited (parallel) script
For_loop_parallel.sh (attached) are not functioning properly.
On the man page, I examined the example they provided:
for x in `cat list` ; do
no_extension=${x%.*};
do_something $x scale $no_extension.jpg
do_step2 <$x $no_extension
done) | process_output
and how they changed it to this:
cat list | parallel "do_something {} scale {.}.jpg ; do_step2 <{} {.}" |
process_output
But in that example, I do not understand how the parallel command will "know"
to iterate through the x variable, as that disappeared in the parallel version.
I believe my problem in my rewritten script just stems from my lack of
understanding of how to tell GNU parallel what to iterate from. In my for loop,
I am trying to iterate through each line using variable $line through a file
called $fName (the Looped_through_file.txt), but currently don't see a way of
including that in proper syntax for the parallel command. If you open up both
of the For_loop scripts, you will see how I attempted to change the syntax of
the (fully functioning) for_loop_regular script into the for_loop_parallel
script based on the online examples.
I did not include the large file that one of the commands references, as it is
extremely large, but I did include the small file that is being referenced and
is actually being looped through.
Any advice on properly rewriting my for loop so that the subsequent iterations
can be sent to different cores will be very helpful. I have yet to see an
example that maintains the iterated variable in the parallel command and do not
understand how parallel can work without that information. However, I am new to
Linux commands and programming in general, so perhaps there is some large
aspect of Linux capabilities/syntax that I am unaware of.
Thank you,
Lauren
For_loop_parallel.sh
Description: For_loop_parallel.sh
For_loop_regular.sh
Description: For_loop_regular.sh
Chr Band Start End 1 1p13.3 108.623405 108.723405 2 2p23.1 31.361287 31.461287 5 5q22.1 110.356742 110.456742 5 5q23.1 116.131428 116.231428 5 5q23.2 125.983906 126.083906 8 8p23.2 4.92478 5.02478 8 8p23.1 10.908539 11.008539 8 8q24.23 139.287546 139.387546 10 10p12.31 20.815157 20.915157 11 11q13.5 76.249194 76.349194 11 11q14.2 86.054495 86.154495 15 15q13.3 31.488542 31.588542 17 17q25.3 76.843135 76.943135 21 21q22.3 45.028556 45.128556 22 22q11.21 21.332482 21.432482 2 AS_rs10189629 102.829464 102.929464 2 AS_rs10497813 198.864072 198.964072 3 AS_rs9860547 188.078979 188.178979 4 AS_rs2101521 38.761551 38.861551 4 AS_rs17388568 123.279362 123.379362 5 AS_rs7720838 40.436896 40.536896 5 AS_rs1438673 110.417499 110.517499 6 AS_rs9266772 31.302113 31.402113 6 AS_rs6906021 32.576311 32.676311 8 AS_rs6473223 81.218155 81.318155 9 AS_rs7032572 6.12238 6.22238 10 AS_rs962993 9.003132 9.103132 11 AS_rs2155219 76.249194 76.349194 11 AS_rs10893845 128.136882 128.236882 14 AS_rs1998359 38.027148 38.127148 15 AS_rs17228058 67.400305 67.500305 16 AS_rs7203459 11.180703 11.280703 16 AS_rs2107357 27.360829 27.460829 17 AS_rs9303280 38.024031 38.124031 20 AS_rs6021270 50.091264 50.191264
