Re: [gentoo-user] Any way to run multiple commands from single script in parallel?
On Monday, March 14, 2022 11:51:44 AM CET Björn Fischer wrote: > Hello Joost, > > > Is there a tool/method to execute multiple lines/commands > > simultaneously? Like having 3 or 4 run together and when 1 is > > finished, it will grab the next one in the list? > > probably, GNU Parallel is what you are looking for: > > https://www.gnu.org/software/parallel/parallel.html#examples > > The tool can handle most variants of batch processing scenarios without > the steep learning curve of fully fledged (clustered) job schedulers. Thanks, this seems to do what I need it to do.
RE: [gentoo-user] Any way to run multiple commands from single script in parallel?
If you don't want to do thread management yourself in bash then you can use
something like GNU Parallel (in the repo) to handle forking and collating
processes for you.
Parallel in particular has the additional advantage that it's capable of
shipping tasks off to other machines via SSH, so if you get to the point where
you need a whole cluster to do your processing it's just a matter of adding a
couple of arguments.
LMP
-Original Message-
From: Ramon Fischer
Sent: Monday, March 14, 2022 3:37 AM
To: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] Any way to run multiple commands from single script
in parallel?
Hello Joost,
I suppose, that you are talking about Bash scripts.
If so, you may put each individual command in a subshell by using an ampersand
("&") at the end of the line.
This example[1] shows it nicely.
-Ramon
[1] 3. Parallelize running commands by grabbing PIDs.:
https://will-keleher.com/posts/5-Useful-Bash-Patterns.html
On 14/03/2022 11:13, J. Roeleveld wrote:
> Hi,
>
> I often put multiple commands into a single file/script to be run in sequence.
> (each line can be executed individually, there is no dependency)
>
> Is there a tool/method to execute multiple lines/commands
> simultaneously? Like having 3 or 4 run together and when 1 is
> finished, it will grab the next one in the list?
>
> I would prefer this over simply splitting the file as the different
> lines/ commands will not take the same amount of time.
>
> Thanks,
>
> Joost
>
>
>
--
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF
Re: [gentoo-user] Any way to run multiple commands from single script in parallel?
On Mon, 14 Mar 2022 11:13:13 +0100
"J. Roeleveld" wrote:
> Hi,
>
> I often put multiple commands into a single file/script to be run in sequence.
> (each line can be executed individually, there is no dependency)
>
> Is there a tool/method to execute multiple lines/commands simultaneously? Like
> having 3 or 4 run together and when 1 is finished, it will grab the next one
> in
> the list?
>
> I would prefer this over simply splitting the file as the different lines/
> commands will not take the same amount of time.
>
> Thanks,
>
> Joost
>
>
>
At the end there's a very rudimentary bash script to do this. I did not
do much debugging (probably it fails already if max_jobs>#list_of_jobs).
Anyway it's just making use of sending jobs to the background and
"communicating" through a FIFO pipe (which you might want to delete at
the end).
#!/bin/bash
list_of_jobs=("sleep 3" "sleep 5" "sleep 1" "sleep 10" "sleep 4")
max_jobs=2
my_fifo=/tmp/my_job_fifo
write_to_fifo="yes"
function run_job () {
eval "$@"
if [[ $write_to_fifo == "yes" ]]; then
echo "Writing to fifo ($@)"
echo 1 > ${my_fifo}
fi
echo Finished job "$@"
}
function read_and_start_job() {
next_job_idx=0
while [[ ${#list_of_jobs[@]} -gt $next_job_idx ]]; do
while IFS= read -r line ; do
echo "next_job_idx=${next_job_idx} total=${#list_of_jobs[@]}"
if [[ $next_job_idx -lt ${#list_of_jobs[@]} ]] ; then
job="${list_of_jobs[${next_job_idx}]}"
echo "Executing: ${job}"
run_job ${job} &
let next_job_idx++
else
echo "Set write_to_fifo=no"
write_to_fifo="no"
fi
done < ${my_fifo}
done
write_to_fifo="no"
wait
}
rm -Rf ${my_fifo}
mkfifo ${my_fifo}
read_and_start_job &
while [[ ${max_jobs} -gt 0 ]] ; do
let max_jobs--
echo 1 > ${my_fifo}
done
wait
Re: [gentoo-user] Any way to run multiple commands from single script in parallel?
Hello Joost, Is there a tool/method to execute multiple lines/commands simultaneously? Like having 3 or 4 run together and when 1 is finished, it will grab the next one in the list? probably, GNU Parallel is what you are looking for: https://www.gnu.org/software/parallel/parallel.html#examples The tool can handle most variants of batch processing scenarios without the steep learning curve of fully fledged (clustered) job schedulers. Cheers, Björn
Re: [gentoo-user] Any way to run multiple commands from single script in parallel?
Hello Joost,
I suppose, that you are talking about Bash scripts.
If so, you may put each individual command in a subshell by using an
ampersand ("&") at the end of the line.
This example[1] shows it nicely.
-Ramon
[1] 3. Parallelize running commands by grabbing PIDs.:
https://will-keleher.com/posts/5-Useful-Bash-Patterns.html
On 14/03/2022 11:13, J. Roeleveld wrote:
Hi,
I often put multiple commands into a single file/script to be run in sequence.
(each line can be executed individually, there is no dependency)
Is there a tool/method to execute multiple lines/commands simultaneously? Like
having 3 or 4 run together and when 1 is finished, it will grab the next one in
the list?
I would prefer this over simply splitting the file as the different lines/
commands will not take the same amount of time.
Thanks,
Joost
--
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF
OpenPGP_signature
Description: OpenPGP digital signature
