Hi, I have a master script that executes two sequences (lists) of child
scripts, i.e. script_1 to script_3, and script_4 to_script_6 (the structure
is attached as a png file).

The execution is sequential, e.g. running script_1, then 2 then 3.

After executing the 1st sequence (script_1 to 3), master will execute the
2nd sequence (script_4 to 6).

Each child script will be calling a multiprocessing function to process a

Master script is like,

for seq in seqs_to_launch:

    for script in seq:


Each child script is like,

import multi_process_update

class Script_n():

    def execute(self, data):
        # some data processing

        multi_process_task(data_task, task_name)

The multiprocessing function is like,

def multi_process_task(tasks, task_name):

    cores_to_use = how_many_core()

    handler = task_handling(task_name)  # task handling class

    task_blocks = slice_list(tasks, cores_to_use)

    for block in task_blocks:
        # spawn processes for each row block assigned to every cpu core
        p = multiprocessing.Process(target=handler.execute, args=(block,))

I like to know how to notify the master when the 2nd sequence is finished
that all multiprocess tasks (seq 1 and 2) are completed, and then maybe
close the multiprocessing.


Reply via email to