Hi all,
I've been experimenting with task prolog/epilog and noticed that task
epilogs have significant overhead. The overhead seems to be independent
of the epilog script content:
$ cat test.sh #!/bin/bash $ time srun --overlap
--task-prolog=$PWD/test.sh --ntasks=64 sleep 1 real 0m1.730s user
0m0.211s sys 0m0.009s $ time srun --overlap --task-prolog=$PWD/test.sh
--task-epilog=$PWD/test.sh --ntasks=2 sleep 1 real 0m3.530s user
0m0.213s sys 0m0.007s $ time srun --overlap --task-prolog=$PWD/test.sh
--task-epilog=$PWD/test.sh --ntasks=64 sleep 1 real 1m5.936s user
0m0.216s sys 0m0.005s
This test was on one node, with 64 tasks in the allocation. The overhead
seems linked only to number of tasks per node, as adding additional
nodes did not increase the overhead. I saw this on
$ srun --version
slurm 23.11.10-BullSequana.1.2.1
and someone else reproduced similar results with |slurm 25.05.7|.
I can live with the overhead, but I am curious as to why it happens and
to understand the design of the implementation. Does the task epilog
script only run when all tasks finish, or does it run immediately at
exit for each task? Are the calls serialised? My use case is a
fusermount that I want to mount in userspace at task start and clean up
at task finish, one task per node is responsible for the mount/unmount
and I want to understand if I have a race condition for the unmount.
Any advice very much appreciated,
Alan
--
slurm-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
