[Users] Einstein Toolkit Meeting Reminder

[users]

Hello,

Please consider joining the weekly Einstein Toolkit phone call at
9:00 am US central time on Thursdays. For details on how to connect
and what agenda items are to be discussed, use the link below.

https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call

--The Maintainers
___
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users

Re: [Users] Einstein toolkit with Sun Grid Engine

[Chris Stevens]

Thanks for this. I have investigated changing the parallel environment, but it 
is not clear how to tell it that each MPI proc has access to n cores. At this 
stage if I request slots for 2 nodes it will allocate them in SGE correctly, 
but everything will run on the one node.

I'll thus try your suggestion and let you know if I run into any other 
difficulties.

Thanks,

Chris


[cid:344dcf4a-7681-4cd8-9c6b-2b279825d64c]


[cid:2111bace-e465-4801-b0ee-9e2ace60acba]



Dr Chris Stevens

Lecturer in Applied Mathematics

Rm 602, Jack Erskine building

School of Mathematics and Statistics

T: +64 3 369 0396 (Internal 90396)

University of Canterbury | Te Whare Wānanga o Waitaha

Private Bag 4800, Christchurch 8140, New Zealand

http://www.chrisdoesmaths.com


Director
SCRI Ltd
http://www.scri.co.nz



From: Roland Haas
Sent: Saturday, October 09, 2021 11:13
To: Erik Schnetter
Cc: Chris Stevens; Users@einsteintoolkit.org
Subject: Re: [Users] Einstein toolkit with Sun Grid Engine

Hello Chris,

the "way"-ness was a TACC thing I believe.

Occasionally, in particular in old files, you will see constructs such
as:

uniq ${PBS_NODEFILE} > ${MPD_NODEFILE}
for node in $(cat ${MPD_NODEFILE}); do
for ((proc=0; $proc<@(@PPN_USED@ / @NUM_THREADS@)@; proc=$proc+1)); do
echo ${node}
done
done > ${MPI_NODEFILE}

mpirun -np @NUM_PROCS@ -machinefile ${MPI_NODEFILE} @EXECUTABLE@ -L 3 @PARFILE@

ie one constructs a custom MPI host list file that manually lists the
nostname as many times as needed to start the correct number of MPI
ranks on the host.

SGE has a similar variable PE_HOSTFILE and if all else fails you can
likely do the same thing replace PBS_NODEFILE by PE_HOSTFILE

Yours,
Roland

> Chris
>
> I am unfamiliar with the details of SGE; I cannot tell whether this
> approach makes sense.
>
> -erik
>
>
> On Thu, Oct 7, 2021 at 5:19 PM Chris Stevens 
> wrote:
>
> > Hi Erik,
> >
> > Thanks for your suggestion.
> >
> > I am happy using these in the scripts, but I think the problem is how to
> > pass these expressions to SGE. From what I can tell, the output of
> > @(@PPN_USED@/@NUM_THREADS@)@way is, for example, "6way", given @PPN_USED@=48
> > and @NUM_THREADS@=8. This means that I have requested the parallel
> > environment called 6way with @PROCS_REQUESTED@ slots. If I requested 48
> > slots, then I would use mpirun -np 6. Thus, from what I gather, for this to
> > work, this specific parallel environment 6way needs to exist. I am now
> > figuring out how to configure parallel environments in such a way, most
> > likely by changing the allocation rule.
> >
> > Let me know if you think this is wrong, as it does seem rather stupid to
> > not be able to just set -ncpus-per-task like in Slurm in the submission
> > script.
> >
> > Cheers,
> >
> > Chris
> >
> >
> >
> >
> >
> >
> >
> >
> > *Dr Chris Stevens*
> >
> > *Lecturer in Applied Mathematics*
> >
> > Rm 602, Jack Erskine building
> >
> > School of Mathematics and Statistics
> >
> > T: +64 3 369 0396 (Internal 90396)
> >
> > University of Canterbury | Te Whare Wānanga o Waitaha
> >
> > Private Bag 4800, Christchurch 8140, New Zealand
> >
> > https://urldefense.com/v3/__http://www.chrisdoesmaths.com__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_vsMaonm$
> >
> > *Director*
> > SCRI Ltd
> > https://urldefense.com/v3/__http://www.scri.co.nz__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_kjY9Edc$
> >
> > --
> > *From:* Erik Schnetter 
> > *Sent:* 08 October 2021 09:40
> > *To:* Chris Stevens 
> > *Cc:* users@einsteintoolkit.org 
> > *Subject:* Re: [Users] Einstein toolkit with Sun Grid Engine
> >
> > Chris
> >
> > It might not be necessary to hard-code the number of threads. You can use
> > arbitrary Python expressions via "@( ... )@" in the templates. See e.g. the
> > template for Blue Waters which uses this to choose between CPU and GPU
> > queues.
> >
> > -erik
> >
> >
> > On Thu, Oct 7, 2021 at 4:04 PM Chris Stevens <
> > chris.stev...@canterbury.ac.nz> wrote:
> >
> > Hi Roland,
> >
> > That's fantastic, thanks for linking those files.
> >
> > It works as expected with only MPI processes. I am careful in compiling
> > and running with the same (and only) OpenMPI installation on the cluster,
> > so this should be OK.
> >
> > Finding a Slurm to SGE conversion table, there is no SGE equivalent to
> > ncpus-per-task from Slurm, rather it is the allocation type of the given
> > parallel environment that does this. I.e. the backend.
> >
> > https://urldefense.com/v3/__https://srcc.stanford.edu/sge-slurm-conversion__;!!DZ3fjg!uIkk67wRbCsh3zElvpI9BgBa5DFSTT4-ejwsw-X1jEDWdsxqLIqqJTCL_iEcWKp1$
> >
> > Further, in the submit script of ranger, the crucial line
> >
> > #$ -pe @(@PPN_USED@/@NUM_THREADS@)@way @PROCS_REQUESTED@
> >
> > shows that you request @PROCS_REQUESTED@ slots (as I currently have) and
> > the