Panayotis, Did you try using --procs=16 --num-threads=1? I never used --cores and I don't know the difference with procs.
Cheers, Bruno Il giorno gio 25 set 2025 alle ore 17:29 IOSIF PANAGIOTIS < [email protected]> ha scritto: > Hi all, > > I am bumping this follow-up question regarding inconsistency warnings when > requesting resources during a job submit, in case someone has some > suggestion. > > Thanks, > Panayotis > > ------------------------------ > *From:* IOSIF PANAGIOTIS <[email protected]> > *Sent:* Thursday, September 18, 2025 3:25 PM > *To:* Roland Haas <[email protected]> > *Cc:* Einstein Toolkit Users <[email protected]> > *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch > [Leonardo cluster] > > Dear Roland, > > I am following up on my question about how to correctly request the > resources that I need. > > Here is a reminder of the context: > > - I have installed ETK on the Leonardo, DCGP partition at CINECA > - I want to run a TOV star example requesting a different number of > cores each time (e.g. 16, 32, 64 etc) to better understand the timescales. > > > - > > I use this command to submit the simulation (sim is the typical alias for > "./simfactory/bin/sim"): > > - sim create-submit tov_test --parfile par/tov_ET.par --cores 16 > --num-threads 1 --walltime 00:20:00 > > > The config files I use are here, and they are based on Bruno Giacomazzo's > originals: > > - https://github.com/piosif/einstein-toolkit-config-leonardo/ > > > I get a *weird error from the CarpetLib thorn*, which complains that: > > > - *The grid structure is inconsistent. It is impossible to continue.* > > > I have attached the simulation .err and .out files. > > FWIW, inspecting the very beginning of the two files, it seems that the > overall setup I used (config files and submit command) results in: > > > - MPI processes: 16 > - OpenMP threads per core: 0.14285714285714285 > > > This is 16 divided by 112 (the total cores in each node), and it does not > seem to be what I want. > I would expect that each OpenMP thread should be assigned to one core. > Furthermore, for that simple test and if my understanding of the basics of > MPI and OpenMP is correct, it would make more sense to have just one MPI > process and 16 OpenMP threads. > > Could you help me understand what is wrong with my submit command above in > connection to the options specified in my config files? > > Best, > Panayotis > > ------------------------------ > *From:* Roland Haas <[email protected]> > *Sent:* Wednesday, September 10, 2025 5:59 PM > *To:* IOSIF PANAGIOTIS <[email protected]> > *Cc:* Einstein Toolkit Users <[email protected]> > *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch > [Leonardo cluster] > > Hello Panayotis, > > "procs" as used by simfactory is (for historical reasons) a misleading > term > ( > https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html > ). > > Really "procs" is the total number of threads that will be started, > which usually will be the same as total number of (logical) cpus being > used, and again usually the same as the number of (phyisal) cores used. > > If using --cores 16 you are asking for a total of 16 threads to be > created which should be assigned to 16 cores. You should then select a > number of threads-per-rank using the `--num-threads` option that is > good for your setup. For 16 cores most likely `--num-threads 1` (ie one > thread per MPI rank) is best. This will most likely fix your issue. > > Note that this will leave some cores unused and a number of simfactory > machine description files will not handle that case gracefully since > they assume that you will always use at least one full node. > > For you questions: > > * this is usually more of an issue with the SubmitScript and RunScript > rather than the ini file and the simfactory command line, though > in your case the init script has `num-threads = 56` which sets the > default number of for `--num-threads` to 56, which is a bit high (but > is overruled by a --num-threads option on your command line) > * yes, specifying --num-threads on the command line will override the > value in the ini file > > Yours, > Roland > > > > [CAUTION: Non-UBC Email] > > > > Hi all, > > > > After having installed the toolkit on Leonardo, CINECA, I wanted to try > the TOV examples. > > > > However, I am getting the following warnings: > > Warning: Too many threads per process specified: specified > num-threads=56 (ppn-used is 112) > > Warning: Total number of threads and number of threads per process are > inconsistent: procs=16, num-threads=56 (procs*num-smt must be an integer > multiple of num-threads) > > Warning: Total number of threads and number of cores per node are > inconsistent: procs=16, ppn-used=112 (procs must be an integer multiple of > ppn-used) > > From those, it seems that I am doing something wrong either in my > machine.ini file or in the resources request when submitting the job. > > > > Here is the command I use to run the TOV example: > > ./simfactory/bin/sim submit tov_ET --parfile par/tov_ET.par --cores 16 > --walltime 00:20:00 > > > > And here is the ini file I am using (the "Cluster characteristics" > section is identical to Bruno Giacomazzo's file< > https://bitbucket.org/simfactory/simfactory2/commits/0d4011bf554ee695f9543e048ae9ff5edc5b5c98 > >: > > > > > https://github.com/piosif/einstein-toolkit-config-leonardo/blob/main/leonardo-dcgp1.ini > > > > The job ran and completed fine, but from what I understand, it ran on 56 > cores instead of the 16 that I requested. > > > > From what I understand, the problem is that I am requesting just 16 > cores for this simple test, but due to the cluster specs in the ini file, > SimFactory expects me to request a multiple of 56. > > Is that so? > > > > So, my question is the following: > > > > * > > I want to do some tests, running a single star e.g. on a single node > using 16, 32, 64 cores (to understand the timescale required). Do I need to > change my ini file? > > * > > Or is there some option to specify when submitting the job that will > override the cluster specs? > > > > I read the simfactory's terminology explanation on nodes, threads etc< > https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html>, > but it is not clear to me how to request the cores I need. > > > > Best, > > Panayotis > > > -- > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from http://pgp.mit.edu . > _______________________________________________ > Users mailing list > [email protected] > http://lists.einsteintoolkit.org/mailman/listinfo/users > -- Prof. Bruno Giacomazzo Deputy Director of the Department of Physics University of Milano-Bicocca Piazza della Scienza 3 20126 Milano Italy email: [email protected] phone: (+39) 02 6448 2321 web: http://www.brunogiacomazzo.org --------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ----------------------------------------------------------------------
_______________________________________________ Users mailing list [email protected] http://lists.einsteintoolkit.org/mailman/listinfo/users
