Hi, Am 06.09.2012 um 11:57 schrieb Schmidt U.:
> I set complex h_vmem consumable with default 2.0G and per exec host the > physical limit to 46G, as well virtual_free (vf) is set consumable, default > 1G and per exec host is as well 46G. Why do you request both? One of them should be sufficient. > Some users have advantage of using openmp and openmpi mixed in their jobs. To > reduce the waste of allocated memory and to reduce the "qw" time of the jobs > I have to understand the behavior of the gridengine in mixed parallel jobs. > First of all h_vmem is per slot ( as well per core ?) Yes. > In case of openmp threads which are not part of the $NSLOTS, must be the > amount of RAM multiplied with OMP_NUM_THREADS or not ? You mean, you start more threads than requested in `-pe smp <N>`? Also an Open MP job is a parallel job and should be submited in this way. For Open MP jobs it's not a problem: on the master node of a parallel job the requested resource (memory) for the jobscript will be multiplied by the amount of slots granted on this machine. If you use `qrsh -inherit <masterhost> ...` in addition, each will get an additional amount of memory granted. Nevertheless you can't oversubscribe the machine by this, as the overall consumption is still monitored by SGE's interal facility and the job killed if you do so. Ex.: qsub -pe smp 4 -l h_vmem=1G => master process is granted 4G. On the slave nodes the problem is, that each `qrsh -inherit <slavehost> ...` will only get a not-multiplied amount set, which is most likely not what you want, in case you want to do a one-time `qrsh -inherit <slavehost> ...` and use threads there. https://arc.liv.ac.uk/trac/SGE/ticket/197 > Secondly, if the user defines in the job as well vf and h_vmem, what > consumable is used by the gridengine to find suitable machines to start the > jobs ? Without > h_vmem, the vf is used this I now and is written in the documentation, but > when I have defined h_vmem as well is then vf ignored ? No, both contrains must be met. > pe: > pe_name openmp_6 > slots 3168 > user_lists standard > xuser_lists NONE > start_proc_args /bin/true > stop_proc_args /bin/true > allocation_rule 6 > control_slaves TRUE > job_is_first_task FALSE > urgency_slots min > accounting_summary FALSE > > job script: > #$ -N test > #$ -cwd > #$ -o $JOB_ID.out > #$ -e $JOB_ID.err > #$ -l h_rt=150:00:00 > #$ -l vf=2.3G > #$ -l h_vmem=3G > #$ -pe openmp_6 72 > export OMP_NUM_THREADS=6 > export MKL_NUM_THREADS=($OMP_NUM_THREADS) > mpirun --mca btl openib,self -pernode -np 12 /my_mixed_job With this you will get 12 machines, and on each you can use 6 threads. As all (threads on a machine) will work on the same memory, this shouldn't be a problem. But you are using MKL with $OMP_NUM_THREADS too, which could create locally 36 processes as a result. Therefore I use unthreaded versions of MKL/ACML/ATLAS usually. -- Reuti > Udo > -- > Udo Schmidt > > Max-Planck-Institut für Mikrostrukturphysik > Weinberg 2, 06120 Halle (Saale), Germany > http://www.mpi-halle.de/~theory > > Phone: +49 345 5582541 Fax: +49 345 5582765 Email: > [email protected] > _______________________________________________ > users mailing list > [email protected] > https://gridengine.org/mailman/listinfo/users _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
