On 17/07/2014 14:07, Satrajit Ghosh wrote:
Re: [slurm-dev] python multiprocessing job on SLURM
This should give you the correct number:
import os
print os.environ['SLURM_JOB_CPUS_PER_NODE']
cheers,
satra
On Thu, Jul 17, 2014 at 4:56 AM, Sheila the angel
<[email protected] <mailto:[email protected]>> wrote:
Hello All,
I am new to SLURM and trying to run Multiprocessing job via python.
import multiprocessing
print multiprocessing.cpu_count()
The number of CPU shown by Python is always same (16) even if I
reserve only 4 CPU.
Does anyone have experience on running Multiprocessing job via
python on SLURM?
How can I run a python multiprocessing job efficiently on SLURM?
Thanks,
--
Best,
Sheila
Shelia,
By default the python multi processing module will use all
the cpus it detects so as hinted above take the slurm environment
variable and pass that to the multiprocessing module to set the amount
of threads you wish to use so a very basic example could be :
import math
import multiprocessing as mp
import os
def f2(x):
return reduce(lambda a, b: math.log(a+b), xrange(10**5), x)
if __name__ == '__main__':
SLURM_CPUS = int(os.environ['SLURM_JOB_CPUS_PER_NODE'])
print "Using %s" % SLURM_CPUS
pool = mp.Pool(SLURM_CPUS)
result = pool.map(f2, range(1,10000))
Regards
Mark
