New submission from M J Harvey <>:


multiprocessing's default assumption about Pool size is os.cpu_count() ie all 
the cores visible to the OS.

This is tremendously unhelpful when running multiprocessing code inside an HPC 
batch system (PBS Pro in my case), as there's no way to hint to the code that 
the # of cpus actually allocated to it may be fewer.

It's quite tedious to have to explain this to every single person trying to use 

Proposal: multiprocessing should look for a hint for default Pool size from the 
environment variable "NCPUS" which most batch systems set. If that's not set, 
or its value is invalid, fall back to os.cpu_count() as before

components: Library (Lib)
messages: 313150
nosy: M J Harvey
priority: normal
severity: normal
status: open
title: multiprocessing, default assumption of Pool size unhelpful
versions: Python 3.6

Python tracker <>
Python-bugs-list mailing list

Reply via email to