[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[miss-islington]

miss-islington  added the comment:


New changeset 43ee0e2ca33ae28d37a3dddf10a648895cd46bda by Miss Islington (bot) 
in branch '3.8':
bpo-33166: Change os.cpu_count to return active (real) processors (GH-15949)
https://github.com/python/cpython/commit/43ee0e2ca33ae28d37a3dddf10a648895cd46bda


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Zachary Ware]

Zachary Ware  added the comment:

Thank you for clarifying this muddy topic, Eryk!

(Dropping bpo-32592 dependency; we've done the update in a basically-compatible 
way.)

--
dependencies:  -Drop support of Windows Vista and 7 in Python 3.9
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[miss-islington]

Change by miss-islington :


--
pull_requests: +15612
pull_request: https://github.com/python/cpython/pull/15979

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Zachary Ware]

Zachary Ware  added the comment:


New changeset aa929273caca2f4e24e3aa9e790272fd4458ad35 by Zachary Ware (Steve 
Dower) in branch 'master':
bpo-33166: Change os.cpu_count to return active (real) processors (GH-15949)
https://github.com/python/cpython/commit/aa929273caca2f4e24e3aa9e790272fd4458ad35


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Steve Dower]

Change by Steve Dower :


--
versions: +Python 3.9 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Steve Dower]

Change by Steve Dower :


--
keywords: +patch
pull_requests: +15584
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/15949

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Cheryl Sabella]

Change by Cheryl Sabella :


--
versions:  -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Cheryl Sabella]

Cheryl Sabella  added the comment:

Added #32592 as a dependency since that is removing the Vista code mentioned 
here.  Once that change is merged, then this would be a simpler change to make.

--
dependencies: +Drop support of Windows Vista in Python 3.7
nosy: +cheryl.sabella

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

That makes sense to me. Thanks for deciphering this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Eryk Sun]

Eryk Sun  added the comment:

> if not available fallback on 
> GetActiveProcessorCount(ALL_PROCESSOR_GROUPS)

The fallback for older versions of Windows is dwNumberOfProcessors from 
GetSystemInfo. This can be removed from 3.7 and 3.8, which no longer support 
Windows versions prior to Windows 7.

> GetLogicalProcessorInformationEx() is what should really be used. 

GetActiveProcessorCount and GetMaximumProcessorCount are implemented via 
GetLogicalProcessorInformationEx (i.e. NtQuerySystemInformation, 
SystemLogicalProcessorAndGroupInformation). They query the RelationGroup 
information. For ALL_PROCESSOR_GROUPS, they respectively sum the 
ActiveProcessorCount and MaximumProcessorCount over all groups.

These functions were added in Windows 7 to support the implementation of 
logical processor groups, which allows up to 64 logical processors per group. 
Each process is created in a single group, which is assigned round-robin. A 
thread can call SetThreadGroupAffinity to manually switch to another group.

Apparently someone at Microsoft advised calling GetMaximumProcessorCount (see 
issue 30581), but I don't follow this decision. Why should os.cpu_count() 
include CPUs that may or may not come online? Also, on POSIX it reports 
sysconf(_SC_NPROCESSORS_ONLN), not sysconf(_SC_NPROCESSORS_CONF), so for 
Windows it should instead call GetActiveProcessorCount. I assume on BSD that 
HW_NCPU is similar, though I'm not sure for MacOS. Also on MacOS it appears to 
be deprecated in favor of HW_LOGICALCPU, HW_LOGICALCPU_MAX, HW_PHYSICALCPU, and 
HW_PHYSICALCPU_MAX.


> which may still report the wrong number of CPUs on 32 bit processes.

32-bit Windows and WOW64 emulation are limited to 32 CPUs. Applications that 
need more logical processors should be 64-bit

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

I created a psutil branch using GetLogicalProcessorInformation() to determine 
both logical and physical CPUs:
https://github.com/giampaolo/psutil/pull/1257
According to https://stackoverflow.com/questions/31209256 basically all Windows 
APIs are unreliable and GetLogicalProcessorInformationEx() is what should 
really be used. 
That is available only starting from Windows 7 though so apparently what we 
want is GetLogicalProcessorInformationEx() and if not available fallback on 
GetActiveProcessorCount(ALL_PROCESSOR_GROUPS) which may still report the wrong 
number of CPUs on 32 bit processes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Adding Chris Wilcox who wrote the original patch using GetMaximumProcessorCount.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

By re-reading
https://bugs.python.org/issue30581 and
https://github.com/giampaolo/psutil/issues/771#issuecomment-264457333 I now 
remember why I haven't fixed this issue in psutil yet: because the whole thing 
(MS APIs and doc basically) is confusing.

GetMaximumProcessorCount (now used by os.cpu_count()) returns "the maximum 
number of logical processors that a processor group or the system CAN have", 
not the actual number. That would explain why in OP's case os.cpu_count() 
returns 128 instead of 40.

As per https://bugs.python.org/issue30581#msg295255 dwNumberOfProcessors wasn't 
good because it doesn't take multiple processor groups into account (hence the 
number may be too small) and GetLogicalProcessorInformationEx may be the way to 
go. This is based on the assumption that os.cpu_count() should report the 
number of CPUs in the system (including the non-usable ones, like in case of 
process groups).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[yanir hainick]

yanir hainick  added the comment:

Ok, no problem.
Just to be sure i'm doing the right thing - this thread will be dedicated to 
the os.cpu_count() issue, and i'll open a new issue on the parallelization 
problem.

makes sense?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> Just to be sure i'm doing the right thing - this thread will be dedicated to 
> the os.cpu_count() issue, and i'll open a new issue on the parallelization 
> problem.

Exactly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Let's not conflate different issues.  The parallelization issue is distinct 
from the os.cpu_count() issue (and I'm skeptical Python is at fault there).

--
versions: +Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[yanir hainick]

yanir hainick  added the comment:

Yes. Both are wrong, and os.cpu_count() is completely off.
Regarding how to determine the number of physical/logical cores in my machine - 
well, not sure what you mean by that. I've attached a screenshot of Windows' 
system information. Also used 'cpu-z'.

It seems like aside from the os.cpu_count() issue, Python itself has some 
problem - it 'sees' only 1 CPU group. It is evident from the fact the when 
parallelizing, utilization level is only 25%.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

Oh! So both os.cpu_count() and psutil.cpu_count() are wrong? How do you 
determine the actual number of logical/physical CPUs on your machine?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[yanir hainick]

yanir hainick  added the comment:

Maybe i'm missing something, and would appreciate clarification.

Perhaps psutil is wrong, but it gives an answer that has something to do with 
the actual situation.

On platform 2, i have 2 Intel Xeon Gold 6138, each with 20 physical processors, 
40 logicals.

you are saying i need to rely on os.cpu_count(), which outputs '128'. Can you 
elaborate on this?

Moreover, when attempting to parallelize on the processors, i reach 25% 
utilization, which suggests Python 'sees' only one processor group.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

The difference between os.cpu_count() and psutil.cpu_count() is because one 
uses GetMaximumProcessorCount() and the other dwNumberOfProcessors.

This is tracked as a bug in psutil bug tracker but it's not fixed yet:
https://github.com/giampaolo/psutil/issues/771

As for Python this is where it was discussed and changed:
https://bugs.python.org/issue30581
https://github.com/python/cpython/commit/c67bae04780f9d7590f9f91b4ee5f31c5d75b3c3

In summary: psutil is wrong and you should rely on os.cpu_count().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[yanir hainick]

yanir hainick  added the comment:

Yup.
Attaching a screenshot.

--
Added file: https://bugs.python.org/file47504/Screenshot.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> os.cpu_count() reports 128 units
> psutil.cpu_count(logical=False) reports 20 units
> psutil.cpu_count(logical=True) reports 40 units

You mean os.cpu_count() reports *more* CPUs than exist on the machine? How can 
that happen?

--
nosy: +giampaolo.rodola, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[yanir hainick]

yanir hainick  added the comment:

wrong number of cpu's is reported on some specific platforms.

***

first platform:
server with X4 Intel Xeon E5-4620 (8 physical, 16 logical), running 
a 64bit Windows Server 2012 R2 Standard.
results:
os.cpu_count() reports 64 units
psutil.cpu_count(logical=False) reports 32 units
psutil.cpu_count(logical=True) reports 64 units

multiprocessing using concurrent.futures able to fully utilize the server;

***

second platform:
server with X2 Intel Xeon Gold 6138 (20 physical, 40 logical), running a 64bit 
Windows Server 2016 Standard.
results:
os.cpu_count() reports 128 units
psutil.cpu_count(logical=False) reports 20 units
psutil.cpu_count(logical=True) reports 40 units

multiprocessing using concurrent.futures able to utilize only 1/4 of the 
server's power;

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33166] os.cpu_count() returns wrong number of processors on specific systems

[yanir hainick]

Change by yanir hainick :


--
components: Windows
nosy: paul.moore, steve.dower, tim.golden, yanirh, zach.ware
priority: normal
severity: normal
status: open
type: behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com