[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
Changes by Antoine Pitrou pit...@free.fr: -- resolution: - out of date stage: patch review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
Antoine Pitrou added the comment: The current os.cpu_count implementation calls sysconf(_SC_NPROCESSORS_ONLN), which is apparently defined under OS X, and returns the number of online CPUs (logical?): https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/sysconf.3.html multiprocessing has been modified to re-use os.cpu_count(), so I suggest closing this issue as out-of-date. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
Yury Selivanov added the comment: bump? -- nosy: +yselivanov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
STINNER Victor added the comment: I like the idea of a new function in the os module because I don't like having to import the multiprocessing module just to known the number of CPUs. I'm using such function to set MAKEFLAGS envrionment variable on Linux: -j8 par example. 2013/3/19 Trent Nelson rep...@bugs.python.org: Trent Nelson added the comment: I remember looking at what multiprocessing did and not really liking it; I ended up writing a C version that works across a wider range of platforms, accessible via posixmodule.c:posix_cpu_count() (os.cpu_count()): http://hg.python.org/sandbox/trent/file/dd1c2fd3aa31/Modules/posixmodule.c#l10213 -- nosy: +trent ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
Ronald Oussoren added the comment: I also like the os.cpu_count() function, the information is useful sometimes outside of multiprocessing, and calling out to external scripts to gather the information (as multiprocessing currently does) feels yucky. That should probably be a new issue, the change in this issue fixes a real problem (the cpu count code in multiprocessing can overestimate the usable CPU count on OSX) and is a bugfix that should be backported to the stable branches. BTW. Trent's os.cpu_count implementation also uses hw.ncpu and is therefore also broken on OSX. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
John Szakmeister added the comment: Actually, Trent's version looks at hw.logicalcpu and then falls back to hw.ncpu, if there was an error. Given the state of the documentation on these parameters, it's hard to say whether it's right or wrong, but at least hw.logicalcpu scales correctly if I disable some of the processors. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
Trent Nelson added the comment: On Tue, Mar 19, 2013 at 01:58:59AM -0700, John Szakmeister wrote: John Szakmeister added the comment: Actually, Trent's version looks at hw.logicalcpu and then falls back to hw.ncpu, if there was an error. Given the state of the documentation on these parameters, it's hard to say whether it's right or wrong, but at least hw.logicalcpu scales correctly if I disable some of the processors. That's pretty much the rationale I used. I tested the fallback on OS X manually (i.e. the _bsd_cpu_count()), and that works, and the hw.logicalcpu definitely works in the first place, so, I figured it was good enough. I'll raise a new issue for os.cpu_count(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
Ronald Oussoren added the comment: I'm not sure if hw.availcpu is the right value to use as it is not documented at all (neither in a manpage, nor in a headerfile). hw.activecpu seems to be the one that should be used: it is documented as The number of processors currently available for executing threads. in the sysctl.h header file and that comment also mentions that it should be used to determine the amount of threads to start in an SMP application. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
John Szakmeister added the comment: Ronald: it is mentioned in some books (a Google search can turn them up), but they don't really offer much description behind the intent. When I looked into this several years ago, it was very unclear what `hw.activecpu` was intended for. It sounded more like a report about how many processors are active, versus targetting your SMP aware application to that number. But since you've turned some information in sysctl.h, I think we should follow that advice and use hw.activecpu. I've attached a new patch with the change. -- Added file: http://bugs.python.org/file29442/use-activecpu.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
STINNER Victor added the comment: Here is an interesting, but old (2007), email on darwin-dev: http://lists.apple.com/archives/darwin-dev/2007/Jun/msg00088.html This can all change in the future, but currently: hw.ncpu is a wart; consider it to be deprecated. hw.physicalcpu is the number of physical CPUs hw.logicalcpu is the number of logical CPUs; this is for SMT, which we don't support (maybe T1s?) hw.availcpu are the number logical CPUs currently online These interfaces are evolving, however, you are unlikely to get a description of these. They are intended for internal ibrary use, and not for use by applications, since applications should use the library abstractions rather than trying to use this information directly themselves. By the way, multiprocessing should use subprocessing directly, not os.popen() (but this is a different issue). -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
Trent Nelson added the comment: I remember looking at what multiprocessing did and not really liking it; I ended up writing a C version that works across a wider range of platforms, accessible via posixmodule.c:posix_cpu_count() (os.cpu_count()): http://hg.python.org/sandbox/trent/file/dd1c2fd3aa31/Modules/posixmodule.c#l10213 -- nosy: +trent ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
New submission from John Szakmeister: While trying to test a fix for Nose, I discovered that multiprocessing is picking up the CPU count incorrectly. It should be using hw.availcpu instead of hw.ncpu. The latter is the number of cpus installed in the system, but the former is the number that are available for processing. The processor pane let's you adjust the available CPUs, which is handy for testing and troubleshooting. -- components: Library (Lib) files: use-availcpu.patch keywords: patch messages: 184367 nosy: jszakmeister priority: normal severity: normal status: open type: behavior Added file: http://bugs.python.org/file29430/use-availcpu.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
Changes by Richard Oudkerk shibt...@gmail.com: -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
Changes by Antoine Pitrou pit...@free.fr: -- stage: - patch review versions: +Python 2.7, Python 3.2, Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17444] multiprocessing.cpu_count() should use hw.availcpu on Mac OS X
Changes by Ned Deily n...@acm.org: -- nosy: +ned.deily, ronaldoussoren ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17444 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
