[Distutils] Status update on the NumPy & SciPy vs SSE problem?

[Nick Coghlan]

While the manylinux PEP brings Linux up to comparable standing with
Windows and Mac OS X in terms of distributing wheel files through
PyPI, that does mean it still suffers from the same problem Windows
does in relation to NumPy and SciPy wheels: no standardisation of the
SSE capabilities of the machines.

I figured that was independent of the manylinux PEP (since it affects
Windows as well), but I'm also curious as to the current status (I
found a couple of apparently relevant threads on the NumPy list, but
figured it made more sense to just ask for an update rather than
trusting my Google-fu)

Cheers,
Nick.

P.S. I'm assuming the existing ability to publish NumPy & SciPy wheels
for Mac OS X is based on Apple's tighter control of their hardware
platform.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Status update on the NumPy & SciPy vs SSE problem?

[Antoine Pitrou]

On Thu, 4 Feb 2016 21:22:32 +1000
Nick Coghlan  wrote:
> 
> I figured that was independent of the manylinux PEP (since it affects
> Windows as well), but I'm also curious as to the current status (I
> found a couple of apparently relevant threads on the NumPy list, but
> figured it made more sense to just ask for an update rather than
> trusting my Google-fu)

While I'm not a Numpy maintainer, I don't think you can go much further
than SSE2 (which is standard under the x86-64 definition).

One factor is support by the kernel. The CentOS 5 kernel doesn't
seem to support AVX, so you can't use AVX there even if your processor
supports it (as the registers aren't preserved accross context
switches). And one design point of manylinux is to support old Linux
setups... (*)

There are intermediate ISA additions between SSE2 and AVX (additions
that don't require OS support), but I'm not sure they help much on
compiler-vectorized code as opposed to hand-written assembly.  Numpy's
pre-compiled loops are typically quite straightforward as far as I've
seen.

One mitigation is to delegate some operations to an optimized library
implementing the appropriate runtime switches: for example linear
algebra is delegated by Numpy and Scipy to optimized BLAS and LINPACK
libraries (which exist in various implementations such as OpenBLAS or
Intel's MKL).

(*) (this is an issue a JIT compiler helps circumvent: it generates
optimal code for the current CPU ;-))

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Status update on the NumPy & SciPy vs SSE problem?

[Nathaniel Smith]

On Feb 4, 2016 3:22 AM, "Nick Coghlan"  wrote:
>
> While the manylinux PEP brings Linux up to comparable standing with
> Windows and Mac OS X in terms of distributing wheel files through
> PyPI, that does mean it still suffers from the same problem Windows
> does in relation to NumPy and SciPy wheels: no standardisation of the
> SSE capabilities of the machines.
>
> I figured that was independent of the manylinux PEP (since it affects
> Windows as well), but I'm also curious as to the current status (I
> found a couple of apparently relevant threads on the NumPy list, but
> figured it made more sense to just ask for an update rather than
> trusting my Google-fu)

I'm not entirely sure what the SSE status is of the numpy OSX wheels. I
think that they may be just following Apple's guidance on this (in the
sense of: we tell their compiler to target a certain OS version and then
use default options beyond that), but I'm not sure. It may even differ
between the 32- and 64-bit "parts" of the fat binaries. Asking on
numpy-discussion might net more details.

Otherwise, yeah, the current plan is to jump to SSE2 as the minimum
required version as the new wheels become usable, since all the evidence
seems to say that it's ubiquitous now.

> P.S. I'm assuming the existing ability to publish NumPy & SciPy wheels
> for Mac OS X is based on Apple's tighter control of their hardware
> platform.

Not particularly. It's based on (a) Linux wheels aren't allowed on pypi
(modulo bugs -- see pypi issue #385), (b) windows wheels are impossible
because on that platform there's no F/OSS-compatible toolchain that can
build cpython-abi-compatible BLAS or scipy. So OSX is what's left after all
the competitors shot themselves in the foot :-)

(manylinux is the solution to (a); mingwpy.github.io is the solution to
(b).)

Once that stuff is solved then yeah, it might be nice to have some better
solution to the problem of ISA variations. But the most important part is
already handled by runtime cpu sniffing in the blas library, and for the
rest there are a variety of possible solutions (doing our own cpu sniffing,
or adding something to wheels, or etc.), and just in general figuring out
which solution is best is just not anywhere near the top of our priority
list when we still can't distribute binaries to most users at all :-)

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nick Coghlan]

On 2 Feb 2016 10:45, "Matthias Klose"  wrote:
>
> On 30.01.2016 00:29, Nathaniel Smith wrote:
>>
>> Hi all,
>>
>> I think this is ready for pronouncement now -- thanks to everyone for
>> all their feedback over the last few weeks!
>
>
> I don't think so.  I am biased because I'm the maintainer for Python in
Debian/Ubuntu.  So I would like to have some feedback from maintainers of
Python in other Linux distributions (Nick, no, you're not one of these).

The RHEL ABI maintainers raised very similar concerns (that's how I found
out about the current status of libabigail), but they're in a similar
position to you: Linux distros provide *much* stronger guarantees of ABI
compatibility than this PEP even attempts to provide.

Here's the key distinction though: not everything is a mission critical
production server, and we know from cross-distro redistributors like
Continuum Analytics & Enthought, as well as our own upstream experience
with the extraordinarily informal ABIs for Windows and Mac OS X wheels that
an underspecified approach like manylinux is likely to be good enough in
practice.

That puts the power in users' hands:

* by default, most users will get to use developer provided builds, rather
than having to build from source (bringing the Linux user experience up to
parity with that for Windows and Mac OS X)
* folks can still force the use of source builds if they prefer (e.g. to
better optimise for their OS and hardware, or to change the threat profile
for their open source project usage)
* folks can still opt to use a third party build service if they prefer
(whether that's a Linux distro or a cross platform redistributor)

Cheers,
Nick.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig