On Thu, Aug 13, 2015 at 7:08 AM, Nathaniel Smith <n...@pobox.com> wrote:
> On Wed, Aug 12, 2015 at 8:10 PM, Robert Collins > <robe...@robertcollins.net> wrote: > > On 13 August 2015 at 12:51, Nathaniel Smith <n...@pobox.com> wrote: > >> On Aug 12, 2015 16:49, "Robert Collins" <robe...@robertcollins.net> > wrote: > >>> > >>> I'm not sure what will be needed to get the PR accepted; At PyCon AU > >>> Tennessee Leuwenberg started drafting a PEP for the expression of > >>> dependencies on e.g. BLAS - its been given number 497, and is in the > >>> packaging-peps repo; I'm working on updating it now. > >> > >> I wanted to take a look at this PEP, but I can't seem to find it. PEP > 497: > >> https://www.python.org/dev/peps/pep-0497/ > >> appears to be something else entirely? > >> > >> I'm a bit surprised to hear that such a PEP is needed. We (= numpy devs) > >> have actively been making plans to ship a BLAS wheel on windows, and > AFAICT > >> this is totally doable now -- the blocker is windows toolchain issues, > not > >> pypa-related infrastructure. > >> > >> Specifically the idea is to have a wheel that contains the shared > library as > >> a regular old data file, plus a stub python package that knows how to > find > >> this data file and how to make it accessible to the linker. So > >> numpy/__init__.py would start by calling: > >> > >> import pyopenblas1 > >> # on Linux modifies LD_LIBRARY_PATH, > >> # on Windows uses ctypes to preload... whatever > >> pyopenblas1.enable() > >> > >> and then get on with things, or the build system might do: > >> > >> import pyopenblas1 > >> pyopenblas1.get_header_directories() > >> pyopenblas1.get_linker_directories() > >> > > Thanks to James for sending on the link! > > Two main thoughts, now that I've read it over: > > 1) The motivating example is somewhat confused -- the draft says: > > + The example provided in the abstract is a > + hypothetical package which needs versions of numpy and scipy, both of > which > + must have been compiled to be aware of the ATLAS compiled set of > linear algebra > + libraries (for performance reasons). This sounds esoteric but is, in > fact, a > + routinely encountered situation which drives people towards using the > + alternative packaging for scientific python environments. > > Numpy and scipy actually work hard to export a consistent, append-only > ABI regardless of what libraries are used underneath. (This is > actually by far our biggest issue with wheels -- that there's still no > way to tag the numpy ABI as part of the ABI string, so in practice > it's just impossible to ever have a smooth migration to a new ABI and > we have no choice but to forever maintain compatibility with numpy > 0.1. But that's not what this proposal addresses.) Possibly part of > the confusion here is that Christoph Gohlke's popular numpy+scipy > builds use a hack where instead of making the wheels self-contained > via statically linking or something like that, then he ships the > actual libBLAS.dll inside the numpy wheel, and then the scipy wheel > has some code added that magically "knows" that there is this special > numpy wheel that it can find libBLAS.dll inside and use it directly > from scipy's own extensions. But this coupling is pretty much just > broken, and it directly motivates the blas-in-its-own-wheel design I > sketched out above. > > (I guess the one exception is that if you have a numpy or scipy build > that dynamically links to a library like BLAS, and then another > extension that links to a different BLAS with an incompatible ABI, and > the two BLAS libraries have symbol name collisions, then that could be > a problem because ELF is frustrating like that. But the obvious > solution here is to be careful about how you do your builds -- either > by using static linking, or making sure that incompatible ABIs get > different symbol names.) > > Anyway, this doesn't particularly undermine the PEP, but it would be > good to use a more realistic motivating example. > > 2) AFAICT, the basic goal of this PEP is to provide machinery to let > one reliably build a wheel for some specific version of some specific > distribution, while depending on vendor-provided libraries for various > external dependencies, and providing a nice user experience (e.g., > telling users explicitly which vendor-provided libraries they need to > install). I say this because strings like "libblas1.so" or "kernel.h" > do not define any fixed ABI or APIs, unless you are implicitly scoping > to some particular distribution with at least some minimum version > constraint. > > It seems like a reasonable effort at solving this problem, and I guess > there are probably some people somewhere that have this problem, but > my concern is that I don't actually know any of those people. The > developers I know instead have the problem of, they want to be able to > provide a small finite number of binaries (ideally six binaries per > Python version: {32 bit, 64 bit} * {windows, osx, linux}) that > together will Just Work on 99% of end-user systems. And that's the > problem that Enthought, Continuum, etc., have been solving for years, > and which wheels already mostly solve on windows and osx, so it seems > like a reasonable goal to aim for. But I don't see how this PEP gets > us any closer to that. Again, not really a criticism -- these goals > aren't contradictory and it's great if pip ends up being able to > handle both common and niche use cases. But I want to make sure that > we're clear that these goals are different and which one each proposal > is aimed at. > > >> This doesn't help if you want to declare dependencies on external, > system > >> managed libraries and have those be automatically somehow provided or > >> checked for, but to me that sounds like an impossible boil-the-ocean > project > >> anyway, while the above is trivial and should just work. > > > > Well, have a read of the draft. > > > > Its a solved problem by e.g. conda, apt, yum, nix and many others. > > None of these projects allow a .deb to depend on .rpms etc. -- they > all require that they own the whole world with some narrow, carefully > controlled exceptions (e.g. anaconda requires some non-trivial runtime > on the host system -- glibc, glib, pcre, expat, ... -- but it's a > single fixed set that they've empirically determined is close enough > to universally available in practice). The "boil the ocean" part is > the part where everybody who wants to distribute wheels has to go > around and figure out every possible permutation of ABIs on every > possible external packaging system and provide separate wheels for > each of them. > > > Uploading system .so's is certainly also an option, and I see no > > reason why we can't do both. > > > > I do know that distribution vendors are likely to be highly allergic > > to the idea of having regular shared libraries present as binaries, > > but thats a different discussion :) > > Yeah, but basically in the same way that they're allergic to all > wheels, period, so ... :-). I think in the long run the only realistic > approach is for most users to either be getting blas+numpy from some > external system like macports/conda/yum/... or else to be getting > blas+numpy from official wheels on pypi. And neither of these two > scenarios seems to benefit from the functionality described in this > PEP. > > (Final emphasis: this is all just my own opinion based on my > far-from-omniscient view of the packaging system, please tell me if > I'm making some ridiculous error, or if well-actually libBLAS is > special and there is some other harder case I'm not thinking of, etc.) > >From my own experience if you have a design that covers blas/lapack issues for numpy/scipy, you've solved a majority of typical binary packaging issues in the python ecosystem. Will you be there at euroscipy ? I will spend some time at Euroscipy to continue the work we started at pycon around some of this (taking into account the PEP as well). David > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig >
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
