> > >> I'm confused -- you don't want a system to be able to install ONE > version > >> of a lib that various python packages can all link to? That's really the > >> key use-case for me.... >
> Are we talking about Python libraries accessed via Python APIs, or > linking to external dependencies not written in Python (including > linking directly to C libraries shipped with a Python library)? > I, at least, am talking about the latter. for a concrete example: libpng, for instance, might be needed by PIL, wxPython, Matplotlib, and who knows what else. At this point, if you want to build a package of any of these, you need to statically link it into each of them, or distribute shared libs with each package -- if you ware using them all together (which I do, anyway) you now have three copies of the same lib (but maybe different versions) all linked into your executable. Maybe there is no downside to that (I haven't had a problem yet), but it seems like a bad way to do it! It's the latter I consider to be out of scope for a language specific > packaging system Maybe, but it's a problem to be solved, and the Linux distros more or less solve it for us, but OS-X and Windows have no such system built in (OS-X does have Brew and macports....) > - Python packaging dependencies are designed to > describe inter-component dependencies based on the Python import > system, not dependencies based on the operating system provided C/C++ > dynamic linking system. I think there is a bit of fuzz here -- cPython, at least, uses the "the operating system provided C/C++ dynamic linking system" -- it's not a totally independent thing. If folks are after the latter, than they want > a language independent package system, like conda, nix, or the system > package manager in a Linux distribution. And I am, indeed, focusing on conda lately for this reason -- but not all my users want to use a whole new system, they just want to "pip install" and have it work. And if you are using something like conda you don't need pip or wheels anyway! > I'm arguing against supporting direct C level dependencies between > packages that rely on dynamic linking to find each other rather than > going through the Python import system, Maybe there is a mid ground. For instance, I have a complex wrapper system around a bunch of C++ code. There are maybe 6 or 7 modules that all need to link against that C++ code. On OS-X (and I think Linux, I haven't been doing those builds), we can statically link all the C++ into one python module -- then, as long as that python module is imported before the others, they will all work, and all use that same already loaded version of that library. (this doesn't work so nicely on Windows, unfortunately, so there, we build a dll, and have all the extensions link to it, then put the dll somewhere it gets found -- a little fuzzy on those details) So option (1) for something like libpng is to have a compiled python module that is little but a something that can be linked to ibpng, so that it can be found and loaded by cPython on import, and any other modules can then expect it to be there. This is a big old kludge, but I think could be done with little change to anything in Python or wheel, or...but it would require changes to how each package that use that lib sets itself up and checks for and install dependencies -- maybe not really possible. and it would be better if dependencies could be platform independent, which I'm not sure is supported now. option (2) would be to extend python's import mechanism a bit to allow it to do a raw "link in this arbitrary lib" action, so the lib would not have to be wrapped in a python module -- I don't know how possible that is, or if it would be worth it. > (Another way of looking at this: if a tool can manage the > Python runtime in addition to Python modules, it's a full-blown > arbitrary software distribution platform, not just a Python package > manager). > sure, but if it's ALSO a Python package manger, then why not? i.e. conda -- if we all used conda, we wouldn't need pip+wheel. > Defining cross-platform ABIs (cf. http://bugs.python.org/issue23966) > This is a mess that you need to deal with for ANY binary package -- that's why we don't distribute binary wheels on pypi for Linux, yes? > I'm > firmly of the opinion that trying to solve both sets of problems with > a single tool will produce a result that doesn't work as well for > *either* use case as separate tools can. > I'm going to point to conda again -- it solves both problems, and it's better to use it for all your packages than mingling it with pip (though you CAN mingle it with pip...). So if we say "pip and friends are not going to do that", then we are saying: we don't support a substantial class of packages, and then I wonder what the point is to supporting binary packages at all? P.S. The ABI definition problem is at least somewhat manageable for > Windows and Mac OS X desktop/laptop environments Ah -- here is a key point -- because of that, we DO support binary packages on PyPi -- but only for Windows and OS-X.. I'm just suggesting we find a way to extend that to pacakges that require a non-system non-python dependency. but beyond > those two, things get very messy, very fast - identifying CPU > architectures, CPU operating modes and kernel syscall interfaces > correctly is still a hard problem in the Linux distribution space right -- but where I am confused is where the line is drawn -- it seem sto be the line is REALLY drawn at "yuo need to compile some C (or Fortran, or???) code, rather than at "you depend on another lib" -- the C code, whether it is a third party lib, or part of your extension, still needs to be compiled to match the host platform. and that's > *without* getting into things like determining which vectorisation > instructions are available). yup -- that's why we don't have binary wheels for numpy up on PyPi at this point..... but the rise of aarch64 and IBM's > creation of the OpenPOWER Foundation is making the data centre space > interesting again, while in the mobile and embedded spaces it's ARM > that is the default, with x86_64 attempting to make inroads. > Are those the targets for binary wheels? I don't think so. > This is why > "statically link all the things" keeps coming back in various guises > but if you statically link, you need to build the static package right anyway -- so it doesn't actually solve the problem at hand anyway. The only solution that is known to work reliably for dynamic linking > is to have a curated set of packages all built by the same build > system, so you know they're using consistent build settings. Linux > distributions provide this, as do multi-OS platforms like nix and > conda. We *might* be able to provide it for Python someday if PyPI > ever gets an integrated build farm, but that's still a big "if" at > this point. > Ah -- here is the issue -- but I think we HAVE pretty much got what we need here -- at least for Windows and OS-X. It depends what you mean by "curated", but it seems we have a (defacto?) policy for PyPi: binary wheels should be compatible with the python.org builds. So while each package wheel is supplied by the package maintainer one way or another, rather than by a central entity, it is more or less curated -- or at least standardized. And if you are going to put a binary wheel up, you need to make sure it matches -- and that is less than trivial for packages that require a third party dependency -- but building the lib statically and then linking it in is not inherently easier than doing a dynamic link. OK -- I just remembered the missing link for doing what I proposed above for third party dynamic libs: at this point dependencies are tied to a particular package -- whereas my plan above would require a dependency ties to particular wheel, not the package as a whole. i.e: my mythical matplotlib wheel on OS-X would depend on a py_libpng module -- which could be provided as separate binary wheel. but matplotlib in general would not have that dependency -- for instance, on Linux, folks would want it to build against the system lib, and not have another dependency. Even on OS-X, homebrew users would want it to build against the homebrew lib, etc... So what would be good is a way to specify a "this build" dependency. That can be hacked in, of course, but nicer not to have to. NOTE; we ran into this with readline and iPython wheels -- I can't remember how that was resolved. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
