On 29 November 2014 at 01:51, Antoine Pitrou <[email protected]> wrote: > On Sat, 29 Nov 2014 01:27:44 +1000 > Nick Coghlan <[email protected]> wrote: >> > >> > Is this not going to be a slippery slope? >> >> Only if folks publish Linux binaries themselves, and that's still a >> bad idea (for the same reason publishing distro binaries is already a >> rare thing for people to do). > > Well, let's not make this a matter of ideology. Everyone knows it's a > bad idea to publish binaries, yet it's often better than nothing, > especially if the software is tedious to compile.
It's not a matter of ideology, but a matter of practicality. Debian stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow enough (and have strong enough ABI compatibility guarantees) to be practical for ISVs to target with prebuilt binaries. Beyond that, the rate of development and breadth of target environments in the Linux world tends to make providing prebuilt binaries in *any* format difficult (unless you statically link everything you need, which works fine until you decide you want a particular package to depend on binaries provided by another one by linking to them dynamically). Regardless of target environment, though, being able to prebuild wheel files in a way that clearly indicates the platform restricted nature of the end result is a useful tool for system integrators. That benefit applies regardless of whether the builds are happening in the context of Linux distro development, or in the context of maintaining a particular set of infrastructure services. >> > How many binary packages will package authors have to provide to cover >> > people's needs? Windows + OS X + Linux multiplied by 32 / 64 multiplied >> > by three or four Python versions is already a lot of binaries to >> > build... >> >> I'd still advise against folks posting Linux wheels on PyPI, just as >> they tend not to post RPM or deb files. This is so we can provide >> wheels at the distro level (or build them internally) without creating >> vast amounts of confusion. > > So do we (software authors) have to wait for that mythical "we" who are > going to build binaries in time for our packages? If the compatibility tagging issue can be resolved, I don't think there should be any restrictions against uploading Linux wheels that avoid the generic Linux platform tags. I just doubt it will make sense for most developers to worry about it, just as most of them don't worry about providing RPMs or Debian packages. > Case in point: can I ask you (the mythical "we") to build packages for > all major distros (including supported LTS releases), and the four most > recent Python versions, of the following piece of software: > https://github.com/numba/llvmlite No, that would be a service provided by the as yet hypothetical PyPI build farm. If/when that happens, it will need to have a way of tagging Linux wheels appropriately, though. Nearer term (and what prompted me to start this thread), the Fedora Environments & Stacks working group is investigating providing prebuilt wheel files for the Fedora ecosystem, and potentially for EPEL as well (see https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement for the broader context of that effort). For other ecosystems, you'll have to ask participants in those ecosystems. >> This isn't really about that - it's about having a way to >> tackle it at the distro level, without introducing significant >> potential for confusion on end user systems > > I'm not sure I understand: distros provide their own packages, they > don't (shouldn't) blindly pull binary wheels from PyPI. Why would they > depend on the wheel tagging format? We don't plan to blindly pull anything from PyPI - we're looking at the feasibility of publishing already reviewed software in ecosystem native formats (with the two pilot projects focusing on Java JAR files and Python wheel files). When I last mentioned that idea here, Marcus pointed out that doing that with the generic "linux_x86_64" compatibility tag on the wheel filenames would be problematic, as there'd be nothing preventing anyone from pulling them down onto inappropriate systems, with no obvious trace of the Fedora or EPEL specific assumptions that went into building them. While that's a valid concern, I also don't want to go invent our own custom compatibility tagging convention just for Fedora & EPEL, but rather work within the limits of what upstream Python packaging natively supports. At the moment there is no such tagging system, which is why I'm interested in pursuing improvements to the definition of the platform tag. However, after thinking further about the situation with EPEL (where we'd likely want a single set of wheel files to cover not only an entire major release series, but also downstream RHEL derivatives like CentOS), the possibility of sharing wheel files between the Fedora & Software Collections Python builds, and the point Matthias raised about the limitations of the current platform tag when it comes to multiarch support on Debian, I'm back to considering the idea of making it possible to override the default platform tag with something more appropriate (I think Daniel may have been the first one to suggest that?). The task of defining what the appropriate platform tag overrides should be would then fall back on the Linux distros as part of declaring our ABI compatibility expectations. Slightly tangentially, I'm also now wondering if we might also be able to address the SOABI problem on Python 2 with an appropriate configuration system design. Consider if the following could be included in the "pyvenv.cfg" file in a virtual environment: [compatibility] python=cp27,cp2,py2 abi=cp27mu platform=linux_x86_64_epel_6 Or for a Python 3 virtual environment: [compatibility] python=cp34,cp3,py3 abi=cp34m,abi3 platform=linux_x86_64_epel_6 If present, these pyvenv.cfg settings would override the normal PEP 425 compatibility tag calculations (I'm OK with the idea of *needing* to be in a virtual environment to gain the power to configure these tags). Note that this would *only* affect the tags used in filenames, and when searching for wheel files. It would still be up I'm not sold on the technical details yet (consider the above to be just a sketch of the idea), but it seems to me that having those virtual environment specific overrides defined in a tool independent way, and updating bdist_wheel to read them when creating wheel files, and pip to read them when deciding which wheel files to consider as candidates for installation, would likely address the use case, without locking us in to anything too specific in the absence of suitable data. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - [email protected] https://mail.python.org/mailman/listinfo/distutils-sig
