On 3 April 2018 at 02:45, Trishank Kuppusamy <trishank.kuppus...@datadoghq.com> wrote: > On Sat, Mar 31, 2018 at 11:06 PM, Nick Coghlan <ncogh...@gmail.com> wrote: >> The main reason PyPI doesn't currently support distro specific wheels is >> because there isn't a compatibility tagging spec for them that's reasonable >> to use on a public index server - they currently end up being tagged as >> generic "linux" wheels. You can live with that on a private index server >> like the one that https://galaxyproject.org/ runs, but it would be >> incredibly confusing on PyPI. >> >> As far as I know, >> https://mail.python.org/pipermail/distutils-sig/2015-July/026617.html is >> still the most complete write-up we have of a potential way to fix that, >> which is to: >> >> 1. extend the default Linux platform tag to include the ID and VERSION_ID >> fields from /etc/os-release (credit to Galaxy Project's Nate Coraor for that >> core concept) >> 2. define a config file under /etc/python/ that distros can use to change >> both the build tag (e.g. allowing CentOS users to emit RHEL compatible >> wheels, or Ubuntu users to emit Debian compatible ones), and a list of >> binary compatible distro tags (e.g. allowing CentOS to consume RHEL tagged >> wheels, Ubuntu to consume Debian tagged wheels, and other musl based distros >> to consume Alpine tagged wheels) >> 3. define an equivalent per-virtualenv file to override the settings from >> (2) >> 4. update pip/setuptools/twine/warehouse/etc to account for the new >> compatibility tag variant
[snip] > If we wanted to contribute towards this effort, how can the community help, > Nick? At the specification level, we'll need an update to PEP 425 that's comparable to Dustin's PEP 566 for the core metadata (which both defined metadata 2.1, and made https://packaging.python.org/specifications/core-metadata/ the stable reference URL for the latest version of the metadata spec). I'd like to see us do the same thing for platform compatibility tags: have https://packaging.python.org/specifications/platform-compatibility-tags/ define the format, and use the PEP process to make changes to that living spec. While the primary motivating use case is distro-specific Linux wheels, we should at least consider the possibility of defining the new optional fields in compatibility tags in such a way that they could also be used for ABI compatibility levels in more centrally controlled platforms (i.e. Windows, macOS, iOS, Android), or to tag an expected deployment environment (e.g. letting the Galaxy Project folks mark all their internal wheels as "galaxyproject", and then configure their deployment environments to only accept wheels tagged that way). Such a spec update would cover points 1->3 above. At the implementation level, I don't have any great insight on how you might best go about pursuing that, but I suspect factoring out some useful helper functions into the low level `packaging` library may make sense (I believe Vinay Sajip's `distlib` already has some helpers along these lines that would potentially require updates). At a minimum, to be useful: * either setuptools+bdist_wheel need to be able to produce wheels with the more detailed tags, or else there has be a suitable wheel renaming utility available (akin to `auditwheel repair`) * twine needs to be able to upload wheels with the more detailed tags * warehouse needs to be able to accept uploads with the more detailed tags (For that last point, I'd personally prefer it to be a project level setting to opt-in to allowing uploads with the more detailed tags, such that the default behaviour is to reject them, and provide an opportunity for us to nudge folks towards manylinux wheels) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig