Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Olivier Grisel]

Strong +1 for all Donald's proposals.

-- 
Olivier
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Donald Stufft]

> On Feb 16, 2016, at 1:33 PM, Barry Warsaw  wrote:
> 
> I don't, but I hope that should not be a problem these days, with modern pip
> on modern Debian/Ubuntu systems.  We prevent pip from installing in system
> locations in various ways, including making --user the default.
> 
> I don't think we're doing anything that isn't at least within the pip
> roadmap.  E.g. --user will probably someday be the default in stock pip, but
> because of reasons it hasn't happened yet.
> 
> IIRC, Donald was also talking about better ways to detect system owned files,
> so pip would also refuse to overwrite them.

Replying to Barry, but also to generally the whole sub-thread!

First off, there is very little chance pip ever removes the ability to install
directly into a system Python. So any proposal to remove that functionality is
unlikely to ever be accepted without an incredibly compelling reason.

As far as some sort of "tainting" goes, that's not really related to this PEP
at all, and as of pip 8.0 we already include information when we install a file
that says that this particular item was installed by pip. That can be accessed
using pkg_resources::

>>> 
pkg_resources.get_distribution("twine").get_metadata("INSTALLER").strip()
'pip'

As far as I know, nothing actually uses that metadata right now, but that
should be plenty for some sort of tooling to be written that can tell a distro
if there are any non-distro provided packages in an enviornment using something
like:

>>> for d in pkg_resources.working_set:
... if d.has_metadata("INSTALLER"):
... print("{}: {}".format(d.project_name, 
d.get_metadata("INSTALLER").strip()))
...
certifi: pip
packaging: pip
pkginfo: pip
pyparsing: pip
pytz: pip
requests: pip
requests-toolbelt: pip
setuptools: pip
twine: pip
virtualenv: pip
wheel: pip

Anyways, where I think [1] we're heading:

* When we don't actually have permission to install into site-packages, then
  default to --user. This will stop leading people towards using sudo because
  they typed ``pip install foo`` as a user and got a permissions error, the
  logical way to solve that being ``sudo !!``.

* Standardize the patches Debian has which splits up a vendor location from the
  site location, giving distros a different place to install things than where
  pip will install by default.

* Provide a mechanism that will enable vendors to say that some particular
  install package is system owned, and then have pip refuse to modify those
  files without some sort of --yes-i-know-what-im-doing flag.


At this time, I don't personally have any plans to add anything like pipsi but
I'm not strictly opposed to it either. The largest problem with that is we
don't have a good way of knowing, when someone does ``pip install foobar`` if
they want to install foobar because it's an isolated bin they want to run, or
if it's something they want to be able to import in their current environment.


[1] These are my opinions, and none of them are set in stone.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Paul Moore]

On 16 February 2016 at 14:55, Petr Viktorin  wrote:
> So, what is the argument against "pip install --user"? Does that not
> leave everyone happy?

See https://github.com/pypa/pip/issues/1668

Long story short, that's the intention, but there are a lot of
questions that need(ed) to be resolved. I'm not sure if we're at a
point where it's practical yet, though. (One big question in my mind
is whether --user installs work as well as we hope they do - as far as
I know very few people actually use them routinely...)

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Glyph Lefkowitz]

> On Feb 17, 2016, at 2:17 AM, David Cournapeau  wrote:
> 
> Sure, but the people I tend to see using `sudo pip` are not the kind of users 
> where that distinction is very useful.

It's hair-splitting but probably correct hair-splitting in terms of how it's 
detected.

> If there were a different simple, reliable way to avoid installing in system 
> python, I would be happy to change my own recommendations during sprints, 
> talks, etc...

Are you recommending 'sudo pip' right now?  Why not 'sudo virtualenv', then?

-g___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Glyph Lefkowitz]

> On Feb 17, 2016, at 12:55 AM, Antoine Pitrou  wrote:
> 
> On Tue, 16 Feb 2016 16:10:34 -0800
> Glyph Lefkowitz  wrote:
>> 
>> I am 100% on board with telling people "don't use `sudo pip install´".  
>> Frankly I have been telling the pip developers to just break this for years 
>> (see https://pip2014.com, which, much to my chagrin, still exists); `sudo 
>> pip install´ should just exit immediately with an error; to the extent that 
>> packagers need it, the only invocation that should work should be `sudo pip 
>> install --i-am-building-an-operating-system´.
> 
> [...] The problem is not the use of "sudo" or the
> invocation under root, it's to install into a system Python. So the
> solution should be to flag the system Python as not suitable for using
> pip into, not to forbid using pip under root.


I didn't mean to suggest that sudo /path/to/venv/bin/pip install should fail, 
so we are in agreement here.  The exact details of how pip detects the 
suitability of a given environment are up for discussion, it's just that the 
default behavior of `sudo pip install´ (install into package-manager-managed 
system prefix) is a bad idea.  Perhaps certain venvs should set this flag as 
well, to indicate that pip should not mess with it any more either.

-glyph
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Glyph Lefkowitz]

> On Feb 16, 2016, at 11:55 PM, Robert T. McGibbon  wrote:
> 
> On Tue, Feb 16, 2016 at 4:10 PM, Glyph Lefkowitz  > wrote:
> This whole section is about a tool to automatically identify possible issues 
> with these wheels - https://www.python.org/dev/peps/pep-0513/#auditwheel 
>  - so I don't even 
> really know what you mean by this comment.  I thought that the existence of 
> this tool is one of the best parts of this PEP!
> 
> Oh cool! Thanks, Glyph! I had a lot of fun writing it.

It really cuts to the heart of the problem with python builds: you can 
accidentally depend on some aspect of the platform in a way which requires 
nuanced understanding of the native build toolchain to understand.  For what 
it's worth this is definitely a problem on OS X and Windows as well 
(accidentally depending on homebrew or chocolatey for example); any chance 
you'll be extending it to deal with 'dumpbin' and 'otool' as well as 'ldd'?

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Antoine Pitrou]

On Wed, 17 Feb 2016 05:12:48 -0500
"Eric V. Smith"  wrote:

> On 2/17/2016 3:55 AM, Antoine Pitrou wrote:
> > On Tue, 16 Feb 2016 16:10:34 -0800
> > Glyph Lefkowitz  wrote:
> >>
> >> I am 100% on board with telling people "don't use `sudo pip install´".  
> >> Frankly I have been telling the pip developers to just break this for 
> >> years (see https://pip2014.com, which, much to my chagrin, still exists); 
> >> `sudo pip install´ should just exit immediately with an error; to the 
> >> extent that packagers need it, the only invocation that should work should 
> >> be `sudo pip install --i-am-building-an-operating-system´.
> > 
> > This is frankly ridiculous. The problem is not the use of "sudo" or the
> > invocation under root, it's to install into a system Python. So the
> > solution should be to flag the system Python as not suitable for using
> > pip into, not to forbid using pip under root.
> 
> I agree that there are uses for running pip as root (I do so myself).
> It's installing into the system Python that needs to be strongly
> discouraged, if not outright prevented. I'm not sure we have a good way
> of identifying the system Python, but that's another issue.

I think it would be reasonable to let vendors do so by placing a
specific file into the "site-packages" (or perhaps even a pip config
file if we want to put more information in it).  Then upstream Python
doesn't need to have a say in it.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[David Cournapeau]

On Wed, Feb 17, 2016 at 8:55 AM, Antoine Pitrou  wrote:

> On Tue, 16 Feb 2016 16:10:34 -0800
> Glyph Lefkowitz  wrote:
> >
> > I am 100% on board with telling people "don't use `sudo pip install´".
> Frankly I have been telling the pip developers to just break this for years
> (see https://pip2014.com, which, much to my chagrin, still exists); `sudo
> pip install´ should just exit immediately with an error; to the extent that
> packagers need it, the only invocation that should work should be `sudo pip
> install --i-am-building-an-operating-system´.
>
> This is frankly ridiculous. The problem is not the use of "sudo" or the
> invocation under root, it's to install into a system Python.
>

Sure, but the people I tend to see using `sudo pip` are not the kind of
users where that distinction is very useful. If there were a different
simple, reliable way to avoid installing in system python, I would be happy
to change my own recommendations during sprints, talks, etc...

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Eric V. Smith]

On 2/17/2016 3:55 AM, Antoine Pitrou wrote:
> On Tue, 16 Feb 2016 16:10:34 -0800
> Glyph Lefkowitz  wrote:
>>
>> I am 100% on board with telling people "don't use `sudo pip install´".  
>> Frankly I have been telling the pip developers to just break this for years 
>> (see https://pip2014.com, which, much to my chagrin, still exists); `sudo 
>> pip install´ should just exit immediately with an error; to the extent that 
>> packagers need it, the only invocation that should work should be `sudo pip 
>> install --i-am-building-an-operating-system´.
> 
> This is frankly ridiculous. The problem is not the use of "sudo" or the
> invocation under root, it's to install into a system Python. So the
> solution should be to flag the system Python as not suitable for using
> pip into, not to forbid using pip under root.

I agree that there are uses for running pip as root (I do so myself).
It's installing into the system Python that needs to be strongly
discouraged, if not outright prevented. I'm not sure we have a good way
of identifying the system Python, but that's another issue.

Eric.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Antoine Pitrou]

On Tue, 16 Feb 2016 16:10:34 -0800
Glyph Lefkowitz  wrote:
> 
> I am 100% on board with telling people "don't use `sudo pip install´".  
> Frankly I have been telling the pip developers to just break this for years 
> (see https://pip2014.com, which, much to my chagrin, still exists); `sudo pip 
> install´ should just exit immediately with an error; to the extent that 
> packagers need it, the only invocation that should work should be `sudo pip 
> install --i-am-building-an-operating-system´.

This is frankly ridiculous. The problem is not the use of "sudo" or the
invocation under root, it's to install into a system Python. So the
solution should be to flag the system Python as not suitable for using
pip into, not to forbid using pip under root.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Robert T. McGibbon]

On Tue, Feb 16, 2016 at 4:10 PM, Glyph Lefkowitz 
wrote:
>
> This whole section is about a tool to automatically identify possible
> issues with these wheels -
> https://www.python.org/dev/peps/pep-0513/#auditwheel - so I don't even
> really know what you mean by this comment.  I thought that the existence of
> this tool is one of the best parts of this PEP!
>

Oh cool! Thanks, Glyph! I had a lot of fun writing it.

-- 
-Robert
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Noah Kantrowitz]

> On Feb 16, 2016, at 4:10 PM, Glyph Lefkowitz  wrote:
> 
>> 
>> On Feb 16, 2016, at 3:05 AM, Matthias Klose  wrote:
>> 
>> On 02.02.2016 02:35, Glyph Lefkowitz wrote:
>>> 
 On Feb 1, 2016, at 3:37 PM, Matthias Klose  wrote:
 
 On 30.01.2016 00:29, Nathaniel Smith wrote:
> Hi all,
> 
> I think this is ready for pronouncement now -- thanks to everyone for
> all their feedback over the last few weeks!
 
 I don't think so.  I am biased because I'm the maintainer for Python in 
 Debian/Ubuntu.  So I would like to have some feedback from maintainers of 
 Python in other Linux distributions (Nick, no, you're not one of these).
>>> 
>>> Possibly, but it would be very helpful for such maintainers to limit their 
>>> critique to "in what scenarios will this fail for users" and not have the 
>>> whole peanut gallery chiming in with "well on _my_ platform we would have 
>>> done it _this_ way".
>>> 
>>> I respect what you've done for Debian and Ubuntu, Matthias, and I use the 
>>> heck out of that work, but honestly this whole message just comes across as 
>>> sour grapes that someone didn't pick a super-old Debian instead of a 
>>> super-old Red Hat.  I don't think it's promoting any progress.
>> 
>> You may call this sour grapes, but in the light of people installing
>> these wheels to replace/upgrade system installed eggs, it becomes an issue. 
>> It's fine to use such wheels in a virtual environment, however people tell 
>> users to use these wheels to replace system installed packages, distros will 
>> have a problem identifying issues.
> 
> I am 100% on board with telling people "don't use `sudo pip install´".  
> Frankly I have been telling the pip developers to just break this for years 
> (see https://pip2014.com, which, much to my chagrin, still exists); `sudo pip 
> install´ should just exit immediately with an error; to the extent that 
> packagers need it, the only invocation that should work should be `sudo pip 
> install --i-am-building-an-operating-system´.

As someone that handles the tooling side, I don't care how it works as long as 
there is an override for tooling a la Chef/Puppet. For stuff like Supervisord, 
it is usually the least broken path to install the code globally.

--Noah



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Glyph Lefkowitz]

> On Feb 16, 2016, at 3:05 AM, Matthias Klose  wrote:
> 
> On 02.02.2016 02:35, Glyph Lefkowitz wrote:
>> 
>>> On Feb 1, 2016, at 3:37 PM, Matthias Klose  wrote:
>>> 
>>> On 30.01.2016 00:29, Nathaniel Smith wrote:
 Hi all,
 
 I think this is ready for pronouncement now -- thanks to everyone for
 all their feedback over the last few weeks!
>>> 
>>> I don't think so.  I am biased because I'm the maintainer for Python in 
>>> Debian/Ubuntu.  So I would like to have some feedback from maintainers of 
>>> Python in other Linux distributions (Nick, no, you're not one of these).
>> 
>> Possibly, but it would be very helpful for such maintainers to limit their 
>> critique to "in what scenarios will this fail for users" and not have the 
>> whole peanut gallery chiming in with "well on _my_ platform we would have 
>> done it _this_ way".
>> 
>> I respect what you've done for Debian and Ubuntu, Matthias, and I use the 
>> heck out of that work, but honestly this whole message just comes across as 
>> sour grapes that someone didn't pick a super-old Debian instead of a 
>> super-old Red Hat.  I don't think it's promoting any progress.
> 
> You may call this sour grapes, but in the light of people installing
> these wheels to replace/upgrade system installed eggs, it becomes an issue. 
> It's fine to use such wheels in a virtual environment, however people tell 
> users to use these wheels to replace system installed packages, distros will 
> have a problem identifying issues.

I am 100% on board with telling people "don't use `sudo pip install´".  Frankly 
I have been telling the pip developers to just break this for years (see 
https://pip2014.com, which, much to my chagrin, still exists); `sudo pip 
install´ should just exit immediately with an error; to the extent that 
packagers need it, the only invocation that should work should be `sudo pip 
install --i-am-building-an-operating-system´.

But `sudo pip install´ of arbitrary packages is now, and always has been, 
basically broken; this PEP doesn't change that in any way I can see.  
Specifically, since there are tools in place to ensure that the extension 
modules will load just fine, this won't be any more broken than `sudo pip 
install´-ing random C extension modules is today.  If anything it will be more 
reliable, since a lot of people already build and ship wheels to their 
production linux environments, and don't always understand the nuances around 
having to build on a system with a native package set that exactly matches 
their target environment.

> There is a substantial amount of extensions built using C++; I didn't check 
> how many of these in c++0x/c++11 mode.  Until GCC 5, the c++11 ABI wasn't 
> stable, and upstream never promised forward compatibility, something that 
> even distros have to care about (usually by rebuilding packages before a 
> release).  So if you want a lowest common denominator, then maybe limit or 
> recommend the use of c++98 only.

Isn't this irrelevant as long as your entry-points are all 'extern "C"' and 
your C++ code statically links libstdc++?  The build toolchain in question 
doesn't include a dynamic libstdc++, does it?  If so, that's a pretty concrete 
problem with this proposal and it should be addressed.

>>> The proposal just takes some environment and declares that as a standard.  
>>> So everybody wanting to supply these wheels basically has to use this 
>>> environment.
>> 
>> There's already been lots of discussion about how this environment is a 
>> lowest common denominator.  Many other similar environments could _also_ be 
>> lowest common denominator.
> 
> sure, but then please call it what it is. centos5 or somelinux1.

The point of the wheel tag is that its output should work on many linuxes.  A 
'centos5' tag would imply that you can use arbitrary dynamic libraries (and 
perhaps even arbitrary packages!) from centos5, of which there are many; you 
can't, because auditwheel will yell at you.  It's the build environment plus 
restrictions around what you can depend on from that environment.

>> In the future, more specific and featureful distro tags sound like a good 
>> idea.  But could we please stop making the default position on distutils-sig 
>> "this doesn't cater to my one specific environment in the most optimal 
>> possible way, so let's give up on progress entirely"?  This is a good 
>> proposal that addresses environment portability and gives Python a 
>> substantially better build-artifact story than it currently has, in the 
>> environment most desperately needing one (server-side linux).  Could it be 
>> better?  Of course.  It could be lots better.  There are lots of use-cases 
>> for dynamically linked wheels and fancy new platform library features in 
>> newer linuxes.  But that can all come later, and none of it needs to have an 
>> impact on this specific proposal, right now.
> 
> I'm unsure how more specific and featureful distro 

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Petr Viktorin]

On 02/16/2016 03:20 PM, Paul Moore wrote:
> On 16 February 2016 at 14:14, Wayne Werner  wrote:
>> I've learned that *usually* linux distro repos lag way behind in updating
>> their Python packages, so unless I *can't* install the package via pip,
>> that's what I do.
> 
> Yeah, and that's what I'd count as an issue between you and your
> distro. If they don't provide sufficiently up to date versions for
> you, and you choose to deal with that in whatever way you prefer,
> that's fine by me.

As a fedora packager, I'd definitely prefer if you used "pip install
--user" instead if "sudo pip install".

> I don't see why the Python community shouldn't provide a solution that
> you can use in such a situation, simply because it's not the solution
> your distro would prefer you to use.

So, what is the argument against "pip install --user"? Does that not
leave everyone happy?

Of course it's your system and you're free to do whatever you want, and
I'm sure you can debug any resulting issues successfully.
But there's a bunch of people spending time to get all kinds of packages
working well together for everyone (even people who don't care about
latest versions of Python packages), and recommending "sudo pip" is
making that job harder.

>> Of course, to my knowledge I've never replaced a system installed version
>> of anything. Though, considering I've been using Python3 since it was 
>> available
>> and most distros use Python 2, that may not really be saying much :)
> 
> I thought the distro "hands off" rules applied even to adding things
> to system-managed directories, not just to overwriting files?

Definitely. You can't know what the distro will add in the future; some
unrelated package might bring in a (possibly newly packaged) dependency
that replaces whatever you installed.

> Anyway, I've already made more inflammatory comments than an outsider
> should, so I'll leave the debate to the Unix users at this point.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nathaniel Smith]

On Feb 16, 2016 3:05 AM, "Matthias Klose"  wrote:
[...]
>
> You may call this sour grapes, but in the light of people installing
> these wheels to replace/upgrade system installed eggs, it becomes an
issue. It's fine to use such wheels in a virtual environment, however
people tell users to use these wheels to replace system installed packages,
distros will have a problem identifying issues.

Like Paul, I don't understand how wheels change any this -- everything in
the above paragraph is just as true if you s/wheel/sdist/, isn't it? "sudo
pip install" was a bad idea before and is a bad now, for sure.

> There is a substantial amount of extensions built using C++; I didn't
check how many of these in c++0x/c++11 mode.  Until GCC 5, the c++11 ABI
wasn't stable, and upstream never promised forward compatibility, something
that even distros have to care about (usually by rebuilding packages before
a release).  So if you want a lowest common denominator, then maybe limit
or recommend the use of c++98 only.

I'm not sure what the situation with the C++11 abi is in detail (do you
mean backwards compatibility?), but in any case your wish is preemptively
granted :-). The libstdc++ in centos5 doesn't contain c++11 support, so in
manylinux1 wheels you are allowed to use C++11 if and only if you can find
some way to provide your own self-contained runtime support that doesn't
rely on the system libstdc++. (The devtoolset compilers are handy in that
they are configured to do this automatically.)

[...]
> I'm unsure how more specific and featureful distro tags will help, unless
you start building more than one binary version of a wheel.  From a distro
point of view I only can discourage using such wheels outside a virtual
environment, and I would like to see a possibility to easily identify such
wheels, something like loading a binary kernel module is tainting the
kernel. This way distros can point users to the provider of such wheels.
>
> This is not a "this doesn't cater to my one specific environment"
position. Of course you probably get less push back from other
distributions closer to the specific environment specified in the pep.  But
please acknowledge that there might be issues that you and me don't yet
even see, and provide a possibility to identify such issues.

I guess it should be possible to implement such a tool by scanning the
.dist-info directories of installed packages?

It's certainly true that installing random packages off pypi into system
directories (and running those packages' setup.py code as root!) can create
arbitrarily terrible breakage. (I guess wheels at least remove the root
code execution. Why does pip even allow itself to be run as root? It's
certainly not written in a paranoid security-first style.) And I agree that
unexpected issues may arise in particular packages.

But again it's not clear to me how wheels create a special/unique problem
that calls for a special/unique countermeasure. Surely the first line of
defense for detecting such things is just "does the traceback mention
/usr/local", which is the same with or without wheels? I assume you are
already used to detecting "sudo pip" users in bug reports and pointing them
to the provider of these wheels or sdists? What about your workflow are you
hoping to change?

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Barry Warsaw]

On Feb 16, 2016, at 06:01 PM, Matthias Klose wrote:

>I know we had such an issue where pip accidentally modified system installed
>files, maybe Barry still remembers the details ...

I don't, but I hope that should not be a problem these days, with modern pip
on modern Debian/Ubuntu systems.  We prevent pip from installing in system
locations in various ways, including making --user the default.

I don't think we're doing anything that isn't at least within the pip
roadmap.  E.g. --user will probably someday be the default in stock pip, but
because of reasons it hasn't happened yet.

IIRC, Donald was also talking about better ways to detect system owned files,
so pip would also refuse to overwrite them.

Cheers,
-Barry


pgpvUxg7OmTnO.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Matthias Klose]

On 16.02.2016 15:43, David Cournapeau wrote:

On Tue, Feb 16, 2016 at 11:05 AM, Matthias Klose  wrote:


On 02.02.2016 02:35, Glyph Lefkowitz wrote:



On Feb 1, 2016, at 3:37 PM, Matthias Klose  wrote:


On 30.01.2016 00:29, Nathaniel Smith wrote:


Hi all,

I think this is ready for pronouncement now -- thanks to everyone for
all their feedback over the last few weeks!



I don't think so.  I am biased because I'm the maintainer for Python in
Debian/Ubuntu.  So I would like to have some feedback from maintainers of
Python in other Linux distributions (Nick, no, you're not one of these).



Possibly, but it would be very helpful for such maintainers to limit
their critique to "in what scenarios will this fail for users" and not have
the whole peanut gallery chiming in with "well on _my_ platform we would
have done it _this_ way".

I respect what you've done for Debian and Ubuntu, Matthias, and I use the
heck out of that work, but honestly this whole message just comes across as
sour grapes that someone didn't pick a super-old Debian instead of a
super-old Red Hat.  I don't think it's promoting any progress.



You may call this sour grapes, but in the light of people installing
these wheels to replace/upgrade system installed eggs, it becomes an
issue. It's fine to use such wheels in a virtual environment, however
people tell users to use these wheels to replace system installed packages,
distros will have a problem identifying issues.



FWIW, I often point out when people put "sudo" and "pip" in the same
sentence.

What about adding some language around this to the PEP ?


that's one thing.  But maybe pip itself could error out on such a situation, and 
maybe overriden with a non-default flag.


I know we had such an issue where pip accidentally modified system installed 
files, maybe Barry still remembers the details ...



In the future, more specific and featureful distro tags sound like a good

idea.  But could we please stop making the default position on
distutils-sig "this doesn't cater to my one specific environment in the
most optimal possible way, so let's give up on progress entirely"?  This is
a good proposal that addresses environment portability and gives Python a
substantially better build-artifact story than it currently has, in the
environment most desperately needing one (server-side linux).  Could it be
better?  Of course.  It could be lots better.  There are lots of use-cases
for dynamically linked wheels and fancy new platform library features in
newer linuxes.  But that can all come later, and none of it needs to have
an impact on this specific proposal, right now.



I'm unsure how more specific and featureful distro tags will help, unless
you start building more than one binary version of a wheel.  From a distro
point of view I only can discourage using such wheels outside a virtual
environment, and I would like to see a possibility to easily identify such
wheels, something like loading a binary kernel module is tainting the
kernel. This way distros can point users to the provider of such wheels.



This sounds like a good idea to me. Do you have a specific idea on how you
would like to see the feature work ?


Not really yet.  lsmod shows you this information on demand for the running 
kernel.  So maybe you want to add such vendor information into the egg, or in 
the compiled extension, and in the interpreter?  Then give a warning when the 
interpreter is called with a new option and the vendor informations don't match, 
or even error out. Of course interested distros would have to backport such a 
patch to older releases, because that's the target market for the pep.


Matthias

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Matthias Klose]

On 16.02.2016 15:20, Paul Moore wrote:

On 16 February 2016 at 14:14, Wayne Werner  wrote:

I've learned that *usually* linux distro repos lag way behind in updating
their Python packages, so unless I *can't* install the package via pip,
that's what I do.


that's how distros work. They provide a stable set of packages which is known to 
work together. Updates are limited usually to bug fixes and security updates. 
Distros do provide updated packages using e.g. backports or e.g. software 
collections. Distros rely themself on a stable python, a lot of system software 
already is written in Python.



Yeah, and that's what I'd count as an issue between you and your
distro. If they don't provide sufficiently up to date versions for
you, and you choose to deal with that in whatever way you prefer,
that's fine by me.


It depends what you are interested in. A Java developer probably will never 
update python modules and just use what it available, while you would not update 
Java libraries. Substitute Java with whatever you are not that interested in.



I don't see why the Python community shouldn't provide a solution that
you can use in such a situation, simply because it's not the solution
your distro would prefer you to use.


Sure, however I gave examples what can go wrong, and best thing, it would be 
good to see these addressed, or at least given better diagnostics if something 
goes wrong.



Of course, to my knowledge I've never replaced a system installed version
of anything. Though, considering I've been using Python3 since it was available
and most distros use Python 2, that may not really be saying much :)


I thought the distro "hands off" rules applied even to adding things
to system-managed directories, not just to overwriting files?


yes, you can do this, and usually each distro at least includes some locations 
to install such things, usually in /usr/local, some in /opt.


For Debian/Ubuntu you can install additional modules into

  /usr/local/lib/pythonX.Y/dist-packages

Yes, this *can* break system installed modules. But it allows you to distinguish 
what was installed by the system, and what by yourself.


And it doesn't get in the way with a python configure with just ./configure.

Matthias

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[David Cournapeau]

On Tue, Feb 16, 2016 at 11:05 AM, Matthias Klose  wrote:

> On 02.02.2016 02:35, Glyph Lefkowitz wrote:
>
>>
>> On Feb 1, 2016, at 3:37 PM, Matthias Klose  wrote:
>>>
>>> On 30.01.2016 00:29, Nathaniel Smith wrote:
>>>
 Hi all,

 I think this is ready for pronouncement now -- thanks to everyone for
 all their feedback over the last few weeks!

>>>
>>> I don't think so.  I am biased because I'm the maintainer for Python in
>>> Debian/Ubuntu.  So I would like to have some feedback from maintainers of
>>> Python in other Linux distributions (Nick, no, you're not one of these).
>>>
>>
>> Possibly, but it would be very helpful for such maintainers to limit
>> their critique to "in what scenarios will this fail for users" and not have
>> the whole peanut gallery chiming in with "well on _my_ platform we would
>> have done it _this_ way".
>>
>> I respect what you've done for Debian and Ubuntu, Matthias, and I use the
>> heck out of that work, but honestly this whole message just comes across as
>> sour grapes that someone didn't pick a super-old Debian instead of a
>> super-old Red Hat.  I don't think it's promoting any progress.
>>
>
> You may call this sour grapes, but in the light of people installing
> these wheels to replace/upgrade system installed eggs, it becomes an
> issue. It's fine to use such wheels in a virtual environment, however
> people tell users to use these wheels to replace system installed packages,
> distros will have a problem identifying issues.
>

FWIW, I often point out when people put "sudo" and "pip" in the same
sentence.

What about adding some language around this to the PEP ?


In the future, more specific and featureful distro tags sound like a good
>> idea.  But could we please stop making the default position on
>> distutils-sig "this doesn't cater to my one specific environment in the
>> most optimal possible way, so let's give up on progress entirely"?  This is
>> a good proposal that addresses environment portability and gives Python a
>> substantially better build-artifact story than it currently has, in the
>> environment most desperately needing one (server-side linux).  Could it be
>> better?  Of course.  It could be lots better.  There are lots of use-cases
>> for dynamically linked wheels and fancy new platform library features in
>> newer linuxes.  But that can all come later, and none of it needs to have
>> an impact on this specific proposal, right now.
>>
>
> I'm unsure how more specific and featureful distro tags will help, unless
> you start building more than one binary version of a wheel.  From a distro
> point of view I only can discourage using such wheels outside a virtual
> environment, and I would like to see a possibility to easily identify such
> wheels, something like loading a binary kernel module is tainting the
> kernel. This way distros can point users to the provider of such wheels.
>

This sounds like a good idea to me. Do you have a specific idea on how you
would like to see the feature work ?

David

>
> This is not a "this doesn't cater to my one specific environment"
> position. Of course you probably get less push back from other
> distributions closer to the specific environment specified in the pep.  But
> please acknowledge that there might be issues that you and me don't yet
> even see, and provide a possibility to identify such issues.
>
> At least Debian/Ubuntu took a long ride to avoid "accidental" interaction
> with local Python installations and local installations into the system
> installed Python.  For me this PEP feels like a step back, promising too
> much (manylinux), not pointing out possible issues, and giving no help in
> identifying possible issues with these wheels.
>
> Matthias
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Paul Moore]

On 16 February 2016 at 14:14, Wayne Werner  wrote:
> I've learned that *usually* linux distro repos lag way behind in updating
> their Python packages, so unless I *can't* install the package via pip,
> that's what I do.

Yeah, and that's what I'd count as an issue between you and your
distro. If they don't provide sufficiently up to date versions for
you, and you choose to deal with that in whatever way you prefer,
that's fine by me.

I don't see why the Python community shouldn't provide a solution that
you can use in such a situation, simply because it's not the solution
your distro would prefer you to use.

> Of course, to my knowledge I've never replaced a system installed version
> of anything. Though, considering I've been using Python3 since it was 
> available
> and most distros use Python 2, that may not really be saying much :)

I thought the distro "hands off" rules applied even to adding things
to system-managed directories, not just to overwriting files?

Anyway, I've already made more inflammatory comments than an outsider
should, so I'll leave the debate to the Unix users at this point.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Wayne Werner]

On Tue, 16 Feb 2016, Paul Moore wrote:


On 16 February 2016 at 11:05, Matthias Klose  wrote:

You may call this sour grapes, but in the light of people installing
these wheels to replace/upgrade system installed eggs, it becomes an issue.
It's fine to use such wheels in a virtual environment, however people tell
users to use these wheels to replace system installed packages, distros will
have a problem identifying issues.


OK, so are you not simply saying that people shouldn't be using (sudo)
pip to install packages into the system environment instead of using
the system packages?

As a non-Linux user my opinion isn't that relevant, but I don't see an
issue with a statement like this. I gather that people typically do
this when the distro packages aren't available, or don't provide a
sufficiently up to date version, but that's a separate issue.


I've learned that *usually* linux distro repos lag way behind in updating 
their Python packages, so unless I *can't* install the package via pip, 
that's what I do.


Of course, to my knowledge I've never replaced a system installed version 
of anything. Though, considering I've been using Python3 since it was 
available and most distros use Python 2, that may not really be saying 
much :)


-W
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Paul Moore]

On 16 February 2016 at 11:05, Matthias Klose  wrote:
> You may call this sour grapes, but in the light of people installing
> these wheels to replace/upgrade system installed eggs, it becomes an issue.
> It's fine to use such wheels in a virtual environment, however people tell
> users to use these wheels to replace system installed packages, distros will
> have a problem identifying issues.

OK, so are you not simply saying that people shouldn't be using (sudo)
pip to install packages into the system environment instead of using
the system packages?

As a non-Linux user my opinion isn't that relevant, but I don't see an
issue with a statement like this. I gather that people typically do
this when the distro packages aren't available, or don't provide a
sufficiently up to date version, but that's a separate issue.

I don't know to what extent using "sudo pip install" to install
packages into the system Python is considered "supported" by the pip
developers. While we're not going to get into a situation of trying to
enforce distro policies, but there's equally an implicit "but your
distro may not support this" around *any* use of "sudo pip", so it's
not as if we're *encouraging* people to do it.

IMO, manylinux wheels should be viewed as a solution for people using
virtualenvs, for people with their own Python builds, or using pyenv,
or similar. As far as interaction with the distro-packaged Python is
concerned, "you should use the system package manager to manage the
system Python" is the basic message - but if (for whatever reason)
people can't or won't do that, then I'm OK with pip supporting people
who choose to install such wheels into the system Python - if they
have issues, then we'll likely fall back on "you should take it up
with your distro", and we'll be perfectly OK with the distro's
response being "get those manylinux wheels out of our managed
environment".

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Matthias Klose]

On 02.02.2016 02:35, Glyph Lefkowitz wrote:



On Feb 1, 2016, at 3:37 PM, Matthias Klose  wrote:

On 30.01.2016 00:29, Nathaniel Smith wrote:

Hi all,

I think this is ready for pronouncement now -- thanks to everyone for
all their feedback over the last few weeks!


I don't think so.  I am biased because I'm the maintainer for Python in 
Debian/Ubuntu.  So I would like to have some feedback from maintainers of 
Python in other Linux distributions (Nick, no, you're not one of these).


Possibly, but it would be very helpful for such maintainers to limit their critique to "in 
what scenarios will this fail for users" and not have the whole peanut gallery chiming in with 
"well on _my_ platform we would have done it _this_ way".

I respect what you've done for Debian and Ubuntu, Matthias, and I use the heck 
out of that work, but honestly this whole message just comes across as sour 
grapes that someone didn't pick a super-old Debian instead of a super-old Red 
Hat.  I don't think it's promoting any progress.


You may call this sour grapes, but in the light of people installing
these wheels to replace/upgrade system installed eggs, it becomes an issue. It's 
fine to use such wheels in a virtual environment, however people tell users to 
use these wheels to replace system installed packages, distros will have a 
problem identifying issues.


There is a substantial amount of extensions built using C++; I didn't check how 
many of these in c++0x/c++11 mode.  Until GCC 5, the c++11 ABI wasn't stable, 
and upstream never promised forward compatibility, something that even distros 
have to care about (usually by rebuilding packages before a release).  So if you 
want a lowest common denominator, then maybe limit or recommend the use of c++98 
only.



The proposal just takes some environment and declares that as a standard.  So 
everybody wanting to supply these wheels basically has to use this environment.


There's already been lots of discussion about how this environment is a lowest 
common denominator.  Many other similar environments could _also_ be lowest 
common denominator.


sure, but then please call it what it is. centos5 or somelinux1.


In the future, more specific and featureful distro tags sound like a good idea.  But 
could we please stop making the default position on distutils-sig "this doesn't 
cater to my one specific environment in the most optimal possible way, so let's give up 
on progress entirely"?  This is a good proposal that addresses environment 
portability and gives Python a substantially better build-artifact story than it 
currently has, in the environment most desperately needing one (server-side linux).  
Could it be better?  Of course.  It could be lots better.  There are lots of use-cases 
for dynamically linked wheels and fancy new platform library features in newer linuxes.  
But that can all come later, and none of it needs to have an impact on this specific 
proposal, right now.


I'm unsure how more specific and featureful distro tags will help, unless you 
start building more than one binary version of a wheel.  From a distro point of 
view I only can discourage using such wheels outside a virtual environment, and 
I would like to see a possibility to easily identify such wheels, something like 
loading a binary kernel module is tainting the kernel. This way distros can 
point users to the provider of such wheels.


This is not a "this doesn't cater to my one specific environment" position. Of 
course you probably get less push back from other distributions closer to the 
specific environment specified in the pep.  But please acknowledge that there 
might be issues that you and me don't yet even see, and provide a possibility to 
identify such issues.


At least Debian/Ubuntu took a long ride to avoid "accidental" interaction with 
local Python installations and local installations into the system installed 
Python.  For me this PEP feels like a step back, promising too much (manylinux), 
not pointing out possible issues, and giving no help in identifying possible 
issues with these wheels.


Matthias

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Donald Stufft]

> On Feb 10, 2016, at 3:18 PM, Matthias Klose  wrote:
> 
> But then why call it manylinux instead of centos5? You build it on this OS, 
> you expect others to build it on this OS. just name it what it is.


Because this is a very specific subset of CentOS 5 that has shown to, in 
practice, work cross distro into the vast majority of glibc using Linux 
distributions. The idea here is that if you restrict yourself to these subset 
of libraries, you’ll produce a wheel that can be installed on say, Debian 
(assuming a recent enough Debian is used, which is why we’re using the very 
ancient CentOS 5, so it’s sufficiently old) and not have ABI issues to contend 
with.

“manylinux” is a nicer name than 
“builtoncentos5butinawaythatisusableonmanyotherlinuxsystems”.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Matthias Klose]

On 02.02.2016 01:30, Donald Stufft wrote:



On Feb 1, 2016, at 6:37 PM, Matthias Klose  wrote:

On 30.01.2016 00:29, Nathaniel Smith wrote:

Hi all,

I think this is ready for pronouncement now -- thanks to everyone for
all their feedback over the last few weeks!


I don't think so.  I am biased because I'm the maintainer for Python in 
Debian/Ubuntu.  So I would like to have some feedback from maintainers of 
Python in other Linux distributions (Nick, no, you're not one of these).

The proposal just takes some environment and declares that as a standard.  So 
everybody wanting to supply these wheels basically has to use this environment. 
Without giving any details, without giving any advise how to produce such 
wheels in other environments. Without giving any hints how such wheels may be 
broken with newer environments.


I’m not sure this is true. It tells you exactly what versions of glibc and 
other libraries it is allowed to link against. It can link against older if it 
wants, it can’t link against newer.


Without mentioning this is am64/i386 only.


First sentence: This PEP proposes the creation of a new platform tag for Python 
package built distributions, such as wheels, calledmanylinux1_{x86_64,i686} 
with external dependencies limited to a standardized, restricted subset of the 
Linux kernel and core userspace ABI.


I read "such as wheels, calledmanylinux1_{x86_64,i686}" as not limited to such 
platforms.



Later on: Because CentOS 5 is only available for x86_64 and i686 architectures, 
these are the only architectures currently supported by the manylinux1 policy.


sorry, didn't see that.


I think it’s a reasonable policy too, AMD64 is responsible for an order of 
magnitude more downloads than all other architectures on Linux combined 
(71,424,040 vs 1,086,527 in current data set). If you compare AMD64+i386 
against everything else then you’re looking at two orders of magnitude 
(72,142,511 vs 368,056). I think we can live with a solution that covers 99.5% 
of all Linux downloads from PyPI.


But then why call it manylinux instead of centos5? You build it on this OS, you 
expect others to build it on this OS. just name it what it is.



There might be more. Pretty please be specific about your environment.  Have a 
look how the LSB specifies requirements on the runtime environment ... and then 
ask yourself why the lsb doesn't have any real value.



Instead of vague references to the LSB, can you tell us why you think the LSB 
doesn’t have any real value, and importantly, how that relates to trying to 
determine a minimum set of binary ABI. In addition, can you clarify why, if 
your assertion is this isn’t going to work, can you state why it won’t work 
when it is working for many user’s in the wild through Anaconda, Enthought, the 
Holy Build Box, etc?


I'm not seeing the LSB used in real life (anymore), and not any recent updates. 
Furthermore, LSB packages were removed in Debian [1].


[1] https://lists.debian.org/debian-lsb/2015/07/msg0.html, 
https://lists.debian.org/debian-lsb/2015/07/msg2.html


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Donald Stufft]

> On Feb 6, 2016, at 4:06 AM, Marius Gedminas  wrote:
> 
> On Fri, Feb 05, 2016 at 12:56:05PM -0500, Nate Coraor wrote:
>> On Fri, Feb 5, 2016 at 12:52 PM, Nathaniel Smith  wrote:
>> 
>>> On Feb 5, 2016 9:35 AM, "Nate Coraor"  wrote:
 Now that pip and wheel both support the Python 3 SOABI tags on 2.X, is
>>> this necessary? The ABI tag should be set correctly on both the build and
>>> installation systems, so is including it as part of the manylinux1 ABI (and
>>> fixing it to wide builds only) overkill?
> ...
>>> Also, just to confirm, the new releases of pip and wheel enable this for
>>> 2.x; is it also available for all 3.x?
>>> 
>>> -n
>>> 
>> 
>> ABI tags always worked with wheel/pip on CPython 3.2+ (it has the SOABI
>> config var), the new change "backports" this functionality to 2.X.
> 
> What are the minimum versions of pip and wheel to have this working
> correctly on 2.x?
> 
> Marius Gedminas
> --
> A "critic" is a man who creates nothing and thereby feels qualified to judge
> the work of creative men. There is logic in this; he is unbiased -- he hates
> all creative people equally.
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

pip 8 and wheel 0.27 (or 0.28, I think there was a bug with 0.27)

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Marius Gedminas]

On Fri, Feb 05, 2016 at 12:56:05PM -0500, Nate Coraor wrote:
> On Fri, Feb 5, 2016 at 12:52 PM, Nathaniel Smith  wrote:
> 
> > On Feb 5, 2016 9:35 AM, "Nate Coraor"  wrote:
> > > Now that pip and wheel both support the Python 3 SOABI tags on 2.X, is
> > this necessary? The ABI tag should be set correctly on both the build and
> > installation systems, so is including it as part of the manylinux1 ABI (and
> > fixing it to wide builds only) overkill?
...
> > Also, just to confirm, the new releases of pip and wheel enable this for
> > 2.x; is it also available for all 3.x?
> >
> > -n
> >
> 
> ABI tags always worked with wheel/pip on CPython 3.2+ (it has the SOABI
> config var), the new change "backports" this functionality to 2.X.

What are the minimum versions of pip and wheel to have this working
correctly on 2.x?

Marius Gedminas
-- 
A "critic" is a man who creates nothing and thereby feels qualified to judge
the work of creative men. There is logic in this; he is unbiased -- he hates
all creative people equally.


signature.asc
Description: Digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nate Coraor]

On Sat, Jan 30, 2016 at 3:37 AM, Nathaniel Smith  wrote:

> On Fri, Jan 29, 2016 at 11:52 PM, Nick Coghlan  wrote:
> > On 30 January 2016 at 09:29, Nathaniel Smith  wrote:
> >> Hi all,
> >>
> >> I think this is ready for pronouncement now -- thanks to everyone for
> >> all their feedback over the last few weeks!
> >>
> >> The only change relative to the last posting is that we rewrote the
> >> section on "Platform detection for installers", to switch to letting
> >> distributors explicitly control manylinux1 compatibility by means of a
> >> _manylinux module.
> >
> > In terms of the proposal itself, I think this version is excellent :)
> >
> > However, I realised that there's an implicit assumption we've been
> > making that really should be spelled out explicitly: manylinux1 wheels
> > targeting CPython 3.2 and earlier need to be compiled against a
> > CPython built in wide Unicode mode, and in those cases, the detection
> > of manylinux1 compatibility at the platform level should include
> > checking for "sys.maxunicode > 0x".
>
> Doh, excellent catch!
>
> I've just pushed the obvious update to handle this directly to the
> copy of the PEP in the manylinux repository.
>
> Diff:
> https://github.com/manylinux/manylinux/commit/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc
>
> New text in full:
>
> https://raw.githubusercontent.com/manylinux/manylinux/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc/pep-513.rst
>
> I haven't sent to the PEP editors, because they already have another
> diff from me sitting in their inboxes and I'm not sure how to do this
> in a way that doesn't confuse things :-)
>

Now that pip and wheel both support the Python 3 SOABI tags on 2.X, is this
necessary? The ABI tag should be set correctly on both the build and
installation systems, so is including it as part of the manylinux1 ABI (and
fixing it to wide builds only) overkill?

--nate


>
> > The main reason we need to spell this out explicitly is that while
> > distros (and I believe other redistributors) build CPython-for-Linux
> > in wide mode as a matter of course, a Linux checkout of CPython 2.7
> > will build in narrow mode by default.
>
> I can confirm that Debian and Anaconda builds of CPython 2.7 both have
> sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
> 0x. Hmm. I guess they should fix that.
>
> Also the manylinux docker image currently has sys.maxunicode ==
> 0x, so we should definitely fix that :-).
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nathaniel Smith]

On Feb 5, 2016 9:35 AM, "Nate Coraor"  wrote:
>
> On Sat, Jan 30, 2016 at 3:37 AM, Nathaniel Smith  wrote:
>>
>> On Fri, Jan 29, 2016 at 11:52 PM, Nick Coghlan 
wrote:
>> > On 30 January 2016 at 09:29, Nathaniel Smith  wrote:
>> >> Hi all,
>> >>
>> >> I think this is ready for pronouncement now -- thanks to everyone for
>> >> all their feedback over the last few weeks!
>> >>
>> >> The only change relative to the last posting is that we rewrote the
>> >> section on "Platform detection for installers", to switch to letting
>> >> distributors explicitly control manylinux1 compatibility by means of a
>> >> _manylinux module.
>> >
>> > In terms of the proposal itself, I think this version is excellent :)
>> >
>> > However, I realised that there's an implicit assumption we've been
>> > making that really should be spelled out explicitly: manylinux1 wheels
>> > targeting CPython 3.2 and earlier need to be compiled against a
>> > CPython built in wide Unicode mode, and in those cases, the detection
>> > of manylinux1 compatibility at the platform level should include
>> > checking for "sys.maxunicode > 0x".
>>
>> Doh, excellent catch!
>>
>> I've just pushed the obvious update to handle this directly to the
>> copy of the PEP in the manylinux repository.
>>
>> Diff:
https://github.com/manylinux/manylinux/commit/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc
>>
>> New text in full:
>>
https://raw.githubusercontent.com/manylinux/manylinux/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc/pep-513.rst
>>
>> I haven't sent to the PEP editors, because they already have another
>> diff from me sitting in their inboxes and I'm not sure how to do this
>> in a way that doesn't confuse things :-)
>
>
> Now that pip and wheel both support the Python 3 SOABI tags on 2.X, is
this necessary? The ABI tag should be set correctly on both the build and
installation systems, so is including it as part of the manylinux1 ABI (and
fixing it to wide builds only) overkill?

Hah, I just asked the same thing :-). Clearly I should finish scrolling
through my inbox before replying...

Anyway, I would like to know this as well. :-)

Also, just to confirm, the new releases of pip and wheel enable this for
2.x; is it also available for all 3.x?

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nick Coghlan]

On 2 Feb 2016 10:45, "Matthias Klose"  wrote:
>
> On 30.01.2016 00:29, Nathaniel Smith wrote:
>>
>> Hi all,
>>
>> I think this is ready for pronouncement now -- thanks to everyone for
>> all their feedback over the last few weeks!
>
>
> I don't think so.  I am biased because I'm the maintainer for Python in
Debian/Ubuntu.  So I would like to have some feedback from maintainers of
Python in other Linux distributions (Nick, no, you're not one of these).

The RHEL ABI maintainers raised very similar concerns (that's how I found
out about the current status of libabigail), but they're in a similar
position to you: Linux distros provide *much* stronger guarantees of ABI
compatibility than this PEP even attempts to provide.

Here's the key distinction though: not everything is a mission critical
production server, and we know from cross-distro redistributors like
Continuum Analytics & Enthought, as well as our own upstream experience
with the extraordinarily informal ABIs for Windows and Mac OS X wheels that
an underspecified approach like manylinux is likely to be good enough in
practice.

That puts the power in users' hands:

* by default, most users will get to use developer provided builds, rather
than having to build from source (bringing the Linux user experience up to
parity with that for Windows and Mac OS X)
* folks can still force the use of source builds if they prefer (e.g. to
better optimise for their OS and hardware, or to change the threat profile
for their open source project usage)
* folks can still opt to use a third party build service if they prefer
(whether that's a Linux distro or a cross platform redistributor)

Cheers,
Nick.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Matthew Brett]

On Sun, Jan 31, 2016 at 11:25 PM, Marius Gedminas  wrote:
> On Sat, Jan 30, 2016 at 12:17:02PM -0800, Matthew Brett wrote:
>> Hi,
>>
>> > I can confirm that Debian and Anaconda builds of CPython 2.7 both have
>> > sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
>> > 0x. Hmm. I guess they should fix that.
>> >
>> > Also the manylinux docker image currently has sys.maxunicode ==
>> > 0x, so we should definitely fix that :-).
>>
>> A quick check on Ubuntu 12.04, Debian sid, Centos 7.2 confirms wide
>> unicode by default.  Are there any known distributions providing UCS2
>> unicode Pythons?
>
> I don't know of any.

I also tested on Debian wheezy 32-bit; Fedora 22 (32-bit packages);
openSUSE 13.2, recent Arch and Gentoo, these are all wide unicode.

> Pyenv, OTOH, deliberately uses upstream defaults and so produces narrow
> unicode builds.

Ouch - good to know.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Matthias Klose]

On 30.01.2016 00:29, Nathaniel Smith wrote:

Hi all,

I think this is ready for pronouncement now -- thanks to everyone for
all their feedback over the last few weeks!


I don't think so.  I am biased because I'm the maintainer for Python in 
Debian/Ubuntu.  So I would like to have some feedback from maintainers of Python 
in other Linux distributions (Nick, no, you're not one of these).


The proposal just takes some environment and declares that as a standard.  So 
everybody wanting to supply these wheels basically has to use this environment. 
Without giving any details, without giving any advise how to produce such wheels 
in other environments. Without giving any hints how such wheels may be broken 
with newer environments.  Without mentioning this is am64/i386 only.
There might be more. Pretty please be specific about your environment.  Have a 
look how the LSB specifies requirements on the runtime environment ... and then 
ask yourself why the lsb doesn't have any real value.


Matthias

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Donald Stufft]

> On Feb 1, 2016, at 6:37 PM, Matthias Klose  wrote:
> 
> On 30.01.2016 00:29, Nathaniel Smith wrote:
>> Hi all,
>> 
>> I think this is ready for pronouncement now -- thanks to everyone for
>> all their feedback over the last few weeks!
> 
> I don't think so.  I am biased because I'm the maintainer for Python in 
> Debian/Ubuntu.  So I would like to have some feedback from maintainers of 
> Python in other Linux distributions (Nick, no, you're not one of these).
> 
> The proposal just takes some environment and declares that as a standard.  So 
> everybody wanting to supply these wheels basically has to use this 
> environment. Without giving any details, without giving any advise how to 
> produce such wheels in other environments. Without giving any hints how such 
> wheels may be broken with newer environments.

I’m not sure this is true. It tells you exactly what versions of glibc and 
other libraries it is allowed to link against. It can link against older if it 
wants, it can’t link against newer.

> Without mentioning this is am64/i386 only.

First sentence: This PEP proposes the creation of a new platform tag for Python 
package built distributions, such as wheels, calledmanylinux1_{x86_64,i686} 
with external dependencies limited to a standardized, restricted subset of the 
Linux kernel and core userspace ABI.

Later on: Because CentOS 5 is only available for x86_64 and i686 architectures, 
these are the only architectures currently supported by the manylinux1 policy.

I think it’s a reasonable policy too, AMD64 is responsible for an order of 
magnitude more downloads than all other architectures on Linux combined 
(71,424,040 vs 1,086,527 in current data set). If you compare AMD64+i386 
against everything else then you’re looking at two orders of magnitude 
(72,142,511 vs 368,056). I think we can live with a solution that covers 99.5% 
of all Linux downloads from PyPI.

> There might be more. Pretty please be specific about your environment.  Have 
> a look how the LSB specifies requirements on the runtime environment ... and 
> then ask yourself why the lsb doesn't have any real value.
> 

Instead of vague references to the LSB, can you tell us why you think the LSB 
doesn’t have any real value, and importantly, how that relates to trying to 
determine a minimum set of binary ABI. In addition, can you clarify why, if 
your assertion is this isn’t going to work, can you state why it won’t work 
when it is working for many user’s in the wild through Anaconda, Enthought, the 
Holy Build Box, etc?


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nathaniel Smith]

On Mon, Feb 1, 2016 at 3:37 PM, Matthias Klose  wrote:
> On 30.01.2016 00:29, Nathaniel Smith wrote:
>>
>> Hi all,
>>
>> I think this is ready for pronouncement now -- thanks to everyone for
>> all their feedback over the last few weeks!
>
>
> I don't think so.  I am biased because I'm the maintainer for Python in
> Debian/Ubuntu.

Thank you for your work! I've been using Debian/Ubuntu as my sole
desktop and for Python development since ~2000, so it's very much
appreciated :-).

> So I would like to have some feedback from maintainers of
> Python in other Linux distributions (Nick, no, you're not one of these).
>
> The proposal just takes some environment and declares that as a standard.
> So everybody wanting to supply these wheels basically has to use this
> environment.

Yeah, pretty much. The spec says "we define the ABI by reference to
this environment" and if you can find some other ways to build these
wheels without using that environment, then go for it, but in practice
I doubt there's much point. CentOS 5 is widely available, free (in all
senses), and works. Even if it does require one to use weird tools
like "yum" and "rpm" whose UIs are constantly confusing me and make no
sense whatsoever [1].

[1] i.e., they aren't identical to apt / dpkg.

> Without giving any details, without giving any advise how to
> produce such wheels in other environments.

There isn't a lot of end-user-focused documentation in the PEP itself,
it's true, but that's because that's how PEPs work.

Here's an example project for building manylinux1 wheels using
Travis-CI (which runs some version of Ubuntu):

  https://github.com/pypa/python-manylinux-demo

Obviously this is all still a work in progress ("latest commit: 40
minutes ago"). Better docs are coming :-)

> Without giving any hints how such
> wheels may be broken with newer environments.

What kind of hints are you suggesting?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Alexander Walters]

On 2/1/2016 18:37, Matthias Klose wrote:

On 30.01.2016 00:29, Nathaniel Smith wrote:

Hi all,

I think this is ready for pronouncement now -- thanks to everyone for
all their feedback over the last few weeks!


I don't think so.  I am biased because I'm the maintainer for Python 
in Debian/Ubuntu.  So I would like to have some feedback from 
maintainers of Python in other Linux distributions (Nick, no, you're 
not one of these).


The proposal just takes some environment and declares that as a 
standard.  So everybody wanting to supply these wheels basically has 
to use this environment. Without giving any details, without giving 
any advise how to produce such wheels in other environments. Without 
giving any hints how such wheels may be broken with newer 
environments.  Without mentioning this is am64/i386 only.
There might be more. Pretty please be specific about your 
environment.  Have a look how the LSB specifies requirements on the 
runtime environment ... and then ask yourself why the lsb doesn't have 
any real value.


Matthias

I... Thought the environment this pep describes is the docker image, and 
only the docker image, and anything not made on that docker image is in 
violation of the pep.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Robert T. McGibbon]

On Mon, Feb 1, 2016 at 3:47 PM, Alexander Walters 
wrote:

>
>
> On 2/1/2016 18:37, Matthias Klose wrote:
>
>> On 30.01.2016 00:29, Nathaniel Smith wrote:
>>
>>> Hi all,
>>>
>>> I think this is ready for pronouncement now -- thanks to everyone for
>>> all their feedback over the last few weeks!
>>>
>>
>> I don't think so.  I am biased because I'm the maintainer for Python in
>> Debian/Ubuntu.  So I would like to have some feedback from maintainers of
>> Python in other Linux distributions (Nick, no, you're not one of these).
>>
>> The proposal just takes some environment and declares that as a
>> standard.  So everybody wanting to supply these wheels basically has to use
>> this environment. Without giving any details, without giving any advise how
>> to produce such wheels in other environments. Without giving any hints how
>> such wheels may be broken with newer environments.  Without mentioning this
>> is am64/i386 only.
>> There might be more. Pretty please be specific about your environment.
>> Have a look how the LSB specifies requirements on the runtime environment
>> ... and then ask yourself why the lsb doesn't have any real value.
>>
>> Matthias
>>
>> I... Thought the environment this pep describes is the docker image, and
> only the docker image, and anything not made on that docker image is in
> violation of the pep.



It's not correct that anything made outside the docker image is in
violation of the PEP. The docker images are just tools that can help you
compile compliant wheels. Nathaniel and I tried to describe this as
precisely as we could. See this section
 of the PEP.

To comply with the policy, the wheel needs to (a not link against any other
external libraries beyond those mentioned in the PEP, (b) *work* on a stock
CentOS 5.11 machine, and (c) not use any narrow-unicode symbols (only
relevant for Python < 3.2). A consequence of requirements (a) and (b) is
that versioned symbols that are referenced in the depended-upon shared
libraries need to use sufficiently old versions of the symbols, which are
noted in the PEP as well. In order to satisfy this aspect of the policy,
the easiest route, from our experience, is to compile the wheel inside a
CentOS 5 environment, but other routes are possible, including

 - statically link everything
 - use your favorite linux distro, but install an older version of glibc
and configure your compiler to point to that.
 - use some inline assembly to instruct the linker to prefer older symbol
versions in libraries like glibc.
 - etc.

I also wrote the auditwheel command line tool that can check to see if a
wheel is manylinux1 compatible, and give you some feedback about what to
fix if it's not.

And furthermore, I've just put up an example project on github that you can
use as a template for compiling manylinux1 wheels using Travis-CI. You can
find it here: https://github.com/pypa/python-manylinux-demo

-Robert
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Marius Gedminas]

On Sat, Jan 30, 2016 at 12:17:02PM -0800, Matthew Brett wrote:
> Hi,
> 
> > I can confirm that Debian and Anaconda builds of CPython 2.7 both have
> > sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
> > 0x. Hmm. I guess they should fix that.
> >
> > Also the manylinux docker image currently has sys.maxunicode ==
> > 0x, so we should definitely fix that :-).
> 
> A quick check on Ubuntu 12.04, Debian sid, Centos 7.2 confirms wide
> unicode by default.  Are there any known distributions providing UCS2
> unicode Pythons?

I don't know of any.

Pyenv, OTOH, deliberately uses upstream defaults and so produces narrow
unicode builds.

Marius Gedminas
-- 
If you are angry with someone, you should walk a mile in their shoes... then
you'll be a mile away from them, and you'll have their shoes.


signature.asc
Description: Digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nick Coghlan]

On 31 January 2016 at 02:49, Donald Stufft  wrote:
>
>> On Jan 30, 2016, at 3:58 AM, Nick Coghlan  wrote:
>>
>> I also think this version covers everything we need it to cover, so
>> I'm going to mark it as Active and point to this post as the
>> resolution :)
>
> Hilariously I just read on Hacker news:
>
> "Incidentay, we have hired Natanael Copa, the awesome creator of Alpine Linux 
> and are in the process of switching the Docker official image library from 
> ubuntu to Alpine.” [1]
>
> So we might end up needing a MUSL based platform tag in the near future ;)

Aye, I saw that, too. Alpine is currently the only MUSL based distro
I'm aware of, so it may be enough to ensure that we include Alpine in
the list of distros we ensure are correctly supported by
distro-specific wheel tags.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Glyph Lefkowitz]

> On Feb 1, 2016, at 3:37 PM, Matthias Klose  wrote:
> 
> On 30.01.2016 00:29, Nathaniel Smith wrote:
>> Hi all,
>> 
>> I think this is ready for pronouncement now -- thanks to everyone for
>> all their feedback over the last few weeks!
> 
> I don't think so.  I am biased because I'm the maintainer for Python in 
> Debian/Ubuntu.  So I would like to have some feedback from maintainers of 
> Python in other Linux distributions (Nick, no, you're not one of these).

Possibly, but it would be very helpful for such maintainers to limit their 
critique to "in what scenarios will this fail for users" and not have the whole 
peanut gallery chiming in with "well on _my_ platform we would have done it 
_this_ way".

I respect what you've done for Debian and Ubuntu, Matthias, and I use the heck 
out of that work, but honestly this whole message just comes across as sour 
grapes that someone didn't pick a super-old Debian instead of a super-old Red 
Hat.  I don't think it's promoting any progress.

> The proposal just takes some environment and declares that as a standard.  So 
> everybody wanting to supply these wheels basically has to use this 
> environment.

There's already been lots of discussion about how this environment is a lowest 
common denominator.  Many other similar environments could _also_ be lowest 
common denominator.

> Without giving any details, without giving any advise how to produce such 
> wheels in other environments. Without giving any hints how such wheels may be 
> broken with newer environments.

They won't be.  That's the whole point.

> Without mentioning this is am64/i386 only.

Wheels already have an architecture tag, separate from the platform tag, so 
this being "am64/i386" is irrelevant.

> There might be more. Pretty please be specific about your environment.  Have 
> a look how the LSB specifies requirements on the runtime environment ... and 
> then ask yourself why the lsb doesn't have any real value.


In the future, more specific and featureful distro tags sound like a good idea. 
 But could we please stop making the default position on distutils-sig "this 
doesn't cater to my one specific environment in the most optimal possible way, 
so let's give up on progress entirely"?  This is a good proposal that addresses 
environment portability and gives Python a substantially better build-artifact 
story than it currently has, in the environment most desperately needing one 
(server-side linux).  Could it be better?  Of course.  It could be lots better. 
 There are lots of use-cases for dynamically linked wheels and fancy new 
platform library features in newer linuxes.  But that can all come later, and 
none of it needs to have an impact on this specific proposal, right now.

-glyph

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[David Cournapeau]

On Sat, Jan 30, 2016 at 8:37 AM, Nathaniel Smith  wrote:

> On Fri, Jan 29, 2016 at 11:52 PM, Nick Coghlan  wrote:
> > On 30 January 2016 at 09:29, Nathaniel Smith  wrote:
> >> Hi all,
> >>
> >> I think this is ready for pronouncement now -- thanks to everyone for
> >> all their feedback over the last few weeks!
> >>
> >> The only change relative to the last posting is that we rewrote the
> >> section on "Platform detection for installers", to switch to letting
> >> distributors explicitly control manylinux1 compatibility by means of a
> >> _manylinux module.
> >
> > In terms of the proposal itself, I think this version is excellent :)
> >
> > However, I realised that there's an implicit assumption we've been
> > making that really should be spelled out explicitly: manylinux1 wheels
> > targeting CPython 3.2 and earlier need to be compiled against a
> > CPython built in wide Unicode mode, and in those cases, the detection
> > of manylinux1 compatibility at the platform level should include
> > checking for "sys.maxunicode > 0x".
>
> Doh, excellent catch!
>
> I've just pushed the obvious update to handle this directly to the
> copy of the PEP in the manylinux repository.
>
> Diff:
> https://github.com/manylinux/manylinux/commit/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc
>
> New text in full:
>
> https://raw.githubusercontent.com/manylinux/manylinux/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc/pep-513.rst
>
> I haven't sent to the PEP editors, because they already have another
> diff from me sitting in their inboxes and I'm not sure how to do this
> in a way that doesn't confuse things :-)
>
> > The main reason we need to spell this out explicitly is that while
> > distros (and I believe other redistributors) build CPython-for-Linux
> > in wide mode as a matter of course, a Linux checkout of CPython 2.7
> > will build in narrow mode by default.
>
> I can confirm that Debian and Anaconda builds of CPython 2.7 both have
> sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
> 0x. Hmm. I guess they should fix that.
>

Yes, they should :) I am not sure why it was built this way (before my
time), it is unfortunately not easy to fix when you have a large existing
customer base.

David


> Also the manylinux docker image currently has sys.maxunicode ==
> 0x, so we should definitely fix that :-).
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nick Coghlan]

On 30 January 2016 at 23:45, Oscar Benjamin  wrote:
> On 30 January 2016 at 08:58, Nick Coghlan  wrote:
>>
>> I applied both this iteration and the previous one to the PEPs repo in
>> order to review it, so modulo caching issues, this latest draft is
>> live now.
>>
>> I also think this version covers everything we need it to cover, so
>> I'm going to mark it as Active and point to this post as the
>> resolution :)
>
> I had to see PEP 1 to understand what "Active" means but now I see
> that it means that this PEP is approved but subject to indefinite
> tinkering:
> https://www.python.org/dev/peps/pep-0001/#pep-review-resolution

Yeah, getting us away from Information PEPs to normal Standards Track
ones is one of the reasons I started tinkering with the way we use the
PEP process. We really do want to be using Standards Track, but that
requires reference specs and implementations that provide the basis
for declaring the PEP "FInal".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nathaniel Smith]

On Sat, Jan 30, 2016 at 12:37 AM, Nathaniel Smith  wrote:
> Also the manylinux docker image currently has sys.maxunicode ==
> 0x, so we should definitely fix that :-).

This is fixed now (or will be in a few minutes when this build
finishes: 
https://quay.io/repository/manylinux/manylinux/build/40961d45-aff6-4ed2-a778-f73413340800)

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nathaniel Smith]

On Sat, Jan 30, 2016 at 12:58 AM, Nick Coghlan  wrote:
> I also think this version covers everything we need it to cover, so
> I'm going to mark it as Active and point to this post as the
> resolution :)

Awesome! Thanks everyone :-)

Of course, as soon as this happened I discovered two small but
substantive bugs in the text...

- the text refers to 32-bit linux as "i386" (because I foolishly
trusted PEP 425); it turns out that on any modern system the platform
tag is "linux_i686".
- there was a small bug in the is_manylinux1_compatible sample code,
because I missed that distutils.util.get_platform() uses a hyphen
separator instead of an underscore. (But now the sample code is
running as part of the test suite for the docker image, so I know it
works.)

Both are fixed in the diff that I just sent to the pep editors:
  https://github.com/manylinux/manylinux/pull/20/files

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Donald Stufft]

> On Jan 30, 2016, at 3:58 AM, Nick Coghlan  wrote:
> 
> I also think this version covers everything we need it to cover, so
> I'm going to mark it as Active and point to this post as the
> resolution :)

Hilariously I just read on Hacker news:

"Incidentay, we have hired Natanael Copa, the awesome creator of Alpine Linux 
and are in the process of switching the Docker official image library from 
ubuntu to Alpine.” [1]

So we might end up needing a MUSL based platform tag in the near future ;)


[1] https://news.ycombinator.com/item?id=11000827

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nathaniel Smith]

On Fri, Jan 29, 2016 at 11:52 PM, Nick Coghlan  wrote:
> On 30 January 2016 at 09:29, Nathaniel Smith  wrote:
>> Hi all,
>>
>> I think this is ready for pronouncement now -- thanks to everyone for
>> all their feedback over the last few weeks!
>>
>> The only change relative to the last posting is that we rewrote the
>> section on "Platform detection for installers", to switch to letting
>> distributors explicitly control manylinux1 compatibility by means of a
>> _manylinux module.
>
> In terms of the proposal itself, I think this version is excellent :)
>
> However, I realised that there's an implicit assumption we've been
> making that really should be spelled out explicitly: manylinux1 wheels
> targeting CPython 3.2 and earlier need to be compiled against a
> CPython built in wide Unicode mode, and in those cases, the detection
> of manylinux1 compatibility at the platform level should include
> checking for "sys.maxunicode > 0x".

Doh, excellent catch!

I've just pushed the obvious update to handle this directly to the
copy of the PEP in the manylinux repository.

Diff: 
https://github.com/manylinux/manylinux/commit/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc

New text in full:
https://raw.githubusercontent.com/manylinux/manylinux/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc/pep-513.rst

I haven't sent to the PEP editors, because they already have another
diff from me sitting in their inboxes and I'm not sure how to do this
in a way that doesn't confuse things :-)

> The main reason we need to spell this out explicitly is that while
> distros (and I believe other redistributors) build CPython-for-Linux
> in wide mode as a matter of course, a Linux checkout of CPython 2.7
> will build in narrow mode by default.

I can confirm that Debian and Anaconda builds of CPython 2.7 both have
sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
0x. Hmm. I guess they should fix that.

Also the manylinux docker image currently has sys.maxunicode ==
0x, so we should definitely fix that :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Robert T. McGibbon]

Oh yes, this is very important! I will have to put a check in auditwheel as
well to verify this in the extensions too.

-Robert
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Oscar Benjamin]

On 30 January 2016 at 08:58, Nick Coghlan  wrote:
>
> I applied both this iteration and the previous one to the PEPs repo in
> order to review it, so modulo caching issues, this latest draft is
> live now.
>
> I also think this version covers everything we need it to cover, so
> I'm going to mark it as Active and point to this post as the
> resolution :)

I had to see PEP 1 to understand what "Active" means but now I see
that it means that this PEP is approved but subject to indefinite
tinkering:
https://www.python.org/dev/peps/pep-0001/#pep-review-resolution

Brilliant, good work everyone! I'm looking forward to this.

So AFAICT the actions are:
1) PyPI allows uploading wheels with manylinux tag.
2) pip updated to recognise these wheels on the appropriate Linux systems.
3) Packagers make and upload wheels

Is there also a policy of informing python-dev once a PEP is approved
here? PEP 1 is ambiguous on that point but something like this really
needs to be known about more widely than this list.

--
Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nick Coghlan]

On 30 January 2016 at 09:29, Nathaniel Smith  wrote:
> Hi all,
>
> I think this is ready for pronouncement now -- thanks to everyone for
> all their feedback over the last few weeks!
>
> The only change relative to the last posting is that we rewrote the
> section on "Platform detection for installers", to switch to letting
> distributors explicitly control manylinux1 compatibility by means of a
> _manylinux module.

In terms of the proposal itself, I think this version is excellent :)

However, I realised that there's an implicit assumption we've been
making that really should be spelled out explicitly: manylinux1 wheels
targeting CPython 3.2 and earlier need to be compiled against a
CPython built in wide Unicode mode, and in those cases, the detection
of manylinux1 compatibility at the platform level should include
checking for "sys.maxunicode > 0x".

The main reason we need to spell this out explicitly is that while
distros (and I believe other redistributors) build CPython-for-Linux
in wide mode as a matter of course, a Linux checkout of CPython 2.7
will build in narrow mode by default.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Matthew Brett]

Hi,

> I can confirm that Debian and Anaconda builds of CPython 2.7 both have
> sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
> 0x. Hmm. I guess they should fix that.
>
> Also the manylinux docker image currently has sys.maxunicode ==
> 0x, so we should definitely fix that :-).

A quick check on Ubuntu 12.04, Debian sid, Centos 7.2 confirms wide
unicode by default.  Are there any known distributions providing UCS2
unicode Pythons?

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nathaniel Smith]

Hi all,

I think this is ready for pronouncement now -- thanks to everyone for
all their feedback over the last few weeks!

The only change relative to the last posting is that we rewrote the
section on "Platform detection for installers", to switch to letting
distributors explicitly control manylinux1 compatibility by means of a
_manylinux module.

-n

---

PEP: 513
Title: A Platform Tag for Portable Linux Built Distributions
Version: $Revision$
Last-Modified: $Date$
Author: Robert T. McGibbon , Nathaniel J. Smith

BDFL-Delegate: Nick Coghlan 
Discussions-To: Distutils SIG 
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 19-Jan-2016
Post-History: 19-Jan-2016, 25-Jan-2016, 29-Jan-2016


Abstract


This PEP proposes the creation of a new platform tag for Python package built
distributions, such as wheels, called ``manylinux1_{x86_64,i386}`` with
external dependencies limited to a standardized, restricted subset of
the Linux kernel and core userspace ABI. It proposes that PyPI support
uploading and distributing wheels with this platform tag, and that ``pip``
support downloading and installing these packages on compatible platforms.


Rationale
=

Currently, distribution of binary Python extensions for Windows and OS X is
straightforward. Developers and packagers build wheels [1]_ [2]_, which are
assigned platform tags such as ``win32`` or ``macosx_10_6_intel``, and upload
these wheels to PyPI. Users can download and install these wheels using tools
such as ``pip``.

For Linux, the situation is much more delicate. In general, compiled Python
extension modules built on one Linux distribution will not work on other Linux
distributions, or even on different machines running the same Linux
distribution with different system libraries installed.

Build tools using PEP 425 platform tags [3]_ do not track information about the
particular Linux distribution or installed system libraries, and instead assign
all wheels the too-vague ``linux_i386`` or ``linux_x86_64`` tags. Because of
this ambiguity, there is no expectation that ``linux``-tagged built
distributions compiled on one machine will work properly on another, and for
this reason, PyPI has not permitted the uploading of wheels for Linux.

It would be ideal if wheel packages could be compiled that would work on *any*
linux system. But, because of the incredible diversity of Linux systems -- from
PCs to Android to embedded systems with custom libcs -- this cannot
be guaranteed in general.

Instead, we define a standard subset of the kernel+core userspace ABI that,
in practice, is compatible enough that packages conforming to this standard
will work on *many* linux systems, including essentially all of the desktop
and server distributions in common use. We know this because there are
companies who have been distributing such widely-portable pre-compiled Python
extension modules for Linux -- e.g. Enthought with Canopy [4]_ and Continuum
Analytics with Anaconda [5]_.

Building on the compability lessons learned from these companies, we thus
define a baseline ``manylinux1`` platform tag for use by binary Python
wheels, and introduce the implementation of preliminary tools to aid in the
construction of these ``manylinux1`` wheels.


Key Causes of Inter-Linux Binary Incompatibility


To properly define a standard that will guarantee that wheel packages meeting
this specification will operate on *many* linux platforms, it is necessary to
understand the root causes which often prevent portability of pre-compiled
binaries on Linux. The two key causes are dependencies on shared libraries
which are not present on users' systems, and dependencies on particular
versions of certain core libraries like ``glibc``.


External Shared Libraries
-

Most desktop and server linux distributions come with a system package manager
(examples include ``APT`` on Debian-based systems, ``yum`` on
``RPM``-based systems, and ``pacman`` on Arch linux) that manages, among other
responsibilities, the installation of shared libraries installed to system
directories such as ``/usr/lib``. Most non-trivial Python extensions will depend
on one or more of these shared libraries, and thus function properly only on
systems where the user has the proper libraries (and the proper
versions thereof), either installed using their package manager, or installed
manually by setting certain environment variables such as ``LD_LIBRARY_PATH``
to notify the runtime linker of the location of the depended-upon shared
libraries.


Versioning of Core Shared Libraries
---

Even if the developers a Python extension module wish to use no
external shared libraries, the modules will generally have a dynamic runtime
dependency on the GNU C library, ``glibc``. While it is possible, statically
linking ``glibc``