Re: [Distutils] Working toward Linux wheel support

[Leonardo Rochael Almeida]

Hi,

Going back in time to this old post, but I think it becomes more relevant
now that Nate's work is being completed:

On 13 August 2015 at 22:47, Nathaniel Smith  wrote:

> On Thu, Aug 13, 2015 at 12:30 PM, Leonardo Rochael Almeida
>  wrote:
> >
> > On 13 August 2015 at 11:07, Nate Coraor  wrote:
> >>
> >> On Wed, Aug 12, 2015 at 9:05 PM, Nathaniel Smith  wrote:
> >>>
> [...]
> >>> (2) the special hard-coded tag "centos5". (That's what everyone
> actually
> >>> uses in practice, right?)
> >>
> >>
> >> The idea here is that we should attempt to install centos5 wheels if
> more
> >> specific wheels for the platform aren't available?
> >
> >
> > Just my opinion, but although I'm +1 on Nate's efforts, I'm -1 on both
> the
> > standard behavior for installation being the exact platform tag, and an
> > automatic fallback to cento5.
> >
> > IMO, on Linux, the default should always be to opt in to the desired
> > platform tags.
> >
> > We could make it so that the word `default` inside
> > `binary-compatibility.cfg` means an exact match on the distro version, so
> > that we could simplify the documentation.
> >
> > But I don't want to upgrade to pip and suddenly find myself installing
> > binary wheels compiled by whomever for whatever platform I have no
> control
> > with, even assuming the best of the package builders intentions.
> >
> > And I certainly don't want centos5 wheels accidentally installed on my
> > ubuntu servers unless I very specifically asked for them.
> >
> > The tiny pain inflicted by telling users to add a one-line text file in a
> > very well known location (or two lines, for the added centos5), so that
> they
> > can get the benefit of binary wheels on linux, is very small compared to
> the
> > pain of repeatable install scripts suddenly behaving differently and
> > installing binary wheels in systems that were prepared to pay the price
> of
> > source installs, including the setting of build environment variables
> that
> > correctly tweaked their build process.
>
> I think there are two issues here:
>
> 1) You don't want centos5 wheels "accidentally" installed on an ubuntu
> server: Fair enough, you're right; we should probably make the "this
> wheel should work on pretty much any linux out there" tag be something
> that distributors have to explicitly opt into (similar to how they
> have to opt into creating universal wheels), rather than having it be
> something you could get by just typing 'pip wheel foo' on the right
> (wrong) machine.
>

I agree that generating something like "this linux binary wheel is
generically installable" should be opt-in, yes. But I also feel strongly
that installing such a generic wheel should also be opt in.

I guess that if we go into the direction of being able to generate wheels
with a libc tag rather than a distro tag, like Nate and Donald are now
discussing, then we could get both kinds of opt-in by specifying the libc
tag in `binary-compatibility.cfg`.


> 2) You want it to be the case that if I type 'pip install foo' on a
> Linux machine, and pip finds both an sdist and a wheel, where the
> wheel is definitely compatible with the current system, then it should
> still always prefer the sdist unless configured otherwise: Here I
> disagree strongly. This is inconsistent with how things work on every
> other platform, it's inconsistent with how pip is being used on Linux
> right now with private wheelhouses, and the "tiny pain" of editing a
> file in /etc is a huge barrier to new users, many of whom are
> uncomfortable editing config files and may not have root access.


Not having root access shouldn't be an issue, as there should be a
user-level and virtualenv level equivalents to a `binary-compatibility.cfg`
on `/etc`, and perhaps it could even be included in `requirements.txt` for
a project, so users of a project might not even have to bother setting up
`binary-compatibility.cfg`.

However, you make an excellent point: not handling binary wheels on Linux
by default (at least with exact platform tag matching) would mean having
different behavior between Linux and Mac/Windows.

Still, I wouldn't want a random binary wheel suddenly finding its way into
my servers, and I would like a way to opt out of it, for "reasons" (ex. I
might have special build flags, or a special compiler, or maybe I'm still
waiting for TUF before trusting other peoples binaries on my servers).

So I'd like to propose that the installation tooling (eg. `pip`, `distlib`)
should allow the user to specify which index servers to trust for receiving
binary wheels and which to trust only for pure python wheels or sdists.

It could trust them all by default, to maintain the current behavior (where
`all` means only pypi unless I specified more, obviously), but I'd like a
switch to limit this trust to a subset of the specified index servers.

Regards,

Leo
___
Distutils-SIG 

Re: [Distutils] Working toward Linux wheel support

[Wes Turner]

On Sep 8, 2015 1:33 PM, "Donald Stufft"  wrote:
>
> On September 8, 2015 at 1:29:53 PM, Nate Coraor (n...@bx.psu.edu) wrote:
> > On Mon, Sep 7, 2015 at 12:02 PM, Donald Stufft wrote:
> >
> > > On September 3, 2015 at 1:23:03 PM, Nate Coraor (n...@bx.psu.edu)
wrote:
> > > > >>>
> > > > >>> I'll create PRs for this against wheel and pip shortly. I can
also
> > > work
> > > > >>> on a PEP for the platform tag - I don't think it's going to
need to
> > > be a
> > > > >>> big one. Are there any preferences as to whether this should be
a
> > > new PEP
> > > > >>> or an update to 425?
> > > > >>>
> > >
> > > Coming back to this, I'm wondering if we should include the libc
> > > implementation/version in a less generic, but still generic linux
wheel.
> > > Right
> > > now if you staticly link I think the only platform ABIs you need to
worry
> > > about
> > > are libc and Python itself. Python itself is handled already but libc
is
> > > not.
> >
> > The only thing I've seen so far is "build on an old enough version of
glibc
> > > that it handles anything sane", but not all versions of Linux even use
> > > glibc at
> > > all.
> >
> >
> > This proposal makes a lot of sense to me. pip will need an update to do
the
> > backwards compatibility, and it may be a little ugly to do this all on
the
> > platform tag. For example, linux_x86_64_ubuntu_12_04 wheels should not
be
> > installed on systems that identify as linux_x86_64_ubuntu_14_04, but
> > linux_x86_64_glibc_2_15 wheels can be installed on systems that
identify as
> > linux_x86_64_glibc_2_19. pip would need to maintain a list of which tag
> > prefixes or patterns should be considered backward compatible, and which
> > should not. Granted, new libcs do not pop up overnight, so it's not
exactly
> > a nightmare scenario.

Could there be shim packages here?
How is this a different dependency?

> >
> > Wheel should be updated to generate the "libc-generic" wheels by default
> > when nothing other than libc is dynamically linked. It'll need libc
> > vendor/version detection.
> >
> > Alternatively, the platform tag could be split in two, in which case you
> > have a "generic" portion (which would probably be what it currently is,
> > distutils.util.get_platform()) and a "specific" portion (the distro or
> > libc), possibly prefixed with something to avoid having to maintain a
list
> > of what's version compatible and what's not, (e.g. 'd_ubuntu_14_04' vs.
> > 'c_glibc_2_19')?
> >
> > I don't think there is a strong case to include the libc version in the
> > specific portion when a distro version will also be specified, because
the
> > distro is supposed to define the ABI (at least in the case of distros
with
> > stable ABIs), and that includes the libc compatibility. So for psycopg2
> > wheels you'd get a "distro" wheel (linux_x86_64-d_ubuntu_14_04) but for
> > SQLAlchemy, you'd get a "libc-generic" wheel
(linux_x86_64-c_glibc_2_19).
> >
> > It's then up to PyPI project owners to build on whatever platforms they
> > wish to support.
> >
>
> I think it's reasonable to not include the libc when the wheel is distro
> specific. I think the barrier to entry on adding new tags is far lower
than
> adding a whole new type of tag. Right now, I think our longest tag is for
OSX
> which is something like macosx_10_10_x86_64 at 19 chars, I don't think
it's
> much worse to have something like linux_glibc_2_19_x86_64 at 23 chars, or
> linux_ubuntu_14_04_x86_64 at 25 chars. I don't think we need the special
c or
> d prefix, we can just treat it as ==, and special case glibc as >= like
we're
> currently special casing the macosx wheels to be >=.
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Daniel Holth]

https://www.python.org/dev/peps/pep-0425/#platform-tag is currently defined
in terms of distutils get_platform(). Instead, it could be defined more
abstractly to read something like "The platform tag expresses which
system(s) might be capable of running or linking with binary components of
the package." This would express what the tag is for rather than the list
of allowed values. Then a "legal" change in the list of allowed values
would not necessarily be effected by changing the distutils get_platform
function.

As for whether a binary is allowed from a particular server the idea of
using a different list of compatible/allowed tags per-package-source has
floated around. Distasteful amount of configuration though. Something like
the Internet Explorer security zones where you have categories of remotes...

On Tue, Sep 8, 2015 at 3:14 PM Wes Turner  wrote:

>
> On Sep 8, 2015 1:33 PM, "Donald Stufft"  wrote:
> >
> > On September 8, 2015 at 1:29:53 PM, Nate Coraor (n...@bx.psu.edu) wrote:
> > > On Mon, Sep 7, 2015 at 12:02 PM, Donald Stufft wrote:
> > >
> > > > On September 3, 2015 at 1:23:03 PM, Nate Coraor (n...@bx.psu.edu)
> wrote:
> > > > > >>>
> > > > > >>> I'll create PRs for this against wheel and pip shortly. I can
> also
> > > > work
> > > > > >>> on a PEP for the platform tag - I don't think it's going to
> need to
> > > > be a
> > > > > >>> big one. Are there any preferences as to whether this should
> be a
> > > > new PEP
> > > > > >>> or an update to 425?
> > > > > >>>
> > > >
> > > > Coming back to this, I'm wondering if we should include the libc
> > > > implementation/version in a less generic, but still generic linux
> wheel.
> > > > Right
> > > > now if you staticly link I think the only platform ABIs you need to
> worry
> > > > about
> > > > are libc and Python itself. Python itself is handled already but
> libc is
> > > > not.
> > >
> > > The only thing I've seen so far is "build on an old enough version of
> glibc
> > > > that it handles anything sane", but not all versions of Linux even
> use
> > > > glibc at
> > > > all.
> > >
> > >
> > > This proposal makes a lot of sense to me. pip will need an update to
> do the
> > > backwards compatibility, and it may be a little ugly to do this all on
> the
> > > platform tag. For example, linux_x86_64_ubuntu_12_04 wheels should not
> be
> > > installed on systems that identify as linux_x86_64_ubuntu_14_04, but
> > > linux_x86_64_glibc_2_15 wheels can be installed on systems that
> identify as
> > > linux_x86_64_glibc_2_19. pip would need to maintain a list of which tag
> > > prefixes or patterns should be considered backward compatible, and
> which
> > > should not. Granted, new libcs do not pop up overnight, so it's not
> exactly
> > > a nightmare scenario.
>
> Could there be shim packages here?
> How is this a different dependency?
>
> > >
> > > Wheel should be updated to generate the "libc-generic" wheels by
> default
> > > when nothing other than libc is dynamically linked. It'll need libc
> > > vendor/version detection.
> > >
> > > Alternatively, the platform tag could be split in two, in which case
> you
> > > have a "generic" portion (which would probably be what it currently is,
> > > distutils.util.get_platform()) and a "specific" portion (the distro or
> > > libc), possibly prefixed with something to avoid having to maintain a
> list
> > > of what's version compatible and what's not, (e.g. 'd_ubuntu_14_04' vs.
> > > 'c_glibc_2_19')?
> > >
> > > I don't think there is a strong case to include the libc version in the
> > > specific portion when a distro version will also be specified, because
> the
> > > distro is supposed to define the ABI (at least in the case of distros
> with
> > > stable ABIs), and that includes the libc compatibility. So for psycopg2
> > > wheels you'd get a "distro" wheel (linux_x86_64-d_ubuntu_14_04) but for
> > > SQLAlchemy, you'd get a "libc-generic" wheel
> (linux_x86_64-c_glibc_2_19).
> > >
> > > It's then up to PyPI project owners to build on whatever platforms they
> > > wish to support.
> > >
> >
> > I think it's reasonable to not include the libc when the wheel is distro
> > specific. I think the barrier to entry on adding new tags is far lower
> than
> > adding a whole new type of tag. Right now, I think our longest tag is
> for OSX
> > which is something like macosx_10_10_x86_64 at 19 chars, I don't think
> it's
> > much worse to have something like linux_glibc_2_19_x86_64 at 23 chars, or
> > linux_ubuntu_14_04_x86_64 at 25 chars. I don't think we need the special
> c or
> > d prefix, we can just treat it as ==, and special case glibc as >= like
> we're
> > currently special casing the macosx wheels to be >=.
> >
> > -
> > Donald Stufft
> > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
> >
> >
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org

Re: [Distutils] Working toward Linux wheel support

[Leonardo Rochael Almeida]

That's nice for singling out some packages (though I only found a , but I
had a different use-case in mind, which I guess I didn't fully articulate:

I might want binary wheels for some packages, just not coming from PyPI,
where I don't necessarily trust whatever was put there. I'm perfectly fine
trusting binary wheels coming from my own wheelhouse, for example.

So, I'd rather have a:

  --accept-binary-from=http://mywheelhouse.example.com

Which would accept binary from all provided indexes if absent

Or perhaps a:

  --no-binary-from=https://pypi.python.org/simple

Regards,

Leo

On 8 September 2015 at 16:22, Donald Stufft  wrote:

> On September 8, 2015 at 3:21:26 PM, Leonardo Rochael Almeida (
> leoroch...@gmail.com) wrote:
> >
> > Still, I wouldn't want a random binary wheel suddenly finding its way
> into
> > my servers, and I would like a way to opt out of it, for "reasons" (ex. I
> > might have special build flags, or a special compiler, or maybe I'm still
> > waiting for TUF before trusting other peoples binaries on my servers).
> >
>
> —no-binary packages,that,have,binaries ?
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Nathaniel Smith]

On Mon, Sep 7, 2015 at 9:02 AM, Donald Stufft  wrote:
> On September 3, 2015 at 1:23:03 PM, Nate Coraor (n...@bx.psu.edu) wrote:
>> >>>
>> >>> I'll create PRs for this against wheel and pip shortly. I can also work
>> >>> on a PEP for the platform tag - I don't think it's going to need to be a
>> >>> big one. Are there any preferences as to whether this should be a new PEP
>> >>> or an update to 425?
>> >>>
>
> Coming back to this, I'm wondering if we should include the libc
> implementation/version in a less generic, but still generic linux wheel. Right
> now if you staticly link I think the only platform ABIs you need to worry 
> about
> are libc and Python itself. Python itself is handled already but libc is not.
> The only thing I've seen so far is "build on an old enough version of glibc
> that it handles anything sane", but not all versions of Linux even use glibc 
> at
> all.

This feels kinda half-baked to me?

"linux" is a useful tag because it has a clear meaning: "there exists
a linux system somewhere that can run this, but no guarantees about
which one, good luck". When building a wheel it's easy to tell whether
this tag can be correctly applied.

Distro-specific tags are useful because they also have a fairly clear
meaning: "here's a specific class of systems that can run this, so
long as you install enough packages to fulfill the external
dependencies". Again, when building a wheel it's pretty easy to tell
whether this tag can be correctly applied. (Of course someone could
screw this up, e.g. by building on a system is technically distro X
but has some incompatible hand-compiled libraries installed, but 99%
of the time we can guess correctly.)

If we define a LSB-style base system and give it a tag, like I don't
know, the "Python base environment", call it "linux_pybe1_core" or
something, that describes what libraries are and aren't available and
their ABIs, and provide docs/tooling to help people explicitly create
such wheels and check whether they're compatible with their system,
then this is also useful -- we have proof that this is sufficient to
actually distribute arbitrary software usefully, given that multiple
distributors have converged on this strategy. (I've been talking to
some people off-list about maybe actually putting together a proposal
like this...)

To me "linux_glibc_2.18" falls between the cracks though. If this
starts being what you get by default when you build a wheel, then
people will use it for wheels that are *not* statically linked, and
what that tag will mean is "there exists some system that can run
this, and that has glibc 2.18 on it, and also some other unspecified
stuff, good luck". Which is pretty useless -- we might as well just
stick with "linux" in this case. OTOH if it's something that builders
have to opt into, then we could document that it's only to be used for
wheels that are statically linked except for glibc, and make it mean
"*any* system which has glibc 2.18 or later on it can run this". Which
would be useful in some cases. But at this point it's basically a
version of the "defined base environment" approach, and once you've
gone that far you might as well take advantage of the various
distributors' experience about what should actually be in that
environment -- glibc isn't enough.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Nate Coraor]

On Mon, Sep 7, 2015 at 12:02 PM, Donald Stufft  wrote:

> On September 3, 2015 at 1:23:03 PM, Nate Coraor (n...@bx.psu.edu) wrote:
> > >>>
> > >>> I'll create PRs for this against wheel and pip shortly. I can also
> work
> > >>> on a PEP for the platform tag - I don't think it's going to need to
> be a
> > >>> big one. Are there any preferences as to whether this should be a
> new PEP
> > >>> or an update to 425?
> > >>>
>
> Coming back to this, I'm wondering if we should include the libc
> implementation/version in a less generic, but still generic linux wheel.
> Right
> now if you staticly link I think the only platform ABIs you need to worry
> about
> are libc and Python itself. Python itself is handled already but libc is
> not.

The only thing I've seen so far is "build on an old enough version of glibc
> that it handles anything sane", but not all versions of Linux even use
> glibc at
> all.


This proposal makes a lot of sense to me. pip will need an update to do the
backwards compatibility, and it may be a little ugly to do this all on the
platform tag. For example, linux_x86_64_ubuntu_12_04 wheels should not be
installed on systems that identify as linux_x86_64_ubuntu_14_04, but
linux_x86_64_glibc_2_15 wheels can be installed on systems that identify as
linux_x86_64_glibc_2_19. pip would need to maintain a list of which tag
prefixes or patterns should be considered backward compatible, and which
should not. Granted, new libcs do not pop up overnight, so it's not exactly
a nightmare scenario.

Wheel should be updated to generate the "libc-generic" wheels by default
when nothing other than libc is dynamically linked. It'll need libc
vendor/version detection.

Alternatively, the platform tag could be split in two, in which case you
have a "generic" portion (which would probably be what it currently is,
distutils.util.get_platform()) and a "specific" portion (the distro or
libc), possibly prefixed with something to avoid having to maintain a list
of what's version compatible and what's not, (e.g. 'd_ubuntu_14_04' vs.
'c_glibc_2_19')?

I don't think there is a strong case to include the libc version in the
specific portion when a distro version will also be specified, because the
distro is supposed to define the ABI (at least in the case of distros with
stable ABIs), and that includes the libc compatibility. So for psycopg2
wheels you'd get a "distro" wheel (linux_x86_64-d_ubuntu_14_04) but for
SQLAlchemy, you'd get a "libc-generic" wheel (linux_x86_64-c_glibc_2_19).

It's then up to PyPI project owners to build on whatever platforms they
wish to support.

--nate


>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Donald Stufft]

On September 8, 2015 at 1:29:53 PM, Nate Coraor (n...@bx.psu.edu) wrote:
> On Mon, Sep 7, 2015 at 12:02 PM, Donald Stufft wrote:
> 
> > On September 3, 2015 at 1:23:03 PM, Nate Coraor (n...@bx.psu.edu) wrote:
> > > >>>
> > > >>> I'll create PRs for this against wheel and pip shortly. I can also
> > work
> > > >>> on a PEP for the platform tag - I don't think it's going to need to
> > be a
> > > >>> big one. Are there any preferences as to whether this should be a
> > new PEP
> > > >>> or an update to 425?
> > > >>>
> >
> > Coming back to this, I'm wondering if we should include the libc
> > implementation/version in a less generic, but still generic linux wheel.
> > Right
> > now if you staticly link I think the only platform ABIs you need to worry
> > about
> > are libc and Python itself. Python itself is handled already but libc is
> > not.
> 
> The only thing I've seen so far is "build on an old enough version of glibc
> > that it handles anything sane", but not all versions of Linux even use
> > glibc at
> > all.
> 
> 
> This proposal makes a lot of sense to me. pip will need an update to do the
> backwards compatibility, and it may be a little ugly to do this all on the
> platform tag. For example, linux_x86_64_ubuntu_12_04 wheels should not be
> installed on systems that identify as linux_x86_64_ubuntu_14_04, but
> linux_x86_64_glibc_2_15 wheels can be installed on systems that identify as
> linux_x86_64_glibc_2_19. pip would need to maintain a list of which tag
> prefixes or patterns should be considered backward compatible, and which
> should not. Granted, new libcs do not pop up overnight, so it's not exactly
> a nightmare scenario.
> 
> Wheel should be updated to generate the "libc-generic" wheels by default
> when nothing other than libc is dynamically linked. It'll need libc
> vendor/version detection.
> 
> Alternatively, the platform tag could be split in two, in which case you
> have a "generic" portion (which would probably be what it currently is,
> distutils.util.get_platform()) and a "specific" portion (the distro or
> libc), possibly prefixed with something to avoid having to maintain a list
> of what's version compatible and what's not, (e.g. 'd_ubuntu_14_04' vs.
> 'c_glibc_2_19')?
> 
> I don't think there is a strong case to include the libc version in the
> specific portion when a distro version will also be specified, because the
> distro is supposed to define the ABI (at least in the case of distros with
> stable ABIs), and that includes the libc compatibility. So for psycopg2
> wheels you'd get a "distro" wheel (linux_x86_64-d_ubuntu_14_04) but for
> SQLAlchemy, you'd get a "libc-generic" wheel (linux_x86_64-c_glibc_2_19).
> 
> It's then up to PyPI project owners to build on whatever platforms they
> wish to support.
> 

I think it's reasonable to not include the libc when the wheel is distro
specific. I think the barrier to entry on adding new tags is far lower than
adding a whole new type of tag. Right now, I think our longest tag is for OSX
which is something like macosx_10_10_x86_64 at 19 chars, I don't think it's
much worse to have something like linux_glibc_2_19_x86_64 at 23 chars, or
linux_ubuntu_14_04_x86_64 at 25 chars. I don't think we need the special c or
d prefix, we can just treat it as ==, and special case glibc as >= like we're
currently special casing the macosx wheels to be >=.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig