Re: python devs complaining about debian packaging

[Donald Stufft]

It's not an accurate characterization that distutils was removed simply because 
it wasn't maintained.

It was as fragile library, and it was difficult to make any changes to it 
because a number of things had implemented themselves by reaching into 
distutils and randomly monkeypatching various aspects. This made it hard to 
maintain it _at all_, because when people had tried to improve it, it caused 
ecosystem wide breakages until things like setuptools, numpy.distutils, etc got 
patched again.

So the unofficial policy became "do not touch distutils", for fear of causing 
these breakages.

In the interim the packaging toolchain evolved to the point that having 
distutils in the stdlib was no longer of general benefit, and in fact made 
things worse because people had grown accustomed to things like `from distutils 
import setup` being transparently monkeypatched to be setuptools under the 
covers.

Ultimately, distutils had diverged so far from modern packaging tooling (due to 
the unofficial policy of not touching it), that it's existence was more or less 
a footgun, and it's only real purpose anymore was to be an implementation 
detail of other libraries, which you had to install from PyPI, so it was 
decided it was better to remove it rather than leave it around.
On 6/3/2024 1:37:32 AM, Salvo Tomaselli  wrote:
Consider that they are the same people that recently removed
"distutils" from the standard library, because it was not maintained.
When they have well enough funding to assign someone to maintain it,
instead of relying on external projects to install packages.

I think they are in the bubble of "people who are here on discuss" and
forgot the 99.9% who is not.

Il giorno lun 3 giu 2024 alle ore 00:08 Paul Boddie
ha scritto:
>
> On Monday, 27 May 2024 04:07:34 CEST Scott Kitterman wrote:
> >
> > While there are technical concerns on both sides, socially I think the
> > Python community isn't that interested in outside perspectives.
>
> I managed to dig up these notes from the packaging summit at PyCon:
>
> https://hackmd.io/@pradyunsg/pycon2024-pack-summit
>
> Here's the summit page itself:
>
> https://us.pycon.org/2024/events/packaging-summit/
>
> There is some fixation on the "system Python" in distributions, and the
> following remarks:
>
> "At least one distro team is working on moving their own Python out of the
> way, so users can install their own Python packages [...] Fedora tried
> platform-python and it broke everything, so it didn't really work"
>
> Given the proliferation of "virtual environments" around Python, where you
> just pick your own Python version and accompanying packages, I find it odd
> that the Python packaging community gets so hung up on the system Python. Do
> they want it to just go away or not be on the path or something? Wouldn't
> having a singular upstream Python just cause the same problems when someone
> finds that it isn't the version they need?
>
> For my own amusement and to confirm my own memories, I went back in time to
> check the Python Web site in the 1990s, and back then there was no problem in
> providing a binary for Linux and the Unix products of choice from that era.
> AIX, FreeBSD, HP-UX, Irix, OSF/1, Solaris, and SunOS 4, plus a Linux binary
> supplied either as an RPM or in a plain archive:
>
> https://web.archive.org/web/20021030010019/http://www.python.org/ftp/python/
> binaries-1.4/
>
> What is stopping anyone from doing that all over again? The user downloads the
> binary, puts it in their current directory, and runs their software. Could it
> be that the burden of support is perhaps a little greater than one might
> expect?
>
> Because from that starting point, you have to build multiple versions, and
> then you have to build accompanying libraries, and then you have to support
> third-party packages which need third-party libraries. It wasn't a surprise
> that things like Sun Freeware (http://www.sunfreeware.com) emerged to cater to
> the proprietary platforms, whereas distributions emerged around Linux to
> manage this complexity and provide all this software.
>
> It is easy for the various language communities to focus only on their own
> ecosystems, but everybody's software has to work together. And then there are
> companies targeting various markets that demand software built on a selection
> of different technologies, so you get perspectives like these:
>
> "Why did the PyPI and Conda ecosystem get created? It was originally created
> as an educational teaching language. If all of your tools are in Python, all
> of the things in your ecosystem are supposed to work well together. However,
> the tools that scientists and data scientists use are very commonly written in
> C, C++, etc. and so there’s something called a “native binary problem”. Making
> this stuff compatible across the board is an incredibly challenging issue!
> Conda was created to resolve those binary compatibility issues."
>
> I honestly don't know what these people 

Re: python devs complaining about debian packaging

[Donald Stufft]

I happen to be subscribed here, so figured I'd comment :)

FWIW I think the way the discussions are going... really in both locations.. is 
needlessly taking shots at each other.

I've commented on discuss.python.org, but figured I'd repeat myself here.

I think the way these discussions devolve into each "side" taking shots at each 
other and arguing over nonsense that doesn't matter does everyone a disservice. 

The facts of the matter, as far as I can tell:

- Distributors ship Python and a number of people find great value in that and 
it works great or at least fine for them.
- A number of users have needs or wants that are not well served by the Python 
that distributors ship, for one reason or another.

People in either of those groups sniping at each other for wanting the "wrong" 
things is completely unproductive. 

It's probably "fine" if Python.org wants to ship a linux binary, I suspect 
it'll have very little impact on distributors of Python (and might even make 
things better-- since some of the problems flow from an impedance mismatch, and 
it provides a different escape hatch to point people towards if the distro 
Python doesn't suit their needs).

On 5/26/2024 9:39:25 AM, Stefano Rivera  wrote:
Hi Ian (2024.05.26_01:33:09_+)
> I am puzzled about some of the responses there, how can anyone expect to
> randomly update packages on the system using pip and not have it go wrong
> on any distribution? This is why things like pipenv exist.

People don't understand that stuff until they dig to the details. And
even then, sometimes they forget and/or assume we have the resources to
massively revamp our stack.

There are long-standing grievances here (see tiran's gist for example).
I have very little experience with Fedora/RH, but from their grumbling,
I assume they solve some problems for developers that we don't:

1. It appears that multiple versions of cPython are trivially available.
Each version has its own site-packages.
2. Their python packaging is more developer-centric than user-centric.
More modules included in the install.

We're pretty constrained on 1 by the debian security team policy. But
maybe there is discussion room there for non-supported Pythons?

We have taken steps to improve 2 by adding the python3-full package.

I could see a long term strategy for having a system-python package that
provides a python3 binary used by Debian packages that need Python. And
a separate python package for developers that installs all the bells and
whistles immediately.

Achieving this would require reorganizing the way we package almost all
Python, and I don't think we have the interest and resources to do that.

If pyenv was packaged in Debian (ITP #978149) it would probably be great
for new Python developers on Debian.

Not sure what other small tactical steps we could take?

> > Perhaps someone else wants to comment on that conversation

Replied. I've engaged with these guys on this stuff before. Let's see
where it goes...

Stefano

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272


[163d8144-a584-4d1d-9d71-f86da864fde2]

Re: can pip be made using local Debian packages for any dependencies

[Donald Stufft]

I'm pretty sure that most if not all debian packages already ship the required 
information for pip to see them as installed, and if they are installed and 
they satisfy the dependency constraints that pip has for those projects, then 
they'll be used.

The question of having pip automatically install a debian package instead of 
using PyPI basically comes down to a few problems:

- pip doesn't know how to invoke apt (and yum, and Conda, and everything else).
- how do we translate "foo" on PyPI to the equivalent package in Debian (and 
Fedora, and Conda, etc)
- how do we handle virtual environments?

Nobody has ever come up with a particularly good answer to those problems, so 
pip upstream hasn't been able to even consider doing something like that.

Of course Debian could patch it's copy of pip to do that, that would make some 
of those questions easier, but not all of them, and would drastically diverge 
it's behavior from the "expected" behavior, which makes it a rough idea in 
general for Debian to diverge that drastically.
On 2/12/2023 3:18:55 AM, Ian Norton  wrote:
https://packaging.python.org/en/latest/specifications/recording-installed-packages/
defines the python spec where a package such as pyparsing would create
a tree of files under:
site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is
essentially a sha256-based manifest of files and some others.

On Sun, 12 Feb 2023 at 08:12, Ian Norton wrote:
>
> You've made me wonder if it would be feasible to have a debian-centric
> tool that populates .dist-info from debs?
>
> On Sun, 12 Feb 2023 at 08:05, Ian Norton wrote:
> >
> > I requested this kind of thing from the pip folks as
> > https://github.com/pypa/pip/issues/11644 and others have requested
> > similar, such as https://github.com/pypa/pip/issues/11607
> >
> > On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon wrote:
> > >
> > > Hey.
> > >
> > > I hope this is not too off topic.
> > >
> > > As far as I understand, dh-python, when building packages somehow
> > > automatically uses the Debian package names and even prevents e.g.
> > > setuptools from downloading any dependencies by setting a (hopefully
> > > not running) proxy.
> > >
> > >
> > > I wondered whether it's possible to make tools like pip and setuptools
> > > directly use the Debian python packages when resolving dependencies.
> > >
> > > The main motivation are security constraints, so I had to configure
> > > pip so that it cannot just download packages from PyPI (which is
> > > rather easy, simply setting no-index in pip.conf).
> > >
> > > But then of course it also fails to e.g. do an editable install of a
> > > locally developed package, when it tries to resolve the dependencies.
> > >
> > > So I wondered whether it's possible to prevent pip from downloading
> > > any remote stuff, while still resolving dependencies (respectively
> > > consider them as being resolved) *if* the package is locally installed
> > > from the Debian archive?
> > > (If a dependency isn't installed from a package it may of course fail.)
> > >
> > >
> > > Thanks,
> > > Philippe.
> > >
> > > PS: Please keep me CCed.
> > >


[88f0dcef-4969-48af-98f1-0c69549b4875]

Re: [RFC] DPT Policy: Canonise recommendation against PyPi-provided upstream source tarballs

[Donald Stufft]

File names on PyPI are write once. Once a specific file name has been used it 
can never be used again (even if the entire project was deleted and recreated). 

Projects can delete uploaded files (and as mentioned they can be yanked, but 
yanking is just extra metadata beside the file), but file content can never 
change, only be removed. 

Sent from my iPhone

> On Jun 25, 2021, at 11:47 PM, Brian Thompson  wrote:
> 
> On Fri, Jun 25, 2021 at 07:01:39PM -0400, Nicholas D Steeves wrote:
>> Does PyPi provide immutable releases?
> 
> From experience, I can tell you that yes, releases cannot be overwritten,
> but they can be "yanked".  Pypi states that a yanked release is:
> 
>  "A release that is always ignored by an installer, unless it is the
>  only release that matches a version specifier (using either '==' or
>  '===)."
> 
> -- 
> Best regards,
> 
> Brian T

Re: the new PyPI, coming next month

[Donald Stufft]

> On Apr 1, 2018, at 2:27 AM, Dominik George  wrote:
> 
> Hi,
> 
>> To be clear, PGP signatures can still be uploaded and they are still
>> available for download, they just don’t appear in the UI anymore.
> 
> So, what does the pypi.debian.net redirector use for uscan?  I imagine it
> used to scrape the website.  Can it be changed to use the JSON API?

The original PoC I wrote used the JSON API, but I don’t think what’s being 
deployed is descendant from my PoC so I have no idea what it uses, but if it’s 
not using the JSON API then yes it can be.

> 
>> Longer term I’d *like* to get rid of PGP signatures, because I think
>> their value here is actually pretty low.
> 
> I partially share this opinion, but that's a question to be discusses with
> the Debian policy people in general.  While checking a GPG signature on the
> source tarball in general is a good idea, I am afraid some developers just
> drop any key they find on first glance into the package and are done with
> it, which actually provides nothing but a false sense of safety.
> 
>> In that case they’d be replaced with TUF, but that’s a longer term
>> project.
> 
> That one?: https://github.com/theupdateframework/tuf 
> 


Yes.


> 
> Well, I can only say *please* do not remove the possibility to upload signed
> source tarballs, but leave that to the developers!
> 
> -nik
> 
> --
> PGP-Fingerprint: 3C9D 54A4 7575 C026 FB17  FD26 B79A 3C16 A0C4 F296
> 
> Dominik George · Hundeshagenstr. 26 · 53225 Bonn
> Phone: +49 228 92934581 · https://www.dominik-george.de/
> 
> Teckids e.V. · FrOSCon e.V. · Debian Developer
> 
> LPIC-3 Linux Enterprise Professional (Security)



signature.asc
Description: Message signed with OpenPGP

Re: GnuPG signatures on PyPI: why so few?

[Donald Stufft]

> On Mar 12, 2017, at 10:35 PM, Paul Wise <p...@debian.org> wrote:
> 
> On Mon, Mar 13, 2017 at 4:28 AM, Jeremy Stanley wrote:
> 
>> upload them to PyPI since the authors of the coming Warehouse
>> replacement for the current CheeseShop PyPI have already indicated
>> that they intend to drop support for signatures entirely.
> 
> Did they give any reasoning for this decision?
> 


https://mail.python.org/pipermail/distutils-sig/2016-May/028933.html 
<https://mail.python.org/pipermail/distutils-sig/2016-May/028933.html>

—
Donald Stufft

Re: GnuPG signatures on PyPI: why so few?

[Donald Stufft]

> On Mar 11, 2017, at 9:23 PM, Brian May <b...@debian.org> wrote:
> 
> Ben Finney <bign...@debian.org> writes:
> 
>> However, this only works if upstream releases are actually accompanied
>> by a valid GnuPG signature each time. The PyPI infrastructure supports
>> this; why isn't it more widely encouraged?
> 
> One reason I have found for myself: I can forget to sign the package
> when uploading to PyPI, and PyPI doesn't let you make any changes after
> the package is uploaded without changing the version (including adding
> signature file). There is a long running bug report on this, it is not
> going to get fixed (TLDR it is not a bug, it is a design feature to
> allow for caching).

I think there is no reason we *couldn’t* allow uploads of signatures after the 
fact, although it sort of makes an already tenuous security claim even more 
tenuous I think. Since distutils/setuptools don’t have reproducible “builds” of 
sdists if you wanted to after the fact upload a signature your choices are:

* Hope you still have the original files you uploaded still laying around, sign 
*them* and then upload just the signature.
* Download the files, open them up and inspect all of the files to ensure they 
match the correct state.
* Download the files, blindly sign them.

Of those options, the first one is no more or less “secure” than uploading 
alongside the original file, but the problem is that it’s impossible to 
differentiate that from the second two, and that could genuinely be worse (for 
example, compromised PyPI gives author a bad file, they sign it and upload the 
signature thus making everyone feel it is a “safe” file). Both the second 
options are kind of bad because it is incredibly easy to smuggle something bad 
into a tarball that a human being would miss.

The larger reason why I personally haven’t implemented it is because I believe 
the GPG signatures on PyPI are largely useless from a practical standpoint and 
I’m not really interested in doing anything to improve them and I would 
personally much rather just disable them completely.


> 
> Maybe there is some way of turning signatures on by default, so I don't
> have to remember for every upload, if so, I haven't been able to work it
> out yet however.

Um, I think you can set something in setup.cfg for ``setup.py upload``. I don’t 
think there is anything like that for Twine. I can’t speak for Ian but I don’t 
personally see anything inherently wrong with adding a environment variable or 
config option to twine that allows it to always sign by default. Ian may feel 
differently though!

—
Donald Stufft

Re: pip for stretch

[Donald Stufft]

> 
> On Nov 21, 2016, at 6:33 PM, Barry Warsaw <ba...@debian.org> wrote:
> 
> I have not started to look at what if anything needs to be done to transition
> to pip 9, but if you have a strong opinion one way or the other, please weigh
> in.
> 


As one might expect, I would prefer it if folks got 9.0.1 as quickly as 
possible. In particular the feature that makes it easier for upstreams to drop
Python 2 support is one that is really only effective when people can consider
pip 9 a "minimum" version of pip to support. Getting it into the hands of folks
as quickly as possible would be a big boon to that.

—
Donald Stufft

Re: /usr/bin/python2 shebangs

[Donald Stufft]

> On Nov 1, 2016, at 6:03 PM, Scott Kitterman <deb...@kitterman.com> wrote:
> 
> Even after python2.7 is removed from Debian, there will still be users who 
> keep a local copy because they couldn't migrate things due to $REASONS.  Re-
> using /usr/bin/python for a python3 version won't accomplish anything more 
> than gratuitously break such setups.


/usr/bin/python3 being Python 4.x is a bit weird though, and it’s likely that 
Python 4.x is not going to be another break the world release.

—
Donald Stufft

Re: New, updated pip coming

[Donald Stufft]

> On Jan 29, 2016, at 5:33 PM, Robert Collins <robe...@robertcollins.net> wrote:
> 
> I am confused. Here's my understanding of things...
> 
> - Pip doesn't need wheels at all - its vendoring technique doesn't use wheels.
> - Virtualenv needs to install pip, wheel, setuptools when it makes a
> new environment, and it does that by some oogly code
> - venv might do the same thing, but I haven't check its actual implementation
> - the pip-whl package was used to give virtualenv a thing to use to
> install pip when making a virtualenv
> 
> If that aligns with your understanding, I'll put it down to the prose
> you wrote rather than us actually having a different worldview; OTOH
> if it doesn't, I'd like to find out where I'm wrong, so as to improve
> my understanding.
> 
> With my understanding in mind - Can I just check something?
> 
> If a new requests package is built, uploaded to the archive and
> apt-get installed on my machine, and I then do:
> virtualenv test
> . test/bin/activate
> pip install foobar
> ^--- what version of requests will be used by this in-virtualenv
> invocation of pip?

Debian doesn’t want to utilize the bundled libraries in pip for various
reasons (some I agree with, some I don't, but suffice it to say it's a
requirement) but there is recognition that pip is a bit special here and can't
just be debundled in the standard way.

In conjunction with Barry, pip has some basic support for debundling by
deleting the contents of pip/_vendor/ except for pip/_vendor/__init__.py,
creating wheel files for all of pip's dependencies, and then making a small
patch to pip/_vendor/__init__.py to turn on the debundling feature and tell it
where those wheel files live. Pip will then add all of those wheel files to
the front of sys.path.

This gives us a few things:

* Pip will *always* use the versions of those libraries from the wheels,
  regardless of what is installed. This makes it so you can't break yourself
  with ``pip install requests`` in any environment (virtual or otherwise).

* Debian gets to have all of the code in pip come from the correct source
  packages.

These wheel files that represent pip's dependencies are generated when
python-pip is being built from the versions of the packages in Debian's
repositories and the python-pip package has a Built-Using header to indicate
which dependencies have been rebundled inside of pip through this process. This
has essentially recreated static linking but for Python. It is my understanding
that all of the wheels for pip's dependencies will exist inside of the
python-pip-whl .deb.

On top of that, virtualenv and venv will both need wheels that it needs to
install for pip, setuptools, and in the case of virtualenv, wheel. I'm not sure
what the plan is for those.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 6:02 PM, Barry Warsaw <ba...@debian.org> wrote:
> 
> On Jan 22, 2016, at 05:50 PM, Donald Stufft wrote:
> 
>> Forget that pip can fetch files from PyPI and install them for a moment and
>> consider the command ``pip install .``. Fundamentally this is similar to the
>> command ``make install`` right?
> 
> Please remind me what the long term plan for this is.  Let's say it's 2021 and
> you and your amazing team of dozens of distutil-sig developers have been
> cranking away the whole time.
> 
> What tool will be used to build Python packages?

Whatever people want! At least, that’s my hope. Ideally we get away from
the idea that any particular build system is the right build system (because
the needs of a pure python thing like six.py is vastly different from Numpy
which needs Fortran and other crazy things).

My end goal is, as long as your build system can consume a sdist (or maybe
a repository too, still up in the air, but some kind of source input) and
produce a wheel (whether this is actually a zipped up wheel file or a properly
formatted directory is also still up in the air). Then you have a good enough
build system and that is all it needs to do, it doesn’t need to also implement
installing and what not, that’s some other tool’s job.

> 
> What tool will be used to upload (built/source?) Python packages to
> WarehouseNG?

Either twine, or we’ll fold that functionality into pip.

> 
> What tool will be used to download Python packages from WarehouseNG?

Whatever tool you want! We’ll have defined formats so as long as your tool
speaks those formats it can install them. In the case of already “compiled”
(pure Python or not) binary .whl files it can just download them, unzip,
move into the correct location + install deps and such. For source files
(in a not-yet thought out or done sdist 2.0 thing) it can download the
sdist 2.0 file, install the build dependencies, invoke the build tool for
this package (from above), get the wheel it produced, and then install
that as if it had downloaded it from PyPI.

> 
> What tool will be used to install packages from WarehouseNG into an importable
> location?

Same answer here.

> 
> Cheers,
> -Barry


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 5:30 PM, Barry Warsaw <ba...@debian.org> wrote:
> 
> Hey Donald, thanks for starting this conversation.  I for one am super
> appreciative of all the consideration you give for Debian's little slice of
> the world.
> 
> There's a lot to unpack in this thread, and I'm a little under the weather[1],
> so hopefully this makes sense.
> 
> Big +1 for recording the files that get installed via the .egg-info/RECORD
> file.  As you know, I've been working on dirtbike, which is a "rewheeling"
> tool to turn an installed package into a .whl.  While Debian tightly controls
> via policy the set of wheels we'll allow into the archive, dirtbike has code
> for parsing the RECORD file.  Unfortunately this is never exercised in
> practice because we don't have RECORD files - at least not for the packages we
> care about[2].
> 
> Big +1 for using setuptools everywhere.  By my count, of the packages that I
> happen to randomly have installed on my Ubuntu 16.04 system, I have 67
> .egg-info files and 113 .egg-info directories.  I'd rather have .egg-info
> directories everywhere.
> 
> +1 for a lintian warning if distutils is used.  I guess I'm +0 on forcing that
> through pybuild because it'll be unobvious and mysterious, and kind of lets
> upstreams off the hook.  I'd mildly prefer to patch packages that use
> distutils because that's much more discoverable, but I can appreciate that
> that's a lot of work we'd be imposing on maintainers, so I won't argue this
> too much (other than to say that if pybuild forces it, let's definitely
> document this in its manpage!).
> 
> On Jan 22, 2016, at 12:40 PM, Scott Kitterman wrote:
> 
>> Currently --record includes the .pyc files which is both unneeded and bad.
>> Before this gets added either in setuptools or by us, this needed to be
>> fixed.
> 
> +1 for (I think) another reason than has already been discussed.  We won't be
> generating .egg-info directories on the end-user's machine, but instead the
> machine the package is built on.  That could be a maintainer's own system or a
> central build machine depending on various conditions.  But because the pycs
> are generated on the end-user's machine, we actually don't know what pycs will
> be generated when the debs are installed, so the egg-info/RECORD file *can't*
> contain them, at least not accurately.

FWIW It appears that if you do setup.py install —no-compile to tell distutils
not to compile the .pyc during .deb build time, I believe that as is it will
just omit the .pyc files. Not sure if that’s workable for Debian or not.

> 
> On Jan 22, 2016, at 11:54 AM, Donald Stufft wrote:
> 
>> Regardless of what happens in this thread, pip is going to stop mucking
>> around in files that are owned by some other tool without some sort of
>> opt-in/--force style flag *and* we're going to be restructuring things to try
>> and guide people away from using pip outside of a virtual environment or
>> through the user of --user as well.
> 
> Of course, I'd still like --user to be the default[3].  I think that's still
> the eventual goal for pip, but isn't yet implemented because $priorities.

Yes, still a goal (it’s not as simple as —user will be the default, but that’s
the outcome for a non root account in a system Python).

> 
>> A more controversial way that comes with possibly some extra benefits (which
>> Debian may not care about) is to use ``pip`` itself as the build tool rather
>> than directly invoking setup.py. In this pip would be responsible for mucking
>> around with the distutils/setuptools mess instead of that needing to be
>> handled by Debian specific tooling.
> 
> I'd like to better understand how this would work.  IIUC, the Fedora ecosystem
> is making or already has made this switch, but I don't know how it works.
> Obviously, we don't want to install wheels into 
> /usr/lib/python3/dist-packages.

I haven’t actually personally seen if they’ve done it, but the Fedora maintainer
of the python-pip told me they were doing it, and they had a patch to pip to
add a feature they needed for it.

Forget that pip can fetch files from PyPI and install them for a moment and
consider the command ``pip install .``. Fundamentally this is similar to the
command ``make install`` right? It builds the files and then installs them
into the final location. As I understand it, the standard way for a project
like Debian to install a project using a “regular” Makefile install like that
is to do ``make DESTDIR=/tmp/some-tempdir install`` so that it invokes the
build system, sets all the paths in the resulting build system to the final
destination, but then right before it actually copies files to the final paths
it just appends a directory in front of them to let you stage the f

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 11:51 AM, Scott Kitterman <deb...@kitterman.com> wrote:
> 
> On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
>>> On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski <pi...@debian.org> wrote:
>>> 
>>> to be honest, I still don't know what you're asking for. What do you
>>> want us to do? Patch 2.7's distutils?
>> 
>> Essentially, ensure that setuptools not distutils is used in a setup.py.
>> There are generally three kinds of setup.py files:
>> 
>> 1) Ones that use setuptools unconditionally - These ones you just leave
>> alone, they are already correct and you should already have a build depends
>> on python-setuptools. 2) Ones that conditionally use setuptools - These
>> ones you just need to satisfy whatever condition the setup.py uses to
>> enable setuptools. Typically this is just checking if setuptools is
>> importable but sometimes they use environment variables or similar. 3) Ones
>> that use distutils unconditionally - These ones you switch to making them
>> use setuptools instead of distutils.
>> 
>> Now, that’s the high level overview, there’s an easier, more automatic way
>> that could maybe just be added to pybuild (Not sure exactly how pybuild
>> works) where instead of invoking the setup.py as:
>> 
>>python setup.py install (or whatever commands/args you’re passing)
>> 
>> You do it as (taken from pip):
>> 
>>python -c "import setuptools,
>> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
>> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
>> (or whatever commands/args you’re passing).
>> 
>> The thing is kind of ugly, but that will install things using setuptools
>> (just like pip does) regardless of if it imports setuptools or distutils in
>> it’s setup.py file.
> 
> I tried this and it works, but what's the big deal?
> 
> It provides a PKG-INFO file that has identical content to the old egg-info 
> file,
> an empty dependency links file, and a top_level.txt file with the one line in
> it.
> 
> Why is this better (I'm not a huge upstream developer of Python stuff, but I 
> do
> do some and I don't see what this gets me)?
> 
> Scott K


This a fair question:

1) distutils is effectively frozen and all improvements to the build system
   are either going to be inside of setuptools, or will be opt in build systems
   that (probably) won't rely on setup.py at all. This includes newer metadata
   formats and the like as well.

2) That top_level.txt is an important file, without it we have no idea that
   $SITE-PACKAGES/requests is in any way associated with the metadata file
   requests-2.9.1.egg-info, that top_level.txt tells us what the top level
   import names are for a particular project.

3) It slipped my mind that you have to pass an additional flag to setuptools
   right now to get the full file list (pip passes that flag unconditionally)
   however I'm going to poke setuptools to see about getting them to add the
   record file unconditionally to the .egg-info directory so it doesn't
   require the --record flag. (Although Debian could add it earlier if they
   wanted, but it's fine to wait for setuptools to change here).

4) Switching it to a directory allows us to add additional files/information to
   the Python level metadata, like the sort of divergent thread about getting
   pip to stop mucking with OS owned files, the most likely path forward is to
   drop an INSTALLER file inside the .egg-info directory that has some special
   value ("system"? "os"? Not sure) which pip will take as a signal that it
   really should not touch.

5) In pip, because of #2 and #3 a regular distutils installed package (which
   gives you an .egg-info file, not directory) we have now way when we're told
   to install a package named "foo" to associate that with any files on disk.
   This means that we're currently lying when we tell a user we've uninstalled
   a distutils installed package, we just remove the .egg-info file but leave
   all the other files behind. We've deprecated the ability to do this, and we
   tried to remove it all together in the recent pip 8.0 release, but we had to
   roll it back because it caused too much breakage, largely due to OS packages
   that were using distutils installed metadata.

   On the surface, it may seem counterproductive for Debian to help pip get
   better metadata about what files belong to a package so that we can
   uninstall it, but we're going to solve the "people modify system files with
   pip when they shouldn't do that" more directly and there exist use cases
   where it's perfectly fine to break your system in this way, because it's an
   emphereal s

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 11:18 AM, Piotr Ożarowski <pi...@debian.org> wrote:
> 
> let's make a deal. If you will make sure pip doesn't touch system files
> (and others will not crucify me for this) - I will make sure pybuild
> uses above line (if setuptools is not detected in setup.py but is listed
> in Build-Depends).

Regardless of what happens in this thread, pip is going to stop mucking around
in files that are owned by some other tool without some sort of opt-in/--force
style flag *and* we're going to be restructuring things to try and guide people
away from using pip outside of a virtual environment or through the user of
--user as well.

The main reason why that hasn't happened yet isn't because I'm holding those
things hostage, it's because I'm one person and my time is spread amongst
several really important tools and I have to prioritize things in accordance to
how bad it's on fire. In addition, pip is a cross platform program and we have
to ensure that whatever we do works in a cross platform manner. You mentioned
/usr/local/.../site-packages/ before, but plenty of systems don't have a
/usr/local/.../site-packages/ that their Python respects because that is a
Debian specific patch, so if we refused to install to /usr/.../site-packages/
we'd break installing at the system level on all platforms but Debian derived
ones.

What we need is a cross platform way to determine if a particular installed
Python package is "owned" by the system or if it's something we're allowed to
modify. That's going to require some effort to work out the exact best way,
but in my head the way forward on that is to use the metadata directory (the
one I'm asking y'all to start using) and add an additional piece of metadata
inside of it that just says "Hey, This is a system install" that we can inspect
and then take additional logic (like refusing to touch it without a --force)
based on that. The change I'm asking for here helps make that possible (among
other things).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski <pi...@debian.org> wrote:
> 
> to be honest, I still don't know what you're asking for. What do you
> want us to do? Patch 2.7's distutils?

Essentially, ensure that setuptools not distutils is used in a setup.py. There 
are generally three kinds of setup.py files:

1) Ones that use setuptools unconditionally - These ones you just leave alone, 
they are already correct and you should already have a build depends on 
python-setuptools.
2) Ones that conditionally use setuptools - These ones you just need to satisfy 
whatever condition the setup.py uses to enable setuptools. Typically this is 
just checking if setuptools is importable but sometimes they use environment 
variables or similar.
3) Ones that use distutils unconditionally - These ones you switch to making 
them use setuptools instead of distutils.

Now, that’s the high level overview, there’s an easier, more automatic way that 
could maybe just be added to pybuild (Not sure exactly how pybuild works) where 
instead of invoking the setup.py as:

python setup.py install (or whatever commands/args you’re passing)

You do it as (taken from pip):

python -c "import setuptools, 
tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open', 
open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install (or 
whatever commands/args you’re passing).

The thing is kind of ugly, but that will install things using setuptools (just 
like pip does) regardless of if it imports setuptools or distutils in it’s 
setup.py file.


-----
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 6:04 PM, Scott Kitterman <deb...@kitterman.com> wrote:
> 
> On Friday, January 22, 2016 05:50:13 PM Donald Stufft wrote:
> ...
>> We already have an option like this, the —root option which will just append
>> a different prefix to all of the installation paths. So essentially instead
>> of invoking ``python setup.py install —root /tmp/something/`` which is what
>> I think you’re doing now, you’d do ``pip install —root /tmp/something/
>> —no-deps .`` and it’d just work similarly to what you have now, except pip
>> would be responsible for interacting with the Python build system.
> ...
> 
> Why would pip interacting with the Python build system instead of setuptools
> be better?
> 
> Scott K

setuptools *is* the build system right now (or distutils if you’re still using
that). So the benefit to Debian right now would be: We maintain the
distutils/setuptools hacks to make them suck less so you don’t have to!

Longer term, we want to enable people to go hog wild and invent their own
build systems instead of trying to do it all one size fits all with everyone
miserable because it doesn’t solve anyones problems nicely. At that point the
benefit to Debian becomes that instead of having to implement the standard
build tool interface that these tools implement (Which will most likely be
geared towards produces wheels, not towards putting files on disk in an
installed location) Debian can just let us maintain that bit of code too.

Of course, if Debian would prefer to interact directly with these tools, it’s
totally fine to do that. The whole point of us doing this work and trying to
define formats and APIs and not bless implementations is so anyone who wants
to can slot in their own tool in the process instead of being forced to use
some blessed tool.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 7:18 PM, Scott Kitterman <deb...@kitterman.com> wrote:
> 
> The Zen of Python says, among other things, "There should be one-- and 
> preferably only one --obvious way to do it".  Build systems seem to me like a 
> great place to apply that.


We have a sliding scale of complexity in what a project needs from it’s build 
system that generally breaks down into a few major classes of projects:

1) Projects that are pure python, single source, whose only real “compilation” 
is shuffling files to the correct location.
2) Projects that are pure python, but require some sort of generation step as 
part of the build process (2to3, etc).
3) Projects that have some basic C code that needs to be compiled, but which 
doesn’t link against anything special besides Python itself.
4) Projects that have some basic C code that needs to be compiled, but which 
links against other libraries like OpenSSL, libpq, etc.
5) Projects that have some basic C code, that needs to be compiled, that links 
against other libraries, and needs to be able to conditionally link against 
different libraries based on the capabilities and what is available in the 
system.
6) Extremely complex projects that need to link against many different 
libraries, possibly hard to build, possibly not C (e.g., Numpy with it’s blas 
libraries, Fortran, etc).

The problem with a one size fits all solution is that it’s very difficult to 
actually cover all of these cases in a way that is not horrible to actually use 
for each particular case. For an example, there is currently a build system 
called flit which doesn’t support anything but building straight to wheels 
(because we don’t have any sort of sdist 2.0 or anything yet). It doesn’t 
attempt to solve anything but the first class of users up there, and because of 
that it is able to create a very simple and easy to use packaging experience 
for authors, you just add a __version__ = “the version” to the top level of 
your package, and then drop in a flit.ini that looks something like:

[metadata]
module=foobar
author=Sir Robin
author-email=ro...@camelot.uk
home-page=http://github.com/sirrobin/foobar

# If you want command line scripts, this is how to declare them.
# If not, you can leave this section out completely.
[scripts]
# foobar:main means the script will do: from foobar import main; main()
foobar=foobar:main

And that’s it. You’re done. From there flit can do the rest of the work for you 
because it didn’t need to concern itself with trying to work on anything 
complex.

In addition to the above types of problems, you also have other things like 
what the “source of truth” is for your metadata. A common thing that people 
want is to be able to not have to duplicate their version in multiple locations 
(sometimes that even extends to the tags in the version control), however it’s 
not currently possible to do that in a particularly easy way. You have systems 
like pbr, setuptools_scm, versioner, etc that all do it but which all rely on 
some level of terribleness to deal with the sort of weird inverted control flow 
we have.

Once you get to all of the possible options for things people reasonably want 
to do, you quickly end up in a place where the only reasonable solution is the 
full complexity of a programming language like what we have in setup.py. 
However, that has it’s own problems which we’ve discovered over two decades of 
doing that :) It tends to end up with people doing badly tested adhoc code that 
they cargo cult from project to project and when there’s a problem there ends 
up being very little understanding why some code did that since it had been 
copy/pasted around so much. Pip has to go to a lot of effort to try and work 
around common mistakes people make in their setup.py files (like, depending on 
it being invoked with the project root as CURDIR) which most people will just 
never notice their slightly broken setup.py.

So basically, by allowing multiple build systems we will enable a world where 
your bog standard pure python project can get an extremely simple tool/ux and 
projects with crazy hard requirements like Numpy can get something more 
complex, without the two groups of users fighting each other over simplicity + 
limited vs complexity + powerful.

In the end, I think it’s likely we’ll end up with 2-3 really popular build 
tools, one that is for the complex projects, one that is for the simpler 
projects with some basic C needs, and *maybe* one that is for pure python (but 
that may be able to be handled by the basic C needs one) though there will be a 
long tail I’m sure.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

Hello!

I'd like to suggest a change to the Debian Policy around Python packages that
will help enable the world of Python packaging to continue to progress forward.

First, a little bit of background:

At the Python level there are three metadata formats for Python packaging:

* The original, setuptools style .egg-info directories.
* The distutils style .egg-info *file* added to distutils at some point.
* The new and improved, wheel based .dist-info directories.

The presence of any of these files will signal to Python tools that a
particular distribution has been installed, however there are two fairly major
and important differences between the distutils style, and the other two.

1. The distutils style has no provisions to record what files on the system
   belong to the installed distribution, making it appear to Python tooling
   that there *are* no files other than the metadata file itself.

2. The distutils style has no provisions to include additional metadata files
   in the metadata, making it impossible to extend the python level metadata
   with additional files.

I have a series of improvements that I'd like to make to the packaging
toolchain that will sort of build on one another, but which is not going to
function correctly with the distutils style metadata and I'm hoping that I can
convince y'all to make it policy to default to generating one of the other two
kinds (with varying methods, more on that later).

Concretely the thing that this is blocking right now, is that with the newly
released pip 8.0 I tried to make it so that pip will refuse to uninstall a
project that is installed with distutils style metadata. This is because we do
not have any way to associate the actual .py (and others) files on disk with
the installed metadata, so all we have ever done is just simply remove the
metadata file, making it appear as if the item is uninstalled but leaving
behind all of the actual files. However I'm going to be reverting this in a
pip 8.0.1 release because it caused a decent amount of breakage amongst pip's
users, almost all of them people who are attempting to upgrade OS provided
packages using pip.

Now, I know that upgrading OS provided packages using pip is less than optimal
and I would greatly prefer that people did not do it (and I'm generally in
agreement) however if we don't enable people to do it, they'll just continue to
use an old version of pip and file bugs. It's a non starter for pip to make it
impossible to do.

In addition to the uninstall bit, it also means that things like pip show -f
return junk information for packages installed in this way.

Beyond just (eventually) enabling pip to disable uninstallations of distutils
based installs this will start to allow some other future changes that I think
will be more interesting to Debian. The uninstallation of distutils based
installs comes hand in hand with pip stomping all over already existing files
willy nilly because the way upgrading a project like that works is pip
uninstalls the metadata file that says X is installed, then it just overwrites
over any of the files that happen to be in it's way when it installs the newer
version. If we can remove the need for pip to gleefully overwrite files to
support these types of installed packages, then we can make it so pip will
hard fail if it attempts to overwrite an already existing file on disk.

An additional benefit here is that by switching to using the directory based
options, we can add additional metadata files to the installed projects, much
like the INSTALLER file from PEP376 (IIRC). This file will likely be the path
to having pip refuse to touch OS owned files all together without some sort of
--force flag to override the safety switch.

As far as compatibility goes, pip has always forced everything to be installed
using setuptools and as far as I am aware, there's no real fallout from doing
so. I think in 2016 it's pretty reasonable to assume that a Python project is
capable of being installed using setuptools instead of distutils.

So without getting into the actual *method* of doing this (of which there are
several different options with different trade offs) does this sound like
something at all that Debian would be interested in?

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 21, 2016, at 9:32 AM, Paul Tagliamonte <paul...@debian.org> wrote:
> 
> Hey Donald!
> 
> As far as using pip to do stuff system-wide, I wrote thoughts on 
> http://notes.pault.ag/debian-python <http://notes.pault.ag/debian-python>

I just want to be clear, that I totally agree with this. The reason I want to
ensure we can enable this isn't because I think doing so is a great idea and
everyone should be doing it. People *are* doing it and we (pip) have to be
careful about what we make impossible to do because a possible result of going
too break happy is people just never upgrade and things start to stagnate
again.

So for pip at the system level, my goal is try and push people away from it as
much as we can without flat out making it impossible *and* try to make it as
"safe" to do so as possible when people decide to do it anyways.

> 
> As for the rest of it, distutils is actually concretely shitty, and replacing 
> it with setuptools
> sounds sane. Sounds like a solid idea.
> 
> Perhaps we can add a Lintian warning for using distutils; it's really nasty 
> cruft,
> and it's going to bite the archive sooner rather than later.
> 

As far as options for how to implement this goes, I'm not sure what best fits
with Debian so I'll leave that up to y'all, but here are a few options that I
can think of.

A lintian check is probably a pretty good thing, I would guess that it would
specifically check for the existence of a .egg-info or .dist-info directory
(or check to the non-existence of a .egg-info file).

In pip the way we make this work is with a bit of a gross little thunk that
ensures setuptools is imported prior to setup.py being invoked which forces all
setup.py calls to use setuptools instead of distutils even if they import from
distutils.

Of course, many setup.py already unconditionally use setuptools, and many more
conditionally use it (typically falling back to distutils) so for many projects
it may be enough to just add python-setuptools to build-depends. For projects
that unconditionally use distutils, either the pip way or patching setup.py
might be a reasonable way forward.

A more controversial way that comes with possibly some extra benefits (which
Debian may not care about) is to use ``pip`` itself as the build tool rather
than directly invoking setup.py. In this pip would be responsible for mucking
around with the distutils/setuptools mess instead of that needing to be handled
by Debian specific tooling. You could even leverage this to remove some of the
runtime dependencies on python-pkg-resources and speed up some python using
software if you let pip build a wheel as part of the buld process [1][2][3]. I
suspect pip would need to grow some additional options to make it suitable for
you in this role, but I would be more than willing to help if this is desired
(or not, this proposal isn't about trying to ram pip down anyones throat!).


[1] To be clear, this isn't about adding wheels for everything to the archive,
it would only be using Wheel as an intermediate step in the build process.
[2] Console scripts generated by setuptools entry wrappers when installed by
setuptools have a dependency on pkg_resources at runtime, however those
same scripts, when installed by pip from a wheel, do not. This would enable
removing the runtime dependency on pkg_resources for anything that only has
it for console scripts.
[3] Import pkg_resources is not the fasest thing in the world, it has to scan
every directory on sys.path looking for things to activate and it comes
with a bunch of side effects. This happens implicitly for any project using
console scripts.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Dealing with flit -- a simplified packaging of python modules

[Donald Stufft]

On September 26, 2015 at 5:30:35 AM, Paul Wise (p...@debian.org) wrote:
> On Fri, 2015-09-25 at 19:25 -0400, Donald Stufft wrote:
> 
> > Because the way Python packaging currently is and historically has
> > been, binary packages are not something that is widely available or
> > viable.
> 
> Hmm, I thought eggs have been around for ages (seems about 8 years)?
> 

Eggs are problematic and hardly anyone publishes them.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Re: Dealing with flit -- a simplified packaging of python modules

[Donald Stufft]

On September 25, 2015 at 7:24:30 PM, Paul Wise (p...@debian.org) wrote:
> On Thu, Sep 24, 2015 at 1:17 PM, Thomas Kluyver wrote:
> 
> > That's my point ;-). From our upstream point of view, it's not a bug
> > that the distributions we put on PyPI contain generated/bundled files -
> > we do it that way deliberately, so that end users can install without
> > needing Javascript developer tools to build those files. If you want a
> > pure source tarball without generated files, that's available from
> > Github.
> 
> Why are end users using source packages instead of binary packages and
> then complaining that the source tarballs aren't ready-to-run binary
> packages?
> 

Because the way Python packaging currently is and historically has been, binary 
packages are not something that is widely available or viable.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Re: Dealing with flit -- a simplified packaging of python modules

[Donald Stufft]

On September 25, 2015 at 8:21:39 PM, Ben Finney (ben+deb...@benfinney.id.au) 
wrote:
> Donald Stufft writes:
>  
> > On September 25, 2015 at 7:24:30 PM, Paul Wise (p...@debian.org) wrote:
> > > Why are end users using source packages instead of binary packages
> > > and then complaining that the source tarballs aren't ready-to-run
> > > binary packages?
> >
> > Because the way Python packaging currently is and historically has
> > been, binary packages are not something that is widely available or
> > viable.
>  
> So, commendable effort has gone toward making that problem reduce,
> perhaps eventually to go away. Congratulations are deserved for getting
> us this far, and thanks for continuing to push for feasible binary
> distributions of Python packages on all supported OSen.
>  
> It seems reasonable, then, to also put effort toward making source
> distributions more targeted toward being bundles of the released
> *source* for the project version, and reducing the justification for
> bundling non-source files in the source distribution.

I think distutils/setuptools is to blame for a lot of this, they don't make it
easy to have generated files that aren't either compiled C code or py(c|o).
Ideally in the future distutils/setuptools will no longer be special (except as
a historical and legacy fallback) and we'll have something far more pluggable
in the future. This will ideally make it a lot easier and sane to have custom
build steps that create other generated files as part of the building of a
package.

>  
> Can we expect the value of “but people expect non-source stuff bundled
> in the source distrubution” to diminish, as some kind of objection to
> making proper source distributions?

It's certainly a goal of mine that we move as much of the upstream Python
packaging onto binary packages so that most people are not installing from
sdists or really, "installing" from an sdist being an outdated concept and
long term, the only thing we do with an sdist is turn it into a binary package,
even if that build step is taken on the end user's computer. I of course, can't
speak to everyone so it would not surprise me if some people simply will never
be willing to not distributed generated files as part of their sdist.

I suspect this to be a long road though, and certainly the nature of *nix makes
it harder on us, since it's hard to make a binary package that works for say,
all of Linux due to ABI problems and such. The more cases where ``pip install``
has end users pulling from a sdist, the more resistence I suspect you'll find
from authors to making sdist have more complicated build steps.

I suspect you'll have an easier time convinving authors of pure Python things
that include some generated files (translations, js, whatever) to rely entirely
on wheels than you will have convincing authors of Python things that include
some C or other compiled code that precludes universal(ish) Wheels.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Re: PyCon BoF: Stretch goals for cPython, PyPy CFFI

[Donald Stufft]

 On Apr 13, 2015, at 4:17 PM, Stefano Rivera stefa...@debian.org wrote:
 
 Matthias and I are planning to have a Debian Python BoF at PyCon,
 tomorrow afternoon. I think lunch is 2pm, so 3pm?
 
 Meet outside the cPython sprint room?
 
 Matthias wants to discuss general stretch goals for Python in Debian.
 I want to make concrete plans for py3k packages that are compatible with
 multiple interpreters.
 
 

Does this mean PyPy too? It’d be great to have a (not just specific to Debian)
standard for how to mark a binary for a particular Python that gracefully 
handles
other interpreters too. Right now we have the de facto standard of binary, 
binaryX,
and binaryX.Y but that really only sanely handles CPython.


---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Debian uscan redirector for PyPI

[Donald Stufft]

 On Feb 5, 2015, at 6:14 PM, Ben Finney ben+deb...@benfinney.id.au wrote:
 
 Ben Finney ben+deb...@benfinney.id.au writes:
 
 Donald Stufft don...@stufft.io writes:
 
 I suggested to #debian-python that a redirector might be better and
 there is now one at pypi.debian.net.
 
 Whe is the VCS for the code running that redirector?
 
 I would like to suggest some patches. To whom should I submit those
 patches?
 

I think Piotr.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/9bd7ebdd-612e-40a8-90e7-08b727e00...@stufft.io

Re: PyPI and debian/watch

[Donald Stufft]

 On Feb 4, 2015, at 11:20 AM, Barry Warsaw ba...@debian.org wrote:
 
 On Feb 04, 2015, at 10:53 AM, Donald Stufft wrote:
 
 That same page also mentions that qa.debian.org runs a number of
 redirectors for sites like SourceForge and GitHub so perhaps a better
 answer is for Debian QA to run a redirector for PyPI instead of PyPI
 implementing a redundant API endpoint with a slightly different layout and
 HTML primarily for Debian.
 
 +1
 
 Cheers,
 -Barry

Dunno the best way to give this to y'all but I wrote a thing:

https://github.com/dstufft/pypi-debian

I can transfer it on github or release it on PyPI or whatever. It shouldn't
really need any maintenance or anything but I'm probably not going to pay
attention to it if it does need any so someone else might want to actually
own it.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/363795e0-f96f-423f-b6a6-d3f3fbf76...@stufft.io

Re: PyPI and debian/watch

[Donald Stufft]

 On Feb 4, 2015, at 3:02 PM, Tianon Gravi admwig...@gmail.com wrote:
 
 On 4 February 2015 at 06:08, Donald Stufft don...@stufft.io wrote:
 If it gets implemented it'll live at /uscan/ because it exists primarily to
 work around the deficiencies that exist in uscan (Particularly the dificulty
 in ignoring url fragments).
 
 This seems like we're building a workaround to a tool we could
 theoretically change. :(
 
 debian/watch has a version=3, which is presumably so that there
 can be a version=4 when deficiencies are discovered -- wouldn't it
 be worthwhile to consider revbumping the watch format and updating
 uscan to have some improved support for edge cases like this?  I know
 uscan has some other open bugs too that could use some thought towards
 a more flexible format to handle cases like this.

We talked about this in #debian-python and there was concern that a new version
of uscan wouldn’t be in Jessie and then wouldn’t cover the people who need it
the most.

I don’t know if that’s true or not but I certainly think that uscan _should_
ignore anything that comes after a # (similarly to how it ignores anything that
comes after a ?). That would solve the largest problem, that the URL fragment
is hard to remove from the d/watch file. The other problem is that 
/simple/whatever/
has files located at /packages/stuff but I believe that’s not very hard to 
work
around.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/bc9910d4-1c34-4856-9f0f-b46a99da1...@stufft.io

Re: Debian uscan redirector for PyPI (was: PyPI and debian/watch)

[Donald Stufft]

I'm on my phone currently but I think Barry is using it in the wheel package 
now. 


 On Feb 4, 2015, at 5:18 PM, Ben Finney ben+deb...@benfinney.id.au wrote:
 
 Donald Stufft don...@stufft.io writes:
 
 I suggested to #debian-python that a redirector might be better and
 there is now one at pypi.debian.net.
 
 Great! Can we please have (from you, or whoerver is best position to
 write it) a reference on how to use this redirector?
 
 I don't know a good location; the Debian wiki doesn't have an obvious
 location, redirectors only seem to be mentioned briefly at
 URL:https://wiki.debian.org/debian/watch#Uncommon_sites.
 
 -- 
 \   Moriarty: “Forty thousand million billion dollars? That money |
  `\must be worth a fortune!” —The Goon Show, _The Sale of |
 _o__)   Manhattan_ |
 Ben Finney
 
 
 -- 
 To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: https://lists.debian.org/8561bhgw2k.fsf...@benfinney.id.au
 


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/dd57c9f2-bab9-41b3-ab51-70bd03fde...@stufft.io

Re: PyPI and debian/watch

[Donald Stufft]

 On Feb 4, 2015, at 4:32 PM, Stefano Rivera stefa...@debian.org wrote:
 
 Hi Donald (2015.02.04_22:06:25_+0200)
 On 4 February 2015 at 06:08, Donald Stufft don...@stufft.io wrote:
 If it gets implemented it'll live at /uscan/ because it exists primarily to
 work around the deficiencies that exist in uscan (Particularly the 
 dificulty
 in ignoring url fragments).
 
 Would it be that hard to have fake directory listings on /simple/?
 I mean, surely keeping compatibility there is simpler than having a
 second endpoint just for Debian.

All the data that uscan needs is already on /simple/, you can make uscan work 
with
it. There is one major problem and one small problem:

1. Major: The /simple/ URLs all have a #md5=hash and it’s non trivial to 
write a
   d/watch file that ignores them and uscan doesn’t by default. You can do it 
but
   it’s ugly and prone to copy/paste bugs.

2. The URLs on /simple/ point to /packages/, so it requires the 2 arg form of
   d/watch instead of the single arg form.

So you can make uscan work right now with /simple/ (and a few people have) but 
#1
means that a few of the #debian-python people were not very happy with that 
solution.
I can’t remove/modify that hash without causing issues with pip/easy_install 
though.
Originally I was going to just make a /uscan/ that was /simple/ without the 
hash,
but instead I suggested to #debian-python that a redirector might be better and 
there
is now one at pypi.debian.net.


 
 We talked about this in #debian-python and there was concern that a new 
 version
 of uscan wouldn’t be in Jessie and then wouldn’t cover the people who need it
 the most.
 
 Who needs it the most? We could fix it in unstable and backports. The
 DEHS data on tracker.debian.org comes from quantz.debian.org. which is
 currently using devscripts from back ports.

No idea, I’m just repeating what folks said in #debian-python, I have no idea 
who
runs uscan and on what platforms. Between fixing uscan and having a redirector I
don’t have an opinion since neither one of those have an impact on what PyPI
does.

 
 
 I don’t know if that’s true or not but I certainly think that uscan _should_
 ignore anything that comes after a # (similarly to how it ignores anything 
 that
 comes after a ?).
 
 Agreed.
 
 SR
 
 -- 
 Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272
 
 
 -- 
 To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: https://lists.debian.org/20150204213208.ga3...@bach.rivera.co.za
 

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/c118b6b9-f56e-46be-9a80-c5e934c1f...@stufft.io

Re: PyPI and debian/watch

[Donald Stufft]

 On Feb 4, 2015, at 10:07 AM, Barry Warsaw ba...@debian.org wrote:
 
 On Feb 04, 2015, at 08:08 AM, Donald Stufft wrote:
 
 If it gets implemented it'll live at /uscan/ because it exists primarily to
 work around the deficiencies that exist in uscan (Particularly the dificulty
 in ignoring url fragments). Everyone else should just use the URLs at 
 /simple/
 which most systems use with no problem because they can parse the URLs and
 ignore the URL fragments (or use them for verifying the hash if need be).
 
 I'll just note that I've found the fragments inconvenient in other settings
 too.  They aren't very user friendly since (IMHO) they add noise that users
 cutting and pasting urls generally don't care about.  They also feel odd in
 that the md5 checksum doesn't fit what I think as a typical fragment.
 Traditionally, they are used to point to an anchor (sub-resource) within the
 parent resource.  That's not the case here.
 
 http://en.wikipedia.org/wiki/Fragment_identifier
 
 has this to say:
 
 
 Several proposals have been made for fragment identifiers for use with plain
 text documents (which cannot store anchor metadata), or to refer to locations
 within HTML documents in which the author has not used anchor tags:
 
 As of September 2012 the Media Fragments URI 1.0 (basic) is a W3C
 Recommendation.[12]
 
 The Python Package Index appends the MD5 hash of a file to the URL as a
 fragment identifier.[13] If MD5 were unbroken (it is a broken hash function),
 it could be used to ensure the integrity of the package.
 
 https://pypi.python.org ... 
 zodbbrowser-0.3.1.tar.gz#md5=38dc89f294b24691d3f0d893ed3c119c
 
 
 So even without the uscan incompatibility (which is just one of the two
 factors leading to noisy d/watch file), I think there's some value in
 fragment-less URLs.  I understand the checksum isn't being used
 cryptographically here, but maybe thinking ahead to the use of more secure
 algorithms in the future can lead to a more flexible design:
 
 Legacy (if it indeed needs to be kept for backward compatibility):
 
 /simple/foo-x.y.z#md5=blah
 
 then:
 
 /simple/plain/foo-x.y.z
 /simple/sha256/foo-x.y.z#sha256=blah
 

Long term PyPI is going to move away from trying to cram a bunch of information
into a hyperlink and relying on HTML parsing and instead is going to move the
installer APIs over to using something more suited to the task, most likely
JSON. At that point we'll be able to design the API to be more extendable in
this regards since we'll be able to do something like:

{
...,
hashes: {
md5: ...,
sha256: ...,
},
...
}

and the client can simply select which hash it wants to use. Long term the
/simple/ API on PyPI will exist only for legacy purposes so people still using
versions of pip, easy_install, etc that only support /simple/ will still be
able to access PyPI.

That doesn't really help uscan at all though since as far as I know uscan has
no ability to parse JSON.

As far as copy/pasting goes, the /simple/ API is an API, it's not designed to
be human consumable but consumable by software. The UI centric pages at /pypi
are the ones designed to be consumable by humans (Although currently PyPI puts
the hash there as well, however Warehouse (aka PyPI 2.0) does not).

The problem here really lies within uscan making assumptions about the
structure of URLs and the content of the HTML on those pages. From looking at
https://wiki.debian.org/debian/watch I'm guessing that it inherited those
assumptions from when FTP was the more common way to distribute files instead
of HTTP(S). That same page also mentions that qa.debian.org runs a number of
redirectors for sites like SourceForge and GitHub so perhaps a better answer
is for Debian QA to run a redirector for PyPI instead of PyPI implementing a
redundant API endpoint with a slightly different layout and HTML primarily for
Debian.

One note in that regard is that the /simple/ indexes don't include the .asc
files if someone has uploaded them however the old URLs that debian/watch used
did. If that is something that is needed we could easily add them to the
/simple/ pages.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/246c48b2-9f15-4827-aab4-b574f95a2...@stufft.io

Re: PyPI and debian/watch

[Donald Stufft]

 On Feb 4, 2015, at 3:05 AM, Ben Finney ben+deb...@benfinney.id.au wrote:
 
 Tristan Seligmann mithra...@mithrandi.net writes:
 
 The debian/watch file I wrote for python-nacl (which also verifies the
 PGP signature) seems to work.
 
 I can't get PGP signature retrieval to rowk (“uscan warning:
 pgpsigurlmangle option exists, but the upstream keyring does not exist”)
 even with your suggested pattern.
 
 But I have also written a working uscan configuration::
 
 opts=filenamemangle=s/\S+\/([^\/]+\.tar\.gz)#md5=[[:alnum:]]+$/$1/ \
https://pypi.python.org/simple/python-daemon/ \
\S+/python-daemon-(\S+)\.tar\.gz#md5=[[:alnum:]]+ \
debian
 
 
 Barry Warsaw ba...@debian.org writes:
 
 I'd love to be able to have something as simple as:
 
 version=3
 https://pypi.python.org/simple/mypkg/mypkg-(.*).tar.gz
 
 which is close to what most packages probably use today, modulo the
 base url path.
 
 That would be great. But remember that the uscan documentation
 recommends a tighter matching pattern, so that would be::
 
version=3
https://pypi.python.org/simple/mypkg/mypkg-(.+).tar.gz
 
 I filed a bug against pypa/warehouse so hopefully we can get something
 better before Jessie is released (which is when I think there will be
 more pressure for a better solution, since most packages won't be
 updated during the freeze).
 
 https://github.com/pypa/warehouse/issues/358
 
 Thanks very much!
 
 I'm not a fan of having it live at “…/uscan/” though. This is not
 specific to Debian, it's a sensible API design for all.
 

If it gets implemented it'll live at /uscan/ because it exists primarily to
work around the deficiencies that exist in uscan (Particularly the dificulty
in ignoring url fragments). Everyone else should just use the URLs at /simple/
which most systems use with no problem because they can parse the URLs and
ignore the URL fragments (or use them for verifying the hash if need be).

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/fa9f43c5-64c5-4cff-a972-30d162099...@stufft.io

Re: Resources/best practices for making a .deb out of a virtualenv

[Donald Stufft]

 On Dec 31, 2014, at 2:21 PM, Tim Chase deb...@tim.thechases.com wrote:
 
 I've got a Python project where the dependencies are
 isolated in a virtualenv and would like to package it up as a .deb
 the purposes of sharing.  Normally I'd include the additional
 libraries as straight-forward dependencies of the Debian package
 itself, but some of the required package-versions conflict with the
 stock Stable or Installation packages (e.g. Django 1.4 in Stable vs.
 1.6 or 1.7 which are more current versions).
 
 Are there any good resources on how to package up a virtualenv for
 distribution so that it can use more recent libraries?
 
 (I'm not currently subscribed to the mailing list, but will if that's
 easer than just CC'ing me or Reply-All'ing me).
 

I’ve used https://github.com/spotify/dh-virtualenv in the past and it worked 
well.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/a9ef950f-ca8b-40e0-9c24-948cdbe2c...@stufft.io

Re: 'deviations from upstream' wiki.debian.org/Python docs

[Donald Stufft]

I hope not, it’s way behind setuptools at this point.

On Jul 2, 2014, at 11:19 AM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote:

  Hi all.
 
  Just wandering if the information from https://wiki.debian.org/Python about 
 'Deviations from upstream' up to date. Not a Debian user myself, but the 
 following notes seem weird:
 
 The python-setuptools package installs the Distribute fork instead
 of the standard setuptools.
 
 The python-virtualenv also uses distribute by default, but can
 enable classic setuptools with an optional switch.
 
  Do debian distributions still actually package `distribute` instead of 
 `setuptools`?
 
  Best regards,
Jurko Gospodnetić
 
 
 -- 
 To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: https://lists.debian.org/53b422f1.9080...@pke.hr
 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Packaging of suds-jurko (was: suds)

[Donald Stufft]

On Jul 1, 2014, at 5:41 PM, Barry Warsaw ba...@debian.org wrote:

 On Jul 01, 2014, at 01:10 PM, Mathias Behrle wrote:
 
 The first tests on suds-jurko are looking very promising. I built the package
 succesfully as a drop-in replacement for the current python-suds package. It
 builds correctly for python2 and python3 with all tests. I tested part of the
 functionality for python2, all was working well. The maintainer of suds-jurko
 is very active and responsive.
 
 Will a Python 3 compatible suds library allow us to make progress on #732644
 without rewriting bts to use REST+JSON wink?
 
 1) Can I drop in the suds-jurko fork into the current suds package as
 proposed by Jordan?
 
 Given that suds on PyPI hasn't been updated in almost 4 years, I think we can
 reasonably assume its upstream is defunct.  We had a sort of analogous
 situation with setuptools, but the distribute and setuptools upstreams did
 eventually merge back together.
 
 A counter example might be oauth which was also abandoned upstream and for
 which a new upstream called oauthlib was released.  However, in that case, the
 replacement was *not* API compatible, so it made sense to make it a different
 Debian package.
 
 I don't really have a strong opinion, as I can see both sides of the coin.
 You're *probably* safe just taking over the source package, but if you woke up
 tomorrow with an extra dose of paranoia, then you might favor a new source
 package, which also wouldn't be objectionable, albeit more work to transition
 dependencies.
 
 2) If not 1) what would be the best alternative?
 
 In this case I would plan
 
 - a new python-suds-jurko package, conflicting with python-suds
 - filing bugs to rdepends to use the new package
 - removing the old package as soon as possible
 
 Yep.  It's a bit ugly though (I don't like the -jurko blarg).  Oh well, do
 what you think is right.
 
 -Barry

*Puts on PyPI Admin Hat*

Probably if suds-jurko or whatever is the unofficial “suds” that people should
be using then there is a good chance that PyPI would be willing to transfer
the name of suds to one of the forks. I’d have to talk to Richard to be sure
about that but it’s potentially an option.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug#750638: ITP: ndg-httpsclient -- enhanced HTTPS support for httplib and urllib2 using PyOpenSSL

[Donald Stufft]

On Jun 5, 2014, at 7:09 AM, Daniele Tricoli er...@mornie.org wrote:

 Hello Julien,
 thanks for packaging ndg-httpsclient!
 
 On Thursday 05 June 2014 12:26:22 Julien Cristau wrote:
 My main interest is to be able to talk to websites using SNI with
 scripts using python-requests.
 
 Once in the archive I will also add ndg-httpsclient into python-requests' 
 Suggests.
 
 Kind regards,
 
 P.S. I'll do the same for python-urllib3:
 https://github.com/shazow/urllib3/pull/156
 
 -- 
 Daniele Tricoli 'Eriol'
 http://mornie.org

You need pyasn1, pyopenssl, and ndg-httpsclient in order for the 
requests/urllib3 stuff to kick in.

It’d probably be a sane idea to use recommends, at least on Python 2.x since 
using that also
prevents CRIME and the like which Python 2.x is vulnerable to else wise IIRC.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug#750638: ITP: ndg-httpsclient -- enhanced HTTPS support for httplib and urllib2 using PyOpenSSL

[Donald Stufft]

On Jun 5, 2014, at 11:47 AM, Daniele Tricoli er...@mornie.org wrote:

 Hello Donald,
 
 On Thursday 05 June 2014 10:24:48 Donald Stufft wrote:
 You need pyasn1, pyopenssl, and ndg-httpsclient in order for the
 requests/urllib3 stuff to kick in.
 
 Yes, of course: I was keeping an eye on all the needed packages.
 
 It’d probably be a sane idea to use recommends, at least on Python 2.x since
 using that also prevents CRIME and the like which Python 2.x is vulnerable
 to else wise IIRC.
 
 Thanks for pointing this: for python-requests I will add to Recommends all of 
 the needed packages to ensure that SNI works as expected and to prevent CRIME.
 
 For python3-requests do you think it's needed to also add them to Reccomends?
 Upstream issue 20994[¹] is still open, but Python3 support SNI, and ssl 
 compression can be disabled, as reported on the issue, using 
 OP_NO_COMPRESSION 
 (on python3 = 3.3, but we have 3.4). I think use Suggests is fine in this 
 case.
 I will add a README.Debian to explain clearly all of it.
 
 Cheers,
 
 
 [¹] http://bugs.python.org/issue20994
 
 -- 
 Daniele Tricoli 'Eriol'
 http://mornie.org

Yea it shouldn’t matter on Python 3.x as the SSLContext stuff urllib3 will use 
to give good defaults there already.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug#750638: ITP: ndg-httpsclient -- enhanced HTTPS support for httplib and urllib2 using PyOpenSSL

[Donald Stufft]

On Jun 5, 2014, at 12:02 PM, Barry Warsaw ba...@debian.org wrote:

 On Jun 05, 2014, at 11:52 AM, Donald Stufft wrote:
 
 Yea it shouldn’t matter on Python 3.x as the SSLContext stuff urllib3 will
 use to give good defaults there already.
 
 Does any of this impact our wheels for virtualenv/pyvenv (Py2 and Py3)?
 
 -Barry

Not really. requests will opportunistically use those three libraries in order 
to
have better TLS on Python 2.x. ensurepip doesn’t include them since It’s
for Python 3.x only, and virtualenv doesn’t because they require a compiler
(well pyopenssl does).

If Debian wanted to make pip in virtualenv safer they could create wheels for
those 3 and install them into python 2.x virtualenvs by default (they can be
installed normally, they don’t require any magic). This would be Debian going
above the “standard” for what upstream or any other OS does afaik.

However I don’t think it’s really a big deal, most of the attacks on TLS affect
the confidentiality portions, however pip doesn’t really care too much about
that and currently relies on TLS mostly for the authenticity portion.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: python-pip and python-virtualenv broken in sid after upgrade to python 2.7.7rc1

[Donald Stufft]

On Jun 3, 2014, at 5:28 AM, Mário Duarte Cruz duartec...@gmail.com wrote:

 On Mon, Jun 2, 2014 at 2:51 PM, Donald Stufft don...@stufft.io wrote:
 
 
 pip 1.5.6 just upgrades the bundled requests FWIW.
 
 
 
 I forgot to mention that it breaks if you're sitting behind a proxy (might 
 not be obvious from a cursory look at the linked bug):
 
 [...]
 Getting page https://pypi.python.org/simple/pip/
 Could not fetch URL https://pypi.python.org/simple/pip/: connection error: 
 hostname 'proxy.company.com' doesn't match either of '*.c.ssl.fastly.net', 
 'c.ssl.fastly.net', [...]
 
 
 @Donald: the currently available get-pip.py seems to also be affected by the 
 same bug
 

Ah yea, I forgot to rebuild that with 1.5.6, I’ll get it fixed in a bit. Thanks.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: python-pip and python-virtualenv broken in sid after upgrade to python 2.7.7rc1

[Donald Stufft]

The fix in 1.5.6 was updating requests. That's the only change. I think Debian 
needs to update urllib3. That's where the actual fix was. 

 On Jun 3, 2014, at 4:07 PM, Barry Warsaw ba...@debian.org wrote:
 
 On Jun 03, 2014, at 11:28 AM, Mário Duarte Cruz wrote:
 
 I forgot to mention that it breaks if you're sitting behind a proxy (might
 not be obvious from a cursory look at the linked bug):
 
 I'm about to upload pip 1.5.6 to unstable.  Please double check that this is
 fixed there, and if not, please open a bug on the python-pip package.
 Remember though that we devendorize all those bundled dependencies.
 
 Cheers,
 -Barry
 
 
 -- 
 To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: https://lists.debian.org/20140603160757.53458...@anarchist.wooz.org
 


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/b799b798-7c22-4f5b-8e5a-6128151a7...@stufft.io

Re: python-pip and python-virtualenv broken in sid after upgrade to python 2.7.7rc1

[Donald Stufft]

On Jun 3, 2014, at 6:55 PM, Barry Warsaw ba...@debian.org wrote:

 On Jun 03, 2014, at 04:31 PM, Donald Stufft wrote:
 
 The fix in 1.5.6 was updating requests. That's the only change. I think
 Debian needs to update urllib3. That's where the actual fix was.
 
 I think you mean we need to update the requests package.  That's on my list.
 
 -Barry

Do you unbundle urllib3 from requests? The fix that broke proxying was a
thing in urllib3, which requests bundles, which pip bundles requests.

Here’s the actual fix for that issue https://github.com/shazow/urllib3/pull/385


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: python-pip and python-virtualenv broken in sid after upgrade to python 2.7.7rc1

[Donald Stufft]

On Jun 2, 2014, at 8:31 AM, Mário Duarte Cruz duartec...@gmail.com wrote:

 Hi,
 
 Since the upgrade to python 2.7.7rc1 in sid, pip and virtualenv's pip have 
 been broken, most likely due to https://bugs.python.org/issue7776 .
 
 Updating pip to 1.5.6 and virtualenv to 1.11.6 should fix the issues (current 
 versions in sid are 1.5.5 and 1.11.4 respectively).
 
 Apologies for not using the bug tracker, too many hoops to jump through. I'd 
 usually just wait it out - sid being sid - but this has been irking me for 
 over a week.
 
 Cheers, 
 
 Mario Cruz
 


pip 1.5.6 just upgrades the bundled requests FWIW.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Proposed changes to python-virtualenv

[Donald Stufft]

On Jun 2, 2014, at 4:35 PM, Barry Warsaw ba...@python.org wrote:

 On May 30, 2014, at 11:47 AM, Barry Warsaw wrote:
 
 So let's say pbundler is the only thing other than tox that does this.
 
 Well, now pbundler has been removed from the archive, so I don't care about
 it anymore. ;)  I can't see any reason not to make virtualenv a Python 3
 application and Python 3-only library.  Therefore, I won't change the binary
 packages built by this, but just retarget them to Python 3.
 
 Cheers,
 -Barry

Sounds reasonable to me, the only “downside” is that virutalenv will default
to Python 3, which is probably not what most people want (however they
can do virtualenv -p python2).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Proposed changes to python-virtualenv

[Donald Stufft]

On May 29, 2014, at 8:15 PM, Scott Kitterman deb...@kitterman.com wrote:

 On May 29, 2014 7:54:53 PM EDT, Barry Warsaw ba...@debian.org wrote:
 I'm looking again at updating tox to the latest upstream 1.7.1.  Along
 the
 way, I'd like to make /usr/bin/tox a Python 3 script.
 
 This requires that virtualenv be importable, e.g. `$python -m
 virtualenv`.  It
 is today in Python 2 since /usr/bin/virtualenv is a Python 2 script and
 we
 only build it for one version of Python.  I want to change that too, so
 that
 at least we build the virtualenv modules for both Python 2 and 3.  I'd
 like to
 switch /usr/bin/virtualenv to be Python 3 as well, and I'd like to
 tackle the
 vendorizing issue similar to how we fixed it with ensurepip, since we
 have all
 the wheels we need now.
 ...
 Thoughts?
 -Barry
 
 I'd rather remove the wheels we have and give up on ensurepip than start down 
 this slippery slope. 
 
 Wheels are an ugly solution for us to work around upstream doing things that 
 can most charitably be described as fixing problems we don't have.  No more. 
 
 Scott K

Eh, it’s not true that users of Debian do not have the problems that ensurepip
solves. Perhaps *you* don’t personally have them but anyone whose ever needed
to install a set of Python packages that Debian either doesn’t package, or
packages the wrong version of them have the problems that virtualenv/venv
solves and virtualenv/venv without pip is practically worthless.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Proposed changes to python-virtualenv

[Donald Stufft]

On May 29, 2014, at 7:54 PM, Barry Warsaw ba...@debian.org wrote:

 I'm looking again at updating tox to the latest upstream 1.7.1.  Along the
 way, I'd like to make /usr/bin/tox a Python 3 script.
 
 This requires that virtualenv be importable, e.g. `$python -m virtualenv`.  It
 is today in Python 2 since /usr/bin/virtualenv is a Python 2 script and we
 only build it for one version of Python.  I want to change that too, so that
 at least we build the virtualenv modules for both Python 2 and 3.  I'd like to
 switch /usr/bin/virtualenv to be Python 3 as well, and I'd like to tackle the
 vendorizing issue similar to how we fixed it with ensurepip, since we have all
 the wheels we need now.
 
 This means however that I need to rejigger the binary packages in the
 python-virtualenv source package.  Right now, python-virtualenv contains both
 the modules and /usr/bin/virtualenv.  I need to at least add a
 python3-virtualenv package to contain the Python 3 modules.  But it doesn't
 make sense to have a /usr/bin/virtualenv with both a Python 2 and Python 3
 shebang line, and thus it doesn't make sense to put Python 3
 /usr/bin/virtualenv in the python-virtualenv binary package.  However, I still
 want to make it obvious and easy which package someone needs to install to get
 the command line script.
 
 My thinking is that I would add a `virtualenv` binary package which would
 contain just the /usr/bin script (with a Python3 shebang) and any other common
 files as makes sense (e.g. the manpage).  `virtualenv` would Replaces/Breaks
 `python-virtualenv` and Depends on python3-virtualenv.
 
 I thought about using a python-virtualenv-common for that but since this will
 be the most likely installed package for end users, I thought `apt-get install
 virtualenv` looked nicer.
 
 Thoughts?
 -Barry

Does anything other than tox depend on virtualenv? Unless something python 2.x
depends on virtualenv the only real benefit to having virtualenv installed in
both 2.x and 3.x is what the default interpreter is whenever you create a
virtual environment. IOW virtualenv is perfectly capable of creating virtual
environments in interpreters other than the one it's installed in.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Dependency of virtual environments on Python wheel (was: Proposed changes to python-virtualenv)

[Donald Stufft]

On May 29, 2014, at 9:18 PM, Ben Finney b...@benfinney.id.au wrote:

 Barry Warsaw ba...@debian.org writes:
 
 On May 29, 2014, at 08:15 PM, Scott Kitterman wrote:
 
 I'd rather [Debian] remove the wheels we have and give up on
 ensurepip than start down this slippery slope.
 
 That means we give up on pyvenv, and given that virtualenv will
 eventually be a wrapper around pyvenv, that means we give up on
 virtual environments.
 
 Could you expand on that? What is it about pyvenv, or wheel, or both,
 that leads from Debian removing wheel packages to Debian “giving up on
 pyvenv”?
 
 Or, if you prefer, what is it about pyvenv that makes it infeasible to
 do without wheel packages?

pyvenv doesn't strictly need Wheels themselves, but it needs something that
is capable of interrogating Python to ask it where it should install the
packages as well as which is capable of understanding virtual environments at
all.

Historically virtualenv has done this by embedding sdists, manually unpacking
them, and then executing setup.py. With the most recent version of virtualenv,
and with pyenv (which uses ensurepip) this has been switched to wheels since
they are really just fancy zip files which means that Python is able to import
them directly. This allows pyenv (again via ensurepip) and virtualenv to use
pip to install both itself and setuptools instead of directly executing
setup.py. This has made the bootstrapping of setuptools/pip into these virtual
environments cleaner, faster, and simpler.

Strictly speaking as well, pyvenv itself doesn't require ensurepip (where the
wheels are being used). However pyvenv does default to bootstrapping pip into
the virtual environment which that particular feature *does* require ensurepip
and the Wheel files. In Python 3.3 there is no ensurepip and pyvenv does not
include pip, but feedback has overwhelmingly been that 3.3's pyvenv is useless
because it does not provide any means of installing things into it like
virtualenv does. This has lead people to continue to use virtualenv instead
of pyvenv which can hopefully be rectified in 3.4+ now.

Additionally it's important to remember that the only reason virtualenv itself
works on Debian is because it violates the Debian policy and ships the Wheel
files that I (as the person whose released the upstream virtualenv) included
within it. The solution that Barry has with pyvenv is more complex than simply
including those files as python-virtualenv does, however it is my understanding
that is allows pyvenv to be compliant with Debian policy.

 
 Wheels are a solution to a problem that is *explicitly* Debian, given
 the policy and DFSG violations vendorizing entails. Upstream doesn't
 have the same concern.
 
 (I think by “to vendorize” you mean “to bundle”. Why the neologism
 “vendorize”, what is the distinction?)

I think the general distinction is that bundling is more generic than vendoring.
Vendoring typically refers to including the code of another package as part of
the code of the primary package. For instance, pip vendors requests (and others)
as pip._vendor.requests. It is effectively a part of the pip namespace now
instead of simply bundling the two packages together so that installing pip
also installs requests.

 
 What do wheels do which address Debian in particular? Why is it that
 Debian can't ignore wheels as superfluous?

Wheels don't address a problem that is specific to Debian, as I explained above
we need a package to install into the virtual environment. As far as I am aware
dpkg is not aware of virtual environments and I cannot install a package (such
as python-pip) multiple times into multiple virtual environments on a single
machine. So more accurately this solves a generic problem for which the
typical Debian machinery does not have a solution for.

 
 -- 
 \  “It is well to remember that the entire universe, with one |
  `\   trifling exception, is composed of others.” —John Andrew Holmes |
 _o__)  |
 Ben Finney
 
 
 -- 
 To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: https://lists.debian.org/85iooo2hez.fsf...@benfinney.id.au
 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: wheel support for Debian?

[Donald Stufft]

Unpacking the Wheel files will trigger pip to see them as installed and it
will attempt to uninstall them if it needs to. If they are going to be unpacked
it’ll need to have the .dist-info directory removed from the unpacked locations
in order to prevent that from happening.

On May 19, 2014, at 7:19 AM, Jakub Wilk jw...@debian.org wrote:

 * Matthias Klose d...@debian.org, 2014-05-19, 12:44:
 we don't unpack jar files either.
 
 Fortunately, we don't have to mimic all the Java misfeatures.
 
 -- 
 Jakub Wilk
 
 
 -- 
 To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: https://lists.debian.org/20140519111953.gc8...@jwilk.net
 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: wheel support for Debian?

[Donald Stufft]

On May 15, 2014, at 6:32 PM, Barry Warsaw ba...@debian.org wrote:

 My thoughts...
 
 On May 16, 2014, at 12:07 AM, Matthias Klose wrote:
 
 - should we add wheels everywhere? I don't think we should,
  but I'd like to state this somewhere, like in the python policy.
 
 Agreed, we should not add wheels everywhere.  I would like to keep it very
 limited to exactly the (small) set of packages we need to devendorize
 ensurepip, recursively.  If some other devendorizing task in the future
 requires the use of wheels, then we have a framework in place, but I would
 like to actively discourage their use.
 
 I do plan to propose an update to policy stating this, but I haven't gotten to
 that yet.  I will of course post the proposed update here first.
 
 - where to put wheels?  /usr/share/python-wheels is an ad-hoc
  decision which was never proposed. I'm aware about universal
  wheels but I'd like to clarify where wheels should be located.
  Do we need /usr/share/python/wheels, and/or /usr/share/python3/wheels?
 
 I proposed /usr/share/python-wheels here:
 
 https://lists.debian.org/debian-python/2014/05/msg00025.html
 
 but it's a detail that was probably easily lost in the wall of text.  I didn't
 see any objections to that specifically though.  We could change it if
 something clearly better is proposed, although it would necessitate some new
 uploads and updated quilt patches.
 
 For the current use case, we only need pure-Python wheels, and in fact Python
 can't currently import extension modules from zips, so architecture dependent
 wheels wouldn't work anyway.  Universal wheels (Python 2 and 3 compatible) are
 used because that's what the ensurepip machinery already uses.  it's just as
 easy to create universal wheels, and all the packages we currently care about
 *are* bilingual, so using them here reduces the upstream delta.  Since I don't
 view the building of wheel packages as general purpose, I think it's fine to
 just put them in a shared directory.
 
 In other words, non-universal wheels YAGNI.

For the record since it’s pip that this is for, we have no plans to ever 
include something
that won’t work with a universal Wheel. Our requirements for us to depend on 
something
is 2.6, 2.7, 3.2+ with single source and pure Python.

 
 - naming of wheel packages.  It's good to see wheels packaged
  in a separate binary package. However there is no proposal
  how to name these packages.
 
 That was also proposed in the above referenced message.  Suggestions welcome,
 but I think python-foo-wheels is as good as anything (it's pretty
 self-descriptive ;).
 
 Cheers,
 -Barry
 
 
 -- 
 To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: https://lists.debian.org/20140515183201.6acc4...@anarchist.wooz.org
 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug #732703 and fixing ensurepip/pyvenv

[Donald Stufft]

On May 9, 2014, at 6:16 PM, Barry Warsaw ba...@python.org wrote:

 A follow up.
 
 I've done a fair bit of prototyping a solution, and with Donald's gracious
 help and feedback, I think I have a plan that will work and should be
 compliant with policy.  I'm beginning to make changes to various DPMT packages
 to build the whole stack, but I'm only uploading the simpler changes for now
 until I have a working local testbed.
 
 The basic approach does involve building a bunch of wheels as explained
 earlier.  We have to build the pip and setuptools wheels from our devendorized
 packages, and install those into the venv's site-packages directory.
 Installation is done by ensurepip, and I have a branch of python3.4 that
 re-enables this.
 
 Now that python3-wheel has cleared NEW, I need to modify the d/rules files to
 build the wheels.  I have this working for pip in svn (along with significant
 packaging changes to use pybuild and update to 1.5.5), but I won't upload this
 just yet.  It installs the wheels into /usr/share/python-wheels.
 
 The tricky bit is what to do about the recursively vendorized packages.  Here
 I think the best course of action is as outlined previously - build wheels of
 these from quilt-patched Debian source at package build time, and install
 those too into /usr/share/python-wheels.  When pyvenv is called, the
 appropriate wheels will be copied into venv/usr/share/python-wheels and the
 venv's pip will know to add those to sys.path.  We cannot unpack these
 dependent wheels into the venv's site-packages so this is to me the simplest
 and best approach.  I think Donald agrees.  The one ugly bit is what to do
 about the venv wheels when you `pip upgrade pip` since this will pull the PyPI
 pip into the venv.  Our unvendorized wheels won't get deleted from the venv,
 but they also won't be used (because PyPI's pip will bundle, and won't contain
 the patch to use the unvendored versions).  IOW, our wheels will stick around
 as unused trash.  Oh well - this only happens if you `pip upgrade pip` in the
 venv.  (We can try to fix that later if we really care.)

Yes I agree. 

For the record I don’t think getting pip to uninstall them during an upgrade 
will
be very difficult but if you’re OK with them sticking around us unused turds
then there’s no real harm to not doing it.

 
 Other than extensive testing wink, what needs to happen now is that I need
 to modify the packaging of the dependencies to build the wheels.  This is
 fairly easy for html5lib, requests, python3-chardet, and urllib3, although
 some of these need their setup.py patched to use setuptools instead of
 distutils.core (because only the former supports the bdist_wheel command
 provided by python3-wheel).  These packages are all team maintained.
 
 It's trickier for distlib, colorama, and six because those are not team
 maintained, so I will generate patches and pester the maintainers once I've
 proven that the whole stack works.  I've reached out to the colorama
 maintainer to see if we can move that into DPMT, since it's only ever seen
 NMUs since the initial upload.  Colin, Matthias, keep an eye out for patches
 to six and distlib respectively.
 
 I won't finish this up this weekend, but will continue working on it early
 next week.
 
 Stay tuned, and best to ping me on IRC if you have questions.
 
 Cheers,
 -Barry


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug #732703 and fixing ensurepip/pyvenv

[Donald Stufft]

On May 7, 2014, at 11:35 AM, Barry Warsaw ba...@debian.org wrote:

  The stuff vendorized inside there don't have to have the
 dist-info directories, so perhaps they don't need to be wheeled at all, but
 they *do* have to be vendorized because of technicalities in the way pip works
 within a virtual environment.

To be specific, they cannot appear as installed to a ``pip list`` command,
where they are imported from doesn’t particular matter.

Acceptable:

1. Putting them unpacked in pip/_vendor/ and modify the imports to import things
   from pip._vendor (aka what pip normally does)
2. Putting them unpacked in pip/_vendor/ (or anywhere really) *without*
   dist-info/egg-info and modifying sys.path to add that path.
3. Putting them packed in a zip file (whl or plain zip) in pip/_vendor
   (or anywhere really) and modifying sys.path to add those zips.

Unacceptable:

4. Installing them directly into the virtual environment
5. Putting them unpacked in pip/_vendor/ (or anywhere really) *with*
   dist-info/egg-info and modifying sys.path to add that path.

Basically if pkg_resources can find a dist-info/egg-info for it, then it'll
see it installed and it'll be available for ``pip uninstall`` (explicitly or
implicitly through an upgrade/downgrade).

Also to be explicit, I've never actually tested what pip will do if there are
*two* copies of requests (for example), one installed into site-packages with a
dist-info/egg-info and one being just the ``requests`` directory that had been
added into sys.path (basically options 2 and 3). I *think* it will work but
that should be tested.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug #732703 and fixing ensurepip/pyvenv

[Donald Stufft]

On May 6, 2014, at 6:28 PM, Barry Warsaw ba...@debian.org wrote:

 PEP 427 does discourage direct import from wheels[1] but for the things we
 care about immediately, the two conditions should not affect us (i.e. no C
 extensions, and no bad use of __file__) so I think we could get away with it.
 I'm open to other good arguments for or against using the wheels directly.

For what it’s worth, I’m the one who pushed hard against ZipImport capabilities
being codified in PEP427 and I can say that we are *incredibly* unlikely to
remove that ability without introducing a better way of achieving that. So even
if the current method breaks, there is almost certainly going to be a path 
forward.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

\o/

 On Apr 4, 2014, at 7:12 PM, Barry Warsaw ba...@python.org wrote:
 
 On Mar 19, 2014, at 05:40 PM, Barry Warsaw wrote:
 
 The current situation in the Python 3.4 package is suboptimal because:
 
 % pyvenv-3.4 /tmp/zz
 Error: Command '['/tmp/zz/bin/python3.4', '-Im', 'ensurepip', '--upgrade', 
 '--default-pip']' returned non-zero exit status 1
 
 Please see issue #732703.  I have a patch that restores usable functionality
 to pyvenv.  Hopefully Matthias will approve and apply the patch.
 
 Cheers,
 -Barry


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/010b1ae2-0d89-4b3d-9ba7-1c35f9d9b...@stufft.io

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 26, 2014, at 9:12 AM, Piotr Ożarowski pi...@debian.org wrote:

 [Barry Warsaw, 2014-03-25]
 One of the things I'd like to see, in addition to supporting
 platform-specified system-level installation directories
 (i.e. /usr/local/lib/pythonX.Y/dist-packages on Debian), is an upstream 
 switch
 to tell Python to ignore this directory, mirroring e.g. -s.  That way, if
 people say Well I just pip installed foo into /usr/local and it broke my
 system we'd be able to respond you better use pythonX.Y --dont-blame-us.
 
 I like the idea (where --dont-blame-us simply disables support for
 .egg/.zip/.whl in the interpreter, assuming pip/easy_install installs
 zips instead of unpacking them)
 -- 
 Piotr Ożarowski Debian GNU/Linux Developer
 www.ozarowski.pl  www.griffith.cc   www.debian.org
 GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
 
 
 -- 
 To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: https://lists.debian.org/20140326131210.gd30...@sts0.p1otr.com
 

Pip does not install as an .egg or a .whl. It doesn’t use .egg at all and 
regardless
of which format it downloads it unpacks it and installs it. There is no 
difference in
format between what pip installs and any debian package I’ve ever looked at.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 26, 2014, at 9:16 AM, Piotr Ożarowski pi...@debian.org wrote:

 [Donald Stufft, 2014-03-26]
 On Mar 26, 2014, at 9:12 AM, Piotr Ożarowski pi...@debian.org wrote:
 [Barry Warsaw, 2014-03-25]
 One of the things I'd like to see, in addition to supporting
 platform-specified system-level installation directories
 (i.e. /usr/local/lib/pythonX.Y/dist-packages on Debian), is an upstream 
 switch
 to tell Python to ignore this directory, mirroring e.g. -s.  That way, if
 people say Well I just pip installed foo into /usr/local and it broke my
 system we'd be able to respond you better use pythonX.Y --dont-blame-us.
 
 I like the idea (where --dont-blame-us simply disables support for
 .egg/.zip/.whl in the interpreter, assuming pip/easy_install installs
 zips instead of unpacking them)
 
 Pip does not install as an .egg or a .whl. It doesn’t use .egg at all and 
 regardless
 of which format it downloads it unpacks it and installs it. There is no 
 difference in
 format between what pip installs and any debian package I’ve ever looked at.
 
 then --dont-blame-us has to mean removing /usr/local from sys.path
 -- 
 Piotr Ożarowski Debian GNU/Linux Developer
 www.ozarowski.pl  www.griffith.cc   www.debian.org
 GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
 
 
 -- 
 To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: https://lists.debian.org/20140326131639.ge30...@sts0.p1otr.com
 

In my half formed idea in my head the way it’d work is there’d be a 
vendor-packages
directory where downstream can install things to, and a flag to the interpreter 
to
remove the typical site-packages. So then you’d get something like:

python -I —no-site-packages -m something

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 26, 2014, at 10:35 AM, Barry Warsaw ba...@debian.org wrote:

 On Mar 26, 2014, at 09:24 AM, Donald Stufft wrote:
 
 In my half formed idea in my head the way it’d work is there’d be a
 vendor-packages directory where downstream can install things to, and a flag
 to the interpreter to remove the typical site-packages. So then you’d get
 something like:
 
   python -I —no-site-packages -m something
 
 There has to be a short option for --no-site-packages (or whatever) so that it
 will work with shebang lines, where we already recommend -Es.

Yea I just didn’t feel like thinking up a short option :)

  But also, -I
 should imply this new option for full isolation.

Not sure about this, I don’t think I saw the original discussion but it looks 
like
-I is to prevent the user from injecting malicious code (so it removes env vars,
the user site packages, the current dir, etc). I don’t think that something
installed by pip by the system administrator falls under that.

 
 Which means for Python 3.4 and beyond we should be recommending system
 services and scripts add -I to the shebang line instead of -Es.  Then we'd get
 this new /usr/local isolation switch for free.
 
 -Barry


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 25, 2014, at 3:11 PM, Barry Warsaw ba...@debian.org wrote:

 On Mar 21, 2014, at 04:30 AM, Bohuslav Kabrda wrote:
 
 I'm on python-dev but not pypa-dev. But if you want to take the discussion
 there, I guess I'm ok with following yet another mailing list.
 
 Gmane makes life on a zillion mailing lists so much more manageable. ;)
 
 Yep, they do that. That's actually another thing that we're working on right
 now - creating an upstream-acceptable patch that would make all system-wide
 pip installations go into /usr/local/lib/..., while system-wide RPM installed
 packages would install into /usr/lib/... We know that Debian already has some
 downstream patches for this and there is also a bug opened for this upstream
 [1].  But yeah, currently users just sudo pip install foo into system
 Python packages path.
 
 Many Pycons ago, Toshio, myself, Tarek, and several others discussed a scheme
 where by (through more indirection of course), various packaging tools could
 query configuration files to determine operating system specific paths.  I
 thought it was quite a nice scheme which would have let us easily drop a conf
 file to declare where Debian wanted certain files installed, and Fedora or OS
 X, etc could declare others.  Mostly this was in the context of the (now
 abandoned) successor to setup.py, but I think tools like pip and such could
 have taken advantage of that.
 
 I'd still like something like that.  On Debian, Python adds a /usr/local
 directory to sys.path, but it's not /usr/local/lib/pythonX.Y/site-packages,
 it's .../dist-packages for reasons we've long hashed out and I don't want to
 re-open.  Suffice to say that we do have a directory where `sudo pip install`
 could install to that would play nice with system Python but wouldn't
 interfere with distro packages.  While I don't much like `sudo pip install` as
 a general recommendation, in principle we support that.
 
 Just to keep in mind when proposing this upstream that distros will have
 similar goals, but different concrete paths.

I have a half formed idea in my head about a PEP that adds more directories
on sys.path and the relevant APIs to query them (pip doesn’t decide where to
install stuff, it just asks distutils where to install stuff).

 
 What will rewheeling prefer when:
 
 * bundled ones are newer than system ones?
 
 That's not currently implemented in our patch, since we'll be removing
 bundled wheels and relying on system versions only. But to solve this
 situation for the upstream patch, we'll do it like this
 - if user runs just python3 -m ensurepip, nothing will happen
 - if user runs python3 -m ensurepip --upgrade, his setuptools/pip will get 
 upgraded
 Note, that this corresponds with current ensurepip documentation [2] of such
 situations.
 
 I read the patch, and I think I get it, but just to be clear, rewheeling
 only covers pip and setuptools, not arbitrary other packages, right?

As far as I understand it yes, also Fedora’s python-pip still bundles
pip._vendor.* 

 
 I still don't like introducing the circular dependency in Debian, so Piotr's
 suggestion is about where I think Debian should head.   That is:
 `python3 -m ensurepip` would suggest (but not execute) to install python3-pip
 (our binary package providing pip for Python 3) if it's not already
 installed.  The key thing is to make sure that pyvenv works in both cases.
 
 I'm not exactly sure about —upgrade.

I assume once someone has installed pip with apt-get they’d still be able
to run pip install —upgrade pip if they wanted too?

 
 I'm not sure what we would do if we wanted avoided the dependency cycle, and
 pip/setuptools wasn't yet installed system wide.
 
 Yeah, that's a tough one and it's actually one of the reasons why we decided
 to introduce the dependency cycle. The obvious (and IMO wrong) solution would
 be subprocess.Popen(['sudo', 'yum', 'install', 'python-pip']), but that just
 isn't something that I'd want to do.
 
 Right, we wouldn't either I think.
 
 Cheers,
 -Barry
 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 25, 2014, at 3:18 PM, Barry Warsaw ba...@debian.org wrote:

 On Mar 21, 2014, at 02:28 PM, Matthias Klose wrote:
 
 - whether to use python3-pip or a freshly downloaded pip.  When using
   pip to install for the system python3, maybe try to use the shipped
   python3-pip.  I have no opinion if another pip is downloaded and
   installed into for environments created by venv or virtualenv.  Some
   package builds use virtual environments, but the population of these
   has to be satisfied by package build dependencies anyway.
 
 In general, I'd like to use the Debian package pip/setuptools for installation
 into system locations, and I am okay with using bundled pip/setuptools for
 installation into virtualenvs.  I think that's pretty much the current state
 of affairs anyway, except for pyvenv.

Doing what you do for system locations and bundling a Python package is
the status quo for virtualenv, which the ensurepip/venv combo copied that
technique.

 
 Cheers,
 -Barry
 
 
 -- 
 To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: https://lists.debian.org/20140325151821.438fa...@limelight.wooz.org
 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

Hey there,

Pip/virtualenv maintainer here and also one of the authors of PEP453. I had
someone complaining to me that Python 3.4’s venv module was broken today and
through investigating that I found this discussion. I'm signed onto the list
now :)

I think I wrote most of those recommendations, I'm not a downstream packager
but I had hoped that if they were wrong someone would have spoke up. I know
the PEP is accepted already but personally I don't have a problem with updating
the recommendations if something else makes sense and still fulfills the
spirit of the PEP.

Also yea pip in the system Python currently kind of sucks. I want to make this
better eventually! I just don't know how yet or have the cycles to spend
investigating it.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 21, 2014, at 3:24 PM, Scott Kitterman deb...@kitterman.com wrote:

 On Wednesday, March 19, 2014 17:40:51 Barry Warsaw wrote:
 Signed by ba...@warsaw.us.   Show Details
 TL;DR: Let's re-enable the ensurepip module in Python 3.4, and possibly
   address some usability issues.  We should descend en masse on Montreal
 and stage a revolt at Pycon. :)
 
 Python 3.4 has an `ensurepip` module[1] which implements the specification
 in PEP 453 regarding the explicit bootstrapping of pip in Python
 installations[2]. This is promoted as a boon to users, especially on
 platforms without OS provided package managers, i.e. not Debian.
 
 The PEP makes some recommendations for downstreams[3], which we do not
 currently adopt, and maybe shouldn't fully.  Our current Python 3.4 package
 disables ensurepip at build time.
 
 The previous discussion on this subject[4] was extensive, and it may not be
 worth rehashing that.  I just reviewed the thread via Gmane, and AFAICT
 there is some general consensus:
 
 * pip should not be used to install packages globally, i.e. in the system
  Python's dist-packages.
 
 Until installing in the user's space is the default,

I’m ok (as a pip maintainer) with that happening in concept. There’s stuff that
needs to be done and figured out related to how it’ll work in practice, 
backwards
compatibility, consistency (especially with Windows which we unfortunately do
have to care about).

 I don't see it as being 
 suitable for installation by default.  I liked Piotr's idea about an option 
 along the lines of -i-know-if-a-break-it-i-keep-both-halves to override that.
 
 Scott K

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 21, 2014, at 3:39 PM, Donald Stufft don...@stufft.io wrote:

 
 On Mar 21, 2014, at 3:24 PM, Scott Kitterman deb...@kitterman.com wrote:
 
 On Wednesday, March 19, 2014 17:40:51 Barry Warsaw wrote:
 Signed by ba...@warsaw.us.  Show Details
 TL;DR: Let's re-enable the ensurepip module in Python 3.4, and possibly
  address some usability issues.  We should descend en masse on Montreal
 and stage a revolt at Pycon. :)
 
 Python 3.4 has an `ensurepip` module[1] which implements the specification
 in PEP 453 regarding the explicit bootstrapping of pip in Python
 installations[2]. This is promoted as a boon to users, especially on
 platforms without OS provided package managers, i.e. not Debian.
 
 The PEP makes some recommendations for downstreams[3], which we do not
 currently adopt, and maybe shouldn't fully.  Our current Python 3.4 package
 disables ensurepip at build time.
 
 The previous discussion on this subject[4] was extensive, and it may not be
 worth rehashing that.  I just reviewed the thread via Gmane, and AFAICT
 there is some general consensus:
 
 * pip should not be used to install packages globally, i.e. in the system
 Python's dist-packages.
 
 Until installing in the user's space is the default,
 
 I’m ok (as a pip maintainer) with that happening in concept. There’s stuff 
 that
 needs to be done and figured out related to how it’ll work in practice, 
 backwards
 compatibility, consistency (especially with Windows which we unfortunately do
 have to care about).
 
 I don't see it as being 
 suitable for installation by default.  I liked Piotr's idea about an option 
 along the lines of -i-know-if-a-break-it-i-keep-both-halves to override that.
 
 Scott K
 
 -
 Donald Stufft
 PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

https://github.com/pypa/pip/issues/1668

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail