Re: Bug#812117: RM: simpletal -- RoQA; unmaintained; low popcon

[Raphael Hertzog]

On Wed, 20 Jan 2016, Mattia Rizzolo wrote:
> loggerhead: loggerhead

And that rdep is relatively important since AFAIK we use loggerhead at least on
alioth to browse bzr repositories.

Maybe it's best to invest a bit of time into updating it rather than
dropping it just for the sake of its low popcon?

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Paul Tagliamonte]

Hey Donald!

As far as using pip to do stuff system-wide, I wrote thoughts on
http://notes.pault.ag/debian-python

As for the rest of it, distutils is actually concretely shitty, and
replacing it with setuptools
sounds sane. Sounds like a solid idea.

Perhaps we can add a Lintian warning for using distutils; it's really nasty
cruft,
and it's going to bite the archive sooner rather than later.

Cheers,
  Paul

On Thu, Jan 21, 2016 at 9:19 AM, Donald Stufft  wrote:

> Hello!
>
> I'd like to suggest a change to the Debian Policy around Python packages
> that
> will help enable the world of Python packaging to continue to progress
> forward.
>
> First, a little bit of background:
>
> At the Python level there are three metadata formats for Python packaging:
>
> * The original, setuptools style .egg-info directories.
> * The distutils style .egg-info *file* added to distutils at some point.
> * The new and improved, wheel based .dist-info directories.
>
> The presence of any of these files will signal to Python tools that a
> particular distribution has been installed, however there are two fairly
> major
> and important differences between the distutils style, and the other two.
>
> 1. The distutils style has no provisions to record what files on the system
>belong to the installed distribution, making it appear to Python tooling
>that there *are* no files other than the metadata file itself.
>
> 2. The distutils style has no provisions to include additional metadata
> files
>in the metadata, making it impossible to extend the python level
> metadata
>with additional files.
>
> I have a series of improvements that I'd like to make to the packaging
> toolchain that will sort of build on one another, but which is not going to
> function correctly with the distutils style metadata and I'm hoping that I
> can
> convince y'all to make it policy to default to generating one of the other
> two
> kinds (with varying methods, more on that later).
>
> Concretely the thing that this is blocking right now, is that with the
> newly
> released pip 8.0 I tried to make it so that pip will refuse to uninstall a
> project that is installed with distutils style metadata. This is because
> we do
> not have any way to associate the actual .py (and others) files on disk
> with
> the installed metadata, so all we have ever done is just simply remove the
> metadata file, making it appear as if the item is uninstalled but leaving
> behind all of the actual files. However I'm going to be reverting this in a
> pip 8.0.1 release because it caused a decent amount of breakage amongst
> pip's
> users, almost all of them people who are attempting to upgrade OS provided
> packages using pip.
>
> Now, I know that upgrading OS provided packages using pip is less than
> optimal
> and I would greatly prefer that people did not do it (and I'm generally in
> agreement) however if we don't enable people to do it, they'll just
> continue to
> use an old version of pip and file bugs. It's a non starter for pip to
> make it
> impossible to do.
>
> In addition to the uninstall bit, it also means that things like pip show
> -f
> return junk information for packages installed in this way.
>
> Beyond just (eventually) enabling pip to disable uninstallations of
> distutils
> based installs this will start to allow some other future changes that I
> think
> will be more interesting to Debian. The uninstallation of distutils based
> installs comes hand in hand with pip stomping all over already existing
> files
> willy nilly because the way upgrading a project like that works is pip
> uninstalls the metadata file that says X is installed, then it just
> overwrites
> over any of the files that happen to be in it's way when it installs the
> newer
> version. If we can remove the need for pip to gleefully overwrite files to
> support these types of installed packages, then we can make it so pip will
> hard fail if it attempts to overwrite an already existing file on disk.
>
> An additional benefit here is that by switching to using the directory
> based
> options, we can add additional metadata files to the installed projects,
> much
> like the INSTALLER file from PEP376 (IIRC). This file will likely be the
> path
> to having pip refuse to touch OS owned files all together without some
> sort of
> --force flag to override the safety switch.
>
> As far as compatibility goes, pip has always forced everything to be
> installed
> using setuptools and as far as I am aware, there's no real fallout from
> doing
> so. I think in 2016 it's pretty reasonable to assume that a Python project
> is
> capable of being installed using setuptools instead of distutils.
>
> So without getting into the actual *method* of doing this (of which there
> are
> several different options with different trade offs) does this sound like
> something at all that Debian would be interested in?
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Scott Kitterman]

On Thursday, January 21, 2016 10:00:53 AM Donald Stufft wrote:
> > On Jan 21, 2016, at 9:32 AM, Paul Tagliamonte  wrote:
> > 
> > Hey Donald!
> > 
> > As far as using pip to do stuff system-wide, I wrote thoughts on
> > http://notes.pault.ag/debian-python 
> I just want to be clear, that I totally agree with this. The reason I want
> to ensure we can enable this isn't because I think doing so is a great idea
> and everyone should be doing it. People *are* doing it and we (pip) have to
> be careful about what we make impossible to do because a possible result of
> going too break happy is people just never upgrade and things start to
> stagnate again.
> 
> So for pip at the system level, my goal is try and push people away from it
> as much as we can without flat out making it impossible *and* try to make
> it as "safe" to do so as possible when people decide to do it anyways.

On Debian, if you try to remove an essential package (which would be really 
bad as a rule), the system won't stop you, but you get this:

You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
 ?]

And you have to put in the entire phrase.

If pip did something like that for system level installs, I doubt most of us 
would object to it anymore.

Scott K

signature.asc
Description: This is a digitally signed message part.

Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

Hello!

I'd like to suggest a change to the Debian Policy around Python packages that
will help enable the world of Python packaging to continue to progress forward.

First, a little bit of background:

At the Python level there are three metadata formats for Python packaging:

* The original, setuptools style .egg-info directories.
* The distutils style .egg-info *file* added to distutils at some point.
* The new and improved, wheel based .dist-info directories.

The presence of any of these files will signal to Python tools that a
particular distribution has been installed, however there are two fairly major
and important differences between the distutils style, and the other two.

1. The distutils style has no provisions to record what files on the system
   belong to the installed distribution, making it appear to Python tooling
   that there *are* no files other than the metadata file itself.

2. The distutils style has no provisions to include additional metadata files
   in the metadata, making it impossible to extend the python level metadata
   with additional files.

I have a series of improvements that I'd like to make to the packaging
toolchain that will sort of build on one another, but which is not going to
function correctly with the distutils style metadata and I'm hoping that I can
convince y'all to make it policy to default to generating one of the other two
kinds (with varying methods, more on that later).

Concretely the thing that this is blocking right now, is that with the newly
released pip 8.0 I tried to make it so that pip will refuse to uninstall a
project that is installed with distutils style metadata. This is because we do
not have any way to associate the actual .py (and others) files on disk with
the installed metadata, so all we have ever done is just simply remove the
metadata file, making it appear as if the item is uninstalled but leaving
behind all of the actual files. However I'm going to be reverting this in a
pip 8.0.1 release because it caused a decent amount of breakage amongst pip's
users, almost all of them people who are attempting to upgrade OS provided
packages using pip.

Now, I know that upgrading OS provided packages using pip is less than optimal
and I would greatly prefer that people did not do it (and I'm generally in
agreement) however if we don't enable people to do it, they'll just continue to
use an old version of pip and file bugs. It's a non starter for pip to make it
impossible to do.

In addition to the uninstall bit, it also means that things like pip show -f
return junk information for packages installed in this way.

Beyond just (eventually) enabling pip to disable uninstallations of distutils
based installs this will start to allow some other future changes that I think
will be more interesting to Debian. The uninstallation of distutils based
installs comes hand in hand with pip stomping all over already existing files
willy nilly because the way upgrading a project like that works is pip
uninstalls the metadata file that says X is installed, then it just overwrites
over any of the files that happen to be in it's way when it installs the newer
version. If we can remove the need for pip to gleefully overwrite files to
support these types of installed packages, then we can make it so pip will
hard fail if it attempts to overwrite an already existing file on disk.

An additional benefit here is that by switching to using the directory based
options, we can add additional metadata files to the installed projects, much
like the INSTALLER file from PEP376 (IIRC). This file will likely be the path
to having pip refuse to touch OS owned files all together without some sort of
--force flag to override the safety switch.

As far as compatibility goes, pip has always forced everything to be installed
using setuptools and as far as I am aware, there's no real fallout from doing
so. I think in 2016 it's pretty reasonable to assume that a Python project is
capable of being installed using setuptools instead of distutils.

So without getting into the actual *method* of doing this (of which there are
several different options with different trade offs) does this sound like
something at all that Debian would be interested in?

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Request to Join

[Harlan Lieberman-Berg]

Hello python modules team!

I'd like to request to join this team to help maintain backports of the Let's 
Encrypt dependencies, especially python-sphinx.

I've read and agree to the Python Modules Team Policy. My Alioth username is 
hlieberman-guest.

Sincerely,
-- 
Harlan Lieberman-Berg
~hlieberman

Python Policy: Things to consider for Stretch

[Scott Kitterman]

I've taken a run through the current Python Policy to see where I think it 
needs to be updated for Stretch.  The updates largely fall into four 
categories:

1.  Update old examples
2.  Clean up old policy test that no longer applies
3.  Simplify things due to there only being one python version
4.  Make the text better match current practice

I've sent this twice now and the list didn't like it for different reasons, so 
I've posted [1] the diff I've come up with so far in rfcdiff format (because I 
think it's easier to read that unified diffs for this kind of thing).

This is just my opinion on stuff I found.  I'm interested in feedback on how 
these changes look and what else ought to be considered (no: rewrite the whole 
thing is not useful feedback unless you provide the text).

While I don't have any specific near-term deadlines for this, at least one of 
the proposals (killing off python:Provides) will take a little while to 
accomplish, so I would like to get consensus on what's reasonable sooner 
rather than later.

Scott K

[1] http://kitterman.com/debian/python-policy.new-from-old.diff.html

signature.asc
Description: This is a digitally signed message part.

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 21, 2016, at 9:32 AM, Paul Tagliamonte  wrote:
> 
> Hey Donald!
> 
> As far as using pip to do stuff system-wide, I wrote thoughts on 
> http://notes.pault.ag/debian-python 

I just want to be clear, that I totally agree with this. The reason I want to
ensure we can enable this isn't because I think doing so is a great idea and
everyone should be doing it. People *are* doing it and we (pip) have to be
careful about what we make impossible to do because a possible result of going
too break happy is people just never upgrade and things start to stagnate
again.

So for pip at the system level, my goal is try and push people away from it as
much as we can without flat out making it impossible *and* try to make it as
"safe" to do so as possible when people decide to do it anyways.

> 
> As for the rest of it, distutils is actually concretely shitty, and replacing 
> it with setuptools
> sounds sane. Sounds like a solid idea.
> 
> Perhaps we can add a Lintian warning for using distutils; it's really nasty 
> cruft,
> and it's going to bite the archive sooner rather than later.
> 

As far as options for how to implement this goes, I'm not sure what best fits
with Debian so I'll leave that up to y'all, but here are a few options that I
can think of.

A lintian check is probably a pretty good thing, I would guess that it would
specifically check for the existence of a .egg-info or .dist-info directory
(or check to the non-existence of a .egg-info file).

In pip the way we make this work is with a bit of a gross little thunk that
ensures setuptools is imported prior to setup.py being invoked which forces all
setup.py calls to use setuptools instead of distutils even if they import from
distutils.

Of course, many setup.py already unconditionally use setuptools, and many more
conditionally use it (typically falling back to distutils) so for many projects
it may be enough to just add python-setuptools to build-depends. For projects
that unconditionally use distutils, either the pip way or patching setup.py
might be a reasonable way forward.

A more controversial way that comes with possibly some extra benefits (which
Debian may not care about) is to use ``pip`` itself as the build tool rather
than directly invoking setup.py. In this pip would be responsible for mucking
around with the distutils/setuptools mess instead of that needing to be handled
by Debian specific tooling. You could even leverage this to remove some of the
runtime dependencies on python-pkg-resources and speed up some python using
software if you let pip build a wheel as part of the buld process [1][2][3]. I
suspect pip would need to grow some additional options to make it suitable for
you in this role, but I would be more than willing to help if this is desired
(or not, this proposal isn't about trying to ram pip down anyones throat!).


[1] To be clear, this isn't about adding wheels for everything to the archive,
it would only be using Wheel as an intermediate step in the build process.
[2] Console scripts generated by setuptools entry wrappers when installed by
setuptools have a dependency on pkg_resources at runtime, however those
same scripts, when installed by pip from a wheel, do not. This would enable
removing the runtime dependency on pkg_resources for anything that only has
it for console scripts.
[3] Import pkg_resources is not the fasest thing in the world, it has to scan
every directory on sys.path looking for things to activate and it comes
with a bunch of side effects. This happens implicitly for any project using
console scripts.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail