Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-23 Thread Scott Kitterman
On Saturday, January 23, 2016 03:13:48 AM Scott Kitterman wrote:
> On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
> > > On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski  wrote:
> > > 
> > > to be honest, I still don't know what you're asking for. What do you
> > > want us to do? Patch 2.7's distutils?
> > 
> > Essentially, ensure that setuptools not distutils is used in a setup.py.
> > There are generally three kinds of setup.py files:
> > 
> > 1) Ones that use setuptools unconditionally - These ones you just leave
> > alone, they are already correct and you should already have a build
> > depends
> > on python-setuptools.
> > 2) Ones that conditionally use setuptools - These ones you just need to
> > satisfy whatever condition the setup.py uses to enable setuptools.
> > Typically this is just checking if setuptools is importable but sometimes
> > they use environment variables or similar.
> > 3) Ones that use distutils unconditionally - These ones you switch to
> > making them use setuptools instead of distutils.
> > 
> > Now, that’s the high level overview, there’s an easier, more automatic way
> > that could maybe just be added to pybuild (Not sure exactly how pybuild
> > 
> > works) where instead of invoking the setup.py as:
> > python setup.py install (or whatever commands/args you’re passing)
> > 
> > You do it as (taken from pip):
> > python -c "import setuptools,
> > 
> > tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
> > open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
> > (or whatever commands/args you’re passing).
> > 
> > The thing is kind of ugly, but that will install things using setuptools
> > (just like pip does) regardless of if it imports setuptools or distutils
> > in
> > it’s setup.py file.
> 
> I decided to work on switching something I'm upstream for from distutils to
> setuptools today (since hey, it's the future) and discovered there aren't
> three kinds as listed above.  There are four:
> 
> 4) Packages that use distutils features that aren't available in setuptools
> and break horribly when switched.
> 
> As a result, I think I global change is a non-starter.
> 
> In my particular case, here's what I have (snipped from setup.py) to install
> man pages and a configuration file for an application written in python3:
> 
> data_files=[(os.path.join('share', 'man', 'man1'),
>   ['policyd-spf.1']), (os.path.join('share', 'man', 'man5'),
>   ['policyd-spf.conf.5']), (os.path.join('/etc', 'python-policyd-spf'),
>   ['policyd-spf.conf']), (os.path.join('share', 'man', 'man5'),
>   ['policyd-spf.peruser.5'])],
> 
> The man pages end up in build/bdist.linux-x86_64/egg as does the config file
> if I remove the leading "/", but it still won't install it.
> 
> codesearch.debian.net claims 720 packages using data_files.  I doubt they
> are all using that pythonically, but they'll need to be checked for
> breakage.
> 
> The standard Google answer to the question of how to install man pages using
> setuptools seems to be use distutils.
> 
> No way can we do any automagic switching.

BTW, it's probably worth mentioning that this particularly burns because the 
setuptools describes this as a feature[1].

> Note, by the way, that this encapsulation of data files means that you can’t
> actually install data files to some arbitrary location on a user’s machine;
> this is a feature, not a bug. You can always include a script in your
> distribution that extracts and copies your the documentation or data files
> to a user-specified location, at their discretion.

Scott K

[1] http://pythonhosted.org/setuptools/setuptools.html#including-data-files

signature.asc
Description: This is a digitally signed message part.


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-23 Thread Scott Kitterman
On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
> > On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski  wrote:
> > 
> > to be honest, I still don't know what you're asking for. What do you
> > want us to do? Patch 2.7's distutils?
> 
> Essentially, ensure that setuptools not distutils is used in a setup.py.
> There are generally three kinds of setup.py files:
> 
> 1) Ones that use setuptools unconditionally - These ones you just leave
> alone, they are already correct and you should already have a build depends
> on python-setuptools. 
> 2) Ones that conditionally use setuptools - These ones you just need to
> satisfy whatever condition the setup.py uses to enable setuptools. Typically
> this is just checking if setuptools is importable but sometimes they use
> environment variables or similar. 
> 3) Ones that use distutils unconditionally - These ones you switch to making
> them use setuptools instead of distutils.
> 
> Now, that’s the high level overview, there’s an easier, more automatic way
> that could maybe just be added to pybuild (Not sure exactly how pybuild
> works) where instead of invoking the setup.py as:
> 
> python setup.py install (or whatever commands/args you’re passing)
> 
> You do it as (taken from pip):
> 
> python -c "import setuptools,
> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
> (or whatever commands/args you’re passing).
> 
> The thing is kind of ugly, but that will install things using setuptools
> (just like pip does) regardless of if it imports setuptools or distutils in
> it’s setup.py file.

I decided to work on switching something I'm upstream for from distutils to 
setuptools today (since hey, it's the future) and discovered there aren't 
three kinds as listed above.  There are four:

4) Packages that use distutils features that aren't available in setuptools 
and break horribly when switched.

As a result, I think I global change is a non-starter.

In my particular case, here's what I have (snipped from setup.py) to install 
man pages and a configuration file for an application written in python3:

data_files=[(os.path.join('share', 'man', 'man1'),
  ['policyd-spf.1']), (os.path.join('share', 'man', 'man5'),
  ['policyd-spf.conf.5']), (os.path.join('/etc', 'python-policyd-spf'),
  ['policyd-spf.conf']), (os.path.join('share', 'man', 'man5'),
  ['policyd-spf.peruser.5'])],

The man pages end up in build/bdist.linux-x86_64/egg as does the config file if 
I remove the leading "/", but it still won't install it.

codesearch.debian.net claims 720 packages using data_files.  I doubt they are 
all using that pythonically, but they'll need to be checked for breakage.

The standard Google answer to the question of how to install man pages using 
setuptools seems to be use distutils.

No way can we do any automagic switching.

Scott K

signature.asc
Description: This is a digitally signed message part.


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Scott Kitterman
On Friday, January 22, 2016 01:32:45 PM Fred Drake wrote:
> On Fri, Jan 22, 2016 at 12:40 PM, Scott Kitterman  
wrote:
> > Currently --record includes the .pyc files which is both unneeded and bad.
> > Before this gets added either in setuptools or by us, this needed to be
> > fixed.
> Why is this bad?  Isn't the point that the record file lists the files
> installed by the installation process?  If the pyc files are installed
> as part of the package, they should be listed.

For Debian it's bad because we don't ship the .pyc files in the package they 
are managed locally by the installed python system.  They are also unnecessary 
because setuptools/pip/python is smart enough to relate .pyc files to their .py 
files without them being listed.

Part of the goal here is to make sure that regardless of the installation 
method, there's a list of files that were installed.  Since Debian packages 
don't install the .pyc files, listing them is wrong.

Scott K

P.S.  I'm subscribed to the list, so there's no need to CC me.



Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Fred Drake
On Fri, Jan 22, 2016 at 12:40 PM, Scott Kitterman  wrote:
> Currently --record includes the .pyc files which is both unneeded and bad.
> Before this gets added either in setuptools or by us, this needed to be fixed.

Why is this bad?  Isn't the point that the record file lists the files
installed by the installation process?  If the pyc files are installed
as part of the package, they should be listed.


  -Fred

-- 
Fred L. Drake, Jr.
"A storm broke loose in my mind."  --Albert Einstein



Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Fred Drake
On Fri, Jan 22, 2016 at 1:41 PM, Scott Kitterman  wrote:
> For Debian it's bad because we don't ship the .pyc files in the package they
> are managed locally by the installed python system.  They are also unnecessary
> because setuptools/pip/python is smart enough to relate .pyc files to their 
> .py
> files without them being listed.

The .pyc files aren't shipped in the .deb, or they aren't installed
when the .deb is installed?

If they're installed, I think they should be listed.  If not installed
when the package is installed, then they definitely shouldn't be
listed.

> Part of the goal here is to make sure that regardless of the installation
> method, there's a list of files that were installed.  Since Debian packages
> don't install the .pyc files, listing them is wrong.

I agree that listing them if they aren't installed is wrong.


  -Fred

-- 
Fred L. Drake, Jr.
"A storm broke loose in my mind."  --Albert Einstein



Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Scott Kitterman
On Friday, January 22, 2016 01:47:28 PM Fred Drake wrote:
> On Fri, Jan 22, 2016 at 1:41 PM, Scott Kitterman  
wrote:
> > For Debian it's bad because we don't ship the .pyc files in the package
> > they are managed locally by the installed python system.  They are also
> > unnecessary because setuptools/pip/python is smart enough to relate .pyc
> > files to their .py files without them being listed.
> 
> The .pyc files aren't shipped in the .deb, or they aren't installed
> when the .deb is installed?

They aren't shipped in the .deb, so the list of files shipped in the .deb 
shouldn't include them.

> If they're installed, I think they should be listed.  If not installed
> when the package is installed, then they definitely shouldn't be
> listed.

No  I don't think they should be.  There's no need for that and it may change.  
Then you have a maintenance nightmare.  Currently we have both python3.4 and 
python3.5 as supported versions so as part of installation *.cpython-34.pyc 
and *.cpython-35.pyc are generated on the local file system.  As the python3 
versions change, those will change and Python knows how to deal with that as 
long as the .py files are present.

Scott K

> > Part of the goal here is to make sure that regardless of the installation
> > method, there's a list of files that were installed.  Since Debian
> > packages
> > don't install the .pyc files, listing them is wrong.
> 
> I agree that listing them if they aren't installed is wrong.
> 
> 
>   -Fred



Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Donald Stufft

> On Jan 22, 2016, at 6:02 PM, Barry Warsaw  wrote:
> 
> On Jan 22, 2016, at 05:50 PM, Donald Stufft wrote:
> 
>> Forget that pip can fetch files from PyPI and install them for a moment and
>> consider the command ``pip install .``. Fundamentally this is similar to the
>> command ``make install`` right?
> 
> Please remind me what the long term plan for this is.  Let's say it's 2021 and
> you and your amazing team of dozens of distutil-sig developers have been
> cranking away the whole time.
> 
> What tool will be used to build Python packages?

Whatever people want! At least, that’s my hope. Ideally we get away from
the idea that any particular build system is the right build system (because
the needs of a pure python thing like six.py is vastly different from Numpy
which needs Fortran and other crazy things).

My end goal is, as long as your build system can consume a sdist (or maybe
a repository too, still up in the air, but some kind of source input) and
produce a wheel (whether this is actually a zipped up wheel file or a properly
formatted directory is also still up in the air). Then you have a good enough
build system and that is all it needs to do, it doesn’t need to also implement
installing and what not, that’s some other tool’s job.

> 
> What tool will be used to upload (built/source?) Python packages to
> WarehouseNG?

Either twine, or we’ll fold that functionality into pip.

> 
> What tool will be used to download Python packages from WarehouseNG?

Whatever tool you want! We’ll have defined formats so as long as your tool
speaks those formats it can install them. In the case of already “compiled”
(pure Python or not) binary .whl files it can just download them, unzip,
move into the correct location + install deps and such. For source files
(in a not-yet thought out or done sdist 2.0 thing) it can download the
sdist 2.0 file, install the build dependencies, invoke the build tool for
this package (from above), get the wheel it produced, and then install
that as if it had downloaded it from PyPI.

> 
> What tool will be used to install packages from WarehouseNG into an importable
> location?

Same answer here.

> 
> Cheers,
> -Barry


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Barry Warsaw
Hey Donald, thanks for starting this conversation.  I for one am super
appreciative of all the consideration you give for Debian's little slice of
the world.

There's a lot to unpack in this thread, and I'm a little under the weather[1],
so hopefully this makes sense.

Big +1 for recording the files that get installed via the .egg-info/RECORD
file.  As you know, I've been working on dirtbike, which is a "rewheeling"
tool to turn an installed package into a .whl.  While Debian tightly controls
via policy the set of wheels we'll allow into the archive, dirtbike has code
for parsing the RECORD file.  Unfortunately this is never exercised in
practice because we don't have RECORD files - at least not for the packages we
care about[2].

Big +1 for using setuptools everywhere.  By my count, of the packages that I
happen to randomly have installed on my Ubuntu 16.04 system, I have 67
.egg-info files and 113 .egg-info directories.  I'd rather have .egg-info
directories everywhere.

+1 for a lintian warning if distutils is used.  I guess I'm +0 on forcing that
through pybuild because it'll be unobvious and mysterious, and kind of lets
upstreams off the hook.  I'd mildly prefer to patch packages that use
distutils because that's much more discoverable, but I can appreciate that
that's a lot of work we'd be imposing on maintainers, so I won't argue this
too much (other than to say that if pybuild forces it, let's definitely
document this in its manpage!).

On Jan 22, 2016, at 12:40 PM, Scott Kitterman wrote:

>Currently --record includes the .pyc files which is both unneeded and bad.
>Before this gets added either in setuptools or by us, this needed to be
>fixed.

+1 for (I think) another reason than has already been discussed.  We won't be
generating .egg-info directories on the end-user's machine, but instead the
machine the package is built on.  That could be a maintainer's own system or a
central build machine depending on various conditions.  But because the pycs
are generated on the end-user's machine, we actually don't know what pycs will
be generated when the debs are installed, so the egg-info/RECORD file *can't*
contain them, at least not accurately.

On Jan 22, 2016, at 11:54 AM, Donald Stufft wrote:

>Regardless of what happens in this thread, pip is going to stop mucking
>around in files that are owned by some other tool without some sort of
>opt-in/--force style flag *and* we're going to be restructuring things to try
>and guide people away from using pip outside of a virtual environment or
>through the user of --user as well.

Of course, I'd still like --user to be the default[3].  I think that's still
the eventual goal for pip, but isn't yet implemented because $priorities.

>A more controversial way that comes with possibly some extra benefits (which
>Debian may not care about) is to use ``pip`` itself as the build tool rather
>than directly invoking setup.py. In this pip would be responsible for mucking
>around with the distutils/setuptools mess instead of that needing to be
>handled by Debian specific tooling.

I'd like to better understand how this would work.  IIUC, the Fedora ecosystem
is making or already has made this switch, but I don't know how it works.
Obviously, we don't want to install wheels into /usr/lib/python3/dist-packages.

FTR, I am still working on updating Debian's pip.  I'm slowly shaving the yak,
but there are still a few things to do.  If you want to help, get in touch.

>[3] Import pkg_resources is not the fasest thing in the world, it has to scan
>every directory on sys.path looking for things to activate and it comes with
>a bunch of side effects. This happens implicitly for any project using
>console scripts.

Which frankly sucks.  It's also fragile.  Every once in a while a broken
package gets uploaded that breaks pkg_resources, and it's not easy to debug or
repair.  I really hope Brett can fix this when/if he builds this functionality
into the stdlib.

Cheers,
-Barry

[1] double meaning: I have a cold and we're in the early stages of an historic
snow storm!

[2] dirtbike has two fallbacks, both of which use `dpkg -L` to get the list of
installed files.  The first fallback uses `dpkg -S` to find the binary package
that contains the Python package's .egg-info file/directory (doesn't matter
for this purpose).  pkg_resources doesn't have one of those.  The second
fallback imports the Python package and looks for the binary package
containing the top-level directory.  It's all rather ugly, and I'd like to
just use the .egg-info/RECORD file, but I suspect I'll still need the import
fallback for pkg_resources.

[3] https://github.com/pypa/pip/issues/1668


pgpcYL6d9Pl7r.pgp
Description: OpenPGP digital signature


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Scott Kitterman
On Friday, January 22, 2016 05:50:13 PM Donald Stufft wrote:
...
> We already have an option like this, the —root option which will just append
> a different prefix to all of the installation paths. So essentially instead
> of invoking ``python setup.py install —root /tmp/something/`` which is what
> I think you’re doing now, you’d do ``pip install —root /tmp/something/
> —no-deps .`` and it’d just work similarly to what you have now, except pip
> would be responsible for interacting with the Python build system.
...

Why would pip interacting with the Python build system instead of setuptools 
be better?

Scott K

signature.asc
Description: This is a digitally signed message part.


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Donald Stufft

> On Jan 22, 2016, at 5:30 PM, Barry Warsaw  wrote:
> 
> Hey Donald, thanks for starting this conversation.  I for one am super
> appreciative of all the consideration you give for Debian's little slice of
> the world.
> 
> There's a lot to unpack in this thread, and I'm a little under the weather[1],
> so hopefully this makes sense.
> 
> Big +1 for recording the files that get installed via the .egg-info/RECORD
> file.  As you know, I've been working on dirtbike, which is a "rewheeling"
> tool to turn an installed package into a .whl.  While Debian tightly controls
> via policy the set of wheels we'll allow into the archive, dirtbike has code
> for parsing the RECORD file.  Unfortunately this is never exercised in
> practice because we don't have RECORD files - at least not for the packages we
> care about[2].
> 
> Big +1 for using setuptools everywhere.  By my count, of the packages that I
> happen to randomly have installed on my Ubuntu 16.04 system, I have 67
> .egg-info files and 113 .egg-info directories.  I'd rather have .egg-info
> directories everywhere.
> 
> +1 for a lintian warning if distutils is used.  I guess I'm +0 on forcing that
> through pybuild because it'll be unobvious and mysterious, and kind of lets
> upstreams off the hook.  I'd mildly prefer to patch packages that use
> distutils because that's much more discoverable, but I can appreciate that
> that's a lot of work we'd be imposing on maintainers, so I won't argue this
> too much (other than to say that if pybuild forces it, let's definitely
> document this in its manpage!).
> 
> On Jan 22, 2016, at 12:40 PM, Scott Kitterman wrote:
> 
>> Currently --record includes the .pyc files which is both unneeded and bad.
>> Before this gets added either in setuptools or by us, this needed to be
>> fixed.
> 
> +1 for (I think) another reason than has already been discussed.  We won't be
> generating .egg-info directories on the end-user's machine, but instead the
> machine the package is built on.  That could be a maintainer's own system or a
> central build machine depending on various conditions.  But because the pycs
> are generated on the end-user's machine, we actually don't know what pycs will
> be generated when the debs are installed, so the egg-info/RECORD file *can't*
> contain them, at least not accurately.

FWIW It appears that if you do setup.py install —no-compile to tell distutils
not to compile the .pyc during .deb build time, I believe that as is it will
just omit the .pyc files. Not sure if that’s workable for Debian or not.

> 
> On Jan 22, 2016, at 11:54 AM, Donald Stufft wrote:
> 
>> Regardless of what happens in this thread, pip is going to stop mucking
>> around in files that are owned by some other tool without some sort of
>> opt-in/--force style flag *and* we're going to be restructuring things to try
>> and guide people away from using pip outside of a virtual environment or
>> through the user of --user as well.
> 
> Of course, I'd still like --user to be the default[3].  I think that's still
> the eventual goal for pip, but isn't yet implemented because $priorities.

Yes, still a goal (it’s not as simple as —user will be the default, but that’s
the outcome for a non root account in a system Python).

> 
>> A more controversial way that comes with possibly some extra benefits (which
>> Debian may not care about) is to use ``pip`` itself as the build tool rather
>> than directly invoking setup.py. In this pip would be responsible for mucking
>> around with the distutils/setuptools mess instead of that needing to be
>> handled by Debian specific tooling.
> 
> I'd like to better understand how this would work.  IIUC, the Fedora ecosystem
> is making or already has made this switch, but I don't know how it works.
> Obviously, we don't want to install wheels into 
> /usr/lib/python3/dist-packages.

I haven’t actually personally seen if they’ve done it, but the Fedora maintainer
of the python-pip told me they were doing it, and they had a patch to pip to
add a feature they needed for it.

Forget that pip can fetch files from PyPI and install them for a moment and
consider the command ``pip install .``. Fundamentally this is similar to the
command ``make install`` right? It builds the files and then installs them
into the final location. As I understand it, the standard way for a project
like Debian to install a project using a “regular” Makefile install like that
is to do ``make DESTDIR=/tmp/some-tempdir install`` so that it invokes the
build system, sets all the paths in the resulting build system to the final
destination, but then right before it actually copies files to the final paths
it just appends a directory in front of them to let you stage the files in
a different location instead.

We already have an option like this, the —root option which will just append
a different prefix to all of the installation paths. So essentially instead
of invoking ``python setup.py install —root 

Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Barry Warsaw
On Jan 22, 2016, at 05:50 PM, Donald Stufft wrote:

>Forget that pip can fetch files from PyPI and install them for a moment and
>consider the command ``pip install .``. Fundamentally this is similar to the
>command ``make install`` right?

Please remind me what the long term plan for this is.  Let's say it's 2021 and
you and your amazing team of dozens of distutil-sig developers have been
cranking away the whole time.

What tool will be used to build Python packages?

What tool will be used to upload (built/source?) Python packages to
WarehouseNG?

What tool will be used to download Python packages from WarehouseNG?

What tool will be used to install packages from WarehouseNG into an importable
location?

Cheers,
-Barry


pgpPQkjaWvhbD.pgp
Description: OpenPGP digital signature


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Scott Kitterman


On January 22, 2016 6:27:08 PM EST, Donald Stufft  wrote:
>
>> On Jan 22, 2016, at 6:04 PM, Scott Kitterman 
>wrote:
>> 
>> On Friday, January 22, 2016 05:50:13 PM Donald Stufft wrote:
>> ...
>>> We already have an option like this, the —root option which will
>just append
>>> a different prefix to all of the installation paths. So essentially
>instead
>>> of invoking ``python setup.py install —root /tmp/something/`` which
>is what
>>> I think you’re doing now, you’d do ``pip install —root
>/tmp/something/
>>> —no-deps .`` and it’d just work similarly to what you have now,
>except pip
>>> would be responsible for interacting with the Python build system.
>> ...
>> 
>> Why would pip interacting with the Python build system instead of
>setuptools
>> be better?
>> 
>> Scott K
>
>setuptools *is* the build system right now (or distutils if you’re
>still using
>that). So the benefit to Debian right now would be: We maintain the
>distutils/setuptools hacks to make them suck less so you don’t have to!
>
>Longer term, we want to enable people to go hog wild and invent their
>own
>build systems instead of trying to do it all one size fits all with
>everyone
>miserable because it doesn’t solve anyones problems nicely. At that
>point the
>benefit to Debian becomes that instead of having to implement the
>standard
>build tool interface that these tools implement (Which will most likely
>be
>geared towards produces wheels, not towards putting files on disk in an
>installed location) Debian can just let us maintain that bit of code
>too.
>
>Of course, if Debian would prefer to interact directly with these
>tools, it’s
>totally fine to do that. The whole point of us doing this work and
>trying to
>define formats and APIs and not bless implementations is so anyone who
>wants
>to can slot in their own tool in the process instead of being forced to
>use
>some blessed tool.

You probably expected something like this back, but here goes anyway...

The Zen of Python says, among other things, "There should be one-- and 
preferably only one --obvious way to do it".  Build systems seem to me like a 
great place to apply that.

Build system proliferation is a nightmare.  In the last several months I've 
gone through roughly all of the several hundred packages in the Debian archive 
that build (or were packed like they build) compiled code (e.g. python C 
extensions, library bindings, etc.).  I was working on two goals:

1) Making sure they built with python3.5

2) Adding support for building with multiple python3 versions

For the former, the issue were sometimes buil system related, but for the 
latter that was almost always the case.  When packages used setup.py (and it 
made no difference if it was distutills or setuputils) it was virtually always 
easy.  For anything else ranged from moderately complex to not worth the 
trouble.

For the things I need it for in the distro, setup.py (with the changes we've 
made) works really well.

Why do we need a bazillion options for Python build tools?  To close with 
another quote from the Zen, "Special cases aren't special enough to break the 
rules."

Scott K

P.S. I worry about having to go through waf again [1].  That was painful.

[1] https://lists.debian.org/debian-dak/2012/01/msg9.html



Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Piotr Ożarowski
[Paul Tagliamonte, 2016-01-22]
> I'm sure if you had a real and honest conversation with Donnald, there'd
> be middleground. I've never found him to be the sort to bully or ignore
> technical arguments.

and why we cannot find a middle ground here? Did I say something bad
about Donald? Quite the contrary, I think he's the best guy to talk to
regarding pip/setuptools/distutils.

> "Give me a thing I want and I'll fix this broken thing otherwise it'll
>  remain broken" is *NOT* how we do things in Debian. Flat out. This
> isn't a game of poker, and the archive is not yours to bargain with.

no, it's: "I already use too many hacks, but I will add another one if
you bribe me" ;P

There's nothing "broken" on our side, so I don't have to fix anything
(other than educating our users to not shoot themselves in the foot)

I want to help Daniel (I will even send a pull request if he wants),
all I need is something for me in return (and I'm also OK with adding
"Q: why don't you allow me to touch system files? A: because evil Piotr
made me" to pip's FAQ)

> I've never used my position on the ftpteam to bully someone into doing a
> thing, and I seriously hope that was a bad joke about using your
> position in DPMT/PAPT to get what you want.

it's my time and I will do in my time whatever I want. If I want to add
a hack to pybuild in my free time, I will do it when I want, you will not
bully me into doing something you think is right!
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645



Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Donald Stufft

> On Jan 22, 2016, at 11:51 AM, Scott Kitterman  wrote:
> 
> On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
>>> On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski  wrote:
>>> 
>>> to be honest, I still don't know what you're asking for. What do you
>>> want us to do? Patch 2.7's distutils?
>> 
>> Essentially, ensure that setuptools not distutils is used in a setup.py.
>> There are generally three kinds of setup.py files:
>> 
>> 1) Ones that use setuptools unconditionally - These ones you just leave
>> alone, they are already correct and you should already have a build depends
>> on python-setuptools. 2) Ones that conditionally use setuptools - These
>> ones you just need to satisfy whatever condition the setup.py uses to
>> enable setuptools. Typically this is just checking if setuptools is
>> importable but sometimes they use environment variables or similar. 3) Ones
>> that use distutils unconditionally - These ones you switch to making them
>> use setuptools instead of distutils.
>> 
>> Now, that’s the high level overview, there’s an easier, more automatic way
>> that could maybe just be added to pybuild (Not sure exactly how pybuild
>> works) where instead of invoking the setup.py as:
>> 
>>python setup.py install (or whatever commands/args you’re passing)
>> 
>> You do it as (taken from pip):
>> 
>>python -c "import setuptools,
>> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
>> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
>> (or whatever commands/args you’re passing).
>> 
>> The thing is kind of ugly, but that will install things using setuptools
>> (just like pip does) regardless of if it imports setuptools or distutils in
>> it’s setup.py file.
> 
> I tried this and it works, but what's the big deal?
> 
> It provides a PKG-INFO file that has identical content to the old egg-info 
> file,
> an empty dependency links file, and a top_level.txt file with the one line in
> it.
> 
> Why is this better (I'm not a huge upstream developer of Python stuff, but I 
> do
> do some and I don't see what this gets me)?
> 
> Scott K


This a fair question:

1) distutils is effectively frozen and all improvements to the build system
   are either going to be inside of setuptools, or will be opt in build systems
   that (probably) won't rely on setup.py at all. This includes newer metadata
   formats and the like as well.

2) That top_level.txt is an important file, without it we have no idea that
   $SITE-PACKAGES/requests is in any way associated with the metadata file
   requests-2.9.1.egg-info, that top_level.txt tells us what the top level
   import names are for a particular project.

3) It slipped my mind that you have to pass an additional flag to setuptools
   right now to get the full file list (pip passes that flag unconditionally)
   however I'm going to poke setuptools to see about getting them to add the
   record file unconditionally to the .egg-info directory so it doesn't
   require the --record flag. (Although Debian could add it earlier if they
   wanted, but it's fine to wait for setuptools to change here).

4) Switching it to a directory allows us to add additional files/information to
   the Python level metadata, like the sort of divergent thread about getting
   pip to stop mucking with OS owned files, the most likely path forward is to
   drop an INSTALLER file inside the .egg-info directory that has some special
   value ("system"? "os"? Not sure) which pip will take as a signal that it
   really should not touch.

5) In pip, because of #2 and #3 a regular distutils installed package (which
   gives you an .egg-info file, not directory) we have now way when we're told
   to install a package named "foo" to associate that with any files on disk.
   This means that we're currently lying when we tell a user we've uninstalled
   a distutils installed package, we just remove the .egg-info file but leave
   all the other files behind. We've deprecated the ability to do this, and we
   tried to remove it all together in the recent pip 8.0 release, but we had to
   roll it back because it caused too much breakage, largely due to OS packages
   that were using distutils installed metadata.

   On the surface, it may seem counterproductive for Debian to help pip get
   better metadata about what files belong to a package so that we can
   uninstall it, but we're going to solve the "people modify system files with
   pip when they shouldn't do that" more directly and there exist use cases
   where it's perfectly fine to break your system in this way, because it's an
   emphereal system (think CI servers) where you just want to get the newest
   version of a package (maybe just one package) into the system in order to
   run some tests before you throw the whole server away or something equally
   as reasonable.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 

Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Paul Tagliamonte
On Fri, Jan 22, 2016 at 05:18:36PM +0100, Piotr Ożarowski wrote:
> let's make a deal. If you will make sure pip doesn't touch system files
> (and others will not crucify me for this) - I will make sure pybuild
> uses above line (if setuptools is not detected in setup.py but is listed
> in Build-Depends).

Oh *come on*. That's insane.


I'm sure if you had a real and honest conversation with Donnald, there'd
be middleground. I've never found him to be the sort to bully or ignore
technical arguments.


"Give me a thing I want and I'll fix this broken thing otherwise it'll
 remain broken" is *NOT* how we do things in Debian. Flat out. This
isn't a game of poker, and the archive is not yours to bargain with.

Your argument is basically:

 - Thing X being broken is preventing Y from doing Z
 - We do Z on our platform
 - So, I'll fix X when you make Y stop doing Z


pip is used outside Debian, and another special case is tech debt. pip
touching the system sucks, but that's unrelated to the fact that thing X
sucks.

Why these issues are intertwined is confusing me, perhaps dstufft would
have been better being dishonest and not mentioned pip in his mail.


I've never used my position on the ftpteam to bully someone into doing a
thing, and I seriously hope that was a bad joke about using your
position in DPMT/PAPT to get what you want.

Cheers,
  Paul


signature.asc
Description: PGP signature


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Scott Kitterman
On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
> > On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski  wrote:
> > 
> > to be honest, I still don't know what you're asking for. What do you
> > want us to do? Patch 2.7's distutils?
> 
> Essentially, ensure that setuptools not distutils is used in a setup.py.
> There are generally three kinds of setup.py files:
> 
> 1) Ones that use setuptools unconditionally - These ones you just leave
> alone, they are already correct and you should already have a build depends
> on python-setuptools. 2) Ones that conditionally use setuptools - These
> ones you just need to satisfy whatever condition the setup.py uses to
> enable setuptools. Typically this is just checking if setuptools is
> importable but sometimes they use environment variables or similar. 3) Ones
> that use distutils unconditionally - These ones you switch to making them
> use setuptools instead of distutils.
> 
> Now, that’s the high level overview, there’s an easier, more automatic way
> that could maybe just be added to pybuild (Not sure exactly how pybuild
> works) where instead of invoking the setup.py as:
> 
> python setup.py install (or whatever commands/args you’re passing)
> 
> You do it as (taken from pip):
> 
> python -c "import setuptools,
> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
> (or whatever commands/args you’re passing).
> 
> The thing is kind of ugly, but that will install things using setuptools
> (just like pip does) regardless of if it imports setuptools or distutils in
> it’s setup.py file.

I tried this and it works, but what's the big deal?

It provides a PKG-INFO file that has identical content to the old egg-info 
file, 
an empty dependency links file, and a top_level.txt file with the one line in 
it.

Why is this better (I'm not a huge upstream developer of Python stuff, but I do 
do some and I don't see what this gets me)?

Scott K

signature.asc
Description: This is a digitally signed message part.


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Donald Stufft

> On Jan 22, 2016, at 11:18 AM, Piotr Ożarowski  wrote:
> 
> let's make a deal. If you will make sure pip doesn't touch system files
> (and others will not crucify me for this) - I will make sure pybuild
> uses above line (if setuptools is not detected in setup.py but is listed
> in Build-Depends).

Regardless of what happens in this thread, pip is going to stop mucking around
in files that are owned by some other tool without some sort of opt-in/--force
style flag *and* we're going to be restructuring things to try and guide people
away from using pip outside of a virtual environment or through the user of
--user as well.

The main reason why that hasn't happened yet isn't because I'm holding those
things hostage, it's because I'm one person and my time is spread amongst
several really important tools and I have to prioritize things in accordance to
how bad it's on fire. In addition, pip is a cross platform program and we have
to ensure that whatever we do works in a cross platform manner. You mentioned
/usr/local/.../site-packages/ before, but plenty of systems don't have a
/usr/local/.../site-packages/ that their Python respects because that is a
Debian specific patch, so if we refused to install to /usr/.../site-packages/
we'd break installing at the system level on all platforms but Debian derived
ones.

What we need is a cross platform way to determine if a particular installed
Python package is "owned" by the system or if it's something we're allowed to
modify. That's going to require some effort to work out the exact best way,
but in my head the way forward on that is to use the metadata directory (the
one I'm asking y'all to start using) and add an additional piece of metadata
inside of it that just says "Hey, This is a system install" that we can inspect
and then take additional logic (like refusing to touch it without a --force)
based on that. The change I'm asking for here helps make that possible (among
other things).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Piotr Ożarowski
Hi,

[Donald Stufft, 2016-01-21]
> I'd like to suggest a change to the Debian Policy around Python packages that
> will help enable the world of Python packaging to continue to progress 
> forward.
[...]
> I have a series of improvements that I'd like to make to the packaging
> toolchain that will sort of build on one another, but which is not going to
> function correctly with the distutils style metadata and I'm hoping that I can
> convince y'all to make it policy to default to generating one of the other two
> kinds (with varying methods, more on that later).

Isn't that something for distutils-sig or python-dev?
We install whatever setup.py creates, no matter if it's egg-info dir,
file or something else.

[...]
> Now, I know that upgrading OS provided packages using pip is less than optimal

it's more than that, it's something that makes me scream: kill it! kill it with
fire!!! (but since we also allow /bin/rm in Debian, I guess we have to
tolerate pip ;)

> and I would greatly prefer that people did not do it (and I'm generally in
> agreement) however if we don't enable people to do it, they'll just continue 
> to
> use an old version of pip and file bugs. It's a non starter for pip to make it
> impossible to do.

did I mention fire? ;)

[...]
> An additional benefit here is that by switching to using the directory based
> options, we can add additional metadata files to the installed projects, much
> like the INSTALLER file from PEP376 (IIRC). This file will likely be the path
> to having pip refuse to touch OS owned files all together without some sort of
> --force flag to override the safety switch.

why pip cannot do that right now? IMO pip should be allowed to touch 3 types
of paths only: /usr/local/pythonX.Y/, ~/.local/lib/pythonX.Y/ and virtual envs.

[...]
> So without getting into the actual *method* of doing this (of which there are
> several different options with different trade offs) does this sound like
> something at all that Debian would be interested in?

to be honest, I still don't know what you're asking for. What do you
want us to do? Patch 2.7's distutils?
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645



Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Donald Stufft

> On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski  wrote:
> 
> to be honest, I still don't know what you're asking for. What do you
> want us to do? Patch 2.7's distutils?

Essentially, ensure that setuptools not distutils is used in a setup.py. There 
are generally three kinds of setup.py files:

1) Ones that use setuptools unconditionally - These ones you just leave alone, 
they are already correct and you should already have a build depends on 
python-setuptools.
2) Ones that conditionally use setuptools - These ones you just need to satisfy 
whatever condition the setup.py uses to enable setuptools. Typically this is 
just checking if setuptools is importable but sometimes they use environment 
variables or similar.
3) Ones that use distutils unconditionally - These ones you switch to making 
them use setuptools instead of distutils.

Now, that’s the high level overview, there’s an easier, more automatic way that 
could maybe just be added to pybuild (Not sure exactly how pybuild works) where 
instead of invoking the setup.py as:

python setup.py install (or whatever commands/args you’re passing)

You do it as (taken from pip):

python -c "import setuptools, 
tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open', 
open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install (or 
whatever commands/args you’re passing).

The thing is kind of ugly, but that will install things using setuptools (just 
like pip does) regardless of if it imports setuptools or distutils in it’s 
setup.py file.


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Piotr Ożarowski
> let's make a deal. If you will make sure pip doesn't touch system files
> (and others will not crucify me for this) - I will make sure pybuild
> uses above line (if setuptools is not detected in setup.py but is listed
> in Build-Depends).

FTR: (after talking about it on IRC)

the "deal" was about my free time (for implementing this) and not about
not accepting patches (and Donald already mentioned he wants to
implement the cookie I want so I simply wanted him to give it higher
priority)
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645



Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Scott Kitterman
On Friday, January 22, 2016 12:11:27 PM Donald Stufft wrote:
...
> 3) It slipped my mind that you have to pass an additional flag to setuptools
> right now to get the full file list (pip passes that flag unconditionally)
> however I'm going to poke setuptools to see about getting them to add the
> record file unconditionally to the .egg-info directory so it doesn't
> require the --record flag. (Although Debian could add it earlier if they
> wanted, but it's fine to wait for setuptools to change here).
...
Just to document this (since I think it's significant and worth memorializing) 
from IRC:

Currently --record includes the .pyc files which is both unneeded and bad.  
Before this gets added either in setuptools or by us, this needed to be fixed.

Scott K

signature.asc
Description: This is a digitally signed message part.


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Donald Stufft

> On Jan 22, 2016, at 6:04 PM, Scott Kitterman  wrote:
> 
> On Friday, January 22, 2016 05:50:13 PM Donald Stufft wrote:
> ...
>> We already have an option like this, the —root option which will just append
>> a different prefix to all of the installation paths. So essentially instead
>> of invoking ``python setup.py install —root /tmp/something/`` which is what
>> I think you’re doing now, you’d do ``pip install —root /tmp/something/
>> —no-deps .`` and it’d just work similarly to what you have now, except pip
>> would be responsible for interacting with the Python build system.
> ...
> 
> Why would pip interacting with the Python build system instead of setuptools
> be better?
> 
> Scott K

setuptools *is* the build system right now (or distutils if you’re still using
that). So the benefit to Debian right now would be: We maintain the
distutils/setuptools hacks to make them suck less so you don’t have to!

Longer term, we want to enable people to go hog wild and invent their own
build systems instead of trying to do it all one size fits all with everyone
miserable because it doesn’t solve anyones problems nicely. At that point the
benefit to Debian becomes that instead of having to implement the standard
build tool interface that these tools implement (Which will most likely be
geared towards produces wheels, not towards putting files on disk in an
installed location) Debian can just let us maintain that bit of code too.

Of course, if Debian would prefer to interact directly with these tools, it’s
totally fine to do that. The whole point of us doing this work and trying to
define formats and APIs and not bless implementations is so anyone who wants
to can slot in their own tool in the process instead of being forced to use
some blessed tool.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-22 Thread Donald Stufft

> On Jan 22, 2016, at 7:18 PM, Scott Kitterman  wrote:
> 
> The Zen of Python says, among other things, "There should be one-- and 
> preferably only one --obvious way to do it".  Build systems seem to me like a 
> great place to apply that.


We have a sliding scale of complexity in what a project needs from it’s build 
system that generally breaks down into a few major classes of projects:

1) Projects that are pure python, single source, whose only real “compilation” 
is shuffling files to the correct location.
2) Projects that are pure python, but require some sort of generation step as 
part of the build process (2to3, etc).
3) Projects that have some basic C code that needs to be compiled, but which 
doesn’t link against anything special besides Python itself.
4) Projects that have some basic C code that needs to be compiled, but which 
links against other libraries like OpenSSL, libpq, etc.
5) Projects that have some basic C code, that needs to be compiled, that links 
against other libraries, and needs to be able to conditionally link against 
different libraries based on the capabilities and what is available in the 
system.
6) Extremely complex projects that need to link against many different 
libraries, possibly hard to build, possibly not C (e.g., Numpy with it’s blas 
libraries, Fortran, etc).

The problem with a one size fits all solution is that it’s very difficult to 
actually cover all of these cases in a way that is not horrible to actually use 
for each particular case. For an example, there is currently a build system 
called flit which doesn’t support anything but building straight to wheels 
(because we don’t have any sort of sdist 2.0 or anything yet). It doesn’t 
attempt to solve anything but the first class of users up there, and because of 
that it is able to create a very simple and easy to use packaging experience 
for authors, you just add a __version__ = “the version” to the top level of 
your package, and then drop in a flit.ini that looks something like:

[metadata]
module=foobar
author=Sir Robin
author-email=ro...@camelot.uk
home-page=http://github.com/sirrobin/foobar

# If you want command line scripts, this is how to declare them.
# If not, you can leave this section out completely.
[scripts]
# foobar:main means the script will do: from foobar import main; main()
foobar=foobar:main

And that’s it. You’re done. From there flit can do the rest of the work for you 
because it didn’t need to concern itself with trying to work on anything 
complex.

In addition to the above types of problems, you also have other things like 
what the “source of truth” is for your metadata. A common thing that people 
want is to be able to not have to duplicate their version in multiple locations 
(sometimes that even extends to the tags in the version control), however it’s 
not currently possible to do that in a particularly easy way. You have systems 
like pbr, setuptools_scm, versioner, etc that all do it but which all rely on 
some level of terribleness to deal with the sort of weird inverted control flow 
we have.

Once you get to all of the possible options for things people reasonably want 
to do, you quickly end up in a place where the only reasonable solution is the 
full complexity of a programming language like what we have in setup.py. 
However, that has it’s own problems which we’ve discovered over two decades of 
doing that :) It tends to end up with people doing badly tested adhoc code that 
they cargo cult from project to project and when there’s a problem there ends 
up being very little understanding why some code did that since it had been 
copy/pasted around so much. Pip has to go to a lot of effort to try and work 
around common mistakes people make in their setup.py files (like, depending on 
it being invoked with the project root as CURDIR) which most people will just 
never notice their slightly broken setup.py.

So basically, by allowing multiple build systems we will enable a world where 
your bog standard pure python project can get an extremely simple tool/ux and 
projects with crazy hard requirements like Numpy can get something more 
complex, without the two groups of users fighting each other over simplicity + 
limited vs complexity + powerful.

In the end, I think it’s likely we’ll end up with 2-3 really popular build 
tools, one that is for the complex projects, one that is for the simpler 
projects with some basic C needs, and *maybe* one that is for pure python (but 
that may be able to be handled by the basic C needs one) though there will be a 
long tail I’m sure.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-21 Thread Paul Tagliamonte
Hey Donald!

As far as using pip to do stuff system-wide, I wrote thoughts on
http://notes.pault.ag/debian-python

As for the rest of it, distutils is actually concretely shitty, and
replacing it with setuptools
sounds sane. Sounds like a solid idea.

Perhaps we can add a Lintian warning for using distutils; it's really nasty
cruft,
and it's going to bite the archive sooner rather than later.

Cheers,
  Paul

On Thu, Jan 21, 2016 at 9:19 AM, Donald Stufft  wrote:

> Hello!
>
> I'd like to suggest a change to the Debian Policy around Python packages
> that
> will help enable the world of Python packaging to continue to progress
> forward.
>
> First, a little bit of background:
>
> At the Python level there are three metadata formats for Python packaging:
>
> * The original, setuptools style .egg-info directories.
> * The distutils style .egg-info *file* added to distutils at some point.
> * The new and improved, wheel based .dist-info directories.
>
> The presence of any of these files will signal to Python tools that a
> particular distribution has been installed, however there are two fairly
> major
> and important differences between the distutils style, and the other two.
>
> 1. The distutils style has no provisions to record what files on the system
>belong to the installed distribution, making it appear to Python tooling
>that there *are* no files other than the metadata file itself.
>
> 2. The distutils style has no provisions to include additional metadata
> files
>in the metadata, making it impossible to extend the python level
> metadata
>with additional files.
>
> I have a series of improvements that I'd like to make to the packaging
> toolchain that will sort of build on one another, but which is not going to
> function correctly with the distutils style metadata and I'm hoping that I
> can
> convince y'all to make it policy to default to generating one of the other
> two
> kinds (with varying methods, more on that later).
>
> Concretely the thing that this is blocking right now, is that with the
> newly
> released pip 8.0 I tried to make it so that pip will refuse to uninstall a
> project that is installed with distutils style metadata. This is because
> we do
> not have any way to associate the actual .py (and others) files on disk
> with
> the installed metadata, so all we have ever done is just simply remove the
> metadata file, making it appear as if the item is uninstalled but leaving
> behind all of the actual files. However I'm going to be reverting this in a
> pip 8.0.1 release because it caused a decent amount of breakage amongst
> pip's
> users, almost all of them people who are attempting to upgrade OS provided
> packages using pip.
>
> Now, I know that upgrading OS provided packages using pip is less than
> optimal
> and I would greatly prefer that people did not do it (and I'm generally in
> agreement) however if we don't enable people to do it, they'll just
> continue to
> use an old version of pip and file bugs. It's a non starter for pip to
> make it
> impossible to do.
>
> In addition to the uninstall bit, it also means that things like pip show
> -f
> return junk information for packages installed in this way.
>
> Beyond just (eventually) enabling pip to disable uninstallations of
> distutils
> based installs this will start to allow some other future changes that I
> think
> will be more interesting to Debian. The uninstallation of distutils based
> installs comes hand in hand with pip stomping all over already existing
> files
> willy nilly because the way upgrading a project like that works is pip
> uninstalls the metadata file that says X is installed, then it just
> overwrites
> over any of the files that happen to be in it's way when it installs the
> newer
> version. If we can remove the need for pip to gleefully overwrite files to
> support these types of installed packages, then we can make it so pip will
> hard fail if it attempts to overwrite an already existing file on disk.
>
> An additional benefit here is that by switching to using the directory
> based
> options, we can add additional metadata files to the installed projects,
> much
> like the INSTALLER file from PEP376 (IIRC). This file will likely be the
> path
> to having pip refuse to touch OS owned files all together without some
> sort of
> --force flag to override the safety switch.
>
> As far as compatibility goes, pip has always forced everything to be
> installed
> using setuptools and as far as I am aware, there's no real fallout from
> doing
> so. I think in 2016 it's pretty reasonable to assume that a Python project
> is
> capable of being installed using setuptools instead of distutils.
>
> So without getting into the actual *method* of doing this (of which there
> are
> several different options with different trade offs) does this sound like
> something at all that Debian would be interested in?
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 

Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-21 Thread Scott Kitterman
On Thursday, January 21, 2016 10:00:53 AM Donald Stufft wrote:
> > On Jan 21, 2016, at 9:32 AM, Paul Tagliamonte  wrote:
> > 
> > Hey Donald!
> > 
> > As far as using pip to do stuff system-wide, I wrote thoughts on
> > http://notes.pault.ag/debian-python 
> I just want to be clear, that I totally agree with this. The reason I want
> to ensure we can enable this isn't because I think doing so is a great idea
> and everyone should be doing it. People *are* doing it and we (pip) have to
> be careful about what we make impossible to do because a possible result of
> going too break happy is people just never upgrade and things start to
> stagnate again.
> 
> So for pip at the system level, my goal is try and push people away from it
> as much as we can without flat out making it impossible *and* try to make
> it as "safe" to do so as possible when people decide to do it anyways.

On Debian, if you try to remove an essential package (which would be really 
bad as a rule), the system won't stop you, but you get this:

You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
 ?]

And you have to put in the entire phrase.

If pip did something like that for system level installs, I doubt most of us 
would object to it anymore.

Scott K

signature.asc
Description: This is a digitally signed message part.


Re: Amend Debian Python Proposal to Include More Python Metadata?

2016-01-21 Thread Donald Stufft

> On Jan 21, 2016, at 9:32 AM, Paul Tagliamonte  wrote:
> 
> Hey Donald!
> 
> As far as using pip to do stuff system-wide, I wrote thoughts on 
> http://notes.pault.ag/debian-python 

I just want to be clear, that I totally agree with this. The reason I want to
ensure we can enable this isn't because I think doing so is a great idea and
everyone should be doing it. People *are* doing it and we (pip) have to be
careful about what we make impossible to do because a possible result of going
too break happy is people just never upgrade and things start to stagnate
again.

So for pip at the system level, my goal is try and push people away from it as
much as we can without flat out making it impossible *and* try to make it as
"safe" to do so as possible when people decide to do it anyways.

> 
> As for the rest of it, distutils is actually concretely shitty, and replacing 
> it with setuptools
> sounds sane. Sounds like a solid idea.
> 
> Perhaps we can add a Lintian warning for using distutils; it's really nasty 
> cruft,
> and it's going to bite the archive sooner rather than later.
> 

As far as options for how to implement this goes, I'm not sure what best fits
with Debian so I'll leave that up to y'all, but here are a few options that I
can think of.

A lintian check is probably a pretty good thing, I would guess that it would
specifically check for the existence of a .egg-info or .dist-info directory
(or check to the non-existence of a .egg-info file).

In pip the way we make this work is with a bit of a gross little thunk that
ensures setuptools is imported prior to setup.py being invoked which forces all
setup.py calls to use setuptools instead of distutils even if they import from
distutils.

Of course, many setup.py already unconditionally use setuptools, and many more
conditionally use it (typically falling back to distutils) so for many projects
it may be enough to just add python-setuptools to build-depends. For projects
that unconditionally use distutils, either the pip way or patching setup.py
might be a reasonable way forward.

A more controversial way that comes with possibly some extra benefits (which
Debian may not care about) is to use ``pip`` itself as the build tool rather
than directly invoking setup.py. In this pip would be responsible for mucking
around with the distutils/setuptools mess instead of that needing to be handled
by Debian specific tooling. You could even leverage this to remove some of the
runtime dependencies on python-pkg-resources and speed up some python using
software if you let pip build a wheel as part of the buld process [1][2][3]. I
suspect pip would need to grow some additional options to make it suitable for
you in this role, but I would be more than willing to help if this is desired
(or not, this proposal isn't about trying to ram pip down anyones throat!).


[1] To be clear, this isn't about adding wheels for everything to the archive,
it would only be using Wheel as an intermediate step in the build process.
[2] Console scripts generated by setuptools entry wrappers when installed by
setuptools have a dependency on pkg_resources at runtime, however those
same scripts, when installed by pip from a wheel, do not. This would enable
removing the runtime dependency on pkg_resources for anything that only has
it for console scripts.
[3] Import pkg_resources is not the fasest thing in the world, it has to scan
every directory on sys.path looking for things to activate and it comes
with a bunch of side effects. This happens implicitly for any project using
console scripts.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail