Re: [Distutils] Entry points: specifying and caching

[Nick Coghlan]

On 20 October 2017 at 21:15, Donald Stufft  wrote:

> Tell you what, I’ll drop everything today and write up a PEP that adds
> metadata for console scripts to the packaging metadata where it belongs,
>

Donald, you're making the same mistake I did with PEP 426: interoperability
specifications are useless without a commitment from tooling developers to
actually provide implementations that know how to read and write them. And
since any new format you come up with won't be supported by existing pip
and pkg_resources installations, there won't be any incentive for
publishers to start using it, which means there's no incentives for runtime
libraries to learn how to read it, etc, etc.

In this case, we already have a perfectly serviceable format
(entry_points.txt), a reference publisher (setuptools.setup) and a
reference consumer (pkg_resources). The fact that the reference consumer is
pkg_resources rather than pip doesn't suddenly take this outside the domain
of responsibility of distutils-sig as a whole - it only takes it outside
the domain of responsibility of PyPI.

So if you want to say it is neither pip's nor PyPI's responsibility to say
anything one way or the other about the entry points format (beyond whether
or not they're used to declare console scripts in a way that pip
understands), then I agree with you entirely. This spec isn't something you
personally need to worry about, since it doesn't impact any of the tools
you work on (aside from giving pip's existing console_scripts
implementation a firmer foundation from an interoperability perpsective).

So the core of our disagreement is whether or not interfaces involving pip
and PyPI represent the limits of distutil-sig's responsibility. They don't,
and that's reflected in the fact we have a split standing delegation from
Guido (one initially to Richard Jones and later to you for changes that
affect PyPI, and one to me for packaging ecosystem interoperability
specifications in general)


> so we can move console_scripts entry point to a legacy footnote as far as
> packaging systems go. Then we can discuss whether an arbitrary plugin
> system is actually a packaging related spec (it’s not) on it’s own merits.
>

Instructing publishing system developers on how to publish pkg_resources
compatible entry points is indeed a Python packaging ecosystem level
concern.

Whether that capability survives into a hypothetical future metadata
specification (whether that's PEP 426+459 or something else entirely) would
then be a different question, but it isn't one we need to worry about right
now (since it purely affects internal interoperability file formats that
only automated scripts and folks maintaining those scripts need to care
about, and we'd expect entry_points.txt and PKG-INFO to coexist alongside
any new format for a *long* time).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Nick Coghlan]

On 20 October 2017 at 21:57, Thomas Kluyver  wrote:

> On Fri, Oct 20, 2017, at 12:50 PM, Donald Stufft wrote:
> > * We stifle innovation (hell just including it in setutools at all does
> > this, but we can’t unopen that can of worms).
>
> I don't think that's true to any significant extent. Having a standard
> does not stop people coming up with something better.
>

entry_points.txt will be hard to change for similar reasons to why PKG-INFO
is hard to change, but that challenge exists regardless of whether we
consider it a setuptools/pkg_resources feature or an ecosystem level
standard, since it relates to coupling between metadata publishers and
consumers of that metadata.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Nick Coghlan]

On 20 October 2017 at 22:10, Donald Stufft  wrote:

> If I could guess, I’d say it hasn’t changed in years because setuptools
> has had bigger things to work on and not enough time to do it in.


Then you'd be wrong - it hasn't changed in years because it's a sensible,
simple solution to the problem of declaring integration points between
independently distributed pieces of software that allows the installed
integration points to be listed *without* importing the software providing
them (unlike most import based plugin systems).

And yes, I know you're attempting to claim that "declaring integration
points between independently distributed pieces of software" isn't
something that's a packaging ecosystem level to concern.

It is an ecosystem level concern, but we haven't had to worry about it
previously, because entry points haven't had problems to be fixed the way
that other aspects of setuptools have (lack of uninstall support in
easy_install, lack of filesystem layout metadata in eggs, ordering quirks
in the versioning scheme). For entry points, by contrast, the only missing
piece is explicit documentation of the file format used in distribution
archives and the installation database.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Thomas Kluyver]

On Fri, Oct 20, 2017, at 01:18 PM, Donald Stufft wrote:
> I guess we shouldn’t have done PEP 517 or PEP 518 because, by your
> logic here, since it won’t be supported by existing tooling, there
> won’t be any incentive for people to use it ever.
I see this as having a similar purpose to those PEPs: reducing
dependence on setuptools. The difference is that for building packages,
pip explicitly uses setuptools, so the practical way forward was to
define an alternative to achieve the same ends. For this, the existing
mechanism does not directly rely on setuptools, so it's sufficient to
document it so that other tools can confidently produce and consume it.
I also get annoyed at times by arguments that it's not worth improving
something because it will be a long time before the change is useful.
But I don't think that's what Nick is saying here.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Donald Stufft]

> On Oct 20, 2017, at 8:23 AM, Nick Coghlan  wrote:
> 
> On 20 October 2017 at 22:10, Donald Stufft  > wrote:
> If I could guess, I’d say it hasn’t changed in years because setuptools has 
> had bigger things to work on and not enough time to do it in.
> 
> Then you'd be wrong - it hasn't changed in years because it's a sensible, 
> simple solution to the problem of declaring integration points between 
> independently distributed pieces of software that allows the installed 
> integration points to be listed *without* importing the software providing 
> them (unlike most import based plugin systems).

I mean no I’m not.

Entry points have a lot of problems and I know of multiple systems that have 
either moved away from them, had to hack around how bad they are, have refused 
to implement them because of previous pain felt by them, are looking for ways 
to eliminate them, or which just regret ever supporting them.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Donald Stufft]

> On Oct 20, 2017, at 8:34 AM, Nick Coghlan  wrote:
> 
> You're acting like you believe you have veto power over this topic. You don't 
> - it's not a PyPI related concern, and it doesn't require any changes to pip 
> or warehouse.
> 
> I'd certainly be *happier* if you were only -0 rather than -1, but your 
> disapproval won't prevent me from accepting Thomas's PR either way.


I’m acting like I have an opinion. You’re obviously free to accept something 
that I think is a bad idea, that doesn’t mean I should just shut up and not 
voice my concerns or objections and I’d appreciate it if you didn’t imply that 
I should.___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Wes Turner]

On Friday, October 20, 2017, Thomas Kluyver  wrote:

> On Fri, Oct 20, 2017, at 01:36 PM, Donald Stufft wrote:
>
> Entry points have a lot of problems and I know of multiple systems that
> have either moved away from them, had to hack around how bad they are, have
> refused to implement them because of previous pain felt by them, are
> looking for ways to eliminate them, or which just regret ever supporting
> them.
>
>
> The fate of the PR notwithstanding, I'd be interested in hearing more
> about what problems projects have experienced with entry points, if you
> have time to describe some examples. We're looking at using them in more
> places than we already do, so it would be useful to hear about drawbacks we
> might not have thought about, and about what other options projects have
> moved to.
>
> Thomas
>

 What were the issues with setuptools entry points here (in 2014, when you
two were opposed to adding them to sendibly list ipython extensions)?

https://github.com/ipython/ipython/pull/4673

https://github.com/ipython/ipython/compare/master...westurner:setuptools_entry_points
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Thomas Kluyver]

On Fri, Oct 20, 2017, at 01:58 PM, Wes Turner wrote:
> What were the issues with setuptools entry points here (in 2014, when
> you two were opposed to adding them to sendibly list ipython
> extensions)?
I'm impressed by your memory! The main issue then was that it implied
that extension authors would have to use setuptools.
Setuptools has got much better since then, we have better tools and
norms for dealing with its rough edges, and there are usable alternative
tools that can be used to distribute entrypoints. But the description
I've written up is still basically trying to solve the same problem: an
application should be able to use entry points without forcing all
plugins to use setuptools.
Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Donald Stufft]

> On Oct 20, 2017, at 7:57 AM, Thomas Kluyver  wrote:
> 
> On Fri, Oct 20, 2017, at 12:50 PM, Donald Stufft wrote:
>> * Since it is a packaging standard, then it is expected that all
>> packaging tools will be updated to work with it.
> 
> Where packaging tools need to know about it,  they already have to.
> Where they don't, writing a standard doesn't imply that every tool has
> to implement it. Documenting it doesn't change either case, it just
> makes life easier for tools that do need to use it.

Packaging tools shouldn’t be expected to know anything about it other than the 
console_scripts feature (which shouldn’t exist as an entry point, but currently 
does for historical reasons).

Publishing tools should have a way for additional files that the publishing 
tool wasn’t even aware might exist someday to get added to the metadata 
directory, installation tools should preserve those files when installing them. 
With those two generic features, then entry points (and other things!) can be 
written on top of the ecosystem *without* needing to standardize on one 
solution for one particular non-packaging problem.

If a publishing tool doesn’t want to provide that mechanism, then that is fine, 
but that limits their audience (in the same way that not building C extensions 
limits their audience, people who need that capability won’t be able to use 
them).

> 
>> * We’re explicitly saying that this is the one true way of solving this
>> problem in the Python ecosystem.
> 
> I don't buy that at all. We're saying that it exists, and this is what
> it is.

It’s literally the way all of our packaging standards are written. Don’t use 
eggs, wheels are the one true way, don’t use YOLO versions, PEP 440 is the one 
true way, don’t add arbitrary extensions to the simple repo format, PEP 503 API 
Is the one true way, etc etc etc.

> 
>> * We stifle innovation (hell just including it in setutools at all does
>> this, but we can’t unopen that can of worms).
> 
> I don't think that's true to any significant extent. Having a standard
> does not stop people coming up with something better.

It doesn’t actively prevent someone from coming up with something better no, 
but what it does do is add a pretty huge barrier to entry for someone who 
wanted to come up with something better. It’s the same way that something being 
added to the stdlib stifles competition. When something is “the standard”, it 
discourages people from even trying to make something better— or if they do 
make other people from trying it, unless “the standard” is really bad.

> 
>> * We make it actively harder to improve the feature (since once it’s part
>> of the purview of packaging standards, all of distutils-sig gets to weigh
>> in on improvements).
> 
> It hasn't changed in years, as far as I know, and it's so widely used
> that any change is likely to break a load of stuff anyway. As we've
> already discussed for caching, we can improve by building *on top* of it
> relatively easily. And ultimately I think that bringing it out into
> daylight leads to a healthier future than leaving it under the stone
> marked 'setuptools''.
> 


If I could guess, I’d say it hasn’t changed in years because setuptools has had 
bigger things to work on and not enough time to do it in.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Donald Stufft]

> On Oct 20, 2017, at 8:06 AM, Nick Coghlan  wrote:
> 
> On 20 October 2017 at 21:15, Donald Stufft  > wrote:
> Tell you what, I’ll drop everything today and write up a PEP that adds 
> metadata for console scripts to the packaging metadata where it belongs,
> 
> Donald, you're making the same mistake I did with PEP 426: interoperability 
> specifications are useless without a commitment from tooling developers to 
> actually provide implementations that know how to read and write them. And 
> since any new format you come up with won't be supported by existing pip and 
> pkg_resources installations, there won't be any incentive for publishers to 
> start using it, which means there's no incentives for runtime libraries to 
> learn how to read it, etc, etc.

Not particularly no.

I can promise you 100% that pip will support it in the next version once I 
write it. I can also promise you that setuptools will have a PR to support it 
(not pkg_resources, because console scripts are a install time feature not a 
runtime feature), and I assume Jason would be happy to merge it.

So there’s commitment from at least one tool.

The “existing installations” is horse shit, because existing implementations 
won’t support *any* new feature of anything so it can literally be used as a 
justification for doing nothing about anything except standardizing what 
already exists. I guess we shouldn’t have done PEP 517 or PEP 518 because, by 
your logic here, since it won’t be supported by existing tooling, there won’t 
be any incentive for people to use it ever.

> 
> In this case, we already have a perfectly serviceable format 
> (entry_points.txt), a reference publisher (setuptools.setup) and a reference 
> consumer (pkg_resources). The fact that the reference consumer is 
> pkg_resources rather than pip doesn't suddenly take this outside the domain 
> of responsibility of distutils-sig as a whole - it only takes it outside the 
> domain of responsibility of PyPI.
> 
> So if you want to say it is neither pip's nor PyPI's responsibility to say 
> anything one way or the other about the entry points format (beyond whether 
> or not they're used to declare console scripts in a way that pip 
> understands), then I agree with you entirely. This spec isn't something you 
> personally need to worry about, since it doesn't impact any of the tools you 
> work on (aside from giving pip's existing console_scripts implementation a 
> firmer foundation from an interoperability perpsective).

My objection has absolutely nothing to do with whether pip is the consumer or 
not. My objection is entirely based on the fact that a plugin system is no .a 
packaging related feature and it doesn’t become one because a packaging tool 
once added a plugin system.

> 
> So the core of our disagreement is whether or not interfaces involving pip 
> and PyPI represent the limits of distutil-sig's responsibility. They don't, 
> and that's reflected in the fact we have a split standing delegation from 
> Guido (one initially to Richard Jones and later to you for changes that 
> affect PyPI, and one to me for packaging ecosystem interoperability 
> specifications in general)

No that’s not the core of our disagreement. The core of our disagreement is 
whether random runtime features suddenly become a packaging concern because 
they were implemented by one packaging tool once.

>  
> so we can move console_scripts entry point to a legacy footnote as far as 
> packaging systems go. Then we can discuss whether an arbitrary plugin system 
> is actually a packaging related spec (it’s not) on it’s own merits.
> 
> Instructing publishing system developers on how to publish pkg_resources 
> compatible entry points is indeed a Python packaging ecosystem level concern. 

No it’s really not.

> 
> Whether that capability survives into a hypothetical future metadata 
> specification (whether that's PEP 426+459 or something else entirely) would 
> then be a different question, but it isn't one we need to worry about right 
> now (since it purely affects internal interoperability file formats that only 
> automated scripts and folks maintaining those scripts need to care about, and 
> we'd expect entry_points.txt and PKG-INFO to coexist alongside any new format 
> for a *long* time).
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com    |   
> Brisbane, Australia

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Nick Coghlan]

On 20 October 2017 at 22:18, Donald Stufft  wrote:

>
> The “existing installations” is horse shit, because existing
> implementations won’t support *any* new feature of anything so it can
> literally be used as a justification for doing nothing about anything
> except standardizing what already exists. I guess we shouldn’t have done
> PEP 517 or PEP 518 because, by your logic here, since it won’t be supported
> by existing tooling, there won’t be any incentive for people to use it ever.
>

No, because PEP 517 and 518 actually change the UX for *publishers* away
from setup.py to pyproject.toml + whatever build system they choose, while
allowing the definition of a *common* setup.py shim for compatibility with
older clients.

By contrast, it's relatively rare for people to edit entry_points.txt by
hand - it's typically a generated file, just like PKG-INFO.

For any *new* console_scripts replacement, you're also going to have define
how to translate it back to entry_points.txt for compatibility with older
pip installations, and that means you're also going to have to define how
to do that without conflicting with any other pkg_resources entry points
already declared by a package.

Those two characteristics mean that entry_points.txt has a lot more in
common with PKG-INFO than it does with setup.py, and that similarity is
further enhanced by the fact that it's a pretty easy format to document.

> So if you want to say it is neither pip's nor PyPI's responsibility to say
> anything one way or the other about the entry points format (beyond whether
> or not they're used to declare console scripts in a way that pip
> understands), then I agree with you entirely. This spec isn't something you
> personally need to worry about, since it doesn't impact any of the tools
> you work on (aside from giving pip's existing console_scripts
> implementation a firmer foundation from an interoperability perpsective).
>
>
> My objection has absolutely nothing to do with whether pip is the consumer
> or not. My objection is entirely based on the fact that a plugin system is
> no .a packaging related feature and it doesn’t become one because a
> packaging tool once added a plugin system.
>

You're acting like you believe you have veto power over this topic. You
don't - it's not a PyPI related concern, and it doesn't require any changes
to pip or warehouse.

I'd certainly be *happier* if you were only -0 rather than -1, but your
disapproval won't prevent me from accepting Thomas's PR either way.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Thomas Kluyver]

On Fri, Oct 20, 2017, at 01:36 PM, Donald Stufft wrote:
> Entry points have a lot of problems and I know of multiple systems
> that have either moved away from them, had to hack around how bad they
> are, have refused to implement them because of previous pain felt by
> them, are looking for ways to eliminate them, or which just regret
> ever supporting them.
The fate of the PR notwithstanding, I'd be interested in hearing more
about what problems projects have experienced with entry points, if you
have time to describe some examples. We're looking at using them in more
places than we already do, so it would be useful to hear about drawbacks
we might not have thought about, and about what other options projects
have moved to.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Daniel Holth]

I like the idea of lifecycle hooks but I worry about the malware problem;
would there need to be a blacklist / whitelist / disable system?
(ignore-scripts=true is now a recommended part of anyone's npm
configuration) That is why we have avoided any kind of (package specific)
hooks to wheel. However hooks would be a very elegant way to avoid worrying
about core pip functionality since it wouldn't be core functionality.

On Fri, Oct 20, 2017 at 4:41 PM Nathaniel Smith  wrote:

> On Oct 19, 2017 11:10, "Donald Stufft"  wrote:
>
>
> EXCEPT, for the fact that with the desire to cache things, it would be
> beneficial to “hook” into the lifecycle of a package install. However I
> know that there are other plugin systems out there that would like to also
> be able to do that (Twisted Plugins come to mind) and that I think outside
> of plugin systems, such a mechanism is likely to be useful in general for
> other cases.
>
> So heres a different idea that is a bit more ambitious but that I think is
> a better overall idea. Let entrypoints be a setuptools thing, and lets
> define some key lifecycle hooks during the installation of a package and
> some mechanism in the metadata to let other tools subscribe to those hooks.
> Then  a caching layer could be written for setuptools entrypoints to make
> that faster without requiring standardization, but also a whole new, better
> plugin system could to, Twisted plugins could benefit, etc [1].
>
>
> In this hypothetical system, how do installers like pip find the list of
> hooks to call? By looking up an entrypoint? (Sorry if this was discussed
> downthread; I didn't see it but I admit I only skimmed.)
>
> -n
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Doug Hellmann]

Excerpts from Nathaniel Smith's message of 2017-10-20 13:41:03 -0700:
> On Oct 19, 2017 11:10, "Donald Stufft"  wrote:
> 
> 
> EXCEPT, for the fact that with the desire to cache things, it would be
> beneficial to “hook” into the lifecycle of a package install. However I
> know that there are other plugin systems out there that would like to also
> be able to do that (Twisted Plugins come to mind) and that I think outside
> of plugin systems, such a mechanism is likely to be useful in general for
> other cases.
> 
> So heres a different idea that is a bit more ambitious but that I think is
> a better overall idea. Let entrypoints be a setuptools thing, and lets
> define some key lifecycle hooks during the installation of a package and
> some mechanism in the metadata to let other tools subscribe to those hooks.
> Then  a caching layer could be written for setuptools entrypoints to make
> that faster without requiring standardization, but also a whole new, better
> plugin system could to, Twisted plugins could benefit, etc [1].

Having post-install and pre-uninstall hooks should be sufficient for
updating a cache, assuming the hook could be given enough information
about the thing being manipulated to probe for whatever data it
needs.

> In this hypothetical system, how do installers like pip find the list of
> hooks to call? By looking up an entrypoint? (Sorry if this was discussed
> downthread; I didn't see it but I admit I only skimmed.)

That's how I would expect it to work. Using setuptools most likely?
That would mean that other plugin systems would have to provide one
setuptools plugin to hook into the installer to build a lookup
cache, but the actual plugins wouldn't have to use setuptools for
anything.

Doug
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[xoviat]

The short answer is that someone complained about the pip API not doing
what they wanted it to do, so we broke the API to make it clear to everyone
that the API is not something that should be depended on.

2017-10-20 17:53 GMT-05:00 Richard Jones :

> Hiya Paul,
>
> There's a bunch of tooling out there using pip's internals to extending
> pip's functionality. Could you please provide a some reasoning as to why
> they're all going to be broken at pip 10, and possibly some guidance on how
> to get that functionality back?
>
>
> Cheers,
>
>  Richard
>
>
> On 21 October 2017 at 00:22, Paul Moore  wrote:
>
>> We're in the process of starting to plan for a release of pip (the
>> long-awaited pip 10). We're likely still a month or two away from a
>> release, but now is the time for people to start ensuring that
>> everything works for them. One key change in the new version will be
>> that all of the internal APIs of pip will no longer be available, so
>> any code that currently calls functions in the "pip" namespace will
>> break. Calling pip's internal APIs has never been supported, and
>> always carried a risk of such breakage, so projects doing so should,
>> in theory, be prepared for such things. However, reality is not always
>> that simple, and we are aware that people will need time to deal with
>> the implications.
>>
>> Just in case it's not clear, simply finding where the internal APIs
>> have moved to and calling them under the new names is *not* what
>> people should do. We can't stop people calling the internal APIs,
>> obviously, but the idea of this change is to give people the incentive
>> to find a supported approach, not just to annoy people who are doing
>> things we don't want them to ;-)
>>
>> So please - if you're calling pip's internals in your code, take the
>> opportunity *now* to check out the in-development version of pip, and
>> ensure your project will still work when pip 10 is released.
>>
>> And many thanks to anyone else who helps by testing out the new
>> version, as well :-)
>>
>> Thanks,
>> Paul
>>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Wes Turner]

On Friday, October 20, 2017, Doug Hellmann  wrote:

> Excerpts from Wes Turner's message of 2017-10-20 10:41:02 -0400:
> > On Friday, October 20, 2017, Donald Stufft  > wrote:
> >
> > >
> > >
> > > On Oct 20, 2017, at 9:35 AM, Nick Coghlan  
> > > ');>>
> wrote:
> > >
> > > On 20 October 2017 at 23:19, Donald Stufft  
> > > ');>>
> wrote:
> > >
> > >> One that I was helping someone debug just the other day is that
> they’re
> > >> super non-debuggable and the behavior when you have two things
> providing
> > >> the same entry point is basically  (If I remember correctly, the
> > >> behavior is that the first thing found is the one that “wins”, which
> means
> > >> the ordering of sys.path and the names of the projects supply it is
> what
> > >> determines it). This got exposed to the end user that they installed
> > >> something that they thought was going to add support for something,
> but
> > >> which silently did nothing because two different project happened to
> pick
> > >> the same name for their entry point (not the group, it was two things
> > >> providing plugins for the same system).
> > >>
> > >
> > > While I agree with this, I think that's a combination of pkg_resources
> > > itself being hard to debug in general, and the fact that pkg_resources
> > > doesn't clearly define the semantics of how it resolves name conflicts
> > > within an entry point group - as far as I know, it's largely an
> accident of
> > > implementation.
> > >
> > > The interoperability spec is going to state that conflict resolution
> when
> > > the same name within a group is declared by multiple packages is the
> > > responsibility of the group consumer, so documenting the format should
> > > actually improve this situation, since it allows for the development of
> > > competing conflict resolution strategies in different runtime
> libraries.
> > >
> > >
> > > I think it makes it *worse*, because now the behavior isn’t just a
> > > entrypoints weirdness, but now it changes based on which runtime
> library
> > > you use (which isn’t something that end users are likely to have much
> > > insight into) and it represents a footgun that package authors are
> unlikely
> > > to be aware of. If mycoolentrypointslib comes out that is faster, but
> > > changes some subtle behavior like this it’ll break people, but that is
> > > unlikely going to be an effect that people expect to happen just
> because
> > > they switched between two things both implementing the same standard.
> > >
> > > So effectively this means that not only is the fact you’re using
> > > entrypoints part of your API, but now which entry point library you’re
> > > using at runtime is now also part of your API.
> > >
> >
> > When should the check for duplicate entry points occur?
> >
> > - At on_install() time (+1)
> > - At runtime
> >
> > Is a sys.path-like OrderedDict preemptive strategy preferable or just as
> > dangerous as importlib?
>
> Having "duplicate" entry points is not necessarily an error. It's
> a different usage pattern.  The semantics of dropping a named plugin
> into a namespace are defined by the application and plugin-point.
> Please do not build assumptions about uniqueness into the underlying
> implementation.


I think that, at least with console_scripts, we already assume uniqueness:
if there's another package which provides a 'pip' console_script, for
example, there's not yet an error message?

Would it be helpful to at least spec that iterated entrypoints are in
sys.path order? And then what about entrypoints coming from the same path
in sys.path: alphabetical? Whatever hash randomization does with it?

Whenever I feel unsure about my data model, I tend to sometimes read the
OWL spec: here, the OWL spec has owl:cardinality OR owl:minCardinality OR
owl:maxCardinality. Some entrypoints may have 0, only one, or n "instances"?

We should throw an error if a given console_script entrypoint has more than
one "instance" (exceeds maxCardinality xsd:string = 1).


>
> The stevedore library wraps up pkg_resources with several such
> patterns. For example, it supports "give me all of the plugins in
> a namespace" (find all the extensions to your app), "give me all
> of the plugins named $name in a namespace" (find the hooks for a
> specific event defined by the app), and "give me *the* plugin named
> $name in a namespace" (load a driver for talking to a backend).
>
> https://docs.openstack.org/stevedore/latest/reference/index.html


https://github.com/openstack/stevedore/blob/master/stevedore/extension.py

https://github.com/openstack/stevedore/blob/master/stevedore/tests/test_extension.py

These tests mention saving discovered entry points in a cache?
___

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Richard Jones]

Hiya Paul,

There's a bunch of tooling out there using pip's internals to extending
pip's functionality. Could you please provide a some reasoning as to why
they're all going to be broken at pip 10, and possibly some guidance on how
to get that functionality back?


Cheers,

 Richard


On 21 October 2017 at 00:22, Paul Moore  wrote:

> We're in the process of starting to plan for a release of pip (the
> long-awaited pip 10). We're likely still a month or two away from a
> release, but now is the time for people to start ensuring that
> everything works for them. One key change in the new version will be
> that all of the internal APIs of pip will no longer be available, so
> any code that currently calls functions in the "pip" namespace will
> break. Calling pip's internal APIs has never been supported, and
> always carried a risk of such breakage, so projects doing so should,
> in theory, be prepared for such things. However, reality is not always
> that simple, and we are aware that people will need time to deal with
> the implications.
>
> Just in case it's not clear, simply finding where the internal APIs
> have moved to and calling them under the new names is *not* what
> people should do. We can't stop people calling the internal APIs,
> obviously, but the idea of this change is to give people the incentive
> to find a supported approach, not just to annoy people who are doing
> things we don't want them to ;-)
>
> So please - if you're calling pip's internals in your code, take the
> opportunity *now* to check out the in-development version of pip, and
> ensure your project will still work when pip 10 is released.
>
> And many thanks to anyone else who helps by testing out the new
> version, as well :-)
>
> Thanks,
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Nathaniel Smith]

On Oct 19, 2017 11:10, "Donald Stufft"  wrote:


EXCEPT, for the fact that with the desire to cache things, it would be
beneficial to “hook” into the lifecycle of a package install. However I
know that there are other plugin systems out there that would like to also
be able to do that (Twisted Plugins come to mind) and that I think outside
of plugin systems, such a mechanism is likely to be useful in general for
other cases.

So heres a different idea that is a bit more ambitious but that I think is
a better overall idea. Let entrypoints be a setuptools thing, and lets
define some key lifecycle hooks during the installation of a package and
some mechanism in the metadata to let other tools subscribe to those hooks.
Then  a caching layer could be written for setuptools entrypoints to make
that faster without requiring standardization, but also a whole new, better
plugin system could to, Twisted plugins could benefit, etc [1].


In this hypothetical system, how do installers like pip find the list of
hooks to call? By looking up an entrypoint? (Sorry if this was discussed
downthread; I didn't see it but I admit I only skimmed.)

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Donald Stufft]

> On Oct 20, 2017, at 1:32 AM, Nick Coghlan  wrote:
> 
> If we want to enable pytest plugin authors to use other build systems like 
> flit, then those build systems need a defined interoperability format that's 
> compatible with what pytest is expecting to see (i.e. entry point definitions 
> that pkg_resources knows how to read).
> 

This is thinking about it wrong IMO.

We could just as easily say if we want tools like flit to be able to package 
Twisted plugins then those build systems need a defined interoperability format 
that is compatible with what Twisted and that ecosystem is expecting.The *ONLY* 
reason we should care at all about defining entry points as a packaging feature 
is console scripts, so we should limit our standardization to that. PBR has a 
runtime feature too where it inserts metadata into the .dist-info directory at 
build time and then a runtime API that reads that.. should we standardize that 
too?

I’m *not* saying that flit doesn’t nee to know how to generate entry points if 
a entry points using project wants to use flit, but what I am saying is that 
entry points isn’t a packaging specification. It’s a setuptools feature that 
should live within setuptools.___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Nick Coghlan]

On 20 October 2017 at 16:43, Thomas Kluyver  wrote:

> I would also be happy to add a section to the document describing the
> specific use of entry points for defining scripts to install.
>

Yeah, it would make sense to include that, as well as reserving the
"console_scripts" name on PyPI so we abide by our own "Only rely on a
category name if you or one of your dependencies controls it on PyPI"
guideline.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Donald Stufft]

> On Oct 20, 2017, at 7:02 AM, Nick Coghlan  wrote:
> 
>  That's the one point where the "de facto standard" status of setuptools is 
> relevant to the question of whether the entry_points.txt format is a PyPA 
> interoperability standard: it is, because providing a functionally equivalent 
> capability is required for publishers to be able to transparently switch from 
> setuptools to something else without their end users noticing the difference.


Nope. Because this isn’t a packaging feature. It’s a runtime feature of 
setuptools, and we do everyone a disservice by trying to move this into the 
purview of distutils-sig just because setuptools included a feature once. Just 
because setuptools included a feature does *NOT* make it a packaging related 
feature.

Tell you what, I’ll drop everything today and write up a PEP that adds metadata 
for console scripts to the packaging metadata where it belongs, so we can move 
console_scripts entry point to a legacy footnote as far as packaging systems 
go. Then we can discuss whether an arbitrary plugin system is actually a 
packaging related spec (it’s not) on it’s own merits.___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Donald Stufft]

> On Oct 20, 2017, at 1:32 AM, Nick Coghlan  wrote:
> 
> 3. Unlike setup.cfg & pyproject.toml, actual humans never touch it - it's 
> written and read solely by software

This is wrong BTW, humans can and do effectively write entry_points.txt, it’s a 
supported feature of setuptools to do:

setuptools.setup(
entry_points=“””
[my_cool_entrypoint]
athing = the.thing:bar
“””,
)


This is documented and I have run into a number of projects that do this.___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Nick Coghlan]

On 20 October 2017 at 20:48, Donald Stufft  wrote:

>
> On Oct 20, 2017, at 1:32 AM, Nick Coghlan  wrote:
>
> If we want to enable pytest plugin authors to use other build systems like
> flit, then those build systems need a defined interoperability format
> that's compatible with what pytest is expecting to see (i.e. entry point
> definitions that pkg_resources knows how to read).
>
>
> This is thinking about it wrong IMO.
>
> We could just as easily say if we want tools like flit to be able to
> package Twisted plugins then those build systems need a defined
> interoperability format that is compatible with what Twisted and that
> ecosystem is expecting.
>

Twisted already defines plugin discovery in an inherently
packaging-friendly way, since it's based on import names rather than
packaging metadata. Other plugin management systems like straight.plugins
are similar: they use Python's import system as their pub/sub channel to
advertise plugin availability, and accept the limitation that this means
all plugin APIs will be module level ones rather than being individual
classes or callables.


> The *ONLY* reason we should care at all about defining entry points as a
> packaging feature is console scripts, so we should limit our
> standardization to that. PBR has a runtime feature too where it inserts
> metadata into the .dist-info directory at build time and then a runtime API
> that reads that.. should we standardize that too?
>

No, because PBR isn't the defacto default build system that pip injects
into setup.py execution by default. That's the one point where the "de
facto standard" status of setuptools is relevant to the question of whether
the entry_points.txt format is a PyPA interoperability standard: it is,
because providing a functionally equivalent capability is required for
publishers to be able to transparently switch from setuptools to something
else without their end users noticing the difference.

Sure we *could* say "We don't want to standardise on that one, we want to
define a different one", but I think entry points are good enough for our
purposes, so inventing something different wouldn't be a good use of
anyone's time (see also: the perpetually deferred status of PEP 426).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Thomas Kluyver]

On Fri, Oct 20, 2017, at 12:15 PM, Donald Stufft wrote:
> Tell you what, I’ll drop everything today and write up a PEP...

Donald, why are you so determined that this spec should not be created?
Your time is enormously valuable, so why would you drop everything to
write a PEP which implies changes to tooling, simply so that we don't
document the status quo? Even if we do make that change, there are
thousands of existing packages using the existing de-facto standard, so
it would still be valuable to document it.
If it makes things easier, I'll host the spec on my own site and add a
'see also' from the specs page of the packaging user guide (because I
think people would expect it to be there, even if it's not the 'right'
place). But I don't think anyone else has expressed any objection to
putting the spec there.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Thomas Kluyver]

On Fri, Oct 20, 2017, at 12:50 PM, Donald Stufft wrote:
> * Since it is a packaging standard, then it is expected that all
> packaging tools will be updated to work with it.

Where packaging tools need to know about it,  they already have to.
Where they don't, writing a standard doesn't imply that every tool has
to implement it. Documenting it doesn't change either case, it just
makes life easier for tools that do need to use it.

> * We’re explicitly saying that this is the one true way of solving this
> problem in the Python ecosystem.

I don't buy that at all. We're saying that it exists, and this is what
it is.

> * We stifle innovation (hell just including it in setutools at all does
> this, but we can’t unopen that can of worms).

I don't think that's true to any significant extent. Having a standard
does not stop people coming up with something better.

> * We make it actively harder to improve the feature (since once it’s part
> of the purview of packaging standards, all of distutils-sig gets to weigh
> in on improvements).

It hasn't changed in years, as far as I know, and it's so widely used
that any change is likely to break a load of stuff anyway. As we've
already discussed for caching, we can improve by building *on top* of it
relatively easily. And ultimately I think that bringing it out into
daylight leads to a healthier future than leaving it under the stone
marked 'setuptools''.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Donald Stufft]

> On Oct 20, 2017, at 7:31 AM, Thomas Kluyver  wrote:
> 
> On Fri, Oct 20, 2017, at 12:15 PM, Donald Stufft wrote:
>> Tell you what, I’ll drop everything today and write up a PEP...
> 
> Donald, why are you so determined that this spec should not be created? Your 
> time is enormously valuable, so why would you drop everything to write a PEP 
> which implies changes to tooling, simply so that we don't document the status 
> quo? Even if we do make that change, there are thousands of existing packages 
> using the existing de-facto standard, so it would still be valuable to 
> document it.
> 
> If it makes things easier, I'll host the spec on my own site and add a 'see 
> also' from the specs page of the packaging user guide (because I think people 
> would expect it to be there, even if it's not the 'right' place). But I don't 
> think anyone else has expressed any objection to putting the spec there.
> 
> Thomas


I mean, it’s a PEP I was already planning on writing at some point, because 
I’ve *never* liked the fact that our console script support was reliant on a 
setuptools feature so all I’d be doing is re-prioritizing work I was already 
planning on doing. I’m also completely happy with documenting the status quo, 
which from a packaging stand point means documenting console_scripts— it 
doesn’t mean pulling in an entire setuptools feature. I’m not even against 
documenting the entire feature, *if* it’s done inside of setuptools where it 
belongs.

What I am against, is moving the entire entry points feature from a setuptools 
feature to a packaging standard. It is at best, tangental to packaging since 
outside of console_scripts it’s only real relation is that it uses features of 
the packaging ecosystem and happened to come from setuptools (but it could have 
just as easily been written externally to setuptools). Making it a packaging 
standard comes with several implications:

* Since it is a packaging standard, then it is expected that all packaging 
tools will be updated to work with it.
* We’re explicitly saying that this is the one true way of solving this problem 
in the Python ecosystem.
* We stifle innovation (hell just including it in setutools at all does this, 
but we can’t unopen that can of worms).
* We make it actively harder to improve the feature (since once it’s part of 
the purview of packaging standards, all of distutils-sig gets to weigh in on 
improvements).

I don’t get why anyone would want to saddle all of the extra implications and 
work that comes with being a packaging standard on a feature that isn’t one and 
doesn’t need to be one. We are at our best when our efforts are on generalized 
mechanisms that allow features such as entry points to be implemented on top of 
us, rather than trying to pull in every tangential feature under the sun into 
our domain.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Thomas Kluyver]

On Fri, Oct 20, 2017, at 05:42 AM, Nick Coghlan wrote:
> I'm wondering if rather than jumping straight to a PEP, it may make
> sense to instead initially pursue this idea as a *non-*standard,
> implementation dependent thing specific to the "entrypoints" project.
> There are a *lot* of challenges to be taken into account for a truly
> universal metadata caching design, and it would be easy to fall into
> the trap of coming up with a design so complex that nobody can
> realistically implement it.
I'd be happy to tackle it like that. Donald's proposed hooks for package
installation and uninstallation would provide all the necessary
interoperation between different tools. As and when it's working, the
cache format can be documented for other consumers to use.
> Right now, the only documented publishing API for that pub/sub channel
> is setuptools.setup(), and the only documented subscription API is
> pkg_resources. Documenting the file format explicitly changes that
> dynamic, such that any publisher that produces a compliant
> `entry_points.txt` file will be supported by pkg_resources, and any
> consumer that can read a compliant `entry_points.txt` file will be
> supported by setuptools.setup()
Yup, this is very much what I'd like :-)

Thanks,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Thomas Kluyver]

I would also be happy to add a section to the document describing the
specific use of entry points for defining scripts to install.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Donald Stufft]

On Oct 20, 2017, at 8:41 AM, Thomas Kluyver  wrote:
> 
> On Fri, Oct 20, 2017, at 01:36 PM, Donald Stufft wrote:
>> Entry points have a lot of problems and I know of multiple systems that have 
>> either moved away from them, had to hack around how bad they are, have 
>> refused to implement them because of previous pain felt by them, are looking 
>> for ways to eliminate them, or which just regret ever supporting them.
> 
> The fate of the PR notwithstanding, I'd be interested in hearing more about 
> what problems projects have experienced with entry points, if you have time 
> to describe some examples. We're looking at using them in more places than we 
> already do, so it would be useful to hear about drawbacks we might not have 
> thought about, and about what other options projects have moved to.
> 

One that I was helping someone debug just the other day is that they’re super 
non-debuggable and the behavior when you have two things providing the same 
entry point is basically  (If I remember correctly, the behavior is that 
the first thing found is the one that “wins”, which means the ordering of 
sys.path and the names of the projects supply it is what determines it). This 
got exposed to the end user that they installed something that they thought was 
going to add support for something, but which silently did nothing because two 
different project happened to pick the same name for their entry point (not the 
group, it was two things providing plugins for the same system).

Of course there is the perennial entrypoints are super slow, which is partially 
the fault of pkg_resources does a bunch of import time logic, but also because 
scanning sys.path for all installed stuff is just slow.

They’re also somewhat fragile since they rely on the packaging metadata system 
at runtime, and a number of tools exclude that information (often times things 
that deploy stuff as a tarball/zipfile) which causes regular issues to be 
opened up for these projects when they get used in those environments.

Those are the ones I remember because they come up regularly (and people 
regularly come to me with issues with any project related to packaging in any 
way even for non packaging related features in those projects). I’m pretty sure 
there were more of them that I’ve encountered and seen projects encounter, but 
I can’t remember them to be sure.

I’m more familiar with why console_scripts entry point is not great and why we 
should stop using it since I regularly try to re-read all of pip’s issues and a 
lot of it’s issues are documented there.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Paul Moore]

We're in the process of starting to plan for a release of pip (the
long-awaited pip 10). We're likely still a month or two away from a
release, but now is the time for people to start ensuring that
everything works for them. One key change in the new version will be
that all of the internal APIs of pip will no longer be available, so
any code that currently calls functions in the "pip" namespace will
break. Calling pip's internal APIs has never been supported, and
always carried a risk of such breakage, so projects doing so should,
in theory, be prepared for such things. However, reality is not always
that simple, and we are aware that people will need time to deal with
the implications.

Just in case it's not clear, simply finding where the internal APIs
have moved to and calling them under the new names is *not* what
people should do. We can't stop people calling the internal APIs,
obviously, but the idea of this change is to give people the incentive
to find a supported approach, not just to annoy people who are doing
things we don't want them to ;-)

So please - if you're calling pip's internals in your code, take the
opportunity *now* to check out the in-development version of pip, and
ensure your project will still work when pip 10 is released.

And many thanks to anyone else who helps by testing out the new
version, as well :-)

Thanks,
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Matthew Brett]

Hi,

On Fri, Oct 20, 2017 at 2:22 PM, Paul Moore  wrote:
> We're in the process of starting to plan for a release of pip (the
> long-awaited pip 10). We're likely still a month or two away from a
> release, but now is the time for people to start ensuring that
> everything works for them. One key change in the new version will be
> that all of the internal APIs of pip will no longer be available, so
> any code that currently calls functions in the "pip" namespace will
> break. Calling pip's internal APIs has never been supported, and
> always carried a risk of such breakage, so projects doing so should,
> in theory, be prepared for such things. However, reality is not always
> that simple, and we are aware that people will need time to deal with
> the implications.
>
> Just in case it's not clear, simply finding where the internal APIs
> have moved to and calling them under the new names is *not* what
> people should do. We can't stop people calling the internal APIs,
> obviously, but the idea of this change is to give people the incentive
> to find a supported approach, not just to annoy people who are doing
> things we don't want them to ;-)
>
> So please - if you're calling pip's internals in your code, take the
> opportunity *now* to check out the in-development version of pip, and
> ensure your project will still work when pip 10 is released.
>
> And many thanks to anyone else who helps by testing out the new
> version, as well :-)

Thanks for the heads-up.

Will y'all be doing a PyPI pre-release so we can test with `pip
install --pre -U pip`?

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Paul Moore]

On 20 October 2017 at 14:26, Matthew Brett  wrote:
> Thanks for the heads-up.
>
> Will y'all be doing a PyPI pre-release so we can test with `pip
> install --pre -U pip`?

We've not yet decided on that. Traditionally I don't think we have
done so, but I'm inclined to think it's a good idea. It might not be
until noticeably closer to the release, though...

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Nick Coghlan]

On 20 October 2017 at 23:19, Donald Stufft  wrote:

> One that I was helping someone debug just the other day is that they’re
> super non-debuggable and the behavior when you have two things providing
> the same entry point is basically  (If I remember correctly, the
> behavior is that the first thing found is the one that “wins”, which means
> the ordering of sys.path and the names of the projects supply it is what
> determines it). This got exposed to the end user that they installed
> something that they thought was going to add support for something, but
> which silently did nothing because two different project happened to pick
> the same name for their entry point (not the group, it was two things
> providing plugins for the same system).
>

While I agree with this, I think that's a combination of pkg_resources
itself being hard to debug in general, and the fact that pkg_resources
doesn't clearly define the semantics of how it resolves name conflicts
within an entry point group - as far as I know, it's largely an accident of
implementation.

The interoperability spec is going to state that conflict resolution when
the same name within a group is declared by multiple packages is the
responsibility of the group consumer, so documenting the format should
actually improve this situation, since it allows for the development of
competing conflict resolution strategies in different runtime libraries.


> Of course there is the perennial entrypoints are super slow, which is
> partially the fault of pkg_resources does a bunch of import time logic, but
> also because scanning sys.path for all installed stuff is just slow.
>

Similar to the above, one of the goals of documenting the entry point file
format is to permit libraries to compete in the development of effective
entrypoint metadata caching strategies without needing to bless any
particular one a priori, and without trying to manage experimental cache
designs across the huge pkg_resources install base.


> They’re also somewhat fragile since they rely on the packaging metadata
> system at runtime, and a number of tools exclude that information (often
> times things that deploy stuff as a tarball/zipfile) which causes regular
> issues to be opened up for these projects when they get used in those
> environments.
>

This is true, and one of the main pragmatic benefits of adopting one of the
purely import based plugin management systems. However, this problem will
impact all packaging metadata based plugin management solutions, regardless
of whether they use an existing file format or a new one.


> Those are the ones I remember because they come up regularly (and people
> regularly come to me with issues with any project related to packaging in
> any way even for non packaging related features in those projects). I’m
> pretty sure there were more of them that I’ve encountered and seen projects
> encounter, but I can’t remember them to be sure.
>
> I’m more familiar with why console_scripts entry point is not great and
> why we should stop using it since I regularly try to re-read all of pip’s
> issues and a lot of it’s issues are documented there.
>

I'm sympathetic to that, but I think even in that case, clearly documenting
the format as an interoperability specification will help tease out which
of those are due to the file format itself, and which are due to
setuptools.setup specifically.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pythonhosted.org doc upload no longer works

[Ronald Oussoren]

> On 7 Oct 2017, at 09:12, Giampaolo Rodola'  wrote:
> 
> Any news about this please? This is becoming a problem as my outdated doc 
> keeps being around and I have no way to remove it. 

I have the same problem with my projects: the documentation on pythonhosted.org 
can no longer be updated, and it is also not possible to change that 
documentation into a link to a new location.

I have new documentation on Read the Docs, but Google searches still prefer the 
old location and I would therefore prefer to have redirects on python host.org.

BTW. Has there been any documentation beyond this list about this change? 

Ronald

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Donald Stufft]

> On Oct 20, 2017, at 9:35 AM, Nick Coghlan  wrote:
> 
> On 20 October 2017 at 23:19, Donald Stufft  > wrote:
> One that I was helping someone debug just the other day is that they’re super 
> non-debuggable and the behavior when you have two things providing the same 
> entry point is basically  (If I remember correctly, the behavior is that 
> the first thing found is the one that “wins”, which means the ordering of 
> sys.path and the names of the projects supply it is what determines it). This 
> got exposed to the end user that they installed something that they thought 
> was going to add support for something, but which silently did nothing 
> because two different project happened to pick the same name for their entry 
> point (not the group, it was two things providing plugins for the same 
> system).
> 
> While I agree with this, I think that's a combination of pkg_resources itself 
> being hard to debug in general, and the fact that pkg_resources doesn't 
> clearly define the semantics of how it resolves name conflicts within an 
> entry point group - as far as I know, it's largely an accident of 
> implementation.
> 
> The interoperability spec is going to state that conflict resolution when the 
> same name within a group is declared by multiple packages is the 
> responsibility of the group consumer, so documenting the format should 
> actually improve this situation, since it allows for the development of 
> competing conflict resolution strategies in different runtime libraries.

I think it makes it *worse*, because now the behavior isn’t just a entrypoints 
weirdness, but now it changes based on which runtime library you use (which 
isn’t something that end users are likely to have much insight into) and it 
represents a footgun that package authors are unlikely to be aware of. If 
mycoolentrypointslib comes out that is faster, but changes some subtle behavior 
like this it’ll break people, but that is unlikely going to be an effect that 
people expect to happen just because they switched between two things both 
implementing the same standard.

So effectively this means that not only is the fact you’re using entrypoints 
part of your API, but now which entry point library you’re using at runtime is 
now also part of your API.

>  
> Of course there is the perennial entrypoints are super slow, which is 
> partially the fault of pkg_resources does a bunch of import time logic, but 
> also because scanning sys.path for all installed stuff is just slow.
> 
> Similar to the above, one of the goals of documenting the entry point file 
> format is to permit libraries to compete in the development of effective 
> entrypoint metadata caching strategies without needing to bless any 
> particular one a priori, and without trying to manage experimental cache 
> designs across the huge pkg_resources install base.

That goal can be achieved if it’s documented in setuptools.

>  
> They’re also somewhat fragile since they rely on the packaging metadata 
> system at runtime, and a number of tools exclude that information (often 
> times things that deploy stuff as a tarball/zipfile) which causes regular 
> issues to be opened up for these projects when they get used in those 
> environments.
> 
> This is true, and one of the main pragmatic benefits of adopting one of the 
> purely import based plugin management systems. However, this problem will 
> impact all packaging metadata based plugin management solutions, regardless 
> of whether they use an existing file format or a new one.
>  
> Those are the ones I remember because they come up regularly (and people 
> regularly come to me with issues with any project related to packaging in any 
> way even for non packaging related features in those projects). I’m pretty 
> sure there were more of them that I’ve encountered and seen projects 
> encounter, but I can’t remember them to be sure.
> 
> I’m more familiar with why console_scripts entry point is not great and why 
> we should stop using it since I regularly try to re-read all of pip’s issues 
> and a lot of it’s issues are documented there.
> 
> I'm sympathetic to that, but I think even in that case, clearly documenting 
> the format as an interoperability specification will help tease out which of 
> those are due to the file format itself, and which are due to 
> setuptools.setup specifically.

All of the ones I’m aware of are due to the file format itself, because they 
exist even without setuptools being involved at all.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Documentation link on PyPI.org

[Ronald Oussoren]

> On 28 Aug 2017, at 00:12, Brett Cannon  wrote:
> 
> If you search the archive of this mailing list you will notice I asked this 
> exact question about a month or 2 ago (I think). The answer I got was it is 
> used on PyPI.org, but I don't know how to set it with setuptools (flit will 
> probably add support after PEP 517).

To get back to this: I’m probably doing something wrong, but I haven’t managed 
yet to make anything show up on PyPI.org  when I use 
“Project-URL” in metadata. 

There’s a pyobjc-core 4.0.1b1 wheel on PyPI.org  that 
includes two Project-URL lines in its metadata, but that doesn’t add anything 
to the sidebar. I’ve manually verified that the metadata is present in the 
wheel file (as I’m using a replacement egg_info command in my setup.py file to 
generate this part of the metadata).

Ronald

P.S. I tried to upload to test.pypi.org , but that 
didn’t work due to an SSL error:

$ twine upload -r testpypi 
dist/pyobjc_core-4.0.1b1-cp27-cp27m-macosx_10_6_intel.whl 
Uploading distributions to https://test.pypi.org/legacy/
Uploading pyobjc_core-4.0.1b1-cp27-cp27m-macosx_10_6_intel.whl
SSLError: HTTPSConnectionPool(host='test.pypi.org', port=443): Max retries 
exceeded with url: /legacy/ (Caused by SSLError(SSLError(1, u'[SSL: 
TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)'),))

> 
> On Sun, Aug 27, 2017, 13:01 Ronald Oussoren  > wrote:
>> On 27 Aug 2017, at 17:33, Wes Turner > > wrote:
>> 
>> You can add a URL in the long_description (e.g. to 
>> https://read-the-docs.readthedocs.org/ 
>>  )
> 
> I know, but that adds the URL to the project description and not the sidebar. 
> PEP 345 appears to specify metadata that could be used for this, and 
> warehouse seems to contain to make use of this (or at least a database model 
> that allows for specifying more links than just the default one). 
> 
> Adding links to the sidebar would be nicer with the new design on PyPI.org 
> , but there doesn’t seem to be a clean way to add 
> “Project-URL” metadata as described in PEP 345 to a wheel file. I could tweak 
> my setup.py files to add Project-URL metadata, but before I do that I’d like 
> to confirm that (a) that metadata is actually used by warehouse and (b) there 
> is no cleaner way to add this metadata using setuptools.
> 
> Ronald
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org 
> 
> https://mail.python.org/mailman/listinfo/distutils-sig 
> 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Wes Turner]

On Friday, October 20, 2017, Donald Stufft  wrote:

>
>
> On Oct 20, 2017, at 9:35 AM, Nick Coghlan  > wrote:
>
> On 20 October 2017 at 23:19, Donald Stufft  > wrote:
>
>> One that I was helping someone debug just the other day is that they’re
>> super non-debuggable and the behavior when you have two things providing
>> the same entry point is basically  (If I remember correctly, the
>> behavior is that the first thing found is the one that “wins”, which means
>> the ordering of sys.path and the names of the projects supply it is what
>> determines it). This got exposed to the end user that they installed
>> something that they thought was going to add support for something, but
>> which silently did nothing because two different project happened to pick
>> the same name for their entry point (not the group, it was two things
>> providing plugins for the same system).
>>
>
> While I agree with this, I think that's a combination of pkg_resources
> itself being hard to debug in general, and the fact that pkg_resources
> doesn't clearly define the semantics of how it resolves name conflicts
> within an entry point group - as far as I know, it's largely an accident of
> implementation.
>
> The interoperability spec is going to state that conflict resolution when
> the same name within a group is declared by multiple packages is the
> responsibility of the group consumer, so documenting the format should
> actually improve this situation, since it allows for the development of
> competing conflict resolution strategies in different runtime libraries.
>
>
> I think it makes it *worse*, because now the behavior isn’t just a
> entrypoints weirdness, but now it changes based on which runtime library
> you use (which isn’t something that end users are likely to have much
> insight into) and it represents a footgun that package authors are unlikely
> to be aware of. If mycoolentrypointslib comes out that is faster, but
> changes some subtle behavior like this it’ll break people, but that is
> unlikely going to be an effect that people expect to happen just because
> they switched between two things both implementing the same standard.
>
> So effectively this means that not only is the fact you’re using
> entrypoints part of your API, but now which entry point library you’re
> using at runtime is now also part of your API.
>

When should the check for duplicate entry points occur?

- At on_install() time (+1)
- At runtime

Is a sys.path-like OrderedDict preemptive strategy preferable or just as
dangerous as importlib?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pythonhosted.org doc upload no longer works

[Nick Coghlan]

On 20 October 2017 at 22:42, Ronald Oussoren  wrote:

>
> On 7 Oct 2017, at 09:12, Giampaolo Rodola'  wrote:
>
> Any news about this please? This is becoming a problem as my outdated doc
> keeps being around and I have no way to remove it.
>
>
> I have the same problem with my projects: the documentation on
> pythonhosted.org can no longer be updated, and it is also not possible to
> change that documentation into a link to a new location.
>
> I have new documentation on Read the Docs, but Google searches still
> prefer the old location and I would therefore prefer to have redirects on
> python host.org.
>
> BTW. Has there been any documentation beyond this list about this change?
>

I don't believe so, as I believe the change from "deprecated" to "no longer
works" was a side effect of disabling uploads via the old service.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Paul Moore]

On 20 October 2017 at 14:55, Jannis Gebauer  wrote:
> Thanks for the heads-up, Paul.
>
> I’m currently using `pip.get_installed_distributions` and as far as I can
> see that has moved into `_internal`, too:
> https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333
>
> Any recommendations?

See https://github.com/pypa/pip/pull/4743

Unfortunately, the "latest" docs build doesn't seem to reflect this (I
don't know why).

I guess you probably want something from pkg_resources? Depends
precisely what you're trying to do, I guess.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Jannis Gebauer]

Thanks for the heads-up, Paul.

I’m currently using `pip.get_installed_distributions` and as far as I can see 
that has moved into `_internal`, too: 
https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333 


Any recommendations?

Thanks,

Jannis

> On 20. Oct 2017, at 15:22, Paul Moore  wrote:
> 
> We're in the process of starting to plan for a release of pip (the
> long-awaited pip 10). We're likely still a month or two away from a
> release, but now is the time for people to start ensuring that
> everything works for them. One key change in the new version will be
> that all of the internal APIs of pip will no longer be available, so
> any code that currently calls functions in the "pip" namespace will
> break. Calling pip's internal APIs has never been supported, and
> always carried a risk of such breakage, so projects doing so should,
> in theory, be prepared for such things. However, reality is not always
> that simple, and we are aware that people will need time to deal with
> the implications.
> 
> Just in case it's not clear, simply finding where the internal APIs
> have moved to and calling them under the new names is *not* what
> people should do. We can't stop people calling the internal APIs,
> obviously, but the idea of this change is to give people the incentive
> to find a supported approach, not just to annoy people who are doing
> things we don't want them to ;-)
> 
> So please - if you're calling pip's internals in your code, take the
> opportunity *now* to check out the in-development version of pip, and
> ensure your project will still work when pip 10 is released.
> 
> And many thanks to anyone else who helps by testing out the new
> version, as well :-)
> 
> Thanks,
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Documentation link on PyPI.org

[Ronald Oussoren]

> On 20 Oct 2017, at 17:51, Wes Turner  wrote:
> 
> 
> 
> On Friday, October 20, 2017, Ronald Oussoren  > wrote:
> 
>> On 28 Aug 2017, at 00:12, Brett Cannon > > wrote:
>> 
>> If you search the archive of this mailing list you will notice I asked this 
>> exact question about a month or 2 ago (I think). The answer I got was it is 
>> used on PyPI.org , but I don't know how to set it with 
>> setuptools (flit will probably add support after PEP 517).
> 
> To get back to this: I’m probably doing something wrong, but I haven’t 
> managed yet to make anything show up on PyPI.org  when I 
> use “Project-URL” in metadata. 
> 
> There’s a pyobjc-core 4.0.1b1 wheel on PyPI.org  that 
> includes two Project-URL lines in its metadata, but that doesn’t add anything 
> to the sidebar. I’ve manually verified that the metadata is present in the 
> wheel file (as I’m using a replacement egg_info command in my setup.py file 
> to generate this part of the metadata).
> 
> https://github.com/pypa/warehouse/search?utf8=✓= 
> "project-url"=Issues
> 
> https://github.com/pypa/warehouse/search?q= 
> "Project-Url"=Commits=✓

I’m note sure what you’re trying to say here, but 
> suggests that Project-URL 
is supported in the PyPI.org  UI (assuming that it is running 
a version of the software from after merging this pull request).  But for some 
reason this doesn’t work for me, and I don’t know why. Either I’m doing 
something wrong, or the code receiving wheel files does not yet store 
project-urls information in the database.

>  
> 
> Ronald
> 
> P.S. I tried to upload to test.pypi.org , but that 
> didn’t work due to an SSL error:
> 
> $ twine upload -r testpypi 
> dist/pyobjc_core-4.0.1b1-cp27-cp27m-macosx_10_6_intel.whl 
> Uploading distributions to https://test.pypi.org/legacy/ 
> 
> Uploading pyobjc_core-4.0.1b1-cp27-cp27m-macosx_10_6_intel.whl
> SSLError: HTTPSConnectionPool(host='test.pypi.org ', 
> port=443): Max retries exceeded with url: /legacy/ (Caused by 
> SSLError(SSLError(1, u'[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert 
> protocol version (_ssl.c:661)'),))
> 
> https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION 
> 
> 
> Looks like TLSv1 (TLS 1.0) is deprecated.
> 
> https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION+twine 
> 
> 
> https://github.com/pypa/twine/issues/273 
> 

Hmm… that issue means I should be slightly worried, the VM I’m using for 
uploading releases might stop working for this some time in the future. 
Hopefully upgrading the python installation on that will avoid that.

Ronald

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Documentation link on PyPI.org

[Wes Turner]

On Friday, October 20, 2017, Ronald Oussoren  wrote:

>
> On 28 Aug 2017, at 00:12, Brett Cannon  > wrote:
>
> If you search the archive of this mailing list you will notice I asked
> this exact question about a month or 2 ago (I think). The answer I got was
> it is used on PyPI.org, but I don't know how to set it with setuptools
> (flit will probably add support after PEP 517).
>
>
> To get back to this: I’m probably doing something wrong, but I haven’t
> managed yet to make anything show up on PyPI.org when I use “Project-URL”
> in metadata.
>
> There’s a pyobjc-core 4.0.1b1 wheel on PyPI.org that includes two
> Project-URL lines in its metadata, but that doesn’t add anything to the
> sidebar. I’ve manually verified that the metadata is present in the wheel
> file (as I’m using a replacement egg_info command in my setup.py file to
> generate this part of the metadata).
>

https://github.com/pypa/warehouse/search?utf8=✓="project-url"=Issues

https://github.com/pypa/warehouse/search?q="Project-Url"=Commits=✓


>
> Ronald
>
> P.S. I tried to upload to test.pypi.org, but that didn’t work due to an
> SSL error:
>
> $ twine upload -r testpypi dist/pyobjc_core-4.0.1b1-cp27-
> cp27m-macosx_10_6_intel.whl
> Uploading distributions to https://test.pypi.org/legacy/
> Uploading pyobjc_core-4.0.1b1-cp27-cp27m-macosx_10_6_intel.whl
> SSLError: HTTPSConnectionPool(host='test.pypi.org', port=443): Max
> retries exceeded with url: /legacy/ (Caused by SSLError(SSLError(1, u'[SSL:
> TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)'),))
>

https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION

Looks like TLSv1 (TLS 1.0) is deprecated.

https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION+twine

https://github.com/pypa/twine/issues/273




>
>
> On Sun, Aug 27, 2017, 13:01 Ronald Oussoren  > wrote:
>
>> On 27 Aug 2017, at 17:33, Wes Turner > > wrote:
>>
>> You can add a URL in the long_description (e.g. to https://read-the-docs.
>> readthedocs.org/ )
>>
>>
>> I know, but that adds the URL to the project description and not the
>> sidebar. PEP 345 appears to specify metadata that could be used for this,
>> and warehouse seems to contain to make use of this (or at least a database
>> model that allows for specifying more links than just the default one).
>>
>> Adding links to the sidebar would be nicer with the new design on
>> PyPI.org , but there doesn’t seem to be a clean way to
>> add “Project-URL” metadata as described in PEP 345 to a wheel file. I could
>> tweak my setup.py files to add Project-URL metadata, but before I do that
>> I’d like to confirm that (a) that metadata is actually used by warehouse
>> and (b) there is no cleaner way to add this metadata using setuptools.
>>
>> Ronald
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> 
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Alex Grönholm]

Perhaps pkg_resources.find_distributions()?

http://setuptools.readthedocs.io/en/latest/pkg_resources.html#getting-or-creating-distributions


Jannis Gebauer kirjoitti 20.10.2017 klo 16:55:

Thanks for the heads-up, Paul.

I’m currently using `pip.get_installed_distributions` and as far as I 
can see that has moved into `_internal`, too: 
https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333


Any recommendations?

Thanks,

Jannis

On 20. Oct 2017, at 15:22, Paul Moore > wrote:


We're in the process of starting to plan for a release of pip (the
long-awaited pip 10). We're likely still a month or two away from a
release, but now is the time for people to start ensuring that
everything works for them. One key change in the new version will be
that all of the internal APIs of pip will no longer be available, so
any code that currently calls functions in the "pip" namespace will
break. Calling pip's internal APIs has never been supported, and
always carried a risk of such breakage, so projects doing so should,
in theory, be prepared for such things. However, reality is not always
that simple, and we are aware that people will need time to deal with
the implications.

Just in case it's not clear, simply finding where the internal APIs
have moved to and calling them under the new names is *not* what
people should do. We can't stop people calling the internal APIs,
obviously, but the idea of this change is to give people the incentive
to find a supported approach, not just to annoy people who are doing
things we don't want them to ;-)

So please - if you're calling pip's internals in your code, take the
opportunity *now* to check out the in-development version of pip, and
ensure your project will still work when pip 10 is released.

And many thanks to anyone else who helps by testing out the new
version, as well :-)

Thanks,
Paul
___
Distutils-SIG maillist  - Distutils-SIG@python.org 


https://mail.python.org/mailman/listinfo/distutils-sig




___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Noah Kantrowitz]

Installing to a temp dir is really not an option for automated tooling (if 
nothing else, it takes way too long). `pip list --outdated` does already get 
fairly close to this (and doesn't install anything I suspect you can actually 
get a lot closer than you think) but it calculates for all packages (read: is 
slow) and doesn't give a good way to restrict things (hence that hack-y script 
which is a modified version of the pip list code). This is 100% a hard 
requirement for config management systems and if not fixed in pip, will require 
continued use of internal APIs. I would recommend just making pip list take a 
set of install-compatible names/version patterns and apply that as a filter in 
a similar way to what I've done there.

--Noah

> On Oct 20, 2017, at 11:35 AM, xoviat  wrote:
> 
> There's no dry-run functionality that I know of so far. However, you could 
> use the following:
> 
> pip install --prefix=tmpdir
> 
> This command is actually about the same speed as a proper implementation, 
> because we can't actually know what we're installing until we build the 
> requirements.
> 
> 2017-10-20 12:42 GMT-05:00 Noah Kantrowitz :
> So as someone on the tooling side, is there any kind of install dry-run yet? 
> I've got 
> https://github.com/poise/poise-python/blob/master/lib/poise_python/resources/python_package.rb#L34-L78
>  which touches a tn of internals. Basically I need a way to know exactly 
> what versions `pip install` would have used in a given situation without 
> actually changing the system. Happy for a better solution!
> 
> --Noah
> 
> > On Oct 20, 2017, at 6:22 AM, Paul Moore  wrote:
> >
> > We're in the process of starting to plan for a release of pip (the
> > long-awaited pip 10). We're likely still a month or two away from a
> > release, but now is the time for people to start ensuring that
> > everything works for them. One key change in the new version will be
> > that all of the internal APIs of pip will no longer be available, so
> > any code that currently calls functions in the "pip" namespace will
> > break. Calling pip's internal APIs has never been supported, and
> > always carried a risk of such breakage, so projects doing so should,
> > in theory, be prepared for such things. However, reality is not always
> > that simple, and we are aware that people will need time to deal with
> > the implications.
> >
> > Just in case it's not clear, simply finding where the internal APIs
> > have moved to and calling them under the new names is *not* what
> > people should do. We can't stop people calling the internal APIs,
> > obviously, but the idea of this change is to give people the incentive
> > to find a supported approach, not just to annoy people who are doing
> > things we don't want them to ;-)
> >
> > So please - if you're calling pip's internals in your code, take the
> > opportunity *now* to check out the in-development version of pip, and
> > ensure your project will still work when pip 10 is released.
> >
> > And many thanks to anyone else who helps by testing out the new
> > version, as well :-)
> >
> > Thanks,
> > Paul
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> 
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
> 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[xoviat]

The "installing collected packages" output is not always correct. The
"successfully installed [package list]" is always correct
(see pypa/pip/issues/4724 for an example).

I understand your requirements, but is the practical implication that pip
should offer a command that outputs packages that are potentially
incorrect? What happens when someone relies on that and then files a bug
report?

2017-10-20 13:48 GMT-05:00 Noah Kantrowitz :

> While I understand that pip itself has to be very careful about edge cases
> and all the pathological things you can do in setup.py, as a higher-level
> tooling author my priorities are on the happy path UX and speed is a big
> factor there. So yes, using PackageFinder is potentially inaccurate, but
> it's also _usually_ accurate :) Anyways, if there is true concern that
> finder-based approaches are too risky, probably don't offer it in the pip
> list output.
>
> --Noah
>
> > On Oct 20, 2017, at 11:43 AM, xoviat  wrote:
> >
> > A correct dry-run implementation will do about the same amount of work
> as installing to a temporary directory right now. In the future, that could
> be optimized, but any patch to the finder doesn't actually detect the
> requirements correctly (as they're not necessarily known until after the
> wheels are built).
> >
> > 2017-10-20 13:41 GMT-05:00 Noah Kantrowitz :
> > Installing to a temp dir is really not an option for automated tooling
> (if nothing else, it takes way too long). `pip list --outdated` does
> already get fairly close to this (and doesn't install anything I suspect
> you can actually get a lot closer than you think) but it calculates for all
> packages (read: is slow) and doesn't give a good way to restrict things
> (hence that hack-y script which is a modified version of the pip list
> code). This is 100% a hard requirement for config management systems and if
> not fixed in pip, will require continued use of internal APIs. I would
> recommend just making pip list take a set of install-compatible
> names/version patterns and apply that as a filter in a similar way to what
> I've done there.
> >
> > --Noah
> >
> > > On Oct 20, 2017, at 11:35 AM, xoviat  wrote:
> > >
> > > There's no dry-run functionality that I know of so far. However, you
> could use the following:
> > >
> > > pip install --prefix=tmpdir
> > >
> > > This command is actually about the same speed as a proper
> implementation, because we can't actually know what we're installing until
> we build the requirements.
> > >
> > > 2017-10-20 12:42 GMT-05:00 Noah Kantrowitz :
> > > So as someone on the tooling side, is there any kind of install
> dry-run yet? I've got https://github.com/poise/
> poise-python/blob/master/lib/poise_python/resources/python_
> package.rb#L34-L78 which touches a tn of internals. Basically I need
> a way to know exactly what versions `pip install` would have used in a
> given situation without actually changing the system. Happy for a better
> solution!
> > >
> > > --Noah
> > >
> > > > On Oct 20, 2017, at 6:22 AM, Paul Moore  wrote:
> > > >
> > > > We're in the process of starting to plan for a release of pip (the
> > > > long-awaited pip 10). We're likely still a month or two away from a
> > > > release, but now is the time for people to start ensuring that
> > > > everything works for them. One key change in the new version will be
> > > > that all of the internal APIs of pip will no longer be available, so
> > > > any code that currently calls functions in the "pip" namespace will
> > > > break. Calling pip's internal APIs has never been supported, and
> > > > always carried a risk of such breakage, so projects doing so should,
> > > > in theory, be prepared for such things. However, reality is not
> always
> > > > that simple, and we are aware that people will need time to deal with
> > > > the implications.
> > > >
> > > > Just in case it's not clear, simply finding where the internal APIs
> > > > have moved to and calling them under the new names is *not* what
> > > > people should do. We can't stop people calling the internal APIs,
> > > > obviously, but the idea of this change is to give people the
> incentive
> > > > to find a supported approach, not just to annoy people who are doing
> > > > things we don't want them to ;-)
> > > >
> > > > So please - if you're calling pip's internals in your code, take the
> > > > opportunity *now* to check out the in-development version of pip, and
> > > > ensure your project will still work when pip 10 is released.
> > > >
> > > > And many thanks to anyone else who helps by testing out the new
> > > > version, as well :-)
> > > >
> > > > Thanks,
> > > > Paul
> > > > ___
> > > > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > > > https://mail.python.org/mailman/listinfo/distutils-sig
> > >
> > > 

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Noah Kantrowitz]

While I understand that pip itself has to be very careful about edge cases and 
all the pathological things you can do in setup.py, as a higher-level tooling 
author my priorities are on the happy path UX and speed is a big factor there. 
So yes, using PackageFinder is potentially inaccurate, but it's also _usually_ 
accurate :) Anyways, if there is true concern that finder-based approaches are 
too risky, probably don't offer it in the pip list output.

--Noah

> On Oct 20, 2017, at 11:43 AM, xoviat  wrote:
> 
> A correct dry-run implementation will do about the same amount of work as 
> installing to a temporary directory right now. In the future, that could be 
> optimized, but any patch to the finder doesn't actually detect the 
> requirements correctly (as they're not necessarily known until after the 
> wheels are built).
> 
> 2017-10-20 13:41 GMT-05:00 Noah Kantrowitz :
> Installing to a temp dir is really not an option for automated tooling (if 
> nothing else, it takes way too long). `pip list --outdated` does already get 
> fairly close to this (and doesn't install anything I suspect you can actually 
> get a lot closer than you think) but it calculates for all packages (read: is 
> slow) and doesn't give a good way to restrict things (hence that hack-y 
> script which is a modified version of the pip list code). This is 100% a hard 
> requirement for config management systems and if not fixed in pip, will 
> require continued use of internal APIs. I would recommend just making pip 
> list take a set of install-compatible names/version patterns and apply that 
> as a filter in a similar way to what I've done there.
> 
> --Noah
> 
> > On Oct 20, 2017, at 11:35 AM, xoviat  wrote:
> >
> > There's no dry-run functionality that I know of so far. However, you could 
> > use the following:
> >
> > pip install --prefix=tmpdir
> >
> > This command is actually about the same speed as a proper implementation, 
> > because we can't actually know what we're installing until we build the 
> > requirements.
> >
> > 2017-10-20 12:42 GMT-05:00 Noah Kantrowitz :
> > So as someone on the tooling side, is there any kind of install dry-run 
> > yet? I've got 
> > https://github.com/poise/poise-python/blob/master/lib/poise_python/resources/python_package.rb#L34-L78
> >  which touches a tn of internals. Basically I need a way to know 
> > exactly what versions `pip install` would have used in a given situation 
> > without actually changing the system. Happy for a better solution!
> >
> > --Noah
> >
> > > On Oct 20, 2017, at 6:22 AM, Paul Moore  wrote:
> > >
> > > We're in the process of starting to plan for a release of pip (the
> > > long-awaited pip 10). We're likely still a month or two away from a
> > > release, but now is the time for people to start ensuring that
> > > everything works for them. One key change in the new version will be
> > > that all of the internal APIs of pip will no longer be available, so
> > > any code that currently calls functions in the "pip" namespace will
> > > break. Calling pip's internal APIs has never been supported, and
> > > always carried a risk of such breakage, so projects doing so should,
> > > in theory, be prepared for such things. However, reality is not always
> > > that simple, and we are aware that people will need time to deal with
> > > the implications.
> > >
> > > Just in case it's not clear, simply finding where the internal APIs
> > > have moved to and calling them under the new names is *not* what
> > > people should do. We can't stop people calling the internal APIs,
> > > obviously, but the idea of this change is to give people the incentive
> > > to find a supported approach, not just to annoy people who are doing
> > > things we don't want them to ;-)
> > >
> > > So please - if you're calling pip's internals in your code, take the
> > > opportunity *now* to check out the in-development version of pip, and
> > > ensure your project will still work when pip 10 is released.
> > >
> > > And many thanks to anyone else who helps by testing out the new
> > > version, as well :-)
> > >
> > > Thanks,
> > > Paul
> > > ___
> > > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> 
> 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Brett Cannon]

On Wed, 18 Oct 2017 at 17:54 Nick Coghlan  wrote:

> On 19 October 2017 at 04:18, Alex Grönholm 
> wrote:
>
>> Daniel Holth kirjoitti 18.10.2017 klo 21:06:
>>
>>
>> http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata
>>
>>
>> http://setuptools.readthedocs.io/en/latest/pkg_resources.html?highlight=pkg_resources#creating-and-parsing
>>
>> It is not very complicated. It looks like the characters are mostly
>> 'python identifier' rules with a little bit of 'package name' rules.
>>
>> I am also concerned about the amount of parsing on startup. A hard
>> problem for certain, since no one likes outdated cache problems either. It
>> is also unpleasant to have too much code with a runtime dependency on
>> 'packaging'.
>>
>> Wasn't someone working on implementing pkg_resources in the standard
>> library at some point?
>>
>
> The idea has been raised, but we've been hesitant for the same reason
> we're inclined to take distutils out: packaging APIs need to be free to
> evolve in line with packaging interoperability standards, rather than with
> the Python language definition.
>
> Barry Warsaw & Brett Cannon recently mentioned something to me about
> working on a potential runtime alternative to pkg_resources that could be
> installed without also installing setuptools, but I don't know any of the
> specifics (and I'm not sure either of them follows distutils-sig).
>

I've been following distutils-sig for a couple of years now. :)

And what Barry and I are working on is only a subset of pkg_resources,
specifically the reading of data files included in a package. We aren't
touching any other aspect of pkg_resources.

Heck, until this discussion, "entry points" == "console scripts" for me so
I don't really know what y'all are talking about standardizing when it
comes to plug-in systems and metadata. Having said that, I do understand
why Donald doesn't want to just go ahead and standardize something by
giving it the level of a spec on packaging.python.org just because it's out
there. But since entry points seem to be used widely enough, having them
written down appropriately also seems reasonable.

As a compromise, could entry points be documented as Thomas is suggesting,
but have a note at the top saying something along the lines of "entry
points are considered a setuptools-specific feature, but their wide spread
use warrants a clear understanding of how they function for other packaging
tools choose on their own to also support them"? Basically acknowledge
there are ad-hoc, folk standards in the community that a decent chunk of
people rely on and thus docs would be helpful, but don't need to be
promoted to full-on, everyone-implements standard.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Noah Kantrowitz]

So as someone on the tooling side, is there any kind of install dry-run yet? 
I've got 
https://github.com/poise/poise-python/blob/master/lib/poise_python/resources/python_package.rb#L34-L78
 which touches a tn of internals. Basically I need a way to know exactly 
what versions `pip install` would have used in a given situation without 
actually changing the system. Happy for a better solution!

--Noah

> On Oct 20, 2017, at 6:22 AM, Paul Moore  wrote:
> 
> We're in the process of starting to plan for a release of pip (the
> long-awaited pip 10). We're likely still a month or two away from a
> release, but now is the time for people to start ensuring that
> everything works for them. One key change in the new version will be
> that all of the internal APIs of pip will no longer be available, so
> any code that currently calls functions in the "pip" namespace will
> break. Calling pip's internal APIs has never been supported, and
> always carried a risk of such breakage, so projects doing so should,
> in theory, be prepared for such things. However, reality is not always
> that simple, and we are aware that people will need time to deal with
> the implications.
> 
> Just in case it's not clear, simply finding where the internal APIs
> have moved to and calling them under the new names is *not* what
> people should do. We can't stop people calling the internal APIs,
> obviously, but the idea of this change is to give people the incentive
> to find a supported approach, not just to annoy people who are doing
> things we don't want them to ;-)
> 
> So please - if you're calling pip's internals in your code, take the
> opportunity *now* to check out the in-development version of pip, and
> ensure your project will still work when pip 10 is released.
> 
> And many thanks to anyone else who helps by testing out the new
> version, as well :-)
> 
> Thanks,
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Doug Hellmann]

Excerpts from Wes Turner's message of 2017-10-20 10:41:02 -0400:
> On Friday, October 20, 2017, Donald Stufft  wrote:
> 
> >
> >
> > On Oct 20, 2017, at 9:35 AM, Nick Coghlan  > > wrote:
> >
> > On 20 October 2017 at 23:19, Donald Stufft  > > wrote:
> >
> >> One that I was helping someone debug just the other day is that they’re
> >> super non-debuggable and the behavior when you have two things providing
> >> the same entry point is basically  (If I remember correctly, the
> >> behavior is that the first thing found is the one that “wins”, which means
> >> the ordering of sys.path and the names of the projects supply it is what
> >> determines it). This got exposed to the end user that they installed
> >> something that they thought was going to add support for something, but
> >> which silently did nothing because two different project happened to pick
> >> the same name for their entry point (not the group, it was two things
> >> providing plugins for the same system).
> >>
> >
> > While I agree with this, I think that's a combination of pkg_resources
> > itself being hard to debug in general, and the fact that pkg_resources
> > doesn't clearly define the semantics of how it resolves name conflicts
> > within an entry point group - as far as I know, it's largely an accident of
> > implementation.
> >
> > The interoperability spec is going to state that conflict resolution when
> > the same name within a group is declared by multiple packages is the
> > responsibility of the group consumer, so documenting the format should
> > actually improve this situation, since it allows for the development of
> > competing conflict resolution strategies in different runtime libraries.
> >
> >
> > I think it makes it *worse*, because now the behavior isn’t just a
> > entrypoints weirdness, but now it changes based on which runtime library
> > you use (which isn’t something that end users are likely to have much
> > insight into) and it represents a footgun that package authors are unlikely
> > to be aware of. If mycoolentrypointslib comes out that is faster, but
> > changes some subtle behavior like this it’ll break people, but that is
> > unlikely going to be an effect that people expect to happen just because
> > they switched between two things both implementing the same standard.
> >
> > So effectively this means that not only is the fact you’re using
> > entrypoints part of your API, but now which entry point library you’re
> > using at runtime is now also part of your API.
> >
> 
> When should the check for duplicate entry points occur?
> 
> - At on_install() time (+1)
> - At runtime
> 
> Is a sys.path-like OrderedDict preemptive strategy preferable or just as
> dangerous as importlib?

Having "duplicate" entry points is not necessarily an error. It's
a different usage pattern.  The semantics of dropping a named plugin
into a namespace are defined by the application and plugin-point.
Please do not build assumptions about uniqueness into the underlying
implementation.

The stevedore library wraps up pkg_resources with several such
patterns. For example, it supports "give me all of the plugins in
a namespace" (find all the extensions to your app), "give me all
of the plugins named $name in a namespace" (find the hooks for a
specific event defined by the app), and "give me *the* plugin named
$name in a namespace" (load a driver for talking to a backend).

https://docs.openstack.org/stevedore/latest/reference/index.html

Doug
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Doug Hellmann]

Excerpts from Nick Coghlan's message of 2017-10-20 14:42:09 +1000:
> On 20 October 2017 at 02:14, Thomas Kluyver  wrote:
> 
> > On Thu, Oct 19, 2017, at 04:10 PM, Donald Stufft wrote:
> > > I’m in favor, although one question I guess is whether it should be a a
> > > PEP or an ad hoc spec. Given (2) it should *probably* be a a PEP (since
> > > without (2), its just another file in the .dist-info directory and that
> > > doesn’t actually need standardized at all). I don’t think that this will
> > > be a very controversial PEP though, and should be pretty easy.
> >
> > I have opened a PR to document what is already there, without adding any
> > new features. I think this is worth doing even if we don't change
> > anything, since it's a de-facto standard used for different tools to
> > interact.
> >
> > https://github.com/pypa/python-packaging-user-guide/pull/390
> >
> > We can still write a PEP for caching if necessary.
> >
> 
> +1 for that approach (PR for the status quo, PEP for a shared metadata
> caching design) from me
> 
> Making the status quo more discoverable is valuable in its own right, and
> the only decisions we'll need to make for that are terminology
> clarification ones, not interoperability ones (this isn't like PEP 440 or
> 508 where we actually thought some of the default setuptools behaviour was
> slightly incorrect and wanted to change it).
> 
> Figuring out a robust cross-platform network-file-system-tolerant metadata
> caching design on the other hand is going to be hard, and as Donald
> suggests, the right ecosystem level solution might be to define
> install-time hooks for package installation operations.
> 
> > > I’m also in favor of this. Although I would suggest SQLite rather than a
> > > JSON file for the primary reason being that a JSON file isn’t
> > > multiprocess safe without being careful (and possibly introducing
> > > locking) whereas SQLite has already solved that problem.
> >
> > SQLite was actually my first thought, but from experience in Jupyter &
> > IPython I'm wary of it - its built-in locking does not work well over
> > NFS, and it's easy to corrupt the database. I think careful use of
> > atomic writing can be more reliable (though that has given us some
> > problems too).
> >
> > That may be easier if there's one cache per user, though - we can
> > perhaps try to store it somewhere that's not NFS.
> >
> 
> I'm wondering if rather than jumping straight to a PEP, it may make sense
> to instead initially pursue this idea as a *non-*standard, implementation
> dependent thing specific to the "entrypoints" project. There are a *lot* of
> challenges to be taken into account for a truly universal metadata caching
> design, and it would be easy to fall into the trap of coming up with a
> design so complex that nobody can realistically implement it.
> 
> Specifically, I'm thinking of a usage model along the lines of the
> updatedb/locate pair on *nix systems: `locate` gives you access to very
> fast searches of your filesystem, but it *doesn't* try to automagically
> keeps its indexes up to date. Instead, refreshing the indexes is handled by
> `updatedb`, and you can either rely on that being run automatically in a
> cron job, or else force an update with `sudo updatedb` when you want to use
> `locate`.
> 
> For a project like entrypoints, what that might look like is that at
> *runtime*, you may implement a reasonably fast "cache freshness check",
> where you scanned the mtime of all the sys.path entries, and compared those
> to the mtime of the cache. If the cache looks up to date, then cool,
> otherwise emit a warning about the stale metadata cache, and then bypass it.
> 
> The entrypoints project itself could then expose a
> `refresh-entrypoints-cache` command that could start out only supporting
> virtual environments, and then extend to per-user caching, and then finally
> (maybe) consider whether or not it wanted to support installation-wide
> caches (with the extra permissions management and cross-process and
> cross-system coordination that may imply).
> 
> Such an approach would also tie in nicely with Donald's suggestion of
> reframing the ecosystem level question as "How should the entrypoints
> project request that 'refresh-entrypoints-cache' be run after every package
> installation or removal operation?", which in turn would integrate nicely
> with things like RPM file triggers (where the system `pip` package could
> set a file trigger that arranged for any properly registered Python package
> installation plugins to be run for every modification to site-packages
> while still appropriately managing the risk of running arbitrary code with
> elevated privileges)
> 
> Cheers,
> Nick.
> 

I have been trying to find time to do something like that within
stevedore for a while to solve some client-side startup performance
issues with the OpenStack client. I would be happy to help add it
to entrypoints instead and use it from there.

Thomas, 

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Donald Stufft]

> On Oct 20, 2017, at 9:30 AM, Paul Moore  wrote:
> 
> On 20 October 2017 at 14:26, Matthew Brett  wrote:
>> Thanks for the heads-up.
>> 
>> Will y'all be doing a PyPI pre-release so we can test with `pip
>> install --pre -U pip`?
> 
> We've not yet decided on that. Traditionally I don't think we have
> done so, but I'm inclined to think it's a good idea. It might not be
> until noticeably closer to the release, though…
> 

I used to cut pre-releases for pip, and after awhile I gave up on doing them 
because it felt like nobody ever actually reported any issues with them 
anyways, and it wasn’t until we cut the final release that we started finding 
bugs with them. I don’t have any problem with us starting to issue them again 
though and seeing if we start catching issues earlier this time.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[xoviat]

There's no dry-run functionality that I know of so far. However, you could
use the following:

pip install --prefix=tmpdir

This command is actually about the same speed as a proper implementation,
because we can't actually know what we're installing until we build the
requirements.

2017-10-20 12:42 GMT-05:00 Noah Kantrowitz :

> So as someone on the tooling side, is there any kind of install dry-run
> yet? I've got https://github.com/poise/poise-python/blob/master/lib/
> poise_python/resources/python_package.rb#L34-L78 which touches a tn
> of internals. Basically I need a way to know exactly what versions `pip
> install` would have used in a given situation without actually changing the
> system. Happy for a better solution!
>
> --Noah
>
> > On Oct 20, 2017, at 6:22 AM, Paul Moore  wrote:
> >
> > We're in the process of starting to plan for a release of pip (the
> > long-awaited pip 10). We're likely still a month or two away from a
> > release, but now is the time for people to start ensuring that
> > everything works for them. One key change in the new version will be
> > that all of the internal APIs of pip will no longer be available, so
> > any code that currently calls functions in the "pip" namespace will
> > break. Calling pip's internal APIs has never been supported, and
> > always carried a risk of such breakage, so projects doing so should,
> > in theory, be prepared for such things. However, reality is not always
> > that simple, and we are aware that people will need time to deal with
> > the implications.
> >
> > Just in case it's not clear, simply finding where the internal APIs
> > have moved to and calling them under the new names is *not* what
> > people should do. We can't stop people calling the internal APIs,
> > obviously, but the idea of this change is to give people the incentive
> > to find a supported approach, not just to annoy people who are doing
> > things we don't want them to ;-)
> >
> > So please - if you're calling pip's internals in your code, take the
> > opportunity *now* to check out the in-development version of pip, and
> > ensure your project will still work when pip 10 is released.
> >
> > And many thanks to anyone else who helps by testing out the new
> > version, as well :-)
> >
> > Thanks,
> > Paul
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Thomas Kluyver]

On Fri, Oct 20, 2017, at 07:24 PM, Doug Hellmann wrote:
> I have been trying to find time to do something like that within
> stevedore for a while to solve some client-side startup performance
> issues with the OpenStack client. I would be happy to help add it
> to entrypoints instead and use it from there.
> 
> Thomas, please me know how I can help.

Thanks Doug! For starters, I'd be interested to hear any plans you have
for how to tackle caching, or any thoughts you have on the rough plan I
described before. If you're happy with the concepts, I'll have a go at
implementing it. I'll probably consider it experimental until there's a
hooks mechanism to trigger rebuilding the cache when packages are
installed or uninstalled.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Marius Gedminas]

On Fri, Oct 20, 2017 at 08:10:06AM -0400, Donald Stufft wrote:
> Packaging tools shouldn’t be expected to know anything about it other
> than the console_scripts feature

Please do not forget about gui_scripts entry points!

Marius Gedminas
-- 
What can I do with Python that I can't do with C#?  You can go home on time at
the end of the day.
-- Daniel Klein


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Thomas Kluyver]

On Fri, Oct 20, 2017, at 07:31 PM, Marius Gedminas wrote:
> Please do not forget about gui_scripts entry points!

I haven't forgotten about them in the draft spec:
https://github.com/pypa/python-packaging-user-guide/pull/390/files#diff-089b079de062f6fdb759bb719b79e6c8R121
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[xoviat]

A correct dry-run implementation will do about the same amount of work as
installing to a temporary directory right now. In the future, that could be
optimized, but any patch to the finder doesn't actually detect the
requirements correctly (as they're not necessarily known until after the
wheels are built).

2017-10-20 13:41 GMT-05:00 Noah Kantrowitz :

> Installing to a temp dir is really not an option for automated tooling (if
> nothing else, it takes way too long). `pip list --outdated` does already
> get fairly close to this (and doesn't install anything I suspect you can
> actually get a lot closer than you think) but it calculates for all
> packages (read: is slow) and doesn't give a good way to restrict things
> (hence that hack-y script which is a modified version of the pip list
> code). This is 100% a hard requirement for config management systems and if
> not fixed in pip, will require continued use of internal APIs. I would
> recommend just making pip list take a set of install-compatible
> names/version patterns and apply that as a filter in a similar way to what
> I've done there.
>
> --Noah
>
> > On Oct 20, 2017, at 11:35 AM, xoviat  wrote:
> >
> > There's no dry-run functionality that I know of so far. However, you
> could use the following:
> >
> > pip install --prefix=tmpdir
> >
> > This command is actually about the same speed as a proper
> implementation, because we can't actually know what we're installing until
> we build the requirements.
> >
> > 2017-10-20 12:42 GMT-05:00 Noah Kantrowitz :
> > So as someone on the tooling side, is there any kind of install dry-run
> yet? I've got https://github.com/poise/poise-python/blob/master/lib/
> poise_python/resources/python_package.rb#L34-L78 which touches a tn
> of internals. Basically I need a way to know exactly what versions `pip
> install` would have used in a given situation without actually changing the
> system. Happy for a better solution!
> >
> > --Noah
> >
> > > On Oct 20, 2017, at 6:22 AM, Paul Moore  wrote:
> > >
> > > We're in the process of starting to plan for a release of pip (the
> > > long-awaited pip 10). We're likely still a month or two away from a
> > > release, but now is the time for people to start ensuring that
> > > everything works for them. One key change in the new version will be
> > > that all of the internal APIs of pip will no longer be available, so
> > > any code that currently calls functions in the "pip" namespace will
> > > break. Calling pip's internal APIs has never been supported, and
> > > always carried a risk of such breakage, so projects doing so should,
> > > in theory, be prepared for such things. However, reality is not always
> > > that simple, and we are aware that people will need time to deal with
> > > the implications.
> > >
> > > Just in case it's not clear, simply finding where the internal APIs
> > > have moved to and calling them under the new names is *not* what
> > > people should do. We can't stop people calling the internal APIs,
> > > obviously, but the idea of this change is to give people the incentive
> > > to find a supported approach, not just to annoy people who are doing
> > > things we don't want them to ;-)
> > >
> > > So please - if you're calling pip's internals in your code, take the
> > > opportunity *now* to check out the in-development version of pip, and
> > > ensure your project will still work when pip 10 is released.
> > >
> > > And many thanks to anyone else who helps by testing out the new
> > > version, as well :-)
> > >
> > > Thanks,
> > > Paul
> > > ___
> > > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[xoviat]

+1 on pre-release wheels. I've seen the process in action
with SciPy, and it helped to catch at least one significant bug.

2017-10-20 8:30 GMT-05:00 Paul Moore :

> On 20 October 2017 at 14:26, Matthew Brett 
> wrote:
> > Thanks for the heads-up.
> >
> > Will y'all be doing a PyPI pre-release so we can test with `pip
> > install --pre -U pip`?
>
> We've not yet decided on that. Traditionally I don't think we have
> done so, but I'm inclined to think it's a good idea. It might not be
> until noticeably closer to the release, though...
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Alex Grönholm]

Yeah, +1 from me too. Pip is one project where people will highly likely 
try out the pre-release versions.



xoviat kirjoitti 20.10.2017 klo 20:34:

+1 on pre-release wheels. I've seen the process in action
with SciPy, and it helped to catch at least one significant bug.

2017-10-20 8:30 GMT-05:00 Paul Moore >:


On 20 October 2017 at 14:26, Matthew Brett
> wrote:
> Thanks for the heads-up.
>
> Will y'all be doing a PyPI pre-release so we can test with `pip
> install --pre -U pip`?

We've not yet decided on that. Traditionally I don't think we have
done so, but I'm inclined to think it's a good idea. It might not be
until noticeably closer to the release, though...

Paul
___
Distutils-SIG maillist  - Distutils-SIG@python.org

https://mail.python.org/mailman/listinfo/distutils-sig





___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Doug Hellmann]

Excerpts from Thomas Kluyver's message of 2017-10-20 19:37:45 +0100:
> On Fri, Oct 20, 2017, at 07:24 PM, Doug Hellmann wrote:
> > I have been trying to find time to do something like that within
> > stevedore for a while to solve some client-side startup performance
> > issues with the OpenStack client. I would be happy to help add it
> > to entrypoints instead and use it from there.
> > 
> > Thomas, please me know how I can help.
> 
> Thanks Doug! For starters, I'd be interested to hear any plans you have
> for how to tackle caching, or any thoughts you have on the rough plan I
> described before. If you're happy with the concepts, I'll have a go at
> implementing it. I'll probably consider it experimental until there's a
> hooks mechanism to trigger rebuilding the cache when packages are
> installed or uninstalled.
> 
> Thomas

I assumed that the user loading the plugins might not be able to
write to any of the directories on sys.path (aside from "." and we
don't want to put a cache file there), so my plan was to build the
cache the first time entry points were scanned and use appdirs [1]
to pick a cache location specific to the user.  I thought I would
use the value of sys.path as a string (joining the paths together
with a separator of some sort) to create a hash for the cache file
ID. Some of that may be obviated if we assume a setuptools hook that
lets us update the cache(s) when a package is installed.

I also thought I'd provide a command line tool to generate the cache
just in case it became corrupted or if someone wanted to update it
by hand for some other reason, similar to Nick's locate/updatedb
parallel UX example (and re-reading your email, I see you mention this,
too).

I hadn't gone as far as deciding on a file format, but sqlite, JSON,
and INI (definitely something built-in) were all on my mind.  I
planned to see if we would actually gain enough of a boost just by
placing a separate file for each dist in a single cache directory,
rather than trying to merge everything into one file. In addition
to eliminating the concurrency issue, that approach might have the
additional benefit of simplifying operating system packages, because
they could just add a new file to the package instead of having to
run a command to update the cache when a package was installed (if
the file is the same format as entry_points.txt but with a different
name, that's even simpler since it's just a copy of a file that
will already be available during packaging).

Your idea of having a cache file per directory on sys.path is also
interesting, though I have to admit I'm not familiar enough with
the import machinery to know if it's easy to determine the containing
directory for a dist to find the right cache to update. I am
interested in hearing more details about what you planned there.

I would also like to compare the performance of a few approaches
(1 file per sys.path hash using INI, JSON, and sqlite; one file per
entry on sys.path using the same formats) using a significant number
of plugins (~100?) before we decide.

I agree with your statement in the original email that applications
should be able to disable the cache. I'm not sure it makes sense
to have a mode that only reads from a cache, but I may just not see
the use case for that.

What's our next step?

Doug

[1] https://pypi.python.org/pypi/appdirs/1.4.3
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig