[Distutils] Re: New packaging security funding & NYU

[Robert Collins]

Indeed congratulations!

Rob

On Sat, 20 Mar 2021, 04:29 Sumana Harihareswara,  wrote:

> Good news!
>
> New York University -- specifically Professor Justin Cappos -- and I
> have successfully asked the US National Science Foundation for a grant
> to improve Python packaging security. The NSF is awarding NYU $800,000
> over two years -- from mid-2021 to mid-2023 -- to further improve the
> pip dependency resolver and to integrate The Update Framework further
> into the packaging toolchain.
>
> https://nsf.gov/awardsearch/showAward?AWD_ID=2054692=false
>
> For what we're planning to do, what this means in the short term, an
> explanation of why NYU and the NSF are involved, and thank-yous, please
> see https://discuss.python.org/t/new-packaging-security-funding-nyu/7792 .
>
> --
> Sumana Harihareswara
> Changeset Consulting
> https://changeset.nyc
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/MUH254XTCE5EUL5YJV7ZD6HSUYNFXUD6/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/QCDWOAMIPCM5XD2YRAEO77CFUKTVNB47/

[Distutils] Re: Archive this list & redirect conversation elsewhere?

[Robert Collins]

On Thu, 30 Jul 2020 at 14:52, Sumana Harihareswara  wrote:

> On 7/29/20 10:14 PM, Jeremy Stanley wrote:
> > On 2020-07-30 07:17:03 +0530 (+0530), Pradyun Gedam wrote:
> >> TL;DR: OK to archive this mailing list? Reply by Aug 30th.
> > [...]
> >
> > I find it disappointing that there will no longer be a mailing list
> > for discussions of Python packaging. Web forums with some E-mail
> > integration are hardly the same. But those of us who still use
> > E-mail (and worse, Usenet) eventually need to get out of the way of
> > the wheels of progress lest they run us over.
> >
> > Many thanks to those who have maintained, moderated, and
> > collaborated through this list over the years. It has been much
> > appreciated.
>
> Jeremy, I'm not sure whether you were serious? If your disappointment is
> only out of nostalgia, then yeah, accepting change makes sense. But if
> your disappointment is because the Discourse experience is/will be worse
> for your participation, then it's totally fine to speak up and tell us how.
>

Discourse requires about 10x the effort to participate in the community.
It's "mailing list mode" sends garbled fragments of interwoved WYSIWYG
documents that are unintelligible - I tried it for some months when the
Pyython Discourse started up but had to turn it off after a while as I
really couldn't effectively tell what was being said in a conversation that
comes in via it.

And its so siloed, I may as well not be in the conversation at all: I have
to actively go and log into the website to look and see what new
conversations are happening, which in my time poor situation just doesn't
happen. So, for all intents and purposes, I'm not participating in any
conversation in Discourse at all, except for a rare helicopter drop-in when
someone pings me on email or Twitter or Slack or Discord to say 'hey, you
should comment on '.

I full well recognise the advantage that these properties have when dealing
with a bulk of (largely) newcomers whose community use case is to sample
discussion: to find one or two things that have been said previously via
search, ask some questions to get a problem solved, and then move on.

Relatively few of those users will be publishing packages though; even with
the rise of docker : consuming Python in a local script or workbook is
still the majority use case I think, so the bulk of the work we do affects
a (large) fraction of users, and most of those users are experienced by the
time they need our assistance.

Pradyun, thanks for starting this conversation.
>
> I am definitely interested in consolidating our conversational channels
> and reducing fragmentation, but I have substantial reservations about
> taking this particular step:
>
> * The majority of information overwhelm in my PyPA-related life is
> because of GitHub repo and issue sprawl -- if we're going to put energy
> into pruning sprawling communications venues, I would prefer that we
> spend some time inventorying all the teams, shutting some down, and
> locking noisy issues/repositories.
>

Agreed with the above.


> * I would like to know, of our ~700 list members, how many of them have
> serious problems using Discourse -- accessibility, user experience,
> sheer tech problems, etc. I suspect that we have several members in that
> category, some who contribute to packaging, some who lurk so they can
> stay apprised and bridge to other communities (distributions, major
> packages, etc.).
>

I have contributed a fair degree in the past; I'm largely if not entirely
emeritus at this point - I get to code only from time to time in my day
job, and then it is rarely Python. I like to stay in touch, both because I
can provide some institutional continuity, but also I do enjoy helping from
time to time, when I can.

I hope these thoughts are useful.

-Rob

On Discourse I've seen
> https://discuss.python.org/t/disappointed-and-overwhelmed-by-discourse/982
> , https://discuss.python.org/t/if-mailing-list-mode-were-better/3951 ,
> and https://discuss.python.org/t/e-mail-settings-are-not-respected/396
> talking about problems people have had keeping up with/watching and
> participating in conversations on Discourse -- including Paul Moore and
> Paul Ganssle, whose opinions I really want to hear from here. I believe
> I've heard Dan Ryan say that he finds Discourse practically unusable,
> and I'd like to hear from him as well.
>
> * There are some things I don't like about how Discourse shapes our
> conversations. Some examples: I think people are chattier on Discourse,
> posting shorter replies more frequently, and that's not always good. In
> the email notifications, Discourse preserves threading so I can see
> better who's replying to whom, but the web view is flat which makes that
> harder to see. And -- as came up in
>
> https://discuss.python.org/t/pep-458-secure-pypi-downloads-with-package-signing/2648/30
> -- people use the heart/"like" button in different ways that have led to
> confusion. “Liking” a post on 

[Distutils] Why lockbot?

[Robert Collins]

Seems like most of the pip bugmail I get now is lockbot messages
telling me that a bug that hasn't received any discussion for a long
time now can't have any more discussion. Is that really needed? The
github UI shows the lock status of bugs itself...

-Rob
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/2Z4THA3D3JQSUYVK7YNKVYAKHVCZJZ7Y/

[Distutils] Re: PEP 541 Regarding emd

[Robert Collins]

On 4 August 2018 at 03:34, Chathika Gunaratne
 wrote:
> Dear Admin,
>
>
>
> I am writing to you regarding the Package Index name emd. I am a PhD student
> at the University of Central Florida and for my dissertation, I have
> recently developed a Python software, Evolutionary Model Discovery, which I
> am assembling into a package, EMD. https://github.com/chathika/EMD
>
>
>
> However, I noticed that there is already a package named emd on pypi.org.
> This package seems abandoned with just one release in February 2012 and no
> contact information about the author,
> https://pypi.org/project/emd/1.0/#history .
>
>
>
> I read about PEP 541 https://www.python.org/dev/peps/pep-0541/ and was
> curious as to if it is still possible that I can acquire the package name
> EMD. I wanted to make sure of this before I went ahead with the name EMD for
> my package, please.

Generally speaking its not whether a package has been updated that
matters, its whether the package is in use: if its published in
debian/suse/fedora, if its getting downloads, then taking over the
name is going to have negative consequences.

That said, this has had 0 downloads per the bigtable statistics (0 for
all recorded time) so in this case that seems unlikely.

-Rob
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QL5V6LJN2ML624OEIXQTENQA5JS2F4JQ/

Re: [Distutils] Any interest in Perforce support in pip?

[Robert Collins]

It won't be testable by most developers, if be inclined to say that making
the VCS layer a supported plugin point would be a better patch to merge.
That would be testable.

On 29 Jan. 2018 13:57, "Barney Gale"  wrote:

> Hi all,
>
> Is there any interest in support for Perforce in pip? Perforce is a
> commercial version control system used in mostly corporate environments.
> It's also used for FreeBSD development. I have a pull request against the
> pypa/pip project (#4857) that allows the following:
>
> pip install p4+p4://host/depot/path
>
> ... but as the pip developers lack experience with Perforce, I've been
> directed to this mailing list to gauge whether this is a feature worth
> adding. I'm aware that Perforce is used in the game and film industries,
> but beyond a github issue requesting support (#2754) I'm not sure who else
> might use this! I'd appreciate any feedback. Thanks.
>
> Barney
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup.cfg - to interpolate or not to interpolate

[Robert Collins]

On 28 July 2017 at 15:34, Nick Coghlan  wrote:
> On 28 July 2017 at 13:30, Nick Coghlan  wrote:
>> On 28 July 2017 at 01:32, Jason R. Coombs  wrote:
>>> I’d like feedback, in either ticket, about whether this change is worth the
>>> trouble and if so if there are any ways to mitigate the risks of introducing
>>> such a change.
>>
>> The one way I can see of doing this with a graceful transition period
>> would be to implement it like this:
>>
>> 1. Switch to raw parsing with no interpolation as the default
>> 2. Check the result for any field values containing either "%()s" or 
>> "%%"
>> 3. If you find any, print a suitable deprecation warning and parse it
>> again with interpolation enabled
>
> Clarifying to add: I think this is a worthwhile change, as it helps to
> ensure that the static config files actually *are* static, and hence
> less dependent on being read specifically with Python's configparser
> library. That's a boon for interoperability, even if it means that
> folks that genuinely want interpolation will need to switch to a
> "setup.cfg.in" style approach where they use the templating tool of
> their choice to read in the input file, substitute values, and then
> write out a completely static setup.cfg file.

Yeah, I'm going to disagree here. While its true that using
interpolation binds the config file semantic interpretation to Python,
its *already* bound that way due to the idiosyncratic continuations
style and nested sections.

Doing a .in -> actual-foo is a great pattern for build systems
building expensive artifacts with complex dependency chains. Not
problems have, and copying their solutions without the problem is just
unnecessary, and unhelpful, complexity.

+1 from me for retaining interpolation.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Robert Collins]

On 1 July 2017 at 22:53, Nathaniel Smith  wrote:
> Hi all,

> If either hook is missing, or returns the built-in constant
> ``NotImplemented``. (Note that this is the object ``NotImplemented``,
> *not* the string ``"NotImplemented"``),

thank you for the clarification.

I am unclear why you *return* that rather than raising
NotImplementedError ? NotImplementedError permits embedding details
about the cause of the failure, whereas the singleton does not.

It seems to me cleaner - thinking in a type sense - to raise than to
return a value from a different domain.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Robert Collins]

Re: returning magic strings to indicate exceptions. Please no. Just no.
Have any pep build host add a small library to Python path with any symbols
we want to define. Hardly an implementation hurdle.

Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The sad and insecure state of commercial private package indexes

[Robert Collins]

On 24 April 2017 at 17:10, Nick Coghlan  wrote:
> On 22 April 2017 at 21:05, Donald Stufft  wrote:
>> I think the biggest barrier to doing it in pip is simply the UX of it. We’re
>> currently constrained by the fact that *all* of our options are available as
>> CLI flags, environment variables, and of course, a config file. This works
>> great for simple key, value configuration but it breaks down with more
>> complex situations like trying to assign a priority to different
>> repositories or selecting which repository a particular package *should*
>> come from (and other more complex situations).
>>
>> Thus far we’ve more or less stuck our fingers in our ears and focused on
>> other problems, but I think we’re going to end up needing to refactor the
>> way pip handles configuration to really make this sort of thing sane.
>
> As much as it annoys me in other ways, the `/etc/yum.repos.d/*`
> approach to managing named repositories seems hard to beat in terms of
> allowing for per-repo configuration settings while limiting command
> line complexity (since the command line mainly deals with repo names
> rather than the individual settings). Debian offers something similar
> now in the form of `/etc/apt/sources.list.d/`.
>
> It doesn't automatically solve the complexity problem around pulling
> components from multiple repositories, but it helps makes the
> complexity more manageable since:
>
> - remote repositories are explicitly modeled as named entities with
> various configurable characteristics
> - other commands then work with the local entity names rather than
> directly with remote URLs
>
> Something I *don't* think either yum/dnf or apt handle well is the
> notion of *relative* priorities for a single command, since they're
> built around the notion of associating static numeric priorities with
> the repository definitions, whereas for development purposes, you
> usually want to express relative priorities like:
>
> --indices=project-specific,org-common,pypi # Continuous deployment
> --indices=dev,staging,stable,pypi # Phased deployment, dev
> --indices=staging,stable # Phased deployment, CI/staging
> --indices=stable # Phased deployment, production

Well apt and similar tools are working in a system global context. So
pipeline stage based rules are a poor fit for the use of system
packages. Of course if you use a system-in-a-container approach you
can have both worlds, which a lot of folk do.

That said apt pinning is pretty capable and can pick arbitrary
packages in arbitrary order across arbitrary repositories.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The Python Packaging Ecosystem (of Nick) Support for other programming languages

[Robert Collins]

I wrote this 5 years ago, and its largely still true as far as I can
tell of the surrounding systems. Snappy-core for instance, isn't
targeting MacOS X or Windows (last I checked anyhow ...)

https://rbtcollins.wordpress.com/2012/08/27/why-platform-specific-package-systems-exist-and-wont-go-away/

-Rob

On 7 April 2017 at 02:32, Thomas Güttler  wrote:
> Dear Nick and other distutils listeners,
>
> Nick wrote this about seven months ago:
>
> http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html
>
> I love Python and I use it daily.
>
> On the other hand there are other interesting programming languages out
> there.
>
> Why not do "thinking in sets" here and see python just as one item in the
> list of a languages?
>
> Let's dream: All languages should be supported in the ever best packaging
> solution of the future.
>
> What do you think?
>
> Regards,
>   Thomas
>
>
>
> --
> Thomas Guettler http://www.thomas-guettler.de/
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] RFC: PEP 541 - Package Index Name Retention

[Robert Collins]

So, this proposition didn't really make sense to me. Folk like Linux
distros will want the source, and you don't need to upload wheels :-
setup.py could quite reasonably limit itself to software installation, vs
configuration. Plenty of pip installable packages are not entirely ready to
use after pip installation.

-Rob

On 17 Jan 2017 08:29, "Dariusz Suchojad"  wrote:

> On 16/01/17 22:19, Nick Timkovich wrote:
> > If you have a non-release release with some description text and a
> > home-page that points to where active development is going on (that could
> > constitute "functionality" in a non-code way), I think that should
> preempt
> > a reasonable person (which is hopefully a superset of maintainers) from
> > deleting it.
>
> Yes, indeed, this is what I'd like to clarify. Someone is simply bound
> to clean up all the packages at one point and this PEP likely will be
> the basis for deciding what to delete or not so I'd very much like to
> ensure ours will not be removed.
>
> regards,
>
> --
> Dariusz Suchojad
>
> https://zato.io
> ESB, SOA, REST, APIs and Cloud Integrations in Python
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Different purposes for Setuptools requirements versus Pip requirements (was: Maintaining a curated set of Python packages)

[Robert Collins]

On 9 Dec 2016 5:45 PM, "Donald Stufft"  wrote:


On Dec 8, 2016, at 10:41 PM, Ben Finney  wrote:

For those who haven't read it, see this post from Donald Stufft for why
those purposes need to be kept distinct


Somewhat funnily, pbr is what triggered me to actually write that post :)

If I recall, at the time it only supported requirements.txt but it’s since
then gotten support for putting it in setup.cfg and I think that is
preferred now?


It always had support for install requires in setup.cfg from its d2to1
heritage.

That said early days of pbr were confused about a bunch of things... It's
much happier and less layer breaking than it was... :)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Different purposes for Setuptools requirements versus Pip requirements (was: Maintaining a curated set of Python packages)

[Robert Collins]

On 9 Dec 2016 4:42 PM, "Ben Finney"  wrote:

Jeremy Stanley  writes:

> [the ‘pbr’ library] does allow you to basically abstract away most
> common configuration into declarative setup.cfg and requirements.txt
> files

Hmm. That description sounds like a mistaken conflation of two things
that should be distinct


It's not, though the name of the file it looks for is indeed evocative of
concrete version lists. Requirements.txt was on the path to being
deprecated in order to reduce confusion, but I'm no longer paid to work on
that tool chain, so it's now in my lengthy best effort to-do list.

Requirements.txt files are mapped directly into install-requires by pbr,
and the replacement is to put the same dependencies into setup.cfg.

* ...
If we're saying ‘pbr’ encourages the use of a single set of declarations
for those quite different purposes, that sounds like an attractive
nuisance


It doesn't, confusing names aside. We even wrote a tool  -
generate-constraints to calculate the transitive closure, python version
specific as needed, for placing into a constraints / requires file for pip
to consume.

Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Robert Collins]

On 29 November 2016 at 07:23, Daniel Holth  wrote:
> Here is the design. The build tool provides the install tool with a
> directory, or if you want to get fancy, a list of directories, that should
> be added to PYTHONPATH. The build tool builds into that path. The install
> tool makes sure Python can find it. This will make anyone who uses
> setuptools happy.
>
> So PEP 517 does not need an editable install hook to support editable
> installs. It can just expose the equivalent of setuptools'
> package_dir={'':'src'}, and include a 'build in place' hook, and the feature
> is preserved.
>
> In this way setuptools users can be happy, and only users of build systems
> that can't work this way are waiting for a more complicated design.

Thats a possible route; note that some situations don't work with
in-place builds, and I don't see any reason to hardcode in-place
builds as the thing; this is why I was trying to punt on it, because
-e is actually somewhat deep in its own right.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Robert Collins]

On 25 November 2016 at 14:04, Nick Coghlan  wrote:
..
>
> The bad reputation of ".pth" doesn't generally stem from their normal
> usage (i.e. just listing absolute or relative directory names that the
> import system will then append to __path__).
>
> Rather, it stems from this little aspect: "Lines starting with import
> (followed by space or tab) are executed." (from
> https://docs.python.org/3/library/site.html )

I think its also worth exploring a targeted, modern namespace aware replacement.

That is - there are two, related, use cases for .pth files vis-a-vis
package installation. One is legacy namespace packages, which AFAICT
are still in use - the migration is "complicated". The second is
arguable a variant of that same thing: putting the current working dir
into the PYTHONPATH without putting it in PYTHONPATH.

The former case may be sufficiently covered by (forget the pep #) that
we don't need to do anything, and the latter is certainly something
that we should be able to deliver without needing the turing complete
capability that you're suggesting. Something data driven rather than
code driven.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Robert Collins]

On 3 November 2016 at 22:10, Nathaniel Smith  wrote:
> On Nov 3, 2016 1:40 AM, "Nick Coghlan"  wrote:

> And then it segfaults because it turns out that your library named  is
> not abi compatible with my library named . Or it would have been if you
> had the right version, but distros don't agree on how to express version
> numbers either. (Just think about epochs.) Or we're on Windows, so it's
> interesting to know that we need a library named , but what are we
> supposed to do with that information exactly?

Well, we weren't trying to fix the problem with incompatible ABI's:
the thoughts that I recall were primarily around getting development
header files in place, to permit building (and then caching) local
binary wheels. The ports system for example, works very very robustly,
even though (it used to) require building everything from scratch.
That was my inspiration.

The manylinux approach seems better to me for solving the 'install a
binary wheel in many places' problem, because of the issues with
variance across higher layered libraries you mention :).

> Again, I don't want to just be throwing around stop energy -- if people want
> to tackle these problems then I wish them luck. But I don't think we should
> be hand waving this as a basically solved problem that just needs a bit of
> coding, because that also can create stop energy that blocks an honest
> evaluation of alternative approaches.

+1.
...> dnf/apt/pacman/chocolatey/whatever and make my wheel work everywhere -- and
> that this will be an viable alternative to conda.

As a distribution of sources, I think its very viable with the
approach above: indeed we do similar things using bindep in OpenStack
and that seems to be working out pretty well.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [proposal] module dependency specification and overriding for packages

[Robert Collins]

So I realise this isn't the specific thing you're dealing with, but
you could use https://pypi.python.org/pypi/junitxml which is mature,
has no external dependencies, and as a LGPL module should be license
compatible with py.test.

-Rob

On 9 September 2016 at 19:48, Ronny Pfannschmidt
 wrote:
>
>
> On 09.09.2016 09:37, Philippe Ombredanne wrote:
>>
>> On Fri, Sep 9, 2016 at 8:46 AM, Ronny Pfannschmidt
>>  wrote:
>>>
>>> while trying to vendor packages that depend on six, i noticed a
>>> problematic
>>> implication
>>>
>>> it is impossible to do this without patching those modules, as such
>>> adding a
>>> fragile maintenance burden.
>>
>>
>> Ronny:
>> I grok the general issue, but would you have a concrete example?
>
> a basic example would be vendoring
>
> https://github.com/kyrus/python-junit-xml and six into pytest
>
> following that we would face 2 problems
>
> a) python-junit-xml imports plain six instead of
> _pytest.vendored_packages.six meaning we have to patch it when including it
> b) debian/fedora will want to remove six and python-junit-xml from their
> pytest-package
> instead linking to python-six-wheel-... and python-junit-xml-wheel...
> packages as part of their deduplication effort
> meaning they will patch around in the pytest package (which tends to break
> the world)
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pep 440 question/clarification Version specifiers no "Or" available

[Robert Collins]

On 9 September 2016 at 15:10, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 9 Sep 2016 12:32 pm, "Robert Collins" <robe...@robertcollins.net> wrote:

>> tl;dr the ramifications of OR are much deeper than those of AND for
>> this problem space, and we don't even have AND solved properly - Issue
>> 988.
>
> Disjoint environment markers provide most of the capabilities we actually
> need (like installing one dependency on Py2 and a different one on Py3)
> without ever forcing the resolver to make an arbitrary choice between them.

Yup :) and we've got that pretty solidly adoptable now - chunk of work
that it was ;).

> There are currently some bugs even in that, though - installing an sdist or
> wheel directly instead of via a requirements file bypasses the environment
> marker checks for its direct dependencies :(
>
> (Steve Kowalik was trying to figure that one out at the PyCon AU sprints,
> but I don't believe he was able to track down where the missing check should
> go)

I didn't mention it before but I should - its my view we should be
really conservative about an OR in the dependency language, for the
reasons in my earlier email.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pep 440 question/clarification Version specifiers no "Or" available

[Robert Collins]

Thats right, there is no 'or' operator available today.

In addition to what Donald says, Or significantly complicates the
resolution job for the resolver: should 'A OR B' prefer A? If B is
already installed should it be uninstalled to permit choosing A (aka
is it a disjunction? What if A and B can coexist and one part of the
graph wants A and one part wants B but there is an A OR B higher up? -
if its a disjunction that becomes uninstallable.

tl;dr the ramifications of OR are much deeper than those of AND for
this problem space, and we don't even have AND solved properly - Issue
988.

-Rob


On 9 September 2016 at 10:47, Matthias Bussonnier
 wrote:
> Hi all,
>
> In the line of recent discussions[1] about releasing some packages
> only for some versions of Python.
>
> I was reading pep 440 and in particular the "Version specifier section".
>
> The following sentence caught my eye:
>
>> The comma (",") is equivalent to a logical and operator: a candidate version 
>> must match all given version clauses in order to match the specifier as a 
>> whole.
>
>
> Is there no way to have an "or" ? It seem to me that it might be
> useful for package wanting to express compatibility with `2.6`, `2.7`
> or `>3.3` for example, in the form `=~2.6  =~3.3`.
>
> I completely agree that the use case is _limited_ and likely rare, I
> was just wondering if I'm missing something, if it was an oversight
> and if not if adding a section detailing that "or" was not considered,
> or that "or" was explicitly not included for X,Y,Z reason would be a
> good thing.
>
> Thanks,
> --
> Matthias
>
>
>
> [1]:https://mail.python.org/pipermail/distutils-sig/2016-August/029604.html
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Robert Collins]

My sense of the current position is:

 - distutils exists for both Python and the broader ecosystem to build
both extension modules and package distributions
 - we recognise that its able to do anything, but many folk find it
hard to work with - or are already invested in other build tools
 - we're afraid of changing it in ways that cause incompatibilities
with e.g. the numpy eco systems heavy customisations, and we're not
sure we can recognise whether a given change is one of those
 - and we're recommending that everyone move away from directly using
distutils to using setuptoools - but setuptools *is* distutils in many
many ways, so this doesn't loosen our dependency and stability
considerations around distutils

So basically the normal 'module goes to the stdlib to freeze' effect.

I haven't read about Sylvain's proposed changes yet - ETHINGSTODO :(.

I can happily agree with a position that making distutils directly
usable is a non-goal... but insofar as making it more usable makes
setuptools more usable, that would be a goal :/.

-Rob




On 29 August 2016 at 05:05, Brett Cannon  wrote:
> The discussion of Sylvain's proposed changes to distutils suggests that
> there isn't a clear-cut agreement or position of this SIG -- and thus Python
> -- on changes to distutils, its future, etc. Is there an official position
> I'm not aware of? If not, could we get one so we know how to handle any more
> proposed changes to distutils?
>
> For me personally (and to start this conversation if necessary), I view
> distutils as the package in the stdlib that handles building extension
> modules in the stdlib for Python. That view means that if something doesn't
> service that explicit goal then it shouldn't go into distutils. Now that we
> are moving down the road towards making the build step configurable I'm fine
> with saying that distutils is not expected to work for people other than
> Python itself and others can use setuptools, enscons, or other projects
> while we continue to improve the situation to where the build system is just
> something you specify in pyproject.toml to generate your wheel. IOW
> distutils is only exposed publicly because Python doesn't hide anything, but
> making it useful for the general case of direct usage by people is a
> non-goal of the package.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] unittest2 issue tracker?

[Robert Collins]

That is the right place for issues with the backport; note that as a
backport new features and issues that also exist in CPython's
master/tip branch should be reported to the python bugtracker, where
Paul pointed you.

Cheers,
Rob

On 22 August 2016 at 21:25, Cosimo Lupo  wrote:
> Hello,
>
> Sorry if this is not the right place to ask this sorts of questions.
>
> The PyPI page for unittest2 package links to
> https://code.google.com/archive/p/unittest-ext/issues as its "issue
> tracker", however the Google Code pages seems to be read-only nowadays, so I
> can't open new issues there.
>
> I've found an auto-exported Github clone for unittest-ext and opened a new
> issue there: https://github.com/testing-cabal/unittest-ext/issues/99
>
> However I'm not sure it's the right one, as there doesn't seem to be much
> activity in there.
>
> It would be good to know where I should report bugs for unittest2.
> Thank you!
>
> --
>
> Cosimo Lupo
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Robert Collins]

On 22 Aug 2016 4:18 AM, "Nick Coghlan"  wrote:
>
>
> Once we have a universal default, unilaterally changing that default
> gets easier, rather than harder - the main precedent being set here is
> that we *don't want* the default format to be platform dependent, we
> want there to be just one default.

Sorry for the separate reply, phone typing.

Monocultures are less flexible, so I'm assuming once we have Just One
format that bit rot in tooling will creep in and it will be harder and
harder to change, not easier.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Robert Collins]

On 22 Aug 2016 4:18 AM, "Nick Coghlan" <ncogh...@gmail.com> wrote:
>
> On 21 August 2016 at 18:21, Robert Collins <robe...@robertcollins.net>
wrote:
> > tl;dr: I think standardising on .tar.gz would be a rather shortsighted
> > thing to do, given how many Windows users Python has and how much of a
> > different supporting .zip makes for workflow on that platform - with
> > no negative impacts on any other platform.
>
> Once we have a universal default, unilaterally changing that default
> gets easier, rather than harder - the main precedent being set here is
> that we *don't want* the default format to be platform dependent, we
> want there to be just one default.
>
> My core assumption in recommending starting with tar.gz as that
> universal default is that Windows users will mostly be interacting
> with sdists through tooling that is closely linked to or at least
> originated in the Python ecosystem (easy_install, pip, PyPM, conda,
> Enthought Canopy, buildout, Christoph Gohlke's Windows installers),
> and that double-clicking a Python sdist in an Explorer window is not
> something most folks are in the habit of doing.

It is certainly wrong for me, when I'm using Windows, and when I've
shoulder surfed others also. E.g. at pycon sprints.

> By contrast, I *know* Linux distros are in the habit of pulling
> release tarballs from PyPI and feeding them directly into their
> release pipelines, so the potential for unanticipated breakage seems
> much higher there, and more likely to fall on community distros rather
> than on commercial Python redistributors (simply because there are
> more volunteers working on Linux based tooling than there are on
> WIndows developer tools).

Those pipelines are mature, poly format capable and centralized, containing
the damage.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Robert Collins]

On 21 August 2016 at 15:22, Nick Coghlan  wrote:
> On 21 August 2016 at 08:40, Chris Barker - NOAA Federal
>  wrote:
>>>
>>> If we're going through all this trouble, isn't it better just to jump to 
>>> .zip files like every other distribution format in existence?
>>
>> Yes. :-)
>
> zip is popular for *user facing* distribution formats, where the file
> format is exposed directly to end users, and end user applications
> (e.g. zipimport, ODF).
>
> That's not where tarballs shine, so it's not where they show up: tar
> is a backend infrastructure format, used for tool-to-tool
> communication, rather than user-to-user.
>
> The trajectory of distutils-sig for the past few years has been
> towards a world where sdists are primarily a tool-to-tool format for
> data interchange between publishing tools (e.g. twine), installation
> tools (e.g. pip) and redistribution tools (e.g. conda, RPM, deb,
> buildout), with wheels (and potentially eggs, or an equally zipimport
> friendly future derivative) being the preferred formats for working
> directly with Python software in a destination environment (whether
> that's a system integrator's build farm, a production software
> deployment, or a Python user's local system).

Jar's are zip's (~= wheels). *source* jars are also zips. (~= sdists).
I'm struggling to think of any two ways that .tar.gz is better than
.zip. The compression is ~=. .tar.lzma and friends can be radically
smaller whereas .zip has per-element compression, so thats one way,
but .tar.gz has no other benefits (it has other differences, but none
are relevant in the context of the sdist use cases AFAICT).

As far as the switching consts Donald raises, here's my take:
 - we're going to break a lot of peoples workflow no matter what: 10%
of a lot is a lot.
 - all setuptools in the while /can/ produce .zip's AIUI, so the
issues w.r.t. the long tail of Linux distros is entirely
uninteresting: the folk depending on the default behaviour will need
to be explict - but thats going to be the case for *everyone* [because
setuptools on windows is inconsistent with setuptools on Linux]: we
are basically telling everyone they have to hardcode to .zip until the
long tail of existing installs *everywhere* has gone away.
 - unzip and friends are as available on Linux distros as tar is.

tl;dr: I think standardising on .tar.gz would be a rather shortsighted
thing to do, given how many Windows users Python has and how much of a
different supporting .zip makes for workflow on that platform - with
no negative impacts on any other platform.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pip maintainers contact

[Robert Collins]

There is no email contact per se. The project tacker is here -
github.com/pypa/pip and there is a google group here -
https://groups.google.com/forum/#!forum/pypa-dev

On 13 August 2016 at 21:28, hyp3rlinx  wrote:
> Can somebody here please tell me the contact email for maintainers of pip.
>
> Thank you,
> John
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Request for comment: Proposal to change behaviour of pip install

[Robert Collins]

On 29 June 2016 at 10:38, Ralf Gommers  wrote:
>
>
> On Wed, Jun 29, 2016 at 12:16 AM, Nick Coghlan  wrote:
>>
>>
>> On 26 Jun 2016 23:37, "Pradyun Gedam"  wrote:
>> >
>> > Hello List!
>> >
>> > I feel it’s fine to hold back the other changes for later but the
>> > upgrade-strategy change should get shipped out to the world as quickly
>> > as
>> > possible. Even how the change is exposed the user can also be discussed
>> > later.
>> >
>> > I request the list members to focus on only the change of the default
>> > upgrade strategy to be non-eager.
>> >
>> > Does anyone have any concerns regarding the change of the default
>> > upgrade
>> > strategy to be non-eager? If not, let’s get just that shipped out as
>> > soon as possible.
>>
>> Pairing that change with an explicit "pip upgrade-all" command would get a
>> +1 from me, especially if there was a printed warning when the new upgrade
>> strategy skips packages the old one would have updated.
>
> Please do not mix upgrade with upgrade-all. The latter is blocked by lack of
> a SAT solver for a long time, and at the current pace that status may not
> change for another couple of years. Also mixing these up is unnecessary, and
> it was discussed last year on this list already to move ahead with upgrade:
> http://article.gmane.org/gmane.comp.python.distutils.devel/24219

I realise the consensus on the ticket is that its blocked, but I don't
actually agree.

Yes, you can't do it *right* without a full resolver, but you can do
an approximation that would be a lot better than nothing (just narrow
the specifiers given across all requirements). That is actually
reasonable when you're dealing with a presumed-good-set of versions
(which install doesn't deal with).

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip behavior

[Robert Collins]

Its almost certainly the version of pip within each environment.

-Rob

On 27 June 2016 at 23:45, Steve Spicklemire  wrote:
> Hi Distutils-SIG Folks,
>
> If this is not the best place for a ‘pip’ question, please direct me to a 
> better destination.
>
> I have several virtual environments. As far as I know they should all be 
> about the same WRT the python binary that was used to create them (home-brew 
> python2.7.11/OSX). However they have different behavior WRT pip install. I’m 
> trying to track down the cues that pip uses to choose a binary wheel, or a 
> source/build install. I’ve been using ‘pip -v’ to get some hints, but it’s 
> still not clear. I’ve saved two log files that illustrate the issue:
>
> https://dl.dropboxusercontent.com/u/20562746/pydev_pip_log.txt
>
> https://dl.dropboxusercontent.com/u/20562746/pytest_pip_log.txt
>
> So my question is: How can I determine why pip is choosing ‘wheel’ in some 
> cases and ‘source’ in others when I have the same python build in both cases?
>
> thanks!
> -steve
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Request for comment: Proposal to change behaviour of pip install

[Robert Collins]

I've replied in the github issue; I don't really want to carry out
*two* conversations, so perhaps we can pick one forum and stick there?

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Request for comment: Proposal to change behaviour of pip install

[Robert Collins]

On 26 June 2016 at 05:16, Donald Stufft  wrote:
>
>> On Jun 25, 2016, at 12:31 PM, Ian Cordasco  
>> wrote:
>>
>> On Sat, Jun 25, 2016 at 5:25 AM, Pradyun Gedam  wrote:
>>> Hello List!
>>>
>>> There is currently a proposal to change the behaviour to pip install to
>>> upgrade a package that is passed even if it is already installed.
>>>
>>> This behaviour change is accompanied with a change in the upgrade strategy -
>>> pip would stop “eagerly” upgrading dependencies and would become more
>>> conservative, upgrading a dependency only when it doesn’t meet lower
>>> constraints of the newer version of a parent. Moreover, the behaviour of pip
>>> install --target would also be changed so that --upgrade no longer affects
>>> it.
>>>
>>> A PEP-style write-up
>>> (https://gist.github.com/pradyunsg/4c9db6a212239fee69b429c96cdc3d73)
>>> documents the reasoning behind this proposed change, what the change is and
>>> some alternate proposals that were rejected in the discussions.
>>>
>>> This is a request for comments on the pull-request implementing this
>>> proposal (https://github.com/pypa/pip/pull/3806) for your views, thoughts
>>> and possible concerns related to it.
>>
>> Having `pip install` implicitly upgrade a package will break the way
>> tooling like ansible, puppet, salt, and chef all work. Most expect
>> that if you do `pip install requests` twice you won't get two
>> different versions. At the very least, there should be an option added
>> that implies that if the version of the specified package is already
>> installed and satisfactory, that it is not upgraded.
>
> So this may be true, but it’s also not particularly hard to work around
> similarly to what they do already for system package managers, they can
> do ``pip list`` or ``pip freeze`` to see if something is already installed.
> Not only will this work across versions, but it’ll also work *better*
> because it won’t involve hitting the network to determine this information.

Would we wait to make this change for the existing config systems to
implement this change, and get it into all their stable versions? If
not, then it doesn't matter whether its easy to work around - we're
causing countless installs of those systems to break if we make the
change.

>> If you want to change the upgrade behaviour of pip, I suggest not
>> modifying the existing `pip install` and instead deprecating that in
>> favor of a new command. Silently pulling the rug out from under people
>> seems like an incredibly bad idea.

+1 ^

> This is going to break things, no matter what we do— and the current
> behavior is also broken in ways that are causing other projects to need
> to do broken things.

How so? I haven't actually seen a bug or issue describing what folk are doing.

> The current solution I think breaks *less* things and once the initial
> pain of breakage is over will end up with a much nicer situation. Importantly
> the vast bulk of the breakage will be centralized in things like Chef where
> one group can make a change and fix it for thousands.

I think the primary outcome of the proposed plan is that we won't hear
about breakage until it shows up on 'haveibeenpawned.com'.

There are I think two discussions to have:
 - what should we be giving users as a UX
 - how to get there

For the how-to-get-there aspect, I see absolutely no reason to make
incompatible changes. We can add new options -O or whatever and leave
the existing ones alone. Changing the behaviour for 'install NAME' I
could potentially see, but I'd hesitate on that too. Our users don't
react well to instability.

For the what-should-we-aim-for question, thats more tricky.

There are I think several different aspects that all interact, and
make comparisons to dnf/apt etc quite meaningless.

Firstly, those systems have automated cron-like systems to keep them
secure; they can afford to not worry about security issues in their
dependency chains because everything will be pushed out automatically.
venvs have no such mechanism today - pip doesn't even have a suitable
command today (though you can of course write one around it). Without
one of these systems, I expect that venvs will accrete *years* old
versions of software, and we'll see the exact inverse breakage that
this proposed change to -U is about: lower minimums in PyPI are often
useless.

Secondly, apt/dnf etc have a presumption that all versions of all
software either work together or explicitly do not (via a combination
of conflicts and dependencies) - because they're dealing with an order
of magnitude less software that we are, and the system is an
integrated system, not a federated system like pip's data model. pip
doesn't have that presumption - hence this whole proposal - but the
consequence is that rather than giving uses the most *likely* to work
combination of software, we're going to give them the *least* likely
to work combination [because there are billions of 

Re: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files

[Robert Collins]

They're definitely separate; there's a draft PEP Tennessee and I
started at PyCon AU 2014 which would make a good basis for such an
effort.

-Rob

On 23 June 2016 at 09:42, Donald Stufft  wrote:
>
> On Jun 22, 2016, at 5:38 PM, Glyph  wrote:
>
>
> On Jun 22, 2016, at 12:21, Nathaniel Smith  wrote:
>
> There are still use cases for distro-specific wheels, though -- some
> examples include Raspbian wheels (manylinux1 is x86/x86-64 only), Alpine
> Linux wheels (manylinux1 is glibc only), internal deploys that want to build
> on Ubuntu 14.04 and deploy on 14.04 and don't need the hassle of generating
> manylinux-style binaries but would like a more meaningful platform tag than
> "linux", and for everyone who wants to extend wheel metadata to allow
> dependencies on external distro packages then having distro-specific wheels
> is probably a necessary first step.
>
> If we want to treat distros as first-class deployment targets I think being
> able to use their platform features in a way that's compatible with PyPI is
> an important next step.  However, wheel tags might be insufficient here; the
> main appeal of creating distro-specific wheels is being able to use
> distro-specific features, but those typically come along with specific
> package dependencies as well, and we don't have a way to express those yet.
>
>
> I don’t think these two things need to be bound together. People can already
> today depend on platform specific things just by not publishing wheels.
> Adding these tags naturally follows that, where people would need to
> manually install items from the OS before they used them. Adding some
> mechanism for automating this would be a good, further addition, but I think
> they are separately useful (and even more useful and powerful when
> combined).
>
>
> I think this is worthwhile to do, but figuring out a way to do it is
> probably going to take a lot of discussion.
>
> -glyph
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> —
> Donald Stufft
>
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Switch PyPA from IRC to Gitter or similar

[Robert Collins]

On 11 June 2016 at 01:22, Jason R. Coombs  wrote:
> In #pypa-dev, I raised the possibility of moving our PyPA support channels 
> from IRC to another hosted solution that enables persistence. Although IRC 
> has served us well, there are systems now with clear feature advantages, 
> which are crucial to my continuous participation:

FWIW, I won't be switching to gitter or slack - I find the push
notifications and always-on experience exhausting. It gets in the way
of thinking or doing anything useful. I often leave my phone in
priority-only mode to shut facebook and twitter and such things up for
hours - or days - at a time.

Thats presuming there is a IRC bridge in whatever the community as a
whole does. If there isn't, then I'd setup an account for use from a
browser when desired, but from the sounds of it I'd need to blacklist
their email address to keep noise down.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Robert Collins]

On 15 May 2016 at 08:21, Ionel Cristian Mărieș  wrote:
>
> On Fri, May 13, 2016 at 9:22 PM, Brett Cannon  wrote:
>>
>> No need to think; the decision is made and it's TOML. I know Chris doesn't
>> mean to stir up trouble, but at this point if someone wants to propose
>> something other than TOML they are going to have to write their own PEP.
>
>
> Not asking for any change but has anyone looked at libconfig? It looks quite
> interesting: simple grammar and nesting support. What do you think of it?

I hadn't, but its certainly irrelevant here, as vendoring it suitably
for pip would be excruciating due to the C dependency.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Robert Collins]

On 12 May 2016 at 08:46, Donald Stufft <don...@stufft.io> wrote:
>
>> On May 11, 2016, at 4:27 PM, Robert Collins <robe...@robertcollins.net> 
>> wrote:
>>
>>> probably too late or out of scope for this pep. However the distro packaging
>>> for python packages recommends to run the testsuite during the package
>>> build. Would it be possible to extend this pep to test depends, or maybe
>>> track this as a separate pep?
>>
>> It's covered as well- this is general purpose 'what is needed to run
>> setup.py' - once you can run setup.py, you can machine interrogate any
>> further dependencies.
>
>
> It’s not *exactly* covered— I mean people could stick test dependencies in
> this new field but I don’t think that setuptools actually exposes the
> test_requires in any meaningful way (nor do I think people actually use
> setuptools test support that consistently). So setuptools could get better
> support for testing dependencies independently of this PEP *or* another PEP
> could add a similar section that layered ontop of this. It’s definitely out
> of scope for this particular PEP though.

Right - in more detail though:
tox using projects:
 - set it to setuptools, wheel and any other 'needed to run setup.py
sdist' things. Then tox will work - and its list is generally plain
text that packagers can introspect, install separately and run the tox
commands directly.

setup.py test using projects:
 -  a small setuptools entrypoint can be written to allow
introspecting test_requires, so we just need to set the requires as
for the tox case

etc.

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Robert Collins]

On 12 May 2016 at 08:22, Matthias Klose <d...@ubuntu.com> wrote:
> On 11.05.2016 02:39, Brett Cannon wrote:
>>
>> Donald, Nathaniel, and I have finished our proposed PEP for specifying a
>> projects' build dependencies. The PEP is being kept at
>> https://github.com/brettcannon/build-deps-pep, so if you find spelling
>> mistakes and grammatical errors please feel free to send a PR to fix them.
>
>
> probably too late or out of scope for this pep. However the distro packaging
> for python packages recommends to run the testsuite during the package
> build. Would it be possible to extend this pep to test depends, or maybe
> track this as a separate pep?

It's covered as well- this is general purpose 'what is needed to run
setup.py' - once you can run setup.py, you can machine interrogate any
further dependencies.

-Rob

-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Robert Collins]

On 12 May 2016 at 06:32, Brett Cannon <br...@python.org> wrote:
> Since you can easily read the PEP at
> https://github.com/brettcannon/build-deps-pep in a nice, rendered format I
> won't repost the whole thing, but here are the changes I have made based on
> the comments so far.
>
> 1. I rephrased throughout the PEP to not explicitly say this is for building
> wheels (for Robert). I was just trying to tie the PEP into what happens
> today, not what we plan to have happen in the future.
>
> 2. Fixed the quotation marks on the TOML examples (for Nathaniel). It's just
> habit from Python why I did it the way I did.
>
> 3. Reworked the Abstract to the following (for Robert):

Thanks Brett.

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Robert Collins]

On 11 May 2016 at 12:39, Brett Cannon <br...@python.org> wrote:
> Donald, Nathaniel, and I have finished our proposed PEP for specifying a
> projects' build dependencies. The PEP is being kept at
> https://github.com/brettcannon/build-deps-pep, so if you find spelling
> mistakes and grammatical errors please feel free to send a PR to fix them.
>
> The only open issue in the PEP at the moment is the bikeshedding topic of
> what to name the sub-section containing the requirements: `[package.build]`
> or `[package.build-system]` (we couldn't reach consensus among the three of
> us on this). Otherwise the three of us are rather happy with the way the PEP
> has turned out and look forward to this being the first step towards
> allowing projects to customize their build process better!

I find calling this build dependencies confusing, but perhaps thats just me.

Right now the work flow is this:

unpack tarball
run setup.py egg-info
introspect that for dependencies
download and install such dependencies (recursively) [ in future, it
would resolve ]
run setup.py install || setup.py wheel
install

1) What would pip do when it encounters one of these files?
unpack tarball
introspect and prepare an isolated environment with the specified dependencies
run setup.py egg_info
download and install such runtime dependencies (recursively) [ in
future, it would resolve ]
run setup.py install || setup.py wheel
install

?

Right now setup.py dependencies are turing complete, and its a useful
escape valve - this design seems to have no provision for that
(previous designs we've had here have had that). If the declared
dependencies are merely those needed to be able to invoke the build
system, rather than those needed to be able to successfully build, it
would preserve that escape valve.

2) what do we expect setuptools to do here? Is it reasonable to
introspect this file and union it with setup_requires?

3) Why does this specify ' what dependencies are required to go from
source checkout to built wheel' ? - build systems also run tests,
generate docs and man pages, produce other artifacts. Perhaps making
either the target more specific (wheel_requires = ...) or the
description less specific ('dependencies required to invoke the build
system') would be good.

I am not suggesting that we model all the things a build system might
do: thats YAGNI at best, scope creep at worst.

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Robert Collins]

Couple thoughts.

Firstly, the human-editable bit: who in the last *decade* has been
writing code using a non-syntax-aware/helping editor? Its a supremely
uninteresting aspect IMO.

On ConfigParser - yes, its horrid. OTOH we do get all the lines
reliably, and setuptools will need to cover the unicode aspect itself
in its own time. All we need to do is permit # inline as a comment -
line a requirements.txt file for pip - and it becomes trivial to parse
in all cases that we need *now*.

Either we are defining the long term thing now, in which case that
huge pile of complexity lands on us, and we have to get everything
right.

Or we are defining a thing which solves the present bug, and as long
as we make sure it does not bind us in future, we're not hamstrung.

E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
as pybuild is a thing in the debian space, and this will confuse the
heck out of folk). https://wiki.debian.org/Python/Pybuild


-Rob



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 5 May 2016 at 21:47, Nathaniel Smith <n...@pobox.com> wrote:
> On Wed, May 4, 2016 at 11:57 PM, Robert Collins
...
>>> I don't think I've ever seen a package that had accurate
>>> setup_requires (outside the trivial case of packages where
>>> setup_requires=[] is accurate). Scientific packages in particular
>>> universally have undeclared setup requirements.
>>
>> Are those requirements pip installable?
>
> Either they are, or they will be soon.

Thats good. It occurs to me that scientific builds may be univerally
broken because folk want to avoid easy-install and the high cost of
double builds of things. E.g. adding bootstrap_requires will let folk
fix things?

But the main question is : why are these packages staying inaccurate?
Even in the absence of a systematic solution I'd expect bug reports
and pull requests to converge on good dependencies.

...
>> So the 10x thing is defining how the thing doing the isolation (e.g.
>> pip) should handle things that can't be installed but are already
>> available on the system.
>>
>> And that has to tunnel all the way out to the user, because its
>> context specific, its not an attribute of the dependencies per se
>> (since new releases can add or remove this situation), nor of the
>> consuming thing (same reason).
>
> # User experience today on i386
> $ pip install foo
> <... error: missing pyqt5 ...>
> $ apt install python-pyqt5
> $ pip install foo
>
> # User experience with build isolation on i386
> $ pip install foo
> <... error: missing pyqt5 ...>
> $ apt install python-pyqt5
> $ pip install --no-isolated-environment foo

So thats one way that isolation can be controlled yes.

> It'd even be straightforward for pip to notice that the requirement
> that it failed to satisfy is already satisfied by the ambient
> environment, and suggest --no-isolated-environment as a solution.
>
>> Ultimately, its not even an interopability question: pip could do
>> isolated builds now, if it chose, and it has no ramifications as far
>> as PEPs etc are concerned.
>
> That's not true. In fact, it seems dangerously naive :-/
>
> If pip just went ahead and flipped a switch to start doing isolated
> builds now, then everything would burst into flame and there would be
> a howling mob in the bug tracker. Sure, there's no PEP saying we
> *can't* do that, but in practice it's utterly impossible.

We've done similarly omg it could be bad changes before - e.g.
introducing wheel caching. A couple of iterations later and we're in
good shape. Paul seems to think we could introduce it gracefully - opt
in, then default with fallback, then solely opt-out.

> If we roll out this feature without build isolation, then next year
> we'll still be in the exact same situation we are today -- we'll have
> the theoretical capability of enabling build isolation, but everything
> would break, so in practice, we won't be able to.
>
> The reason I'm being so intense about this is that AFAICT these are all true:
>
> Premise 1: Without build isolation enabled by default, then in
> practice everyone will putter along putting up with broken builds all
> the time. It's *incredibly* easy to forget to declare a build
> dependency, it's the kind of mistake that every new user makes, and
> experienced users too.

I know lots of projects that already build in [mostly] clean
environments all the time - e.g. anything using tox[*], most things
that use Travis, Appveyor, Jenkins etc. Yes, if its not there by
default it requires effort to opt-in, and there will be a tail of some
sort. I don't disagree with the effect of the premise, but I do
disagree about the intensity.

[*]: yes, to runs setup.py sdist outside the environment, so
setup_requires doesn't get well exercised. IIRC tox is adding a
feature to build in a venv so they will be exercised.

> Premise 2: We can either enable build isolation together with the new
> static bootstrap requirements, or we can never enable build isolation
> at all, ever.

I don't agree with this one at all. We can enable it now if we want:
yes, its a behavioural change, and we'd need to phase it in, but the
logistics are pretty direct.

> Conclusion: If we want to ever reach a state where builds are
> reliable, we need to tie build isolation to the new static metadata.
>
> If you have some clever plan for how we could practically transition
> to build isolation without having them opt-in via a new feature, then
> that would be an interesting counter-argument; or an alternative plan
> for how to reach a point where build requirements are accurate without
> being enforced; or ...?

As Paul said, add it off by default, phase it in over a couple of releases.
0: opt-in
1: opt-out but when builds fail disable and

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 5 May 2016 at 18:32, Nathaniel Smith <n...@pobox.com> wrote:
> On Wed, May 4, 2016 at 10:42 PM, Robert Collins
>...
>> Yes, things will break: anyone using this will need a new pip, by
>> definition. Not everyone will be willing to wait 10 years before using
>> it :).
>
> Just to clarify (since we seem to agree): I meant that if pip starts
> interpreting an existing setup.cfg thing, then the new-pip/old-package
> situation could break, which would be bad.

No. Old pip new package will break, new pip old package is entirely safe AFAICT.

>>> - IMO an extremely valuable aspect of this new declarative
>>> setup-requirements thing is that it gives us an opportunity to switch
>>> to enforcing the accuracy of this metadata. Right now we have a swamp
>>> we need to drain, where there's really no way to know what environment
>>> any given setup.py needs to run. Sometimes there are setup_requires,
>>> sometimes not; if there are setup_requires then sometimes they're
>>
>> Huh? I've not encountered any of this, ever. I'd love some examples to
>> go look at. The only issue I've ever had with setup_requires is the
>> easy_install stuff it ties into.
>
> I don't think I've ever seen a package that had accurate
> setup_requires (outside the trivial case of packages where
> setup_requires=[] is accurate). Scientific packages in particular
> universally have undeclared setup requirements.

Are those requirements pip installable?
..
>> I'm very much against forcing isolated build environments as part of
>> this effort. I get where you are coming from, but really it conflates
>> two entirely separate things, and you'll *utterly* break building
>> anything with dependencies that are e.g. SWIG based unless you
>> increase the size of the PEP by about 10-fold. (Thats not hyperbole, I
>> think).
>
> Okay, now's my turn to be baffled :-). I literally have no idea what
> you're talking about here. What would this 10x longer PEP be talking
> about? Why would this be needed?

Take an i386 linux machine, and build something needing pyqt5 on it
:). Currently, you apt-get/yum/dnf/etc install python-pyqt5, then run
pip install. If using a virtualenv you enable system site packages.

When you introduce isolation, the build will only have the standard
library + whatever is declared as a dep: and pyqt5 has no source on
PyPI.

So the 10x thing is defining how the thing doing the isolation (e.g.
pip) should handle things that can't be installed but are already
available on the system.

And that has to tunnel all the way out to the user, because its
context specific, its not an attribute of the dependencies per se
(since new releases can add or remove this situation), nor of the
consuming thing (same reason).

Ultimately, its not even an interopability question: pip could do
isolated builds now, if it chose, and it has no ramifications as far
as PEPs etc are concerned.
...
> What are these things that aren't pip-installable and why isn't the
> solution to fix that? I definitely don't want to break things that
> work now, but providing new features that incentivize folks to clean
> up their stuff is a good thing, surely? Yeah, it means that the
> bootstrap-requirements stuff will take some time and cleanup to
> spread, but that's life.

We've a history in this group of biting off too much and things not
getting executed. We're *still* in the final phases of deploying
PEP-508, and it was conceptually trivial. I'm not arguing that we
shouldn't make things better, I'm arguing that tying two separate
things together because we *can* seems, based on the historical
record, to be unwise.

> We've spent a huge amount of effort on reaching the point where pretty
> much everything *can* be made pip installable. Heck, *PyQt5*, which is
> my personal benchmark for a probably-totally-unpackageable package,
> announced last week that they now have binary wheels on pypi for all
> of Win/Mac/Linux:
>
>   https://pypi.python.org/pypi/PyQt5/5.6
>
> I want to work towards a world where this stuff just works, not keep
> holding the entire ecosystem back with compromise hacks to work around
> a minority of broken packages.

Sure, but the underlying problem here is that manylinux is a 90%
solve: its great for the common cases but it doesn't actually solve
the actual baseline problem: we can't represent the actual system
dependencies needed to rebuild many Python packages.

pyqt5 not having i386 is just a trivial egregious case. ARM32 and 64
is going to be super common, Power8 another one, let alone less common
but still extant and used architectures like PPC, itanium, or new ones
like x86_32 [If I remember the abbreviation right - no, its not i386].

Solve that underlying problem - great, then isolation becomes an
opti

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 5 May 2016 at 16:22, Nathaniel Smith  wrote:
...
> I'm sympathetic to the general approach, but on net I think I prefer a
> slightly different proposal.
>
> Downsides to just standardizing [setup_requires]:

If I write a PEP, it won't be standardising setup_requires, it will be
standardising bootstrap requirements. The distinction is nuance but
critical:

 - we don't expect setuptools to ever need to honour it (of course, it
could, should it choose to, and ease of adoption may suggest that it
should)
 - as a new feature, making it opt in allows folk to adopt it when
they are ready; if it was literally just a new home for
setup_requires, we may see build systems auto-populating it, and the
potential swamp you describe below would then apply.

> - if projects have existing ones, that's actually kinda good / kinda
> bad -- pip has never been respecting these before, so if we suddenly
> start doing that then existing stuff could break. I don't know how
> likely this is, but experience suggests that *something* will break
> and make someone angry. (I'm still blinking at getting angry
> complaints arguing that uploading a wheel to pypi, where before there
> was only an sdist, should be treated as a compatibility-breaking
> change that requires a new version etc.)

Yes, things will break: anyone using this will need a new pip, by
definition. Not everyone will be willing to wait 10 years before using
it :).

> - IMO an extremely valuable aspect of this new declarative
> setup-requirements thing is that it gives us an opportunity to switch
> to enforcing the accuracy of this metadata. Right now we have a swamp
> we need to drain, where there's really no way to know what environment
> any given setup.py needs to run. Sometimes there are setup_requires,
> sometimes not; if there are setup_requires then sometimes they're

Huh? I've not encountered any of this, ever. I'd love some examples to
go look at. The only issue I've ever had with setup_requires is the
easy_install stuff it ties into.
...
> instead of punishing them -- is if we make the rule be "projects that
> use the declarative setup-requirements feature also get isolated build
> environments". (Then the messaging is: "see, this helps you check that
> you actually set it up right! if your new metadata works for you in
> testing, it'll also work for your users!) But this again would mean we
> can't reuse the existing [setup_requires] section.

I'm very much against forcing isolated build environments as part of
this effort. I get where you are coming from, but really it conflates
two entirely separate things, and you'll *utterly* break building
anything with dependencies that are e.g. SWIG based unless you
increase the size of the PEP by about 10-fold. (Thats not hyperbole, I
think).

Working around that will require a bunch of UX work, and its
transitive: you have to expose
how-to-workaround-the-fact-that-all-our-deps-are-not-installable all
the way up the chain. That's probably a good thing to do (e.g. see
bindep, or the aborted callout-to-system-packages we proposed after
PyCon AU last year), but tying these two things together is not
necessary, though I can certainly see the appeal; the main impact I
see is that it will just impede adoption.

The reality, AFAICT, is that most projects with undeclared build deps
today get corrected fairly quickly: a bug is filed, folk fix it, and
we move on. A robotic system that isolates everything such that folk
*cannot* fix it is much less usable, and I'm very much in favour of
pragmatism here.

> - setup.cfg is kind of a terrible format for standardizing things
> because the only definition of the format is "read the ConfigParser
> source". You cannot parse setup.cfg from a non-Python language. And
> the *only* benefit is that it already exists; teaching pip to read
> pybuild.json or pybuild.toml instead would be completely trivial. So
> if we aren't going to try and grandfather in existing [setup_requires]
> sections, then we might as well switch to a better file format at the
> same time -- this is not the hard part.

The patch to read a single list-valued key out of setup.cfg is trivial
and shallow. We've not managed to settle on consensus on a file format
choice in a year of debate. I hold zero confidence we will going
forward either. If the BDFL delegate makes a call - fine. I read
Nick's earlier email in the thread as such a thing TBH :).

> So I like the idea of splitting out the declarative setup-requirements
> PEP from the new build system hook interface PEP, but I think that the
> way we should do this is by defining a new pybuild.whatever file like
> the ones in PEP 516 / PEP 517 (they're identical in this regard) that
> *only* has schema-version and bootstrap-requirements, and then we can
> add the build backend key as a second step.

I think the risk there is that that presumes the answer, and without
the abstract build effort actually moving forward we may be years
before we actually 

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

Ok so, if i draft a pep for said proposal, will it die under the weight of
a thousand bike sheds?
On 5 May 2016 3:09 PM, "Nick Coghlan" <ncogh...@gmail.com> wrote:

> On 5 May 2016 at 06:28, Robert Collins <robe...@robertcollins.net> wrote:
> > the only reason I got involved in build system discussions was
> > pushback 18months or so back when I implemented a proof of concept for
> > pip that just used setup.cfg. I'd be very happy to ignore all the
> > build system stuff and just do bootstrap requirements in setup.cfg.
>
> I know I'm one of the folks that has historically been dubious of the
> "just use setup.cfg" idea, due to the assorted problems with the
> ini-style format not extending particularly well to tree-structured
> data (beyond the single level of file sections).
>
> However, my point of view has changed over that time:
>
> 1. We've repeatedly run up against the "JSON is good for programs
> talking to each other, but lousy as a human-facing interface" problem
> 2. By way of PEPs 440 and 508, we've got a lot more experience in
> figuring out how to effectively bless de facto practices as properly
> documented standards (even when it makes the latter more complicated
> than we'd like)
> 3. The ongoing popularity of setup.cfg shows that while ini-style may
> not be perfect for this use case, it clearly makes it over the
> threshold of "good enough"
> 4. Folks that *really* want a different input format (whether that's
> YAML, TOML, or something else entirely) will still be free to treat
> setup.cfg as a generated file, just as setup.py can be a generated
> file today
>
> The last couple of years have also given me a whole range of
> opportunities (outside distutils-sig) to apply the mantra "the goal is
> to make things better than the status quo, not to make them perfect",
> and that has meant getting better at distinguishing what I would do
> given unlimited development resources from what makes the most sense
> given the development resources that are actually available.
>
> So when I ask myself now "What's the *simplest* thing we could do that
> will make things better than the status quo?", then the answer I come
> up with today is your original idea: bless setup.cfg (or at least a
> subset of it) as a standardised interface.
>
> Don't get me wrong, I still think that answer has significant
> downsides - I've just come around to the view that "is likely to be
> easy to implement and adopt" are upsides that can outweigh a whole lot
> of downsides :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 4 May 2016 at 22:33, Donald Stufft <don...@stufft.io> wrote:
>
..> I also believe that we can't provide a replacement for setup.py
without either
> purposely declaring we no longer support something that people used from it or
> providing a way to support that in the new, setup.py-less format.

The thunk I wrote being insufficient?

> One thing I remarked to Nataniel yesterday was that it might be a good idea to
> drop the build system aspect of these for right now (since afaict all the
> invested parties are currently overloaded and/or have a lack of time) and 
> focus
> soley on the part of the proposal that enables us to get a real setup_requires
...

the only reason I got involved in build system discussions was
pushback 18months or so back when I implemented a proof of concept for
pip that just used setup.cfg. I'd be very happy to ignore all the
build system stuff and just do bootstrap requirements in setup.cfg.

-Rob

-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 4 May 2016 at 19:39, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 4 May 2016 at 16:03, Robert Collins <robe...@robertcollins.net> wrote:
>> The edits I'd expect to make if the conclusions I suggested in
>> https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
>> are adopted are:
>>
>>  - change to a Python API
>>  - BFDL call on the file format and name
>>
>> There is no need to issue a new sdist thing, because sdists today are
>> *already* documented across PEPs 241, 314 and 345.
>
> I already +1'ed using a Python API, but on the file name & format
> side, we have the following candidates and prior art floating around:
>
> pypa.json in PEP 516
> pypackage.json in PEP 517
> pydist.json in PEP 426
> METADATA (Key: Value) in sdists and wheels
> WHEEL (Key: Value) in wheels
>
> My impression is that we're generally agreed on wanting to move from
> Key:Value to JSON as the baseline for interoperability formats, so my
> suggestion is to use the name "pybuild.json".
>
> The problem I have with pypa/pypackage/pydist is that they're all too
> broad - we're moving towards an explicitly multi-stage pipeline (tree
> -> sdist -> wheel -> installed) and additional metadata gets added at
> each step. The "pybuild.json" metadata specifically covers how to get
> from a source tree or sdist to a built wheel file, so I think it makes
> sense to use a name that reflects that.

I don't think we have anything resembling consensus on that pipeline idea.

pybuild.json would be fine as a name though :).

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 4 May 2016 at 05:10, Paul Moore  wrote:

> Nick - do you have the time to pick this up? Or does it need someone
> to step up as BDFL-delegate? Robert, Nathaniel, do you have time to
> spend on a final round of discussion on this, on the assumption that
> the goal will be a final decision at the end of it? Donald, do you
> have the time and interest to complete and publish your proposal?

I'm currently going through a redundancy process @ HPE - while I
remain convinced that sorting out the issues with packaging is crucial
for Python going forward, my time to work on it is now rather more
limited than it was. I'm not sure where I'm going to end up, nor how
much work time I'll have for this going forward: it may end up being a
personal-time-only thing. Planning wise, we have to work on 'personal
time only' going forward - which means I"m going to be very careful
about biting off more than I can chew :)

Right now, I don't see any point updating the PEP at this point.

The edits I'd expect to make if the conclusions I suggested in
https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html
are adopted are:

 - change to a Python API
 - BFDL call on the file format and name

There is no need to issue a new sdist thing, because sdists today are
*already* documented across PEPs 241, 314 and 345.

So - if Nick is ready to rule, there is basically about an hour of
editing to switch the CLI to a Python API and update the file format
bits, and then we can call it provisional and encourage
implementations, update the thunk implementation and so on.

If there's more debate to be had, thats fine too, but editing the PEP
won't achieve that.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Basic Markdown Readme Support

[Robert Collins]

On 3 May 2016 4:19 PM, "Alexander Walters"  wrote:
>
> I am -1 on this on the basis that the services mentioned also happily
support restructured text READMEs

I don't understand why that makes you say no to the ability to support
markdown.

Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] abstract build system approaches redux

[Robert Collins]

Just to set expectations: this whole process seems stalled to me; I'm
going to context switch and focus on things that can move forward.
Someone please ping me when its relevant to put effort in again :).

-Rob

On 4 March 2016 at 03:11, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 3 March 2016 at 23:44, Donald Stufft <don...@stufft.io> wrote:
>> * We make it easier to use a non Python based build system. This is also a 
>> nice
>>   thing, however I don't think it should be a major decider in what API we
>>   provide. Any reasonable build system is going to have to be available via
>>   ``pip install`` in some fashion, so even if you write your build system in
>>   Go or Rust, you're going to have to create a Python package for it anyways,
>>   and if you're doing that, adding a tiny shim is pretty trivial, something
>>   like:
>>
>> import os.path
>> import subprocess
>>
>> BIN = os.path.join(os.path.dirname(__file__), "mytool")
>>
>> def some_api_function(*args, **kwargs):
>> flags = convert_args_kwargs_to_flags(*args, **kwargs)
>> subprocess.run(BIN, *flags, check=True)
>>
>>   I don't believe it to be a substantial burden to need to write a tiny 
>> wrapper
>>   if you're going to do something which I believe is going to be very 
>> unlikely.
>
> Ah, you're right, I hadn't accounted for the fact the same shim that
> makes a non-Python tool installable as a build dependency could also
> handle the adaptation from a Python API to a CLI or FFI based
> approach, so putting the standardised interface in Python doesn't
> raise any insurmountable barriers to cross-language interoperability -
> they just move the additional complexity to the less common case.
>
> Given that, I'm going to go back to reserving judgement on *all* of
> the points Robert mentioned, at least until you've had a chance to
> finish writing up your own proposal - the determining factor I thought
> I had found turned out not to be so determining after all :)
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] abstract build system approaches redux

[Robert Collins]

This is my attempt to consolidate the three different proposals we
have at the moment into a single set of design choices.

We have three draft designs - from Nathaniel (517), Donald (unnumberd)
and mine (516).

For purposes of comparison, I've skipped all rationale here, and only
listed the highlights that differ across the proposals. If you haven't
read all three, you should before continuing.

My goal is to let us pick a final shape - and then we can shave the
rough edges off of that shape to get a final PEP (or PEPs)

517:
 - Python interface to the system
 - build requires, optional wheel metadata (to directory on disk),
build wheel. install editable commands

516:
 - CLI interface to the system
 - build requires, mandatory wheel metadata, build wheel, install
editable commands

Donald's:
 - Python interface to the system
 - defines new 'source wheel' concept
 - source_metadata, source_wheel, binary_metadata, binary_wheel commands
 - 'develop' // editable not addressed currently
 - outputs directories on disk, rather than zipped up things


Ok, now here's my view on the differences...

Python vs CLI - I don't care enough to argue. I think a CLI is better
in this space, as running commands is the lingua franca of CLI's;
previously Donald has advocated for that as well, but his proposal is
now a Python API. *shrug*. I want the BDFL-Delegate to choose.

516 and 517 are otherwise identical in all meaningful ways - there's
plenty of stuff to bikeshed on (e.g. should wheel metadata be optional
or mandatory), but I don't think it is crucial: we can always change
this in future by issuing a new version. Its not free, but if we can't
agree, we can at least find out later. Donalds and 516 both have the
ability to get metadata out explicitly as a mandatory thing, so I
think we should make it mandatory and weaken it in future if e.g. flit
eventually come back saying its a burden. [I don't believe it will
be].

File formats - BDFL-Delegate can pick. I so don't care :).

New source format. I'm quite strongly against this at the moment:
Building from VCS is now a fairly major thing in the community, and we
can't mandate new URLs for all sources. We're going to thus be forcing
a change to all tooling (e.g. Debian and Fedora and pip and others)
that know how to build things from an arbitrary VCS url. A new source
extension in pypi doesn't reduce the impact of that at all, and the
necessary code to handle that will also handle sdists's on pypi that
have the new config file and may be missing a setup.py. If there is a
new source extension in pypi, we should expect that folk using e.g.
flit will *not* upload old style sdists to pypi - they don't at the
moment, and the objection to doing so will *remain*. -> We don't save
any grief or friction for the adoption of the new thing, nor enable
new build systems any more easily. A shim to the new shiny would
enable current sdists for anything adopting the new thing in exactly
the same manner as it would if we reuse the extension. I can see an
argument for a new iteration of the sdist format for better metadata,
and to be able to tell what metadata is trustable etc, but I don't see
that as related to the problem of enabling third party build systems -
and since all the easy cases will be solved by binary wheels I think
we're in rapidly diminishing territory here - if something can't
upload a binary wheel, its quite likely to have complex dependencies -
and we haven't solved the numpy ABI problem yet, so I'd like to
actually focus on that well and solve it - and once solved see if we
can retrofit it into the existing sdist format, or if a new one is
actually needed.

Develop is essential to tackle IMO. We can either fully PEP define it
now - and if so we should stop talking about this and start talking
about that. If we don't have bandwidth to do that yet (and AFAICT we
don't, based on IRC discussions and the headaches of interactions with
the three different namespace package possibilities, virtualenvs and
user installs).

Outputting the wheel as an unzipped directly on disk would be a nice
optimisation for big trees, I think, so perhaps we should apply that
to 516/517 to reduce the set of things we're discussing.

-Rob

516: 
https://github.com/pypa/interoperability-peps/blob/master/pep-0516-build-system-abstraction.rst
517: 
https://github.com/pypa/interoperability-peps/blob/master/pep-0517-build-system-abstraction.rst
Donalds: 
https://github.com/pypa/interoperability-peps/compare/master...dstufft:build-pipeline




-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] abstract build system PEP update

[Robert Collins]

On 18 February 2016 at 01:44, Donald Stufft <don...@stufft.io> wrote:
>
>> On Feb 17, 2016, at 6:20 AM, Paul Moore <p.f.mo...@gmail.com> wrote:
>>
>> Currently, however, people using distutils can create sdists using
>> "setup.py sdist". For other tools, this PEP does not specify how a
>> sdist should be created, but it does imply that it is sufficient to
>> make an archive of a source directory, including a "pypa.json" file,
>> and pip will be able to consume that as a sdist. Whether tools provide
>> a command to do this is out of scope for this PEP.
>
>
> I'm writing my own alternative to this problem (was hoping to have it ready
> yesterday, but I ended up with a migraine and spent most of the day MIA
> instead) and it finally clicked to me why I felt that sdist needed to be a 
> part
> of this, and part of why I felt mildly uncomfortable by the PEP.
>
> Right now, distutils/setuptools provides a number of functions:
>
> * Installation from Sdist
> * Building a Wheel
> * Fetching Metadata
> * Building a sdist
> * Uploading
>
> It's fair to say that I think we're trying to get away from the first of those
> and it's a good thing that neither of these proposals include it. In addition
> both proposals have fetching metadata and building wheels built into them.
>
> Where they fall short is that there isn't a way to build a sdist or upload 
> said
> sdist to PyPI. The assumption is that people will just package up their local
> directories and then publish them to PyPI. However I don't think that
> assumption is reasonable.

I don't understand why that should be part of a PEP about *consuming*
sdists. They're entirely different use cases.

> You could say that using twine to handle the uploading is a thing people 
> should
> do (and I agree!) but that currently relies on having static metadata inside 
> of
> the sdist that twine can parse, static metadata that isn't going to exist if
> you just simply tarball up a directory on disk.

So thats a good reason to have an sdist format newer than PEP314, one
that includes all the data we want to have (and that will hopefully be
usually-static). But the existing sdist PEP's are sufficient to cover
twine's needs today.

> So I think just tarballing up a directory on disk is *not* a reasonable way to
> create a sdist, however you can't reasonably (IMO) create an independent tool
> to handle this task with either of these build proposals either.

Sure you can - PEP 314 already documents PKG-INFO; setup.py has been
defacto - and thats been the blocker for flit- AIUI it at least.

> For instance,
> building a sdist requires getting the version of the project, that is going to
> require some logic which is (in both of these PEPs) build system specific. So
> taking flit for example, in order to build a sdist of a flit using package, a
> tool is going to have to implement the same version detection logic that flit
> itself is using (and the same goes for other metadata like name, and
> long_description). As far as I can tell, any tool to create a sdist is going 
> to
> have to be specific to a particular build tool in order to have the same
> metadata that you would get building a wheel.
>
> The closest that either of these PEPs make it to being able to interrogate a
> directory for the metadata is by asking it for the metadata that would occur
> if we created a wheel file right now. While that is theortically enough
> information, it's also going to contain things which do not belong in a sdist
> at all.
>
> In the end, I think *both* of the current proposals, as written, will cause
> people to not uploads sdists because neither of them really address the fact
> that their intent is to remove the standard interface for doing that, without
> including a replacement to that interface.

I disagree that thats the intent though.


The interface for a developer has the things you listed. The interface
for an installer has been defined in an adhoc fashion as a subset of
'whatever setup.py does' - and we're fixing *that*, but - and see the
preamble for my PEP - the whole point is to decouple the rich do
whatever developers need from the very narrow thing installers like
pip need.

-Rob



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] abstract build system PEP update

[Robert Collins]

On 17 February 2016 at 20:13, Marius Gedminas <mar...@gedmin.as> wrote:
> On Tue, Feb 16, 2016 at 04:10:43PM +1300, Robert Collins wrote:
>> diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst
>> index a6e4712..56464f1 100644
>> --- a/build-system-abstraction.rst
>> +++ b/build-system-abstraction.rst
>> @@ -68,12 +68,15 @@ modelled on pip's existing use of the setuptools
>> setup.py interface.
>>  pypa.json
>>  -
>>
>> -The file ``pypa.json`` acts as neutron configuration file for pip and other
>> +The file ``pypa.json`` acts as neutral configuration file for pip and other
>>  tools that want to build source trees to consult for configuration. The
>>  absence of a ``pypa.json`` file in a Python source tree implies a setuptools
>>  or setuptools compatible build system.
>>
>> -The JSON has the following schema. Extra keys are ignored.
>> +The JSON has the following schema. Extra keys are ignored, which permits the
>> +use of ``pypa.json`` as a configuration file for other related tools. If 
>> doing
>> +that the chosen keys must be namespaced - e.g. ``flit`` with keys under that
>> +rather than (say) ``build`` or other generic keys.
>
> Is this going to be a file that human beings are expected to edit by
> hand?
>
> If so, can we please not use JSON?  JSON is rather hostile to humans: no
> trailing commas, no possibility to add comments.

Find another format thats ideally in the standard library, with an as
clean language-neutral schema. Yaml isn't. Toml isn't. $bikeshed.

Honestly - If this is the bit we bog down on, great - someone can
spend the time finding a thing to use here, but as discussed
previously, I don't care: the PEP editor that accepts the PEP can tell
me what format should be there, and I'll put it there. Until then, I
don't want to think about it because its not interesting.

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] abstract build system PEP update

[Robert Collins]

On 18 February 2016 at 10:30, Paul Moore <p.f.mo...@gmail.com> wrote:
> I think the key point here is that twine expects a *sdist*, that is,
> an archive, containing a PKG-INFO file conforming to the Metadata 1.1
> spec. See PEP 314. (Note, I only just read PEP 314 myself - it's as
> much news to me that it defines how sdists have to contain metadata as
> it probably is to everyone else...)
>
> One of the problems here is that the older packaging PEPs got into
> such a mess, that people have stopped believing in them at all. I just
> looked at PEPs 241 and 314, and they actually do a pretty good job of
> specifying how sdists must include metadata, and I think it's fair to
> say that this should be our staring point.

Hmm - I certainly didn't stop believing it.. its just that the
metadata *specified* by the PEP's doesn't include anything distutils
didn't know about: e.g. dependencies.

>> I think both Robert and my proposal basically see their scope as being
>> strictly restricted to "here's how we want to replace pip-calling-setup.py
>> with pip-calling-something-else", while keeping everything else the same or
>> at least delegating any other changes to other PEPs. So we envision that
>> build system authors will provide some way to package up source into an
>> sdist, whatever that means; that could be a current-style sdist with
>> metadata requirements reverse-engineered from twine and setuptools, or it
>> could be done kind of new and improved sdist that is about to get its own
>> PEP... either way, it's orthogonal to replacing setup.py.
>
> Robert more or less did that, with the proviso that his PEP left open
> the possibility that build systems can upload things that the PEP says
> can be processed by pip, even though in practice they aren't sdists
> (according to PEP 314). But given that no-one had really dug back into
> the implications of the older PEPs, that's more of a loophole than a
> deliberate choice. On the other hand, your proposal explicitly allows
> a "version 1" sdist to not contain PEP 314 metadata, precisely because
> it defines a new format of sdist *without* mandating that it must
> conform to PEP 314 (no blame here - as I say, we'd all omitted to
> check back at the older PEPs).

Well, I'm adamant that my PEP should not make anything worse! My
specific concern w.r.t. sdists is that its an orthogonal consideration
to 'not having a setup.py'. Not having a setup.py applies to any
source tree irrespective of how you obtained it. Sdists and their
constraints apply to well - sdists.

> IMO, we should either bite the bullet and properly define a "new
> sdist" format (which won't be an easy job!), or we should couch the
> "build system abstraction" PEPs in terms of adding a new "build system
> interface" metadata file to the existing sdist format. In the
> interests of openness, I'd include comments pointing out that the
> existing sdist format is defined as conforming to PEP 314, and that
> even though pip itself (and the build system interface) doesn't use
> the metadata mandated by PEP 314, other tools do.
>
> The latter option is far easier (we're basically almost there) but it
> will *not* suit build tools that can't (or won't) generate a sdist
> that conforms to PEP 314 (i.e., contains PKG-INFO).

I still don't understand how *making* an sdist is tied into this PEP?
AIUI flit's only concern with sdists was that they didn't want a
setup.py with all the stuff we've discussed about that. Perhaps a flit
author can weigh in - generating good PEP 314 PKG-INFO shouldn't be a
problem for flit, has no Python2/3 implication, and doesn't require a
setup.py.

PEP 241 and 314 do not require a setup.py to be present AFAICT.

I think all we need to do is make sure that its crystal clear that
we're not changing any specified part of the contract of sdists: you
should still make them, they still need to meet PEP 314 setup.py
was never PEP'd as part of the contract.

We are changing the defacto contract, and so we should make that clear
in the prose.

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Alternative build system abstraction PEP

[Robert Collins]

Cool. I've replied on the PR as well, but my understanding from Donald
was that from pip's perspective, the spawn interface being the
contract was preferred; the other two differences are I think mostly
cosmetic - except that I'm very worried about the injection
possibilities of the calling code being responsible for preserving
metadata between 'metadata' and 'build-wheel'. That seems like an
unnecessary risk to me.

-Rob

On 17 February 2016 at 14:32, Nathaniel Smith  wrote:
> Hi all,
...
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Robert Collins]

On 13 February 2016 at 03:54, Steve Dower <pyt...@stevedower.id.au> wrote:
> I was also planning to use it in an upcoming project that has to "do its
> own" package management. The aim was to install different versions of
> packages in different directories and use sys.path modifications to resolve
> them at runtime (kind of like what setuptools did in the older days).
>
> An alternative would be great, though I can probably fake things somehow for
> my purposes.

Sounds similar to Daniel's need - and again, --prefix + setting PATH
and PYTHONPATH would be better.

-Rob



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Robert Collins]

On 13 February 2016 at 04:12, Daniel Holth <dho...@gmail.com> wrote:
> My setup-requires wrapper, adding pre-setup.py dependency installation to
> setup.py, relies on this feature. It needs to install something in a
> directory that is only added to PYTHONPATH during the installation and does
> not interfere with the normal environment.

Python packages that create (and need) scripts, or use data_files,
won't work with --target. Using a --prefix and setting PYTHONPATH
*and* PATH appropriately would be better.

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] abstract build system PEP update

[Robert Collins]

On 16 February 2016 at 22:40, Paul Moore <p.f.mo...@gmail.com> wrote:
> On 16 February 2016 at 03:10, Robert Collins <robe...@robertcollins.net> 
> wrote:
>> -The file ``pypa.json`` acts as neutron configuration file for pip and other
>> +The file ``pypa.json`` acts as neutral configuration file for pip and other
>
> Aw, I was looking forward to controlling my nuclear power plant with pip :-(
>
> Oh, and "acts as a" rather than just "acts as".

Fixed.

>> +We discussed having an sdist verb. The main driver for this was to make sure
>> +that build systems were able to produce sdists that pip can build - but 
>> this is
>> +circular: the whole point of this PEP is to let pip consume such sdists
>> +reliably and without requiring an implementation of setuptools. Further, 
>> while
>> +most everyone agrees that encouraging sdists to be uploaded to PyPI, there
>
> s/most/almost/
>
> "to be uploaded to PyPI"... what? The phrase isn't complete.
> Presumably "is a good thing".
>
>> +wasn't complete consensus on that.
>
> And I didn't think there was any dispute over this. There were people
> who didn't want to disallow binary-only projects, but that's hardly
> the same as not encouraging people who *are* making sources public to
> put them in the same place as the binaries.

Yes, badly phrased. I've removed that whole bit, and just kept it to
the core: the PEP describes how to consume source trees, it doesn't
change pip's relationship to source trees - sdist or vcs or other.

> I thought the key point was that we'd agreed to Nick's suggestion that
> we add some comments to the existing specifications to note that you
> could bundle up a source tree with a pypa.json and get something
> sufficient for new pips to install, so this provided a sufficiently
> well-defined "source upload format" to work until discussions on a new
> source format came to fruition?
>
> Specifically, my expectation is that this PEP require that the
> specification changes proposed by Nick be implemented. Sure, it's an
> informational change to a document, but it's important that this PEP
> acknowledge that the action was part of the consensus.

So, I don't know what note you're looking for. The PEP /already/
documents that pip will be able to consume this, and the context is
from sdists / source trees.

Yes, we're going to document in the other relevant specs that we
expect folk to upload sdists - thats also my understanding. My note in
the PEP was specifically about the inclusion or not of an sdist verb
in the interface.

-Rob

+The file ``pypa.json`` acts as a neutral configuration file for pip and other
 tools that want to build source trees to consult for configuration. The
 absence of a ``pypa.json`` file in a Python source tree implies a setuptools
 or setuptools compatible build system.
@@ -414,10 +414,13 @@ the difference older pip versions require.

 We discussed having an sdist verb. The main driver for this was to make sure
 that build systems were able to produce sdists that pip can build - but this is
-circular: the whole point of this PEP is to let pip consume such sdists
-reliably and without requiring an implementation of setuptools. Further, while
-most everyone agrees that encouraging sdists to be uploaded to PyPI, there
-wasn't complete consensus on that.
+circular: the whole point of this PEP is to let pip consume such sdists or VCS
+source trees reliably and without requiring an implementation of setuptools.
+Being able to create new sdists from existing source trees isn't a thing pip
+does today, and while there is a PR to do that as part of building from
+source, it is contentious and lacks consensus. Rather than impose a
+requirement on all build systems, we are treating it as a YAGNI, and will add
+such a verb in a future version of the interface if required.

 References
 ==

Press ENTER or type command to continue
robertc@lifeless-z140:~/work/interoperability-peps$
robertc@lifeless-z140:~/work/interoperability-peps$ git diff
diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst
index 56464f1..a69c150 100644
--- a/build-system-abstraction.rst
+++ b/build-system-abstraction.rst
@@ -68,7 +68,7 @@ modelled on pip's existing use of the setuptools
setup.py interface.
 pypa.json
 -

-The file ``pypa.json`` acts as neutral configuration file for pip and other
+The file ``pypa.json`` acts as a neutral configuration file for pip and other
 tools that want to build source trees to consult for configuration. The
 absence of a ``pypa.json`` file in a Python source tree implies a setuptools
 or setuptools compatible build system.
@@ -414,10 +414,13 @@ the difference older pip versions require.

 We discussed having an sdist verb. The main driver for this was to make sure
 that build

[Distutils] abstract build system PEP update

[Robert Collins]

I've just pushed this up.

- provide space for config of other things in pypa.json via convention
- document PATH as a reliable variable
- capture the sdist discussion
- remove --root from develop: it isn't needed after discussion on IRC.

-Rob

diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst
index a6e4712..56464f1 100644
--- a/build-system-abstraction.rst
+++ b/build-system-abstraction.rst
@@ -68,12 +68,15 @@ modelled on pip's existing use of the setuptools
setup.py interface.
 pypa.json
 -

-The file ``pypa.json`` acts as neutron configuration file for pip and other
+The file ``pypa.json`` acts as neutral configuration file for pip and other
 tools that want to build source trees to consult for configuration. The
 absence of a ``pypa.json`` file in a Python source tree implies a setuptools
 or setuptools compatible build system.

-The JSON has the following schema. Extra keys are ignored.
+The JSON has the following schema. Extra keys are ignored, which permits the
+use of ``pypa.json`` as a configuration file for other related tools. If doing
+that the chosen keys must be namespaced - e.g. ``flit`` with keys under that
+rather than (say) ``build`` or other generic keys.

 schema
 The version of the schema. This PEP defines version "1".  Defaults to "1"
@@ -130,6 +133,9 @@ Available environment variables
 These variables are set by the caller of the build system and will always be
 available.

+PATH
+The standard system path.
+
 PYTHON
 As for format variables.

@@ -176,7 +182,7 @@ wheel -d OUTPUT_DIR

 flit wheel -d /tmp/pip-build_1234

-develop [--prefix PREFIX] [--root ROOT]
+develop [--prefix PREFIX]
 Command to do an in-place 'development' installation of the project.
 Stdout and stderr have no semantic meaning.

@@ -185,8 +191,11 @@ develop [--prefix PREFIX] [--root ROOT]
 that doing so will cause use operations like ``pip install -e foo`` to
 fail.

-The prefix option is used for defining an alternative prefix within the
-installation root.
+The prefix option is used for defining an alternative prefix for the
+installation. While setuptools has ``--root`` and ``--user`` options,
+they can be done equivalently using ``--prefix``, and pip or other
+tools that accept ``--root`` or ``--user`` options should translate
+appropriately.

 The root option is used to define an alternative root within which the
 command should operate.
@@ -403,6 +412,13 @@ the setuptools shim is in use (with older pip
versions), or an environment
 marker ready pip is in use. The setuptools shim can take care of exploiting
 the difference older pip versions require.

+We discussed having an sdist verb. The main driver for this was to make sure
+that build systems were able to produce sdists that pip can build - but this is
+circular: the whole point of this PEP is to let pip consume such sdists
+reliably and without requiring an implementation of setuptools. Further, while
+most everyone agrees that encouraging sdists to be uploaded to PyPI, there
+wasn't complete consensus on that.
+
 References
 ==



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 11 February 2016 at 11:12, Barry Warsaw <ba...@python.org> wrote:
> On Feb 10, 2016, at 10:08 AM, Paul Moore wrote:
>
>>But those people will then find that distributing their sources isn't
>>something that flit covers, so they'll make up their own approach (if it were
>>me, I'd probably just point people at the project's github account).
>>
>>Once people get set up with a workflow that goes like this (build
>>wheels and point people to github for source) it'll be harder to
>>encourage them later to switch *back* to a process of uploading
>>sources to PyPI.
>>
>>And that I do think is bad - that we end up pushing people who would
>>otherwise happily use PyPI for source and binary hosting, to end up
>>with a solution where they host binaries only on PyPI and make the
>>source available via another (non-standardised) means.
>
> That worries me a lot.  Think of the downstream consumers who aren't end
> users, e.g. Linux distro developers.  Some distros have strict requirements on
> the availability of the source, reproducibility of builds, and so on, along
> with stacks of tooling that are built on downloading tarballs from PyPI.
>
> It's not impossible to migrate to something else, but it's impractical to
> migrate to dozens of something elses.  Right now, if we can count on PyPI
> having the source in an easily consumable lowest common denominator format,
> the friction of providing those packages to *our* end users, and updating them
> in a timely manner, is often minimal.  Changing that ecosystem upstream of us,
> either deliberately or otherwise, will likely result in more out of date
> packages in the distros.

But again, as we already covered, none of the draft peps encourage
sourceless uploads to PyPI: a big chunk of it is in fact largely about
enabling source uploads without requiring implementing an undocumented
changed-at-will interface.

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] deprecating pip install --target

[Robert Collins]

This is fairly broken - it doesn't handle platlib vs purelib (see pip
PR 3450), doesn't handle data files, or any other layout. Donald says
pip uses it to maintain the _vendor subtrees only, which doesn't seem
like a particularly strong use case.

Certainly the help description for it is misleading - since what we're
actually doing is copying only a subset of what the package installed
- so at a minimum we need to be much clearer about what it does.

But, I think it would be better to deprecate it and remove it... so
I'm pinging here to see if anyone can explain a sensible use case for
it in the context of pip :)

-Rob



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

[Robert Collins]

I'm a little over this particular subthread of the topic - we did it
to death late last year. So apologies in advance if I get a little
terse.

On 12 February 2016 at 05:08, M.-A. Lemburg <m...@egenix.com> wrote:
> On 10.02.2016 19:46, Robert Collins wrote:
>> On 11 February 2016 at 02:36, M.-A. Lemburg <m...@egenix.com> wrote:
>>
>>>> Currently what pip does is to
>>>> invoke
>>>>
>>>> $ python setup.py egg_info --egg-base $TEMPDIR
>>>>
>>>> to get the metadata. It is not possible to get the metadata without
>>>> executing the setup.py which is problematic for many applications.
>>>> Providing a static pypa.json file is much better: tools can read a
>>>> static file to get the metadata.
>>>
>>> Depends on which kind of meta data you're after. sdist packages
>>> do include the static PKG-INFO file which has the version 1.0
>>> meta data. This doesn't include dependencies or namespace
>>> details, but it does have important data such as version,
>>> package name, description, etc.
>>
>> For pip to use it, it needs to include - reliably - version, name, and
>> dependencies; for it to be in any way better, we also need
>> setup-requires or a functional equivalent.
>>
>> Today, we can't rely on the PKG-INFO being complete, so we assume they
>> are all wrong and start over. One of the things we'll get by being
>> strict in subsequent iterations is the ability to rely on things.
>
> Then why not fix distutils' sdist command to add the needed
> information to PKG-INFO and rely on it ?

How can we tell it is reliable? There's some 100K's of sdists that
aren't reliable on PyPI already. We don't want to break installing
those.

> Or perhaps add a new distutils command which outputs the needed
> information as JSON file and fix the sdist command to call this
> by default ?

We already have a command which outputs the needed info (as egg info)
- and my draft PEP has a similar one, using PEP427 wheel METADATA
format.

> There are many ways to address such issues, but defining a new
> standard for every issue we have instead of fixing the existing
> distutils implementation is not the best way to approach this.

I think you need to make a much stronger case for this, given how
consistent the disagreement in this forum with that position has been.
I understand you consider it to be not-the-best way, but so far, noone
else seems to agree.

Specifics that Donald raised with any 'fix distutils' approach:
 - how do we fix people running pip install on Python 2.7, 3.4, 3.5,
for the next 5 years?
 - how do we address the needs of folk who are using bento or flit
 - and how do we address the needs of the *authors* of bento or flit?

Plus the one I've already mentioned

 - how do we fix the bootstrap issue setup.py has ?

>>> In the end, you'll just be defining a different standard
>>> to express the same thing in different ways.
>>>
>>> The setup.py interface was never designed with integration in mind
>>> (only with the idea to provide an extensible interface; and I'm not
>>> going get into the merits of setuptools additions such as
>>> --single-version-externally-managed :-)) but it's still
>>> quite usable for the intended purpose.
>>
>> However we *are defining an integration point*, which is yet another
>> reason not to use the setup.py interface.
>
> https://xkcd.com/927/ :-)
>
> setup.py is the current standard and even though it's not
> necessarily nice, it works well and it does support adding
> different build systems (among many other things).

I don't think 'well' is a useful adjective here. setup.py has some
very sharp limits as an interface:
 - bootstrap dependencies are broken in non-standard networking environments
 - bootstrap dependencies are very slow due to multiple builds
 - setup.py is very complex for both implementors and package authors

Folk that want/need/like setup.py can keep using it! The draft we're
discussing can obviously thunk back through to setup.py for setuptools
projects, and no harm occurs to them. For other projects, they gain
choice and the ecosystem can expand.

> mxSetup.py, for example, includes a build system for C libraries
> completely outside the distutils system, based on the standard
> Unix configure/make dance. It simply hooks into distutils, takes
> a few parameters and goes off to build things, feeding the
> results back into the distutils machinery.

Thats great; and mxSetup.py can keep doing that, while adding compat
for pypa.json very easily.

Can I ask - why the push-back on a new, clean, crisp, separate
interface here? All the drafts had this in common, perhaps we're just
too close to it!

What harm is it going to cause?

-Rob



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 12 February 2016 at 08:45, Wes Turner <wes.tur...@gmail.com> wrote:
>
> On Feb 11, 2016 1:23 PM, "Robert Collins" <robe...@robertcollins.net> wrote:

>> We already have a command which outputs the needed info (as egg info)
>> - and my draft PEP has a similar one, using PEP427 wheel METADATA
>> format.
>
> with JSON-LD.org, warehouse can be built with RDFJS.org for the UI; and
> anything could index the catalog metadata (e.g. pip, as it does things)

This is a non-sequitor and entirely unrelated to the discussion with
Marc-Andre about reusing setup.py as the interface vs a new interface.

>> Plus the one I've already mentioned
>>
>>  - how do we fix the bootstrap issue setup.py has ?
>
> https://bootstrap.pypa.io/get-pip.py

How does that address the issue? I think perhaps you are thinking of
'how do you install pip'. The thing I'm referring to is 'how do you
install flit', or 'numpy', or other build time requirements. It must
be automated, such that pip can do it.

-Rob

-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 11 February 2016 at 01:21, M.-A. Lemburg <m...@egenix.com> wrote:
> On 10.02.2016 12:10, Paul Moore wrote:
>> On 10 February 2016 at 10:23, M.-A. Lemburg <m...@egenix.com> wrote:
>>> IMO, that's easy to achieve, though, with the existing de-facto
>>> standard interface we already have: the setup.py command line API.
>>> We'd just need to publish the minimal set of commands and options,
>>> installer will want to see implemented in order to initiate
>>> the builds.
>>
>> No-one who's investing time in writing PEPs is willing to thrash out
>> the details of how to use the setup.py interface in a formal proposal
>> that sticks to the sort of "minimum required" spec that alternative
>> tools would be willing to support. And there's no indication that tool
>> developers are willing to implement a setup.py compatible interface
>> format as you suggest. And finally, you'd need a way to declare that
>> pip installs tool X before trying to run setup.py.
>
> I don't think that installing 3rd party tools is within the scope
> of such a proposal. The setup.py of packages using such tools would
> have to either define a dependency to have the installer get the
> extra tool, download and install it directly when needed, or tell
> the user how to install the tool.
>
> Alternatively, the package distro could simply ship the tool
> embedded in the package. That's what we're doing with
> mxSetup.py.
>
>> So "easy to achieve" still needs someone to take the time to deal with
>> these sorts of issue. It's the usual process of the people willing to
>> put in the effort get to choose the direction (which is also why I
>> just provide feedback, and don't tend to offer my own proposals,
>> because I'm not able to commit that sort of time).
>
> Wait. You are missing the point that the setup.py interface
> already does work, so no extra effort is needed. All that's
> needed is some documentation of what's currently being used,
> so that other tools can support the interface going forward.
>
> At the moment, pip this interface is only defined by
> "what pip uses" and that's a moving target.

I disagree with the claim that setup.py already works.

Right now the vast majority of setup.py's will end up invoking
easy-install, which has entirely separate configuration to pip for
networking. It also means that pip is utterly incapable of caching and
reusing setup_requires dependencies, so when e.g. numpy is a
setup-requires dependency, build times can be arbitrarily high as
numpy gets built 3, 4, 5 or more times.

Secondly, the setup.py interface includes 'install' as a verb, and
there is pretty solid consensus that that is a bug - any definition of
'what pip uses' has to include that as a fallback, for the same
reasons pip has to fallback to that - but there is a substantial
difference between the end user UX setuptools provides, the interface
pip is forced to use, and the smaller one pip would /like/ to use.

So all the variants of the PEP we've been discussing are about a
modest step forward, capturing pip's existing needs and addressing the
setup-requires/easy-install bug as well as the use of install bug.

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Idea: Positive/negative extras in general and as replacement for features

[Robert Collins]

Sorry for not replying for so long.

On 16 December 2015 at 07:54, Ronny Pfannschmidt
<opensou...@ronnypfannschmidt.de> wrote:
> Hello everyone,
>
> the talk about the sqlalchemy feature extra got me thinking
>
> what if i could specify extras that where installed by default, but
> users could opt out
>
> a concrete example i have in mind is the default set of pytest plugins
> i would like to be able to externalize legacy support details of pytest
> without needing people to suddenly depend on pytest[recommended] instead
> of pytest to have what they know function as is
>
> instead i would like to be able to do something like a dependency on
> pytest[-nose,-unittest,-cache] to subtract the items


So the challenge here will be defining good, useful, and predictable
behaviour when the dependency graph is non-trivial.

Using your example, what should pip do when told

pip install pytest[nose, -unittest] proj2

and proj2 depends on pytest[unittest]

?

If it installs pytest[unittest], then the first pytest dependency was
no honoured. If it does not install pytest[unittest], then the proj2
dependencies were not honoured. So it must error.

-> Which means that using a [-THING] clause anywhere is going to be
super fragile, as indeed 'never install with X' things are in
distributions - its much better to find ways to express things purely
additively IMO.

There are many more complex interactions possible with the + / - DSL
you've sketched - and I don't think they are reducible - that is, its
not the DSL per se, but the basic feature of allowing cuts in the
graph traversal that lead to that complexity. If we can come up with
good, consistent, predictable answers, even considering three-party
interactions, then I've no objection per se: but I think that will be
very hard to do. I certainly don't have time at the moment to help -
sorry :(.

-Rob




-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 27 January 2016 at 22:24, Paul Moore <p.f.mo...@gmail.com> wrote:
> On 27 January 2016 at 05:39, Robert Collins <robe...@robertcollins.net> wrote:
>>> - You and Paul were strongly in favor of splitting up "working directories"
>>> and "sdists"; Robert and I didn't like the idea. The argument in favor is
>>> that we have to support working directory -> sdist and sdist -> wheel, so
>>> adding working directory -> wheel as an additional pathway creates larger
>>> test matrices and more opportunities for things to go wrong; the argument
>>> against is that this requires a new sdist format definition (currently it's
>>> basically "a zipped up working dir"), and it breaks like incremental
>>> rebuilds, which are a critical feature for developer adoption.
>>
>> That was something that occurred in the rabbit-hole discussion of your
>> first draft; The PR Donald is proposing to accept is I think the
>> fourth major iteration? Two from you, two from me - building on
>> feedback each time. While I don't think Donalds position here has
>> changed, the draft in question doesn't alter any semantics of anything
>> - it captures a subset of the semantics such that the existing
>> behaviour of pip can be modelled on the resulting interface. In
>> particular, for this question, it retains 'develop', which is the
>> primary developer mode in use today: it has no implications for or
>> against incremental builds - it doesn't alter pips copy-before-build
>> behaviour, which pip already has for non-develop installs. E.g. its
>> moot; we can debate that further - and i'm sure we shall - but it has
>> no impact on the interface.
>
> I'm still not clear on what the position is here. The PEP as I read it
> still offers no means to ask the build system to produce a sdist, so
> I'm concerned as to what will happen here.


> In the absence of a "pip sdist" command I can see that you're saying
> that we can implement pip's functionality without caring about this.
> But fundamentally, people upload sdists and wheels to PyPI. A build
> system that doesn't support sdists (which this PEP allows) will
> therefore have to publish wheel-only builds to PyPI, and I am very
> much against the idea of PyPI hosting "binary only" distributions.

As you know - 
https://mail.python.org/pipermail/distutils-sig/2015-March/026013.html
- flit /already/ does this. If we want to require sdists as a thing,
PyPI needs to change to do that. I don't care whether it does or
doesn't myself: folk that want to not ship sources will always find a
way. I predict empty sdists, for instance.

> If project foo has no sdist, "pip wheel foo" for a platform where
> there's no compatible wheel available will fail, as there's no way for
> pip to locate the source.

Thats true; but pip isn't a build system itself - and so far pip has
no role in the uploading of wheels to PyPI. Nor does - last I checked
- twine build things for you. The pattern is that developers prepare
there artifacts, then ask twine to upload them.

> So can we please revisit the question of whether build systems will be
> permitted to refuse to generate sdists? Note that I don't care whether
> we formally define a new sdist format, or go with something adhoc, or
> whatever. All I care about is that the PEP states that build systems
> must support generating a file that can be uploaded to PyPI and used
> by pip to build a wheel as described above (not "git clone this
> location and do 'pip wheel .'"). I think that making build systems
> expose how you make such a file by requiring a "sdist" subcommand is a
> reasonable approach, but I'm OK with naming the subcommand
> differently.

I truely have no opinion here. I don't think it harms the abstract
build system to have it, but I do know that Donald very much does not
want pip to have to know or care about building sdists per se, and he
may see this as an attractive nuisance.

Who / what tool do we expect to use the sdist command in the abstract interface?

> Thanks,
> Paul
>
> PS I agree with Nathaniel, insofar as there were definitely a number
> of unresolved points remaining after the various public discussions,
> and I don't think it's reasonable to say that any proposal is "ready
> for implementation" without a public confirmation that those issues
> have been resolved.

I suspect it had been quescient too long and folk had paged out the
state. Certainly I don't think Donald intended to bypass any community
debate, and neither did I.

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 10 February 2016 at 11:40, Paul Moore <p.f.mo...@gmail.com> wrote:
> On 9 February 2016 at 21:38, Robert Collins <robe...@robertcollins.net> wrote:
>>> In the absence of a "pip sdist" command I can see that you're saying
>>> that we can implement pip's functionality without caring about this.
>>> But fundamentally, people upload sdists and wheels to PyPI. A build
>>> system that doesn't support sdists (which this PEP allows) will
>>> therefore have to publish wheel-only builds to PyPI, and I am very
>>> much against the idea of PyPI hosting "binary only" distributions.
>>
>> As you know - 
>> https://mail.python.org/pipermail/distutils-sig/2015-March/026013.html
>> - flit /already/ does this. If we want to require sdists as a thing,
>> PyPI needs to change to do that. I don't care whether it does or
>> doesn't myself: folk that want to not ship sources will always find a
>> way. I predict empty sdists, for instance.
>
> I know flit does this. We're obviously never going to be able to
> *stop* people generating wheels any way they like. But there is value
> in having a standard invocation that builds the wheel - anyone who has
> ever used make is aware that a standardised build command has value.
>
> At the moment, that standard is "pip wheel foo". Or maybe it's
> "setup.py bdist_wheel", but I prefer the layer of build system
> abstraction offered by pip.

But 'pip wheel foo' already reuses existing wheels: things like flit
that upload universal wheels work with 'pip wheel foo' just fine (as
long as --no-binary has not been passed). Things that upload an sdist
will also still work, unchanged, with this proposal.

>>> If project foo has no sdist, "pip wheel foo" for a platform where
>>> there's no compatible wheel available will fail, as there's no way for
>>> pip to locate the source.
>>
>> Thats true; but pip isn't a build system itself - and so far pip has
>> no role in the uploading of wheels to PyPI. Nor does - last I checked
>> - twine build things for you. The pattern is that developers prepare
>> there artifacts, then ask twine to upload them.
>
> As someone who frequently builds wheels for multiple projects, where
> those projects don't have complex build requirements but the projects
> themselves do not upload wheels, I find significant value in being
> able to just do "pip wheel foo". Maybe it's not pip's core
> functionality, maybe in the long term it'll be replaced by a separate
> utility, but the key point is that one command, given a project name,
> locates it on PyPI, downloads it and builds the source into a wheel.
>
> Having that capability is a *huge* benefit (I remember the days before
> distutils and PyPI, when every single package build was a process of
> locating files on obscure websites, and struggling with custom build
> processes - I'm admittedly over-sensitive about the risks of going
> back to that).
>
> I don't care in the slightest how build systems implement
> " sdist". They can write a custom "setup.py" script that
> downloads the real sources from github and bulids them using flit, for
> all I care.

I'm struggling to connect 'pip wheel foo' to 'they can do an sdist'.
'pip wheel foo' does not invoke sdist. There is a bug proposing that
it should ( https://github.com/pypa/pip/issues/2195 and
https://github.com/pypa/pip/pull/3219 ) - but that proposal is
contentious - see the ongoing related debate about editable mode /
incremental builds and also the discussion on PR 3219.

> What I do care about is ensuring that if a projects hasn't published a
> wheel for some particular environment, and the project is *not*
> actively trying to avoid publishing sources, then we provide them with
> a means to publish those sources in such a way that "pip wheel
> theproject" (or whatever command replaces it as the canonical "build a
> wheel" command) can automate the process of generating a wheel for the
> user's platform.

Sure, they should have that. I don't understand why pip's
*abstraction* over project *building* should know about that. Issue
2195 aside, which seems to be entirely about avoiding one particular
developer failure mode, and other than that makes no sense to me.


> Of course projects who want to make their sources unavailable can, and
> will. I'm not trying to force them not to.
> Of course users can manually locate the sources and build from them. I
> believe that is a major step back in usability.
> Of course projects can upload wheels. They will never cover all platforms.
>
> Longer term, do we want a standard "source distribution" format? I
> thought we did (Natha

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 10 February 2016 at 13:09, Paul Moore <p.f.mo...@gmail.com> wrote:
> [I need to read and digest the rest of this, but it's late here, so
> that will be tomorrow]

OK, cool.

> On 9 February 2016 at 23:19, Robert Collins <robe...@robertcollins.net> wrote:
>>>> Who / what tool do we expect to use the sdist command in the abstract 
>>>> interface?

>>> I do not accept any proposal that removes "pip wheel " without
>>> providing *any* replacement.
>>
>> But the current proposal *DOES NOT REMOVE IT*.
>
> By  I had in mind "project name", implying "download from
> PyPI". And by "remove" i meant "open up the possibility of people
> using tools that don't support easy creation of source artifacts that
> can be uploaded to PyPI, resulting in pip not being able to find
> something to download".

Sure - but Nathaniel and I both seem to think that the PEP doesn't
make it any easier to do that - and in fact should allow flit to start
uploading source artifacts (by allowing pip to consume it's sdists),
optionally with a setuptools_shim style setup.py.

>> We're clearly miscommunicating about something :).
>
> Yes, and that's probably my fault. I need to go back and reread the
> PEP and the thread.
>
> But as I said in my response to Nathaniel, it may be that all that is
> needed is some context in the PEP explaining how we require[1] people
> to upload source to PyPI in the new world where we support build
> systems which don't have a "sdist" command like setuptools does.
>
> Paul
>
> [1] I say "require" in the sense of "you have to follow these rules if
> pip is to be able to use your source", not "you must upload source" -
> although I hope that the number of people actually preferring to *not*
> include source in their PyPI uploads is vanishingly small...

So, I'm not against us making a statement like that, but I don't think
it belongs in this PEP - it should be in the main PyPI docs/rules,
surely?

-Rob

-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Fwd: How to get pip to really, really, I mean it -- rebuild this damn package!

[Robert Collins]

Bah, offlist by mistake.


-- Forwarded message --
From: Robert Collins <robe...@robertcollins.net>
Date: 30 January 2016 at 09:25
Subject: Re: [Distutils] How to get pip to really, really, I mean it
-- rebuild this damn package!
To: Chris Barker - NOAA Federal <chris.bar...@noaa.gov>


Please try pip 7.1? Latest before 8; we're not meant to be caching
wheels of by-location things from my memory, but it may have
regressed/changed with the cache changes made during the 8 development
cycle.

-Rob

On 30 January 2016 at 04:48, Chris Barker - NOAA Federal
<chris.bar...@noaa.gov> wrote:
>>>  Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3 from
>>> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in
>>> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>>
>> I think this is saying that pip thinks it has found an
>> already-installed version of gsw 3.0.3 in sys.path, and that the
>> directory in your sys.path where it's already installed is
>>
>> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>
> That is the temp dir conda sets up to unpack downloaded files, and do
> its work in -- hence the name. I'll look and see what's there. I'm
> pretty sure conda build starts out with an empty dir, however. And
> that dir should not be on sys.path.
>
>> I think this means that that directory is (a) in sys.path, and (b)
>> contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a)
>> seems weird and broken.
>
> Indeed. And I get the same symptoms with a clean environment that I've
> set up outside conda build. Though with the same source dir. But with
> conda build, it's a fresh unpack of the tarball.
>
>> Do you have "." in your PYTHONPATH or anything like that?
>
> God no!
>
>> Don't know why it seems to be building a wheel for it, if it already
>> thinks that it's installed... this is also odd.
>
> Yes it is. But it doesn't install it :-(
>
>>
>> $PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./
>>
>> ? Though I'd think that -I would have the same affect as --force-reinstall...
>>
> So did I, and I think I tried --force-reinstall already, but I will again.
>
>> (It doesn't look like the cache dir is your problem here, but you do
>> probably want to use --no-cache-dir anyway just as good practice, just
>> because you don't want to accidentally package up a stale version of
>> the software that got pulled out of your cache instead of the version
>> you thought you were packaging in the tree in front of you.
>
> Exactly. Doesn't seem to make a difference, though.
>
>> Also, I think it's a bug in pip that it caches builds of source trees
>> -- PyPI can enforce the rule that each (package name, version number)
>> sdist is unique, but for a work-in-progress VCS checkout it's just not
>> true that (package name, version number) uniquely identifies a
>> snapshot of the whole tree. So in something like 'pip install .', then
>> requirement resolution code should treat this as a special requirement
>> that it wants *this exact tree*, not just any package that has the
>> same (package name, version number) as this tree; and the resulting
>> wheel should not be cached.
>
> Absolutely! In fact, I'll bet that approach is the source of the
> problem here. If not automagically, there should be a flag, at least.
>
> However, what seems to be happening is that pip is looking outside the
> current Python environment somewhere to see if this package needs to
> be installed. It may be something that works with virtualenv, but
> doesn't with conda environments for some reason.
>
> I guess on some level pip simply isn't designed to build and install
> from local source :-(
>
> In the end, I'm still confused: does pip install give me anything that:
>
> setup.py install single-version-externally-managed
>
> Doesn't? Other that support for non-setuptools installs, anyway.
>
> CHB
>
>
>> I don't know if there are any bugs filed
>> in pip on this...)
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig



--
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

 correct ;-), while the third is more confusing and more of a
> judgement call.
>
> I haven't looked at the current version of the PEP, so the above may be out
> of date, but I think it's accurate WRT what's been discussed on the mailing
> list. Have there been some private video meetings where all these issues got
> hashed out somehow, or...?

No; there was the one video call, which was advertised here, and then
the subsequent drafts. The PR hasn't been changed in a couple of
months: in that time we've had Xmas, various work things, hacking
older pip version compat stuff into pbr, and me nagging Donald a lot
to get a review of the PR :).

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

snapshot of earlier drafts
of PEP-426, so its really not all that well defined, and before folk
can iterate on that further we'll need to do something about that -
either by a rewrite of PEP 426 into a much more pithy thing now,
without the enterprise feel and the moon-shot aspects, or by issuing a
PEP that specifies exactly what is in METADATA.

> Well, at least now it looks like we can forget about that whole debate
> about the need to quickly get metadata from scipy sdists, since
> apparently scipy will have wheels on all supported platforms before
> pip implements either the new build system interface or the resolver
> :-). Of course there will still be an issue for more obscure platforms
> and for projects whose developers can't be arsed to provide wheels.

There is no install command in the interface; if projects can't
provide wheels, they can't migrate to the new abstract system - it may
be that some projects need an updated wheel spec to be able to
migrate, and it may also be that pip retains the ability to run
setup.py install for unmigrated projects forever (since pypi has deep
history), but from a design perspective, we are explicitly assuming
'can make wheels'. - Thats reasonable. 'making wheels is cheap' is not
so reasonable :).

> [...]
>> No; there was the one video call, which was advertised here, and then
>> the subsequent drafts. The PR hasn't been changed in a couple of
>> months: in that time we've had Xmas, various work things, hacking
>> older pip version compat stuff into pbr, and me nagging Donald a lot
>> to get a review of the PR :).
>
> Plus all your great work on getting PEP 508 sorted out so it wouldn't
> block this build system stuff :-).

Yah - that was a group effort though, I was just the ring leader :0

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

Cool - thank you.

Right now it looks like there is one edit (add PATH as a variable folk
can depend on being set) and one undecided (metadata file format).

I've pushed a shim that permits having a setup.py for non-setuptools
projects using solely the abstract interface to
https://pypi.python.org/pypi/setuptools_shim and
https://github.com/rbtcollins/setuptools_shim

It depends on https://github.com/pypa/packaging/pull/50 - the test
suite is hardcoded to locate that in ../packaging at the moment - but
I believe that that will be merging soon, so it should be generally
usable from there on out.

I'd love it if someone put together a flit implementation of the
abstract build system and checked that the shim does let pip work :).
Theory and practice and all that.

-Rob


On 27 January 2016 at 14:08, Donald Stufft <don...@stufft.io> wrote:
> As many know there has been an effort to get a generalized interface that a 
> build system can implement so that we can break the hard dependency on 
> setuptools that Python packaging currently has. After a lot of deliberation 
> and threads back and forth (as well as video meetings!) I think that the 
> version of the PEP that is currently on the PR in 
> https://github.com/pypa/interoperability-peps/pull/54 looks like it’s 
> generally the right thing to move forward with. I made a few little comments 
> but overall I think it’s there and we’re ready to move forward on trying to 
> get a reference implementation done that can validate the decisions made in 
> that PEP (and then, hopefully finalize the PEP and merge those 
> implementations).
>
> So many thanks to everyone involved in hammering this out thus far :) I’m 
> nervous but excited about the possibility of making setuptools just another 
> build system.
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Robert Collins]

On 23 January 2016 at 07:10, Donald Stufft <don...@stufft.io> wrote:
> PEP 376 added a file to the .dist-info directory called "INSTALLER" which was
> supposed to be:
>
> This option is the name of the tool used to invoke the installation.
>
> However, nothing has really ever implemented it and it's gone largely ignored
> until just recently pip 8.0 started writing the INSTALLER file into the
> metadata directories with a value of "pip".
>
> I'd like to propose adding a special cased value to add to the installer file
> that will tell projects like pip that this particular installed thing is being
> managed by someone else, and we should keep our hands off of it. According to
> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I think
> since nobody has ever implemented it, we could expand that so that it so you
> can also have a special value, of "dpkg (system)" or maybe that's not worth it
> and we could just have "system" as a special value.
>
> The benefit of doing this, is that with a special value in that file that says
> "this file belongs to the OS", then pip could start looking for that file and
> require a --force flag before it modifies any files belonging to that project.
> Then distributors like Debian, Fedora, etc could simply write out the 
> INSTALLER
> file with the correct value, and pip would start to respect their files by
> default.
>
> Thoughts? Does this sound reasonable to everyone? Do we think it needs a new
> PEP or can we just amend PEP 376 to add this extra bit of information?

I think asking other distros to export the installing information to
us is a good thing.

I think requiring a force flag to replace files installed by another
packaging tool is an unbreakme option as things stand, and so I'm very
concerned about the resulting user experience.

If someone runs 'sudo pip install -U pip' they are already signalling
precisely what they want, and as a user I very much doubt that:

"Refusing to uninstall pip because it was not installed by pip"

will make even 5% of them pause longer than required to look up the
--force-alternative-installer flag (or whatever it is) and pass it in.

The only reason we're touching these other installer installed files
is because there is no location on the Python and exec paths where we
can install the new pip without overwriting the one dpkg (or conda or
whatever) installed. If there was such a place we could just write
(and upgrade things) there and not have to deal with shadowing, or
ever uninstalling dpkg etc installed files. In the absence of such a
place, removing the other-install-system's installed files is the
right thing to do when a user asks us to, without needing an
additional option.

We could add an option folk could set to turn *on* such a safety belt
- but I don't know who it would /really/ be helping. (I'm fairly sure
I know the places folk think it would help, I just don't agree that it
would :)).

Having a interface to the system package manager as has been
previously mooted - and I'm going to get back to that - might help a
little, but uninstalls are quite different to installs. Uninstalls get
blocked by things like 'app-foo requires requests', and I would be
super suprised (and delighted!) if we were able to override that when
upgrading requests via pip :)

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 440 ad PEP 508 requirement formats

[Robert Collins]

Oh that reminds me, I noticed the other day that PEP 426 references
James' draft markers PEP that actually ended up in 508. Whats the
process for fixing that? Just do it in hg?

On 17 January 2016 at 04:46, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 16 January 2016 at 00:12, Paul Moore <p.f.mo...@gmail.com> wrote:
>> Pip refers to PEP 440 when defining the format of a requirement (see
>> https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers).
>> But PEP 508 *also* defines a requirement - the title implies that it's
>> for dependency specification, but the content of the PEP says "the
>> language defined is a compact line based format which is already in
>> widespread use in pip requirements files".
>>
>> So what's the relationship between PEP 440 and PEP 508? Which one is
>> the definitive definition of what is a "requirement"? The "packaging"
>> library implements specifiers which conform (according to the docs) to
>> PEP 440.
>
> As Donald already noted, 508 is the higher level specification, that
> includes 440 by reference to cover the "version specifier" section of
> a full dependency specifier
>
> The key parts that 508 adds above and beyond 440 are:
>
> - package names (the current rules were previously only given in the 426 
> draft)
> - extras
> - environment markers
>
> The main benefit of keeping the two levels separate is that there's a
> *lot* of arcana in PEP 440 around version ordering and how that
> relates to version comparison operators that you don't really need to
> think about when working at the level of processing a list of
> dependency specifiers.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup.py install using pip

[Robert Collins]

Thanks for digging that up.

-Rob

On 24 December 2015 at 06:35, Erik Bray <erik.m.b...@gmail.com> wrote:
> On Thu, Dec 10, 2015 at 5:12 PM, Robert Collins
> <robe...@robertcollins.net> wrote:
>> On 8 December 2015 at 09:14, Erik Bray <erik.m.b...@gmail.com> wrote:
>>> On Mon, Dec 7, 2015 at 2:40 PM, Paul Moore <p.f.mo...@gmail.com> wrote:
>>>> On 7 December 2015 at 18:58, Erik Bray <erik.m.b...@gmail.com> wrote:
>>>>> I wasn't able to produce this problem.  Even with --no-binary
>>>>> specified pip installs (by default) with
>>>>> --single-version-externally-managed.  My prototype implicitly disables
>>>>> the --pip flag if --single-version-externally-managed was specified
>>>>> (true to the purpose of that flag).
>>>>
>>>> Ah - that was the bit I was missing, the
>>>> --single-version-externally-managed flag can be used to trigger
>>>> ignoring --pip.
>>>>
>>>>> What *is* a problem is if --pip is in setup.cfg, and one invokes `pip
>>>>> install --egg .`.  I wasn't quite able to make that go into an
>>>>> infinite loop, but it did invoke pip.main recursively, and stuff broke
>>>>> on the second invocation for reasons not clear to me.
>>>>
>>>> Yeah, but honestly I don't think pip install --egg is that important a
>>>> use case. I may be wrong (there's lots of ways people use pip that I
>>>> know nothing of :-)) but as a starting point it might be OK just to
>>>> say that at the same time as the --pip flag was introduced, "pip
>>>> install --egg" was deprecated (and we explicitly document that pip
>>>> install --egg is known to interact badly with setup.py --pip).
>>>
>>> I'd be fine with that too.  IIRC pip install --egg was introduced in
>>> part to work around problems with namespace packages.  This doesn't
>>> completely eliminate the need for that workaround, but it does reduce
>>> it.
>>
>> Huh? No, my understanding was that it was introduced solely to support
>> interop with folk using 'easy-install', and its considered deprecated
>> and delete-as-soon-as-practical.
>
> The original issue that motivated it did have to do with (lack of)
> interoperability of different ways namespace packages are implemented:
>
> https://github.com/pypa/pip/issues/3
>
> The fact that it introduced general backwards-compat for
> easy-install-like installation was a side "benefit", useful I'm sure
> to a few people.  But otherwise as you say, was intended to be deleted
> as soon as practical.
>
> Erik



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pypi simple index Inbox x

[Robert Collins]

As I answered to your python-dev post, this is due to easy-install not
supporting wheel. The fix is easy - don't use easy-install.

-Rob

On 18 December 2015 at 06:27, Carlos Barera <carlos.bar...@gmail.com> wrote:

> Hi,
>
> I'm using install_requires in setup.py to specify a specific package my
> project is dependant on.
> When running python setup.py install, apparently the simple index is used
> as an older package is taken from pypi. While in
> https://pypi.python.org/pypi, there's a newer package.
> When installing directly using pip, the latest package is installed
> successfully.
> I noticed that the new package is only available as a wheel and older
> versions of setup tools won't install wheels for install_requires.
> However, upgrading setuptools didn't help.
>
> Several questions:
> 1. What's the difference between the pypi simple index and the general
> pypi index?
> 2. Why is setup.py defaulting to the simple index?
> 3. How can I make the setup.py triggered install use the main pypi index
> instead of simple
>
> Thanks!
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Metadata 2.0: Warning if optional features are missing

[Robert Collins]

 - in widespread deployment - entirely and
only a list of additional distributions.
 - there's no concept of 'default extra', and there is no clear path
for bringing it in compatibly, at least so far
 - we haven't worked through the ui implications about which end of
the relation this should be configured: should consumers be specifying
them, or providers?
 - negative operators on extras are as yet undefined, and due to the
dependencies of an install being a graph, not a tree, a naive
definition is likely very hard to use IMO

Recommends and suggests are an interesting way of modelling this, and
its possible we don't need an exclude relation- rather users should
blacklist them globally in the target environment somehow, which would
contain that partcular complexity.

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] workflow recommendations to update requirements.txt

[Robert Collins]

You might find 
https://github.com/openstack/requirements/blob/master/openstack_requirements/cmds/generate.py
useful.

Basically it takes a open-ended (or partly so) set of requirements,
expressed as in requirements.txt format, and generates an exact-match
requirements.txt file as output.

So, have one file, say 'dependencies.txt', which is open ended, and
your requirements.txt which is exact, then run generate from time to
time. There are other such tools around, but I forget their names ;)

-Rob

On 16 December 2015 at 17:56, Chris Jerdonek <chris.jerdo...@gmail.com> wrote:
> Hi,
>
> I have a development workflow question I was wondering if people on
> this list had a recommended solution for.
>
> Say you're working on a web application that you deploy using a
> requirements.txt file.  And say you have a set of "abstract
> dependencies" that your application depends on.
>
> What are some convenient ways of storing your abstract dependencies in
> source control and periodically generating an updated requirements
> file from that information (e.g. when your dependencies come out with
> new versions)?
>
> The main idea that occurs to me is making a setup.py for the purposes
> of representing your abstract dependencies (e.g. using
> "install_requires," etc), creating a new virtualenv, running "pip
> install .", and then "pip freeze."
>
> One problem with this approach is that the pip freeze output includes
> an entry for the setup.py application itself, when the output should
> only include the _dependencies_ of the application and not the
> application itself.  It also seems clunky to me to create a virtualenv
> and install dependencies only for the purposes of computing
> dependencies.
>
> Thanks for any help or suggestions.
>
> --Chris
> ___________
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup.py install using pip

[Robert Collins]

On 8 December 2015 at 09:14, Erik Bray  wrote:
> On Mon, Dec 7, 2015 at 2:40 PM, Paul Moore  wrote:
>> On 7 December 2015 at 18:58, Erik Bray  wrote:
>>> I wasn't able to produce this problem.  Even with --no-binary
>>> specified pip installs (by default) with
>>> --single-version-externally-managed.  My prototype implicitly disables
>>> the --pip flag if --single-version-externally-managed was specified
>>> (true to the purpose of that flag).
>>
>> Ah - that was the bit I was missing, the
>> --single-version-externally-managed flag can be used to trigger
>> ignoring --pip.
>>
>>> What *is* a problem is if --pip is in setup.cfg, and one invokes `pip
>>> install --egg .`.  I wasn't quite able to make that go into an
>>> infinite loop, but it did invoke pip.main recursively, and stuff broke
>>> on the second invocation for reasons not clear to me.
>>
>> Yeah, but honestly I don't think pip install --egg is that important a
>> use case. I may be wrong (there's lots of ways people use pip that I
>> know nothing of :-)) but as a starting point it might be OK just to
>> say that at the same time as the --pip flag was introduced, "pip
>> install --egg" was deprecated (and we explicitly document that pip
>> install --egg is known to interact badly with setup.py --pip).
>
> I'd be fine with that too.  IIRC pip install --egg was introduced in
> part to work around problems with namespace packages.  This doesn't
> completely eliminate the need for that workaround, but it does reduce
> it.

Huh? No, my understanding was that it was introduced solely to support
interop with folk using 'easy-install', and its considered deprecated
and delete-as-soon-as-practical.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP, take #2

[Robert Collins]

On 10 December 2015 at 22:28, M.-A. Lemburg <m...@egenix.com> wrote:
> In all this discussion, please don't forget that distutils and
> setuptools differentiate between building the code and creating
> a distribution file.



> The later is not restricted to just the sdist and wheel formats.
> distutils can create a wide variety of other distribution formats
> such as MSI files, Windows .exe installers, binary in-place
> distributions. With extensions it's possible to create the
> ActiveState package format, RPMs, DEBs, Solaris PKG files and
> other formats such as our eGenix prebuilt format or web
> installers.

None of which can be installed by pip, which is the blessed
installation path, nor can most of them be uploaded to PyPI, which is
the index format we're working too.

> Apart from that it's also possible to have distutils
> completely drop the distribution file generation and go
> straight to installation after the build step, e.g.
> when not using a package manager at all.

This is something there has been clear consensus on getting away from.
Different distributions (e.g. RPM/Deb/Conda/etc) may have different
requirements around how installation is done, and getting away from
the cross product to a single interop point is a key design point.

> IMO, it would be better to use the existing "setup.py build"
> and "setup.py bdist_wheel" APIs to create a build system
> abstraction, since that'll keep the other existing distribution
> methods working in the same context, which is important since
> pip is not the only way to distribution Python packages.

Since 'build' and 'bdist_wheel' are defacto, and since setuptools is
infinitely variable... I'd like it if you could specify what you want,
rather than just referring to those commands. As it stands I've read
your mail but don't understand what is missing from the proposed
contract, nor what things are present but objectionable.

> The PEP's ideas as well as many other approaches to building
> packages can be implemented using a "setup.py build"
> and "setup.py bdist_wheel" interface ("bdist_wheel" will
> implicitly run "build").

A second key design input was that competing build system authors *do
not want* to have a setup.py in the tree of projects using them
(specifically the flit folk put this forward) - because they aren't
setuptools, and blindly running setuptools commands against their UI
won't work... and they don't want to offer all the complexity of
setuptools commands because 99% of it is irrelevant to them and their
users.

> To specifically use the PEP's mechanism, a package could be
> created which overrides and implements the PEP's build strategy,
> e.g. "pep778build" (assuming the PEP number is 778 as example).
>
> The dependency of a package on this pep778build package
> would then signal the compatibility of the package with
> the PEP's build mechanism.

That would fail to be bootstrappable, since there is no programmatic
API to get build requirements out of setuptools packages today without
triggering easy-install. At a minimum we'd have to address that before
we could consider utilising setup.py as the contract itself.

> In summary: As long as the final result of a call to
> "setup.py bdist_wheel" is a .whl file in dist/, pip should be
> able to continue working as normal (and as it does today),
> regardless of what "setup.py build" does under the hood.

pip will be able to keep working if folk don't opt into the new
system, because backwards compat with the 10's of thousands of
distributions on PyPI is a big deal - I don't see that being
deliberately broken ever.

And, a setuptools_pepxxx adapter should be easy to write - in fact I
plan to write a pbr_pepxxx adapter which will be nearly identical, as
a proof of concept - for folk that want to allow easy-install to not
be involved but still use setuptools.

-Rob

-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP, take #2

[Robert Collins]

On 10 December 2015 at 08:59, Ralf Gommers <ralf.gomm...@gmail.com> wrote:
>
>
> On Wed, Dec 9, 2015 at 2:55 AM, Robert Collins <robe...@robertcollins.net>
> wrote:
>>
>> Updated - tl;dr:
>>
>> The thing I'm least happy about is that implementing install support
>> will require recursively calling back into pip, that or reimplementing
>> the installation of wheels logic from within pip - because
>> sufficiently old pip's won't call wheel at all.
>
>
> You're specifying a new interface here, and updating pip itself is quite
> easy. So why would you do things you're not happy about to support
> "sufficiently old pip's"?

Because the goal is to make the barrier to entry for new build systems
low. Right now, for instance, flit doesn't support building an sdist
(From its pypi page : "

Flit only creates packages in the new ‘wheel’ format. People using
older versions of pip (<1.5) or easy_install will not be able to
install them. People may also want a traditional sdist for other
reasons, such as Linux distro packaging. I hope that these problems
will diminsh over time.

")

The current defacto contract is:
 - egg-info
 - install
 - wheel
 - develop

The issues with this are:
 - install is a responsibility for pip/conda etc, not build systems
 - ditto develop
 - there's no way to bootstrap the build system
 - there's no way to determine build requirements

Of those four things, two are very low hanging fruit (the last two),
one is nontrivial - develop mode - and one is inbetween - since we're
requiring 'wheel', "install" is something pip can do itself with a
trivial change... but it shuffles the complexity onto the setuptools
shim, which we likely want to enable rapid adoption.

So we can choose:
 - delay the ability for pip/conda etc to own the install stage [by
putting install in the contract]
 - centralise that complexity by having setuptools_shim wear it

So - so far the second is better I think, I just don't like it :)

>> And even modern pips
>> can be told *not to call wheel*.
>
>
> Isn't that something you can ignore? If the plan for pip anyway is to always
> go sdist-wheel-install, why support this flag for a new build interface?

Well, there's still debate about that. I think its waste and will piss
developers off (heck, even in tox OpenStack folk find sdist too long
and disable it routinely - we've added CI checks that sdist doesn't
error to allow keeping the local developer workflow smooth).

-Rob



-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP, take #2

[Robert Collins]

On 10 December 2015 at 10:07, Donald Stufft <don...@stufft.io> wrote:
>
>> On Dec 9, 2015, at 3:56 PM, Robert Collins <robe...@robertcollins.net> wrote:
>>
>> On 10 December 2015 at 08:59, Ralf Gommers <ralf.gomm...@gmail.com> wrote:
>>>
>>>> And even modern pips
>>>> can be told *not to call wheel*.
>>>
>>>
>>> Isn't that something you can ignore? If the plan for pip anyway is to always
>>> go sdist-wheel-install, why support this flag for a new build interface?
>>
>> Well, there's still debate about that. I think its waste and will piss
>> developers off (heck, even in tox OpenStack folk find sdist too long
>> and disable it routinely - we've added CI checks that sdist doesn't
>> error to allow keeping the local developer workflow smooth).
>>
>
> I’m in process of moving so I’m a bit scattered brained at the moment and I 
> don’t have the time to look into the specifics but if this is for the build 
> interface (vs the shim) then I don’t think we should support plain 
> ``install``. Opting into the new format should mandate the capability of 
> producing a wheel and then installing from that instead of being able to 
> directly install.

It is neither; Ralf was referring to the long term pip internals
stuff. The new format does mandate wheel and does not support direct
'install', nor require building an sdist.

> If we consider the setuptools/distutils era to be “Make it Work”, then we’re 
> now at “Make it Right”, making it fast can come later but sacrificing 
> correctness for speed isn’t something I think we should be doing and so speed 
> arguments (vs why it’s more correct to do X instead of Y) don’t matter much 
> to me.

Developer speed is a correctness issue: this took ages to get my head
fully around, but at the heart of it, there's a very narrow window
between in-flow and breaking-flow and the reason developers care so
much about latency of local operations is staying in-flow.

Yes, there are a wide set of correctness issues to preserve, but if we
can't do that and retain a certain minimum performance level,
developers will route around us, and we'll be legislating things folk
will ignore. The current interface, preserving develop, is sufficient
for now, I was commenting in my mail on the longer term view that Ralf
was suggesting: tl;dr - this whole sub-bit is not a subject for now.

-Rob


-- 
Robert Collins <rbtcoll...@hpe.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Partial installs for in-process distribution upgrades (was: Installing packages using pip)

[Robert Collins]

On 10 December 2015 at 08:39, Erik Bray  wrote:
> Apologies for resending this--my original message got buried in
> unrelated commentary about how *nix filesystems work.  Resending with
> new subject line...

I think its going to break in nasty ways for users on Windows, on a
regular basis.

For instance: start two interpreters at once with a partially
installed thing present and watch them race.

For instance: put a .dll extension in a partially installed thing and
start a second interpreter without closing the one that triggered the
install. Watch the move error because the .dll is locked.

(And this is ignoring the questions around module namespaces in
sys.modules and the poorly-supported-in-practice semantics of
reload()).

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP, take #2

[Robert Collins]

Updated - tl;dr:
- specify encoding of stdout streams that matter
- change build command template to a list of strings so we don't need
to attempt to address shell escaping for paths with spaces(or other
such things)
- be clear about which variables apply in command templating, and
which are environment variables [we dont' want an implicit dependency
on shell, since this is for Windows too.

I've written a proof of concept of a  setuptools shim (looks like
setup.py, calls the build system from pypa.json) - right now it only
handles 'develop', so its missing wheel support, and for pip < 6 also
install support.

It is however sufficient to let us reason about things.
https://pypi.python.org/pypi/setuptools_shim

The thing I'm least happy about is that implementing install support
will require recursively calling back into pip, that or reimplementing
the installation of wheels logic from within pip - because
sufficiently old pip's won't call wheel at all. And even modern pips
can be told *not to call wheel*.

-Rob

diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst
index 762cd88..a6e4712 100644
--- a/build-system-abstraction.rst
+++ b/build-system-abstraction.rst
@@ -84,18 +84,19 @@ bootstrap_requires
 Optional list of dependency specifications [#dependencyspec] that must be
 installed before running the build tool. For instance, if using flit, then
 the requirements might be::
-
+
 bootstrap_requires: ["flit"]

 build_command
-A mandatory Python format string [#strformat]_ describing the command to
-run. For instance, if using flit then the build command might be::
+A mandatory key, this is a list of Python format strings [#strformat]_
+describing the command to run. For instance, if using flit then the build
+command might be::

-build_command: "flit"
+build_command: ["flit"]

 If using a command which is a runnable module fred::

-{PYTHON} -m fred
+build_command: ["{PYTHON}", "-m", "fred"]

 Process interface
 -
@@ -114,14 +115,23 @@ redirected and no communication with the user is possible.

 As usual with processes, a non-zero exit status indicates an error.

-Available variables

+Available format variables
+--

 PYTHON
 The Python interpreter in use. This is important to enable calling things
 which are just Python entry points.

-${PYTHON} -m foo
+{PYTHON} -m foo
+
+Available environment variables
+---
+
+These variables are set by the caller of the build system and will always be
+available.
+
+PYTHON
+As for format variables.

 PYTHONPATH
 Used to control sys.path per the normal Python mechanisms.
@@ -133,9 +143,9 @@ There are a number of separate subcommands that
build systems must support.
 The examples below use a build_command of ``flit`` for illustrative purposes.

 build_requires
-Query build requirements. Build requirements are returned as a JSON
-document with one key ``build_requires`` consisting of a list of
-dependency specifications [#dependencyspec]_. Additional keys must be
+Query build requirements. Build requirements are returned as a UTF-8
+encoded JSON document with one key ``build_requires`` consisting of a list
+of dependency specifications [#dependencyspec]_. Additional keys must be
 ignored. The build_requires command is the only command run without
 setting up a build environment.

@@ -145,9 +155,9 @@ build_requires

 metadata
 Query project metadata.  The metadata and only the metadata should
-be output on stdout. pip would run metadata just once to determine what
-other packages need to be downloaded and installed. The metadata is output
-as a wheel METADATA file per PEP-427 [#pep427]_.
+be output on stdout in UTF-8 encoding. pip would run metadata just once to
+determine what other packages need to be downloaded and installed. The
+metadata is output as a wheel METADATA file per PEP-427 [#pep427]_.

 Note that the metadata generated by the metadata command, and the metadata
 present in a generated wheel must be identical.
@@ -169,7 +179,7 @@ wheel -d OUTPUT_DIR
 develop [--prefix PREFIX] [--root ROOT]
 Command to do an in-place 'development' installation of the project.
 Stdout and stderr have no semantic meaning.
-
+
 Not all build systems will be able to perform develop installs. If a build
 system cannot do develop installs, then it should error when run. Note
 that doing so will cause use operations like ``pip install -e foo`` to
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP, take #2

[Robert Collins]

Updated with:
 - the pep 0508 reference
 - reviews from github
 - document --prefix and --root to develop.

-Rob

diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst
index d36b7d5..762cd88 100644
--- a/build-system-abstraction.rst
+++ b/build-system-abstraction.rst
@@ -40,9 +40,9 @@ easy-install.
 As PEP-426 [#pep426]_ is draft, we cannot utilise the metadata format it
 defined. However PEP-427 wheels are in wide use and fairly well specified, so
 we have adopted the METADATA format from that for specifying distribution
-dependencies. However something was needed for communicating bootstrap
-requirements and build requirements - but a thin JSON schema is sufficient
-when overlaid over the new dependency specification PEP.
+dependencies and general project metadata. PEP-0508 [#pep508] provides a self
+contained language for describing a dependency, which we encapsulate in a thin
+JSON schema to describe bootstrap dependencies.

 Motivation
 ==
@@ -123,6 +123,8 @@ PYTHON

 ${PYTHON} -m foo

+PYTHONPATH
+Used to control sys.path per the normal Python mechanisms.

 Subcommands
 ---
@@ -164,7 +166,7 @@ wheel -d OUTPUT_DIR

 flit wheel -d /tmp/pip-build_1234

-develop
+develop [--prefix PREFIX] [--root ROOT]
 Command to do an in-place 'development' installation of the project.
 Stdout and stderr have no semantic meaning.

@@ -173,6 +175,20 @@ develop
 that doing so will cause use operations like ``pip install -e foo`` to
 fail.

+The prefix option is used for defining an alternative prefix within the
+installation root.
+
+The root option is used to define an alternative root within which the
+command should operate.
+
+For instance::
+
+flit develop --root /tmp/ --prefix /usr/local
+
+Should install scripts within `/tmp/usr/local/bin`, even if the Python
+environment in use reports that the sys.prefix is `/usr/` which would lead
+to using `/tmp/usr/bin/`. Similar logic applies for package files etc.
+
 The build environment
 -

@@ -326,7 +342,7 @@ had.  Minutes from that were posted to the list [#minutes]_.
 This specification is a translation of the consensus reached there into PEP
 form, along with some arbitrary choices on the minor remaining questions.

-The basic heuristic for the design has to been to focus on introducing an
+The basic heuristic for the design has been to focus on introducing an
 abstraction without requiring development not strictly tied to the
 abstraction. Where the gap is small to improvements, or the cost of using the
 existing interface is very high, then we've taken on having the improvement as
@@ -343,7 +359,7 @@ CLI).
 The use of 'develop' as a command is because there is no PEP specifying the
 interoperability of things that do what 'setuptools develop' does - so we'll
 need to define that before pip can take on the responsibility for doing the
-'develop' step. Once thats done we can issue a successor PEP to this one.
+'develop' step. Once that's done we can issue a successor PEP to this one.

 The use of a command line API rather than a Python API is a little
 contentious. Fundamentally anything can be made to work, and the pip
@@ -410,8 +426,8 @@ References
 .. [#strformat] The Python string formatting syntax.
(https://docs.python.org/3.1/library/string.html#format-string-syntax)

-.. [#dependencyspec] Dependency specification language PEP.
-   (https://github.com/pypa/interoperability-peps/pull/56)
+.. [#pep508] Dependency specification language PEP.
+   (https://www.python.org/dev/peps/pep-0508/)

 Copyright
 =


-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

I didn't realise PEP 440 refered to 426, though the reference is weak,
its enumerating one valid sort of content to refer to (urls that are
valid as source urls).

-Rob

On 19 November 2015 at 05:44, Marcus Smith <qwc...@gmail.com> wrote:
> as it is, this PEP defers the concept of a "Direct Reference URL" to PEP440.
>
> but then PEP440 partially defers to PEP426's "source_url" concept, when it
> says  "a direct URL reference may be a valid source_url entry"
>
> do we expect PEP440 to be updated to fully own what a "Direct Reference URL"
> can be?, since referring to PEP426 is now a dead-end path (and partially
> replaced by this PEP)
>
> On Mon, Nov 16, 2015 at 12:46 PM, Robert Collins <robe...@robertcollins.net>
> wrote:
>>
>> :PEP: XX
>> :Title: Dependency specification for Python Software Packages
>> :Version: $Revision$
>> :Last-Modified: $Date$
>> :Author: Robert Collins <rbtcoll...@hp.com>
>> :BDFL-Delegate: Donald Stufft <don...@stufft.io>
>> :Discussions-To: distutils-sig <distutils-sig@python.org>
>> :Status: Draft
>> :Type: Standards Track
>> :Content-Type: text/x-rst
>> :Created: 11-Nov-2015
>> :Post-History: XX
>>
>>
>> Abstract
>> 
>>
>> This PEP specifies the language used to describe dependencies for
>> packages.
>> It draws a border at the edge of describing a single dependency - the
>> different sorts of dependencies and when they should be installed is a
>> higher
>> level problem. The intent is provide a building block for higher layer
>> specifications.
>>
>> The job of a dependency is to enable tools like pip [#pip]_ to find the
>> right
>> package to install. Sometimes this is very loose - just specifying a name,
>> and
>> sometimes very specific - referring to a specific file to install.
>> Sometimes
>> dependencies are only relevant in one platform, or only some versions are
>> acceptable, so the language permits describing all these cases.
>>
>> The language defined is a compact line based format which is already in
>> widespread use in pip requirements files, though we do not specify the
>> command
>> line option handling that those files permit. There is one caveat - the
>> URL reference form, specified in PEP-440 [#pep440]_ is not actually
>> implemented in pip, but since PEP-440 is accepted, we use that format
>> rather
>> than pip's current native format.
>>
>> Motivation
>> ==
>>
>> Any specification in the Python packaging ecosystem that needs to consume
>> lists of dependencies needs to build on an approved PEP for such, but
>> PEP-426 [#pep426]_ is mostly aspirational - and there are already existing
>> implementations of the dependency specification which we can instead
>> adopt.
>> The existing implementations are battle proven and user friendly, so
>> adopting
>> them is arguably much better than approving an aspirational, unconsumed,
>> format.
>>
>> Specification
>> =
>>
>> Examples
>> 
>>
>> All features of the language shown with a name based lookup::
>>
>> requests [security,tests] >= 2.8.1, == 2.8.* ; python_version <
>> "2.7.10"
>>
>> A minimal URL based lookup::
>>
>> pip @
>> https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
>>
>> Concepts
>> 
>>
>> A dependency specification always specifies a distribution name. It may
>> include extras, which expand the dependencies of the named distribution to
>> enable optional features. The version installed can be controlled using
>> version limits, or giving the URL to a specific artifact to install.
>> Finally
>> the dependency can be made conditional using environment markers.
>>
>> Grammar
>> ---
>>
>> We first cover the grammar briefly and then drill into the semantics of
>> each
>> section later.
>>
>> A distribution specification is written in ASCII text. We use a parsley
>> [#parsley]_ grammar to provide a precise grammar. It is expected that the
>> specification will be embedded into a larger system which offers framing
>> such
>> as comments, multiple line support via continuations, or other such
>> features.
>>
>> The full grammar including annotations to build a useful parse tree is
>> included at the end of the PEP.
>>
>> Versions may be specified according to the PEP-440 [#pep440]_ rules.
>&

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

On 19 November 2015 at 08:40, Marcus Smith <qwc...@gmail.com> wrote:
>>
>> > Will "direct references" ever be well-defined? or open to whatever any
>> > tool
>> > decides can be an artifact reference?
>>
>> We can define the syntax without capturing all the tool support, which
>> is what PEP-440 and thus this PEP does.
>
>
> so, to be clear, what syntax for the URI portion does it define or require?
> (beyond it just being a valid URI)
>
> it sounds like you're saying nothing?  i.e. although PEP440 says things like
> it "may" be a sdist or a wheel target or a "source_url", its wide open to
> whatever a tool may decide is a unique artifact reference?

Thats my understanding from PEP-440 today.

And given that an arbitrary url can contain wheel content, I'd be
cautious about trying to place semantic constraints on the syntax (vs
the content).

-Rob



-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

On 19 November 2015 at 07:30, Marcus Smith <qwc...@gmail.com> wrote:
>>
>> Its included in the complete grammar, otherwise it can't be tested.
>> Note that that the PEP body refers to the IETF document for the
>> definition of URIs. e.g. exactly what you suggest.
>
>
> doesn't this imply any possible URI can theoretically be a PEP440 direct
> reference URI ?
>
> Is that true?
>
> It's unclear to me what PEP440's definition really is with words like "The
> exact URLs and targets supported will be tool dependent"
>
> Will "direct references" ever be well-defined? or open to whatever any tool
> decides can be an artifact reference?

We can define the syntax without capturing all the tool support, which
is what PEP-440 and thus this PEP does.

What I mean here is that e.g. pip may not support a given VCS today,
but it can be added without changing the definition of a URI. We can't
demand that all tools support all VCS's reasonably, but we can demand
that all tools be able to recognise that its a direct reference and
act accordingly (e.g. try to clone it, error because direct references
aren't allowed in that context, etc).

-Rob


-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

I think we should start supporting that, yes.

On 19 November 2015 at 09:14, Marcus Smith <qwc...@gmail.com> wrote:
>
>
> On Wed, Nov 18, 2015 at 11:42 AM, Donald Stufft <don...@stufft.io> wrote:
>>
>>
>> On Nov 18, 2015, at 2:40 PM, Marcus Smith <qwc...@gmail.com> wrote:
>>
>>>
>>> > Will "direct references" ever be well-defined? or open to whatever any
>>> > tool
>>> > decides can be an artifact reference?
>>>
>>> We can define the syntax without capturing all the tool support, which
>>> is what PEP-440 and thus this PEP does.
>>
>>
>> so, to be clear, what syntax for the URI portion does it define or
>> require?  (beyond it just being a valid URI)
>>
>> it sounds like you're saying nothing?  i.e. although PEP440 says things
>> like it "may" be a sdist or a wheel target or a "source_url", its wide open
>> to whatever a tool may decide is a unique artifact reference?
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>>
>> Only half way thinking about this right this moment, but I think so yes.
>> It’s largely designed for private use cases which is why it’s not allowed on
>> PyPI. It’s essentially a replacement for dependency_links.
>
>
> practically speaking, isn't it also a future replacement for
> "#egg=name" syntax in pip vcs urls?... i.e.  using  "name@"
> instead?



-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

On 18 November 2015 at 05:51, Antoine Pitrou <solip...@pitrou.net> wrote:
> On Wed, 18 Nov 2015 05:40:33 +1300
> Robert Collins <robe...@robertcollins.net> wrote:
>>
>> Its included in the complete grammar, otherwise it can't be tested.
>> Note that that the PEP body refers to the IETF document for the
>> definition of URIs. e.g. exactly what you suggest.
>
> What I suggest is that the grammar doesn't try to define URIs at all,

We don't. We consume the definition the IETF give.

> and instead includes a simple rule that is a superset of URI matching.
> It doesn't make sense for Python packaging tools to detect what is a
> valid URI or not. It's not their job, and the work will probably be
> replicated by whatever URI-loading library they use anyway (since they
> will pass it the URI by string, not by individual components).
>
> The only place where URIs are used seem to be the "urlspec" rule, and
> probably you can accept any opaque string there.

Uhm, why are you making this suggestion? What problem will we solve by
using a proxy rule?

-Rob


-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

On 18 November 2015 at 02:59, Antoine Pitrou <solip...@pitrou.net> wrote:
> On Tue, 17 Nov 2015 09:46:21 +1300
> Robert Collins <robe...@robertcollins.net> wrote:
>>
>> URI_reference = 
>> URI   = scheme ':' hier_part ('?' query )? ( '#' fragment)?
>> hier_part = ('//' authority path_abempty) | path_absolute |
>> path_rootless | path_empty
>> absolute_URI  = scheme ':' hier_part ( '?' query )?
>> relative_ref  = relative_part ( '?' query )? ( '#' fragment )?
>> relative_part = '//' authority path_abempty | path_absolute |
>> path_noscheme | path_empty
>> scheme= letter ( letter | digit | '+' | '-' | '.')*
>> authority = ( userinfo '@' )? host ( ':' port )?
>> userinfo  = ( unreserved | pct_encoded | sub_delims | ':')*
>> host  = IP_literal | IPv4address | reg_name
>> port  = digit*
>> IP_literal= '[' ( IPv6address | IPvFuture) ']'
>> IPvFuture = 'v' hexdig+ '.' ( unreserved | sub_delims | ':')+
>> IPv6address   = (
>>   ( h16 ':'){6} ls32
>>   | '::' ( h16 ':'){5} ls32
>>   | ( h16 )?  '::' ( h16 ':'){4} ls32
>>   | ( ( h16 ':')? h16 )? '::' ( h16 ':'){3} ls32
>>   | ( ( h16 ':'){0,2} h16 )? '::' ( h16 ':'){2} ls32
>>   | ( ( h16 ':'){0,3} h16 )? '::' h16 ':' ls32
>>   | ( ( h16 ':'){0,4} h16 )? '::' ls32
>>   | ( ( h16 ':'){0,5} h16 )? '::' h16
>>   | ( ( h16 ':'){0,6} h16 )? '::' )
>
> It seems weird that the PEP tries to include an entire subgrammar for
> URIs, including even the parsing various kinds of IP addresses.  Why not
> be lenient in their detection and leave actual definition of valid URIs
> to the IETF?
>
> It doesn't seem there is any point to embed/duplicate such knowledge in
> Python packaging tools.

Its included in the complete grammar, otherwise it can't be tested.
Note that that the PEP body refers to the IETF document for the
definition of URIs. e.g. exactly what you suggest.

-Rob
-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP, take #2

[Robert Collins]

On 18 November 2015 at 03:27, Antoine Pitrou <solip...@pitrou.net> wrote:
> On Tue, 17 Nov 2015 15:24:04 +1300
> Robert Collins <robe...@robertcollins.net> wrote:
>>
>> The programmatic interface allows decoupling of pip from its current
>> hard dependency on setuptools [#setuptools]_ able for two
>> key reasons:
>>
>> 1. It enables new build systems that may be much easier to use without
>>requiring them to even appear to be setuptools.
>
> And yet:
>
>> There are a number of separate subcommands that build systems must support.
>
> I wonder how desirable and viable this all is. Desirable, because you
> are still asking the build system to appear as setuptools *in some way*.
> Viable, because pip may some day need to ask more from setuptools and
> then third-party build tools will have to adapt and implement said
> command-line options, defeating the "abstraction".
>
> In other words, this PEP seems to be only solving a fraction of the
> problem.
>
>
>> The file ``pypa.json`` acts as neutron configuration file for pip and other
>> tools that want to build source trees to consult for configuration.
>
> What is a "neutron configuration file"?

Typo - neutral.

> Why is it called "pypa.json" and not the more descriptive
> "pybuild.json" (or, if you prefer, "pip.json")?
> "pypa", as far as I know, is the name of an informal group, not a
> well-known utility, operation or command.

I don't care about the name. As I don't want to be the arbiter of a
bikeshedding thing, the BDFL delegate can choose: until they do, I'm
going to stay with the current name to avoid edit churn.

>> bootstrap_requires
>> Optional list of dependency specifications [#dependencyspec] that must be
>> installed before running the build tool.
>
> How are the requirements gathered and installed? Using setuptools?

By whatever tool is consuming the build system. E.g. pip, or perhaps
conda. Or some new tool. How they do the install is not an
interoperability issue, and thus out of scope for the PEP.

-Rob


-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP, take #2

[Robert Collins]

On 18 November 2015 at 04:20, Daniel Holth <dho...@gmail.com> wrote:
> LGTM
>
> Q: Why is build_command a list?

Because the dependency spec framing we established doesn't describe
multiple dependencies in one string (and we chose to write one that
can be embedded in different higher layer things specifically to allow
wider reuse) - we could define a wrapper here, or, since we have a
structured config file, use that structure. It read a bit nicer in
YAML, but see JSON under rationale.

> Q: Why isn't the file name venezuelanbeavercheese.json instead of pypa.json?

:)

-Rob



-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP, take #2

[Robert Collins]

On 18 November 2015 at 05:28, Leonardo Rochael Almeida
 wrote:
> On 17 November 2015 at 13:20, Daniel Holth  wrote:
>>
>> LGTM
>>
>> Q: Why is build_command a list?
>> Q: Why isn't the file name venezuelanbeavercheese.json instead of
>> pypa.json?
>
>
> Or why not just use a specific key in setup.cfg instead of a pypa.json file?
> ISTM that this PEP expects to find in pypa.json some keys that are supposed
> to be entered manually by humans, even though json is a format more easily
> written by machines than by humans...

I believe this is fully described in the rationale section - if not,
let me know which bit didn't make sense and I'll edit it.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

On 18 November 2015 at 06:35, Antoine Pitrou <solip...@pitrou.net> wrote:
> On Wed, 18 Nov 2015 06:32:35 +1300
> Robert Collins <robe...@robertcollins.net> wrote:
>> >
>> > The only place where URIs are used seem to be the "urlspec" rule, and
>> > probably you can accept any opaque string there.
>>
>> Uhm, why are you making this suggestion? What problem will we solve by
>> using a proxy rule?
>
> Making the PEP simpler, making implementations simpler, avoiding bugs
> in implementations which may otherwise try to implement full URI
> matching, avoiding having to issue a PEP update whenever the IETF
> updates the definition.
>
> These are the four I can think about, anyway :-)

Taking them in reverse order:
We reference the URI standard, so when that changes, our definition
changes - we can update the combined grammar, but we don't need to
update anything we've defined (unless the URI change is something that
invalidates our symbol delimitation, in which case we'd have to
anyway.

The grammar is a reference, meant to act as a test in the event of
disagreement should two implementations differ from each other. The
packaging implementation for instance uses pyparsing and thus by
necessity a different grammar. Implementations don't have to use any
of it - I'm not sure, unless we require that implementations use
OMeta, how we're causing (or preventing) bugs by changing the combined
grammar. Note that previous PEPs have either had no grammar (and
interop issue) or partially defined grammar's (and logical issues -
see PEP-426 for example). I think its very important we be able to
test what we're saying should happen well.

Implementations don't have to use the grammar, they just have to
accept the same inputs and interpret it in the same ways. packaging's
implementation doesn't use the same grammar, for instance. (It uses
pyparsing and a mix of regexes and objects).

Since the bit you're complaining about is basically an appendix, I can
see that it makes the PEP shorter, but not how it makes it simpler: we
still depend on the definition of URI, because we consume URI's - and
thats a PEP-440 choice, so changing that is something I'd seek a very
strong reason for, since its accepted.

-Rob


-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

On 18 November 2015 at 07:10, Paul Moore <p.f.mo...@gmail.com> wrote:
> On 17 November 2015 at 17:32, Robert Collins <robe...@robertcollins.net> 
> wrote:
>>> The only place where URIs are used seem to be the "urlspec" rule, and
>>> probably you can accept any opaque string there.
>>
>> Uhm, why are you making this suggestion? What problem will we solve by
>> using a proxy rule?
>
> I think the point here is syntax vs semantics. It is simpler to parse
> if we make the *syntax* state that an opaque string is allowed here.
> The *semantics* can then say that the string is to be handled as a
> URL, meaning that any string that isn't a valid URL will fail when we
> try to pass it to urllib or whatever.
>
> The only advantage of *parsing* it as a URL is that we get to reject
> foo/bar:baz as a syntax error. But we'd still reject foo:/bar as
> an invalid URL (unknown protocol) later in the processing, so why
> bother trying to trap the specific error of "doesn't look like a URL"
> early?
>
> By including the URL syntax, we're mandating that conforming
> implementations *have* to trap malformed URLs early, and can't defer
> that validation to the URL library being used to process the URL.

I don't understand how we're mandating that.

-Rob

-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pip is not a library was: FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

On 18 November 2015 at 00:26, Thomas Güttler
<guettl...@thomas-guettler.de> wrote:
..
> I want to parse a single dependency of a python
> package.

Presumably from Python; in which case you can use the packaging
library once the PEP is pronounced on and the code has been added.

If you want to do it from another language, you can use the grammar
from the PEP to write an implementation.

Please do remember though, the *goal* of the PEP process here is to
define the interoperation of multiple competing implementations. So
its actually the intent that you be able to write a library function
from this specification and have good confidence that it will
interoperate with the one in pip - *whether or not that is in / from a
library*.

-Rob


-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

Another hopefully last tweak - adding in more punctuation to the
strings in markers. I'm still holding off of defining \ escapes for
now.

-Rob

diff --git a/dependency-specification.rst b/dependency-specification.rst
index 8a632ba..b115b66 100644
--- a/dependency-specification.rst
+++ b/dependency-specification.rst
@@ -96,7 +96,9 @@ environments::

 marker_op = version_cmp | (wsp* 'in') | (wsp* 'not' wsp+ 'in')
 python_str_c  = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' |
- '-' | '_' | '*')
+ '-' | '_' | '*' | '#' | ':' | ';' | ',' | '/' | '?' |
+ '[' | ']' | '!' | '~' | '`' | '@' | '$' | '%' | '^' |
+ '&' | '=' | '+' | '|' | '<' | '>' )
 dquote= '"'
 squote= '\\''
 python_str= (squote (python_str_c | dquote)* squote |
@@ -361,7 +363,9 @@ The complete parsley grammar::
 urlspec   = '@' wsp* 
 marker_op = version_cmp | (wsp* 'in') | (wsp* 'not' wsp+ 'in')
 python_str_c  = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' |
- '-' | '_' | '*' | '#')
+ '-' | '_' | '*' | '#' | ':' | ';' | ',' | '/' | '?' |
+ '[' | ']' | '!' | '~' | '`' | '@' | '$' | '%' | '^' |
+ '&' | '=' | '+' | '|' | '<' | '>' )
 dquote= '"'
 squote= '\\''
 python_str= (squote <(python_str_c | dquote)*>:s squote |

On 18 November 2015 at 09:51, Robert Collins <robe...@robertcollins.net> wrote:
> Addressed along with some minor quirks and github feedback.
>
> diff --git a/dependency-specification.rst b/dependency-specification.rst
> index a9953ed..8a632ba 100644
> --- a/dependency-specification.rst
> +++ b/dependency-specification.rst
> @@ -18,7 +18,7 @@ Abstract
>  This PEP specifies the language used to describe dependencies for packages.
>  It draws a border at the edge of describing a single dependency - the
>  different sorts of dependencies and when they should be installed is a higher
> -level problem. The intent is provide a building block for higher layer
> +level problem. The intent is to provide a building block for higher layer
>  specifications.
>
>  The job of a dependency is to enable tools like pip [#pip]_ to find the right
> @@ -85,7 +85,7 @@ Versions may be specified according to the PEP-440
> [#pep440]_ rules. (Note:
>  URI is defined in std-66 [#std66]_::
>
>  version_cmp   = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | '==='
> -version   = wsp* ( letterOrDigit | '-' | '_' | '.' | '*' )+
> +version   = wsp* ( letterOrDigit | '-' | '_' | '.' | '*' | '+' | '!' 
> )+
>  version_one   = version_cmp version wsp*
>  version_many  = version_one (wsp* ',' version_one)*
>  versionspec   = ( '(' version_many ')' ) | version_many
> @@ -94,7 +94,7 @@ URI is defined in std-66 [#std66]_::
>  Environment markers allow making a specification only take effect in some
>  environments::
>
> -marker_op = version_cmp | 'in' | 'not' wsp+ 'in'
> +marker_op = version_cmp | (wsp* 'in') | (wsp* 'not' wsp+ 'in')
>  python_str_c  = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' |
>   '-' | '_' | '*')
>  dquote= '"'
> @@ -108,10 +108,14 @@ environments::
>   'implementation_name' | 'implementation_version' |
>   'extra' # ONLY when defined by a containing layer
>   )
> -marker_var= env_var | python_str
> -marker_expr   = ('(' wsp* marker wsp* ')'
> - | (marker_var wsp* marker_op wsp* marker_var))
> -marker= wsp* marker_expr ( wsp* ('and' | 'or') wsp* marker_expr)*
> +marker_var= wsp* (env_var | python_str)
> +marker_expr   = marker_var marker_op marker_var
> +  | wsp* '(' marker wsp* ')'
> +marker_and= marker_expr wsp* 'and' marker_expr
> +  | marker_expr
> +marker_or = marker_and wsp* 'or' marker_and
> +  | marker_and
> +marker= marker_or
>  quoted_marker = ';' wsp* marker
>
>  Optional components of a distribution may be specified using the extras
> @@ -304,7 +308,7 @@ The ``implementation_version`` marker variable is
> derived from
>  Backwards Compatibility
>  ===
>
> -Most of this PEP is already widely deployed and thus offers no compatibiltiy
> +Most of this PEP is already widely deployed and thus offers no compatibility
>  concerns.
>
>  There are however a few points where the PEP differs from the deployed base.
> @@ -355,7 +359,7 @@ The complete parsley grammar::
>  version_many  = version_one:v1 (wsp* ','

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

ame=='fred'",
+"name; os_name=='a' or os_name=='b'",
+# Should parse as (a and b) or c
+"name; os_name=='a' and os_name=='b' or os_name=='c'",
+# Overriding precedence -> a and (b or c)
+"name; os_name=='a' and (os_name=='b' or os_name=='c')",
+# should parse as a or (b and c)
+"name; os_name=='a' or os_name=='b' and os_name=='c'",
+# Overriding precedence -> (a or b) and c
+"name; (os_name=='a' or os_name=='b') and os_name=='c'",
 ]

 def format_full_version(info):
@@ -502,8 +515,8 @@ A test program - if the grammar is in a string ``grammar``::

 compiled = makeGrammar(grammar, {'lookup': bindings.__getitem__})
 for test in tests:
-  parsed = compiled(test).specification()
-  print(parsed)
+parsed = compiled(test).specification()
+print("%s -> %s" % (test, parsed))

 References
 ==

On 17 November 2015 at 16:59, Robert Collins <robe...@robertcollins.net> wrote:
> On 17 November 2015 at 16:37, Nathaniel Smith <n...@pobox.com> wrote:
>
> Blah. Shall address.
>
> -Rob
>
>>
>> --
>> Nathaniel J. Smith -- http://vorpus.org
>
>
>
> --
> Robert Collins <rbtcoll...@hp.com>
> Distinguished Technologist
> HP Converged Cloud



-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP, take #2

[Robert Collins]

On 18 November 2015 at 04:20, Daniel Holth <dho...@gmail.com> wrote:
> LGTM
>
> Q: Why is build_command a list?

I misread the question - fixed.

diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst
index 8eb0681..d36b7d5 100644
--- a/build-system-abstraction.rst
+++ b/build-system-abstraction.rst
@@ -91,7 +91,7 @@ build_command
 A mandatory Python format string [#strformat]_ describing the command to
 run. For instance, if using flit then the build command might be::

-build_command: ["flit"]
+build_command: "flit"

 If using a command which is a runnable module fred::

@@ -254,7 +254,7 @@ Examples
 An example 'pypa.json' for using flit::

   {"bootstrap_requires": ["flit"],
-   "build_command": ["flit"]}
+   "build_command": "flit"}

 When 'pip' reads this it would prepare an environment with flit in it before
 trying to use flit.

-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pip is not a library was: FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

On 17 November 2015 at 20:57, Thomas Güttler
<guettl...@thomas-guettler.de> wrote:
>> The job of a dependency is to enable tools like pip [#pip]_ to find the right
>> package to install.
>
> My worries: AFAIK pip is not a library.
>
> I don't want to re-implement code to handle this pep.
>
> I would like to re-use.
>
> But AFAIK pip is not a library.
>
> I am stupid and don't know how to proceed.
>
> Please tell me what to do.

What are you trying to accomplish?

-Rob

-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

On 18 November 2015 at 08:53, Paul Moore <p.f.mo...@gmail.com> wrote:
> On 17 November 2015 at 18:43, Robert Collins <robe...@robertcollins.net> 
> wrote:
>>> By including the URL syntax, we're mandating that conforming
>>> implementations *have* to trap malformed URLs early, and can't defer
>>> that validation to the URL library being used to process the URL.
>>
>> I don't understand how we're mandating that.
>
> urlspec   = '@' wsp* 
>
> combined with
>
>  URI_reference = 
>  URI   = scheme ':' hier_part ('?' query )? ( '#' fragment)?
> (etc)
>
> implies that conforming parsers have to validate that what follows '@'
> must conform to the URI definition. So they have to reject @:
> because : is not a valid URI. But why bother? It's extra work, and
> given that all an implementation will ever do with the URI_reference
> is pass it to a function that treats it as a URI, and that function
> will do all the validation you need.
>
> I'd argue that the spec can simply say
>
> URI_reference = 
>
> The discussion of how a urlspec is used can point out that the string
> will be assumed to be a URI.
>
> A library that parsed any non-whitespace string as a URI_reference
> would be just as useful for all practical purposes, and much easier to
> write (and test!) But it would technically be non-conformant to this
> PEP.
>
> Personally, I don't actually care all that much, as I probably won't
> ever write a library that implements this spec. The packaging library
> will be fine for me. But given that the point of writing the
> interoperability PEPs is to ensure people *can* write alternative
> implementations, I'm against adding complexity and implementation
> burden that has no practical benefit.


I'm still struggling to understand.

I see two angles; the first is on what is accepted or not by an implementation:
The reference here is not the implementation - its a *reference*. An
implementation whose URI handling can't handle std-66 URI's that
another ones can would lead to interop issues : and thats what we're
trying to avoid. An alternative implementation whose URI handling has
some extension that means it handles things that other implementations
don't would accept everything the PEP mandates but also accept more -
leading to interop issues. Some interop issues (e.g. pip handles
git+https:// urls, setuptools doesn't) are not covered yet, but thats
a pep-440 issue (at least, the way things are split up today) - so I
don't want to dive into that.

The second is on whether the implementation achieves that acceptance
up front in its parsing, or on the backend in its URI library. And I
could care way less which way around it does it. We're not defining
implementation, but we are defining the language.

As I understand it, you and Antoine are saying that the current PEP
*does* define implementation because folk can't trust their URI
library to error appropriately - and thats the bit I don't understand.
Just parse however you want as an author, and cross check against the
full grammar here in case of doubt.

-Rob

-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

On 18 November 2015 at 10:22, R. David Murray <rdmur...@bitdance.com> wrote:
...
>> As I understand it, you and Antoine are saying that the current PEP
>> *does* define implementation because folk can't trust their URI
>> library to error appropriately - and thats the bit I don't understand.
>> Just parse however you want as an author, and cross check against the
>> full grammar here in case of doubt.
>
> OK, so it *is* the case that the PEP is mandating that a conforming
> implementation has to accept valid and reject invalid URLs according
> to the grammar in the PEP, but not *how* or *when* it does that (the
> implementation).  So "trap malformed URLs early" is false, but "trap
> malformed URLs" is true, if you want to be a conformant implementation.

Yes.

-Rob


-- 
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] FINAL DRAFT: Dependency specifier PEP

[Robert Collins]

:PEP: XX
:Title: Dependency specification for Python Software Packages
:Version: $Revision$
:Last-Modified: $Date$
:Author: Robert Collins <rbtcoll...@hp.com>
:BDFL-Delegate: Donald Stufft <don...@stufft.io>
:Discussions-To: distutils-sig <distutils-sig@python.org>
:Status: Draft
:Type: Standards Track
:Content-Type: text/x-rst
:Created: 11-Nov-2015
:Post-History: XX


Abstract


This PEP specifies the language used to describe dependencies for packages.
It draws a border at the edge of describing a single dependency - the
different sorts of dependencies and when they should be installed is a higher
level problem. The intent is provide a building block for higher layer
specifications.

The job of a dependency is to enable tools like pip [#pip]_ to find the right
package to install. Sometimes this is very loose - just specifying a name, and
sometimes very specific - referring to a specific file to install. Sometimes
dependencies are only relevant in one platform, or only some versions are
acceptable, so the language permits describing all these cases.

The language defined is a compact line based format which is already in
widespread use in pip requirements files, though we do not specify the command
line option handling that those files permit. There is one caveat - the
URL reference form, specified in PEP-440 [#pep440]_ is not actually
implemented in pip, but since PEP-440 is accepted, we use that format rather
than pip's current native format.

Motivation
==

Any specification in the Python packaging ecosystem that needs to consume
lists of dependencies needs to build on an approved PEP for such, but
PEP-426 [#pep426]_ is mostly aspirational - and there are already existing
implementations of the dependency specification which we can instead adopt.
The existing implementations are battle proven and user friendly, so adopting
them is arguably much better than approving an aspirational, unconsumed, format.

Specification
=

Examples


All features of the language shown with a name based lookup::

requests [security,tests] >= 2.8.1, == 2.8.* ; python_version < "2.7.10"

A minimal URL based lookup::

pip @ 
https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686

Concepts


A dependency specification always specifies a distribution name. It may
include extras, which expand the dependencies of the named distribution to
enable optional features. The version installed can be controlled using
version limits, or giving the URL to a specific artifact to install. Finally
the dependency can be made conditional using environment markers.

Grammar
---

We first cover the grammar briefly and then drill into the semantics of each
section later.

A distribution specification is written in ASCII text. We use a parsley
[#parsley]_ grammar to provide a precise grammar. It is expected that the
specification will be embedded into a larger system which offers framing such
as comments, multiple line support via continuations, or other such features.

The full grammar including annotations to build a useful parse tree is
included at the end of the PEP.

Versions may be specified according to the PEP-440 [#pep440]_ rules. (Note:
URI is defined in std-66 [#std66]_::

version_cmp   = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | '==='
version   = wsp* ( letterOrDigit | '-' | '_' | '.' | '*' )+
version_one   = version_cmp version wsp*
version_many  = version_one (wsp* ',' version_one)*
versionspec   = ( '(' version_many ')' ) | version_many
urlspec   = '@' wsp* 

Environment markers allow making a specification only take effect in some
environments::

marker_op = version_cmp | 'in' | 'not' wsp+ 'in'
python_str_c  = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' |
 '-' | '_' | '*')
dquote= '"'
squote= '\\''
python_str= (squote (python_str_c | dquote)* squote |
 dquote (python_str_c | squote)* dquote)
env_var   = ('python_version' | 'python_full_version' |
 'os_name' | 'sys_platform' | 'platform_release' |
 'platform_system' | 'platform_version' |
 'platform_machine' | 'python_implementation' |
 'implementation_name' | 'implementation_version' |
 'extra' # ONLY when defined by a containing layer
 )
marker_var= env_var | python_str
marker_expr   = ('(' wsp* marker wsp* ')'
 | (marker_var wsp* marker_op wsp* marker_var))
marker= wsp* marker_expr ( wsp* ('and' | 'or') wsp* marker_expr)*
quoted_marker = ';' wsp* marker

Optional components of a distribution may be specified using the extras
field::

identifier= letterOrDigit (
letterOrDigit |