Re: [Distutils] comparison of configuration languages

[Brett Cannon]

Based on this email and Nathaniel's evaluation I've gone ahead and taken it
upon myself to start writing a PEP so we have something concrete to work
from. I'm hoping to have it done some time this week.

On Sun, 8 May 2016 at 08:43 Donald Stufft  wrote:

>
> > On May 8, 2016, at 9:13 AM, Nick Coghlan  wrote:
> >
> > So let's reduce our scope to: "We want *current users* of d2to1 and
> > pbr to be able to declare those dependencies to pip and other
> > installation tools in a way that avoids the implicit invocation of
> > easy_install by setuptools"
>
> I think it's less pbr and d2to1 that we're designing for here (although
> they'll
> benefit from it) but more like numpy.distutils. The reasoning is that pbr
> and
> such have already been able to architecture themselves so that they work
> within
> the current system. One of the goals here is to enable projects that
> haven't
> been able to (or can't reasonably) be written to work as a setuptools
> extension
> to be used here without requiring a manual pre-installation step.
>
> The other side of this is anytime you have dependencies that aren't
> installed
> by pip (such as setup_requires being installed by setuptools) you end up
> with
> a confusing situation where settings don't get passed down into the "inner"
> installer (like ``--index-url``) or when they have different resolution
> algorithms (like setuptools supporting pre-releases by default) or when
> they
> support different formats (like pip not supporting eggs, but setuptools not
> supporting wheels).
>
> However, while I think that a new format is more work just to make sure we
> don't back ourselves into a corner, I also don't think it's so much more
> work
> that it's really worth it to skip doing it now. Their isn't a whole lot of
> ways
> to solve the very narrow problem of "we need some dependencies installed
> prior
> to building". We need a field in a file that takes a list of PEP 508 style
> specifiers. We can skip any of the build system support right now and add
> it in
> later (and when we add it in, we can just make it so that a lack of a
> declaration is an implicit setuptool). So the only real things we need to
> decide on are:
>
> 1) What format to use? For this I think that it's been pretty clearly that
> TOML
>has had the greatest amount of support here and I think it represents
> the
>best set of trade offs for us.
>
> 2) What do we want to call this file? This one is pure bikeshed but also a
> bit
>important since this is one of the things that will be the hardest to
> change
>once we actually pick the name. Ideas I've seen so far (using the toml
>extension) are:
>
>* pypa.toml - This one is specific to us which is good, but it's also a
> bit
>  bad in that I don't think it's very widely known what the PyPA even
> is and
>  I think that the PyPA works better if it's just sort of the thing in
> the
>  background.
>
>* pybuild.toml - This one might be a bit too oriented towards building
>  rather than all of the possible uses of this file, but the bigger
> problem
>  with it I think is that hte name clashes with pybuild on Debian which
> is
>  their tool used for building Python packages.
>
>* pip.toml - Also specific to us which is good, but I think it's a bit
> too
>  overly specific maybe? One of our goals has been to make this stuff
> not
>  dependent on a specific implementation so that we can replace it if we
>  need to (as we more or less replaced distutils with setuptools, or
>  easy_install with pip) however this also isn't exactly depending on an
>  implementation-- just the name of a particular implementation. It does
>  have the benefit that for a lot of people they associate "pip" with
>  everything packaging related (e.g. I see people calling them
>  "pip packages" now).
>
>* pymeta.toml - This one might be reasonable, it's a bit generic but at
>  least it has the ``py`` prefix to tie it to us. Not much to say about
> it
>  otherwise.
>
>Overall from those names, I think I probably like pymeta.toml the best
> (or
>maybe just ``meta.toml`` if we don't like the py prefix) but maybe other
>people have ideas/opinions that I haven't seen yet.
>
> 3) How do we structure the file and what keys do we use for just this part
> of
>the overall feature we're hoping to eventually get (and what semantics
> do
>we give these keys). We could just call it ``setup_requires``, but I
> think
>that's a bad name for something we're not planning on getting rid of
> since
>one of the eventual goals is to make ``setup.py`` itself optional.
> Another
>option is ``build_requires`` but that isn't quite right because we don't
>*just* need these dependencies for the build step (``setup.py
> bdist_wheel``)
>but also for the metadata step (``setup.py egg_info``). Although
> perhaps it
>doesn't really matter. 

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 8 May 2016 at 22:38, Nicholas Chammas  wrote:
> Perhaps a more limited YAML library like Poyo would address some of the
> concerns we have about using PyYAML.

I'm -1 on using "a subset of" a standard format. It immediately
invites debate and confusion over *precisely* which subset is
supported.

I like (basic) YAML, but IMO it made the mistake of allowing itself to
get too complex.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 8, 2016, at 5:38 PM, Nicholas Chammas  
> wrote:
> 
> Poyo in particular may be too limited, but I wonder how much it would sway 
> people if we used YAML but not PyYAML.


If we found a reasonable library for parsing YAML it might be a more reasonable 
alternative. Though I think we really need something that supports most if not 
all of YAML otherwise I think it starts to get confusing because you have to 
figure out what the specific subset of YAML that the library that whatever 
packaging tool you’re using supports.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nicholas Chammas]

On Sat, May 7, 2016 at 6:23 AM Paul Moore p.f.mo...@gmail.com
 wrote:

- YAML ought to be wonderful, but it ended up over-engineered (yes, we
> can ignore the bits we don't care about). Also, pyYAML is a bit of an
> annoying dependency (big, reportedly slow unless you use the C
> version) - not something I'd want pip to have to vendor.
>
Perhaps a more limited YAML library like Poyo would address some of the
concerns we have about using PyYAML.

https://github.com/hackebrot/poyo#readme

Poyo in particular may be *too* limited, but I wonder how much it would
sway people if we used YAML but not PyYAML.

Anyway, here are some relevant snippets from Poyo’s README:

Please note that Poyo supports only a chosen subset of the YAML format.
It can only read but not write and is not compatible with JSON.
…
Poyo is 100% Python and does not require any additional libs.

Nick
​
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 8, 2016, at 9:13 AM, Nick Coghlan  wrote:
> 
> So let's reduce our scope to: "We want *current users* of d2to1 and
> pbr to be able to declare those dependencies to pip and other
> installation tools in a way that avoids the implicit invocation of
> easy_install by setuptools"

I think it's less pbr and d2to1 that we're designing for here (although they'll
benefit from it) but more like numpy.distutils. The reasoning is that pbr and
such have already been able to architecture themselves so that they work within
the current system. One of the goals here is to enable projects that haven't
been able to (or can't reasonably) be written to work as a setuptools extension
to be used here without requiring a manual pre-installation step.

The other side of this is anytime you have dependencies that aren't installed
by pip (such as setup_requires being installed by setuptools) you end up with
a confusing situation where settings don't get passed down into the "inner"
installer (like ``--index-url``) or when they have different resolution
algorithms (like setuptools supporting pre-releases by default) or when they
support different formats (like pip not supporting eggs, but setuptools not
supporting wheels).

However, while I think that a new format is more work just to make sure we
don't back ourselves into a corner, I also don't think it's so much more work
that it's really worth it to skip doing it now. Their isn't a whole lot of ways
to solve the very narrow problem of "we need some dependencies installed prior
to building". We need a field in a file that takes a list of PEP 508 style
specifiers. We can skip any of the build system support right now and add it in
later (and when we add it in, we can just make it so that a lack of a
declaration is an implicit setuptool). So the only real things we need to
decide on are:

1) What format to use? For this I think that it's been pretty clearly that TOML
   has had the greatest amount of support here and I think it represents the
   best set of trade offs for us.

2) What do we want to call this file? This one is pure bikeshed but also a bit
   important since this is one of the things that will be the hardest to change
   once we actually pick the name. Ideas I've seen so far (using the toml
   extension) are:

   * pypa.toml - This one is specific to us which is good, but it's also a bit
 bad in that I don't think it's very widely known what the PyPA even is and
 I think that the PyPA works better if it's just sort of the thing in the
 background.

   * pybuild.toml - This one might be a bit too oriented towards building
 rather than all of the possible uses of this file, but the bigger problem
 with it I think is that hte name clashes with pybuild on Debian which is
 their tool used for building Python packages.

   * pip.toml - Also specific to us which is good, but I think it's a bit too
 overly specific maybe? One of our goals has been to make this stuff not
 dependent on a specific implementation so that we can replace it if we
 need to (as we more or less replaced distutils with setuptools, or
 easy_install with pip) however this also isn't exactly depending on an
 implementation-- just the name of a particular implementation. It does
 have the benefit that for a lot of people they associate "pip" with
 everything packaging related (e.g. I see people calling them
 "pip packages" now).

   * pymeta.toml - This one might be reasonable, it's a bit generic but at
 least it has the ``py`` prefix to tie it to us. Not much to say about it
 otherwise.

   Overall from those names, I think I probably like pymeta.toml the best (or
   maybe just ``meta.toml`` if we don't like the py prefix) but maybe other
   people have ideas/opinions that I haven't seen yet.

3) How do we structure the file and what keys do we use for just this part of
   the overall feature we're hoping to eventually get (and what semantics do
   we give these keys). We could just call it ``setup_requires``, but I think
   that's a bad name for something we're not planning on getting rid of since
   one of the eventual goals is to make ``setup.py`` itself optional. Another
   option is ``build_requires`` but that isn't quite right because we don't
   *just* need these dependencies for the build step (``setup.py bdist_wheel``)
   but also for the metadata step (``setup.py egg_info``). Although perhaps it
   doesn't really matter. Another option is to call them ``bootstrap_requires``
   though it's a bit wonky to cram *build* requirements into that.

   I guess if I were deciding this I would just call it build requirements
   because the nature of ``setup.py`` is that pretty much anything you need to
   execute ``setup.py`` is going to be a requirement for all ways you invoke
   ``setup.py`` (metadata and build and the legacyish direct to install). We
   could then say that if a project is missing this new 

Re: [Distutils] comparison of configuration languages

[Nick Coghlan]

On 8 May 2016 at 08:46, Donald Stufft  wrote:
>
>> On May 7, 2016, at 5:05 PM, Robert Collins  wrote:
>>
>> Either we are defining the long term thing now, in which case that
>> huge pile of complexity lands on us, and we have to get everything
>> right.
>>
>> Or we are defining a thing which solves the present bug, and as long
>> as we make sure it does not bind us in future, we're not hamstrung.
>>
>> E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
>> as pybuild is a thing in the debian space, and this will confuse the
>> heck out of folk). https://wiki.debian.org/Python/Pybuild
>
> I think this is roughly true, we could either do the simplest thing and just
> add ``setup_requires`` to ``setup.cfg`` and teach pip how to understand them
> and then worry about a new format later, or we can do a new format now and add
> a bit of complexity to what we need to specify (though I don't think _too_ 
> much
> complexity, we don't have to define the build system stuff now, just make sure
> we don't back ourselves into a corner with that).
>
> I think either answer is OK, just the second one is a bit more work and we
> might either get the start of a better format _now_ or end up regretting what
> we pick when we add more things to it.

Hmm, are we perhaps forcing a false choice on ourselves here?

1. setup_requires in setup.py is the de facto baseline

2. d2to1 and pbr already use setup.cfg, so any arguments about the
readability of the format are moot for users of those projects

3. we're not sure we're comfortable mandating the use of an ini-style
format for all *future* Python projects

So let's reduce our scope to: "We want *current users* of d2to1 and
pbr to be able to declare those dependencies to pip and other
installation tools in a way that avoids the implicit invocation of
easy_install by setuptools"

For *that* design goal, the conclusions I'd draw from Nathaniel's
write-up would be those Łukasz suggested:

- use https://pypi.python.org/pypi/configparser in Python 2.7
- explicitly define how to handle end-of-line comments

We should perhaps also recommend that "#" be used for comments (even
if the parser didn't enforce that), as I haven't seen anyone using ';'
for comments since I stopped writing assembly code by hand.

Improving the status quo for d2to1 and pbr users doesn't lock us into
anything in terms of the "future of Python packaging as a whole".
Individual build system authors may *choose* to take advantage of the
setup.cfg mechanism, but they wouldn't be obliged to - they could keep
using setup.py based bootstrapping if they preferred to do so.

This approach would also provide the ability to iterate on a TOML
based dependency declaration system *independent of pip itself*:
setup.cfg could be used to bootstrap that system while it was still
experimental, letting the idiosyncrasies get worked out before we
commit to anything in pip itself.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

[Nick Coghlan]

On 8 May 2016 at 09:23, Brett Cannon  wrote:
> On Sat, May 7, 2016, 12:16 Chris Barker  wrote:
>> how is this about "bootstrapping" -- to me, bootstrapping is when you need
>> X to build X. Isn't this just regular old configuration: you need x,y to
>> build z?
>
> Sure, if you don't like the term "bootstrap" then you can call it "build
> requirements". We have not been calling it " configuration" in a general
> sense as this doesn't cover how to invoke the build step (that will probably
> be the next PEP), just what needs to be installed to even potentially do a
> build.

The reason I think "bootstrap" is a better name at this point than
"build" is that there are actually three commands we're installing the
relevant dependencies for:

* egg_info/dist_info (i.e. metadata generation)
* sdist (i.e. archive generation)
* bdist_wheel (i.e. building)

The bootstrapping at the moment is taken care of by "assume everything
uses setuptools, install setuptools by default, if you want to use
something else, use setuptools to define and retrieve it".

The new metadata aims to take the place of setuptools in that
bootstrapping process: if the software publisher so chooses, they'll
be able to both create an sdist from a source tree and a wheel archive
from an sdist without ever installing setuptools. (Of course, one or
more of their dependencies are likely to bring in setuptools anyway
for the foreseeable future, but at the level of their own project
they'll be able to ignore it)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 8 May 2016 at 04:15, Chris Barker  wrote:
> On Sat, May 7, 2016 at 6:51 AM, Nick Coghlan  wrote:
>>
>> On 7 May 2016 01:55, "Chris Barker"  wrote:
>> > So my point is about scope-creep -- if you want the PyPa tools to solve
>> > all these problems, then you are re-inventing conda -- better to simply
>> > adopt conda (or fork it and fix issues that I'm sure are there)
>>
>> conda doesn't solve these problems either - it solves the *end user*
>> problem for data analysts (install the Python library they want to use),
>
> I really need to make this clear -- there is NOTHING "data analyst" specific
> about these problems -- they do come up more in the computational
> programming domain, but there are any number of other application that have
> the same problems (pyQT has come up in this conversation, yes?) -- and we're
> finding conda to be a good solution for our web development needs, too -- a
> conda environment  is kinda like a lighter-weight, platform independent
> docker container. And of course, there is more an more data analysis going
> on behind web services these days too -- any python developer is going to
> run into these issues eventually...

Aye, I know - conda was one of the systems I evaluated as a possible
general purpose tool for user level package management in Fedora and
derivatives (see
https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement#Directions_to_be_Explored
for details).

The problem with it is a practical one related to barriers to
adoption: to push any of the three systems I looked at, we'd face
significant challenges on the system administrator side (providing
capabilities comparable to what they're used to with RPMs) *and* on
the developer side (interoperating with the existing language specific
tooling for all of the language runtimes provided in Fedora).

Hence my comment about conda not currently solving *system integrator*
problems in the general case - it's fine as a platform for running
software *on top of* Fedora and for DIY integration within an
organisation, but as a tool for helping to *build Fedora*, it turned
out to introduce a whole slew of new problems (like needing to add
support to COPR/Koji/Pulp), without giving us any significant
capabilities that can't be addressed just as well by increasing the
level of automation in the upstream -> RPM pipeline and simplifying
the end user experience for container images.

However, the kinds of enhancements we're now considering upstream in
pip should improve things for conda users as well - just as some folks
in Fedora are exploring the feasibility of automatically rebuilding
the whole of PyPI as RPMs (twice, actually, anchored on
/usr/bin/python and /usr/bin/python3), it should eventually be
feasible to have the upstream->conda pipeline fully automated as well.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Wes Turner]

On Sat, May 7, 2016 at 4:05 PM, Robert Collins 
wrote:

> Couple thoughts.
>
> Firstly, the human-editable bit: who in the last *decade* has been
> writing code using a non-syntax-aware/helping editor? Its a supremely
> uninteresting aspect IMO.
>
> On ConfigParser - yes, its horrid. OTOH we do get all the lines
> reliably, and setuptools will need to cover the unicode aspect itself
> in its own time. All we need to do is permit # inline as a comment -
> line a requirements.txt file for pip - and it becomes trivial to parse
> in all cases that we need *now*.
>

ConfigObj is read/write with full Unicode support (and type validation)


| Src: https://github.com/DiffSK/configobj
| PyPI: https://pypi.python.org/pypi/configobj/5.0.6
| Docs: https://configobj.readthedocs.io/en/latest/

*Having read/write support avoids the primary issue (injection) with
templating config files*

>From https://github.com/DiffSK/configobj/blob/master/setup.py

LONG_DESCRIPTION = """**ConfigObj** is a simple but powerful config
file reader and writer: an *ini
file round tripper*. Its main feature is that it is very easy to use, with a
straightforward programmer's interface and a simple syntax for config files.
It has lots of other features though :

* Nested sections (subsections), to any level
* List values
* Multiple line values
* Full Unicode support
* String interpolation (substitution)
* Integrated with a powerful validation system

- including automatic type checking/conversion
- and allowing default values
- repeated sections

* All comments in the file are preserved
* The order of keys/sections is preserved
* Powerful ``unrepr`` mode for storing/retrieving Python data-types



>
> Either we are defining the long term thing now, in which case that
> huge pile of complexity lands on us, and we have to get everything
> right.
>
> Or we are defining a thing which solves the present bug, and as long
> as we make sure it does not bind us in future, we're not hamstrung.
>
> E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
> as pybuild is a thing in the debian space, and this will confuse the
> heck out of folk). https://wiki.debian.org/Python/Pybuild
>
>
> -Rob
>
>
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig