Re: [Distutils] comparison of configuration languages

[Greg Ewing]

Having looked over the TOML spec, I like the simplicity
of it (and I cringe from the complexity of YAML).
The only thing I don't like about TOML is the way it
cops out on nesting.

The only reason it does that as far as I can see is
because of a dislike for significant indentation.
But that doesn't apply to us -- we're Python folks,
we're not afraid of indentation! On the contrary,
for us it's the One Obvious Way to do nesting.

So I'd like a format that is very similar to TOML
except that you're allowed to represent nesting using
indented blocks.

--
Greg

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Nathaniel Smith]

On May 10, 2016 20:53, "Randy Syring"  wrote:
>
[...]
>
> I suppose, if its possible the file would ever have a different top-level
config namespace other than "package" so that information related to
non-packaging issues could possible end up here (like maybe tox or flake8
starting to read config from here instead of from setup.cfg) then I don't
think I'd feel as strongly about "pyproject.toml" being too generic.

Look again :-)

"All other top-level keys and tables are reserved for future use by other
PEPs except for the ``[tool]`` table. Within that table, tools can have
users specify configuration data as long as they use a sub-table within
``[tool]``, e.g. the `flit `_ tool might
store its configuration in ``[tool.flit]``.

We need some mechanism to allocate names within the ``tool.*`` namespace,
to make sure that different projects don't attempt to use the same
sub-table and collide. Our rule is that a project can use the subtable
``tool.$NAME`` if, and only if, they own the entry for ``$NAME`` in the
Cheeseshop/PyPI.
"

(Maybe should be more prominent I guess?)

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Randy Syring]

 On 05/10/2016 08:39 PM, Brett Cannon wrote:

The build dependencies will be stored in a file named
``pyproject.toml``

[...snip...]

A top-level ``[package]`` table will represent details specific to the
package that the project contains.

[...snip...]

Rejected Ideas
==

pypackage.toml & pypackaging.toml
   Name conflation of what a "package" is (project versus namespace).


I know this is in the rejected ideas, but I'd like to point out that 
there is an inconsistency here that bothers me.  If the top-level 
namespace in the file is going to be "package" then it makes sense to me 
that the file would also be named "py*package*.yaml".


It seems like the term "package", while possibly running into the the 
"conflation issue", is a pretty solid term to build on.


 * Python *Packaging* Authority (PyPA)
 * http://python-*packaging*-user-guide.readthedocs.io/en/latest/
 * 
http://the-hitchhikers-guide-to-*packaging*.readthedocs.io/en/latest/introduction.html
 * docs.python-guide.org/en/latest/shipping/*packaging*/
 * Top-level [*package*] namespace for the pep
 *  From the PEP "This PEP specifies how Python software*packages*
   should specify their build dependencies"
 * "As of March 2013, the Python *packaging* ecosystem is currently in
   a rather confusing state." [1]

Given the proliferation of the term "package" in describing exactly what 
this file is about, I'd really like to see the file name reconsidered to 
be  "pypackag[e|ing].toml".  Python "packaging", at least in the 
discussions I follow and am involved is, is becoming _the term_ to refer 
to this part of the Python ecosystem.


IMO, the term "project" is more ambiguous, it could possibly refer to a 
lot of things.  So, IMO, if you are just weighing the two terms on 
possibility for misunderstanding, "packaging" may get docked a point for 
being conflated with a python top-level module namespace but project 
should equally get docked a point for just being too generic AND another 
point for not matching the top-level config namespace of the file.  If 
you add in the fact that "packaging" is used frequently in the Python 
ecosystem now to refer to the process of bundling up and installing a 
python source tree, that's like +5 for "pypackag[e|ing].toml".


And, if the case of conflation still feels strong, then surly using 
"pypackaging.toml" would have to remove most of the confusion.  
Although, I'd prefer "pypackage.toml" myself.


I suppose, if its possible the file would ever have a different 
top-level config namespace other than "package" so that information 
related to non-packaging issues could possible end up here (like maybe 
tox or flake8 starting to read config from here instead of from 
setup.cfg) then I don't think I'd feel as strongly about 
"pyproject.toml" being too generic.


Although, we could be a bit cagy and make a play on "crate" by using a 
synonym "carton" (carton.toml, pycarton.toml) which, interestingly, 
sometimes hold eggs.  ;)


Thanks.

[1] 
http://python-notes.curiousefficiency.org/en/latest/pep_ideas/core_packaging_api.html



*Randy Syring*
Husband | Father | Redeemed Sinner

/"For what does it profit a man to gain the whole world
and forfeit his soul?" (Mark 8:36 ESV)/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Nathaniel Smith]

On Tue, May 10, 2016 at 6:45 PM, Donald Stufft  wrote:
[...]
> I think the way this will eventually end up going, is that when we get to the
> point of adding that next PEP that allows us to swap out setup.py for a 
> totally
> different interface, is that we'll add something like bootstrap_requires or
> something, and do something like "If package.build.requires exists, we'll use
> that for build dependencies, else we'll invoke some API in the ABS to get
> them".

Hmm, it's not something we have to decide now, but given this roadmap:

Now:
- a way to specify static requirements before running setup.py

Later:
- (definitely) a static method for specifying requirements that need
to be fulfilled before running the build backend at all ("bootstrap
requirements")
- (probably) a way to dynamically query the build backend for extra
requirements before building a wheel
- (maybe) some sort of optimization where we skip the dynamic query if
the right static configuration is provided

...I think the simplest way to manage the transition is if we make
the-thing-we're-adding-now map to the future "bootstrap requirements".
That way we won't have to change the semantics at all. And if we
decide we want some sort of static-build-requirements-optimization
thing we can make that the new key that we add later.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Ethan Furman]

On 05/10/2016 05:39 PM, Brett Cannon wrote:


Donald, Nathaniel, and I have finished our proposed PEP for specifying a
projects' build dependencies.


Looks great!  Thanks to all three of you!

+1 to build-system.

It's entirely possible to have other sections with the word 'build' in 
them, so it's better to be more explicit now.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Ian Cordasco]

On Tue, May 10, 2016 at 8:47 PM, Donald Stufft  wrote:
>
>> On May 10, 2016, at 9:43 PM, Ian Cordasco  wrote:
>> I think "build-system" is more descriptive and the more descriptive we
>> can be, the better. (Think of choosing descriptive method and
>> attribute names as well as variables.)
>
> I’m in favor of “build”, mostly because I think
>
> [package.build-system]
> requires = [“setuptools”, “wheel”]
>
> is uglier than

Oh, absolutely it's ugly, but reading it aloud is what makes me prefer
it. "The package's build-system requires setuptools, and wheel"
Granted, Ruby is the perfect example of trying to make something read
perfectly like English being a terrible idea when taken to extremes,
but I don't think that will happen here.

Other than that, the PEP looks fine. I'm anti-TOML but for reasons
that don't deserve being brought up on the mailing list. It's the best
option of a bunch of bad options. The rest of the PEP looks great.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Nathaniel Smith]

On Tue, May 10, 2016 at 6:27 PM, Robert Collins
 wrote:
[...]
> If the declared
> dependencies are merely those needed to be able to invoke the build
> system, rather than those needed to be able to successfully build, it
> would preserve that escape valve.

My understanding is that this is exactly the intention -- these are
the requirements you need before you can run setup.py or whatever
replaces setup.py; it doesn't stop us from providing something like
PEP 516/517's build requirements. [Reminder for those following along:
in the current PEP 516/517 drafts, the sequence goes like: (1) pip
consults the static list of "bootstrap" requirements, and makes those
available, (2) pip invokes the build backend to ask it if it has any
dynamic "build" requirements, (3) pip makes those available, (4) pip
invokes the build backend again to run the actual build. So Robert's
being bothered at the implication that this PEP might do away with
steps (2)-(3).]

The awkward bit is that this distinction really only makes sense in
"phase 2", once we add the pluggable build backend stuff. The PEP's
written to target "phase 1", where we're still assuming setup.py as
the only build system, so we're going from "no working build
requirements" to "working static build requirements". That doesn't
stop us from taking the next step later.

I guess the PEP text introduction could be clearer about this somehow.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Donald Stufft]

> On May 10, 2016, at 9:27 PM, Robert Collins  wrote:
> 
> On 11 May 2016 at 12:39, Brett Cannon  wrote:
>> Donald, Nathaniel, and I have finished our proposed PEP for specifying a
>> projects' build dependencies. The PEP is being kept at
>> https://github.com/brettcannon/build-deps-pep, so if you find spelling
>> mistakes and grammatical errors please feel free to send a PR to fix them.
>> 
>> The only open issue in the PEP at the moment is the bikeshedding topic of
>> what to name the sub-section containing the requirements: `[package.build]`
>> or `[package.build-system]` (we couldn't reach consensus among the three of
>> us on this). Otherwise the three of us are rather happy with the way the PEP
>> has turned out and look forward to this being the first step towards
>> allowing projects to customize their build process better!
> 
> I find calling this build dependencies confusing, but perhaps thats just me.
> 
> Right now the work flow is this:
> 
> unpack tarball
> run setup.py egg-info
> introspect that for dependencies
> download and install such dependencies (recursively) [ in future, it
> would resolve ]
> run setup.py install || setup.py wheel
> install
> 
> 1) What would pip do when it encounters one of these files?
> unpack tarball
> introspect and prepare an isolated environment with the specified dependencies
> run setup.py egg_info
> download and install such runtime dependencies (recursively) [ in
> future, it would resolve ]
> run setup.py install || setup.py wheel
> install
> 
> ?

Yes

> 
> Right now setup.py dependencies are turing complete, and its a useful
> escape valve - this design seems to have no provision for that
> (previous designs we've had here have had that). If the declared
> dependencies are merely those needed to be able to invoke the build
> system, rather than those needed to be able to successfully build, it
> would preserve that escape valve.


I think the way this will eventually end up going, is that when we get to the
point of adding that next PEP that allows us to swap out setup.py for a totally
different interface, is that we'll add something like bootstrap_requires or
something, and do something like "If package.build.requires exists, we'll use
that for build dependencies, else we'll invoke some API in the ABS to get
them".

Thus this lets us do the simple thing now, and in the future we can layer on
additional things in a backwards compatabile way that *also* keeps the simple
case simple and allows the advanced case to happen.


> 
> 2) what do we expect setuptools to do here? Is it reasonable to
> introspect this file and union it with setup_requires?

I think setuptools should just ignore this file when invoking setup.py directly
(maybe raise an error if something doesn’t exist) because I think the common
pattern is going to be using real, top level imports of the stuff depended on
here so it won’t have a chance to setup_requires them prior to executing.

I think that easy_install should behave similarly to pip though.

> 
> 3) Why does this specify ' what dependencies are required to go from
> source checkout to built wheel' ? - build systems also run tests,
> generate docs and man pages, produce other artifacts. Perhaps making
> either the target more specific (wheel_requires = ...) or the
> description less specific ('dependencies required to invoke the build
> system') would be good.

I think the description just wasn't fully thought out. I believe that
"dependencies to invoke the build system" is closer to what I've been viewing
this as (particularly since with setup.py the way it is, it's hard to
differentiate between the different "roles" of why you might be invoking
setup.py).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Donald Stufft]

> On May 10, 2016, at 9:43 PM, Ian Cordasco  wrote:
> 
> On Tue, May 10, 2016 at 8:24 PM, Nathaniel Smith  wrote:
>> On Tue, May 10, 2016 at 5:39 PM, Brett Cannon  wrote:
>>> Donald, Nathaniel, and I have finished our proposed PEP for specifying a
>>> projects' build dependencies. The PEP is being kept at
>>> https://github.com/brettcannon/build-deps-pep, so if you find spelling
>>> mistakes and grammatical errors please feel free to send a PR to fix them.
>> 
>> Thanks Brett!
>> 
>>> The only open issue in the PEP at the moment is the bikeshedding topic of
>>> what to name the sub-section containing the requirements: `[package.build]`
>>> or `[package.build-system]` (we couldn't reach consensus among the three of
>>> us on this).
>> 
>> To maybe help nudge initial bikeshedding on this in useful directions,
>> the main arguments (IIUC) were:
>> 
>> In favor of "build-system": setup.py is used for more than just the
>> strict "build" (source tree/sdist -> wheel) phase. For example,
>> setup.py is also used to do VCS checkout -> sdist. And it seems likely
>> that the new build system abstraction thing will grow similar
>> capabilities at some point. So calling the section just "build" might
>> be misleading.
>> 
>> In favor of "build": it's just shorter and reads better.
>> 
>> Maybe there's a third option that's even better -- [package.automation] ?
>> 
>> Maybe it doesn't matter that much :-)
> 
> I think "build-system" is more descriptive and the more descriptive we
> can be, the better. (Think of choosing descriptive method and
> attribute names as well as variables.)

I’m in favor of “build”, mostly because I think

[package.build-system]
requires = [“setuptools”, “wheel”]

is uglier than

[package.build]
requires = [“setuptools, “wheel”]

and I think for 99% of people the distinction is going to be lost anyways.

That being said, I think either one is OK.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Ian Cordasco]

On Tue, May 10, 2016 at 8:24 PM, Nathaniel Smith  wrote:
> On Tue, May 10, 2016 at 5:39 PM, Brett Cannon  wrote:
>> Donald, Nathaniel, and I have finished our proposed PEP for specifying a
>> projects' build dependencies. The PEP is being kept at
>> https://github.com/brettcannon/build-deps-pep, so if you find spelling
>> mistakes and grammatical errors please feel free to send a PR to fix them.
>
> Thanks Brett!
>
>> The only open issue in the PEP at the moment is the bikeshedding topic of
>> what to name the sub-section containing the requirements: `[package.build]`
>> or `[package.build-system]` (we couldn't reach consensus among the three of
>> us on this).
>
> To maybe help nudge initial bikeshedding on this in useful directions,
> the main arguments (IIUC) were:
>
> In favor of "build-system": setup.py is used for more than just the
> strict "build" (source tree/sdist -> wheel) phase. For example,
> setup.py is also used to do VCS checkout -> sdist. And it seems likely
> that the new build system abstraction thing will grow similar
> capabilities at some point. So calling the section just "build" might
> be misleading.
>
> In favor of "build": it's just shorter and reads better.
>
> Maybe there's a third option that's even better -- [package.automation] ?
>
> Maybe it doesn't matter that much :-)

I think "build-system" is more descriptive and the more descriptive we
can be, the better. (Think of choosing descriptive method and
attribute names as well as variables.)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Robert Collins]

On 11 May 2016 at 12:39, Brett Cannon  wrote:
> Donald, Nathaniel, and I have finished our proposed PEP for specifying a
> projects' build dependencies. The PEP is being kept at
> https://github.com/brettcannon/build-deps-pep, so if you find spelling
> mistakes and grammatical errors please feel free to send a PR to fix them.
>
> The only open issue in the PEP at the moment is the bikeshedding topic of
> what to name the sub-section containing the requirements: `[package.build]`
> or `[package.build-system]` (we couldn't reach consensus among the three of
> us on this). Otherwise the three of us are rather happy with the way the PEP
> has turned out and look forward to this being the first step towards
> allowing projects to customize their build process better!

I find calling this build dependencies confusing, but perhaps thats just me.

Right now the work flow is this:

unpack tarball
run setup.py egg-info
introspect that for dependencies
download and install such dependencies (recursively) [ in future, it
would resolve ]
run setup.py install || setup.py wheel
install

1) What would pip do when it encounters one of these files?
unpack tarball
introspect and prepare an isolated environment with the specified dependencies
run setup.py egg_info
download and install such runtime dependencies (recursively) [ in
future, it would resolve ]
run setup.py install || setup.py wheel
install

?

Right now setup.py dependencies are turing complete, and its a useful
escape valve - this design seems to have no provision for that
(previous designs we've had here have had that). If the declared
dependencies are merely those needed to be able to invoke the build
system, rather than those needed to be able to successfully build, it
would preserve that escape valve.

2) what do we expect setuptools to do here? Is it reasonable to
introspect this file and union it with setup_requires?

3) Why does this specify ' what dependencies are required to go from
source checkout to built wheel' ? - build systems also run tests,
generate docs and man pages, produce other artifacts. Perhaps making
either the target more specific (wheel_requires = ...) or the
description less specific ('dependencies required to invoke the build
system') would be good.

I am not suggesting that we model all the things a build system might
do: thats YAGNI at best, scope creep at worst.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Nathaniel Smith]

On Tue, May 10, 2016 at 5:39 PM, Brett Cannon  wrote:
> Donald, Nathaniel, and I have finished our proposed PEP for specifying a
> projects' build dependencies. The PEP is being kept at
> https://github.com/brettcannon/build-deps-pep, so if you find spelling
> mistakes and grammatical errors please feel free to send a PR to fix them.

Thanks Brett!

> The only open issue in the PEP at the moment is the bikeshedding topic of
> what to name the sub-section containing the requirements: `[package.build]`
> or `[package.build-system]` (we couldn't reach consensus among the three of
> us on this).

To maybe help nudge initial bikeshedding on this in useful directions,
the main arguments (IIUC) were:

In favor of "build-system": setup.py is used for more than just the
strict "build" (source tree/sdist -> wheel) phase. For example,
setup.py is also used to do VCS checkout -> sdist. And it seems likely
that the new build system abstraction thing will grow similar
capabilities at some point. So calling the section just "build" might
be misleading.

In favor of "build": it's just shorter and reads better.

Maybe there's a third option that's even better -- [package.automation] ?

Maybe it doesn't matter that much :-)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP for specifying build dependencies

[Brett Cannon]

Donald, Nathaniel, and I have finished our proposed PEP for specifying a
projects' build dependencies. The PEP is being kept at
https://github.com/brettcannon/build-deps-pep, so if you find spelling
mistakes and grammatical errors please feel free to send a PR to fix them.

The only open issue in the PEP at the moment is the bikeshedding topic of
what to name the sub-section containing the requirements: `[package.build]`
or `[package.build-system]` (we couldn't reach consensus among the three of
us on this). Otherwise the three of us are rather happy with the way the
PEP has turned out and look forward to this being the first step towards
allowing projects to customize their build process better!

-

PEP: NNN
Title: Specifying build dependencies for Python Software Packages
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon ,
Nathaniel Smith ,
Donald Stufft 
BDFL-Delegate: Nick Coghlan
Discussions-To: distutils-sig 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: NN-Mmm-2016
Post-History: NN-Mmm-2016


Abstract


This PEP specifies how Python software packages should specify their
build dependencies (i.e. what dependencies are required to go from
source checkout to built wheel). As part of this specification, a new
configuration file is introduced for software packages to use to
specify their build dependencies (with the expectation that the same
configuration file will be used for future configuration details).


Rationale
=

When Python first developed its tooling for building distributions of
software for projects, distutils [#distutils]_ was the chosen
solution. As time went on, setuptools [#setuptools]_ gained popularity
to add some features on top of distutils. Both used the concept of a
``setup.py`` file that project maintainers executed to build
distributions of their software (as well as users to install said
distribution).

Using an executable file to specify build requirements under distutils
isn't an issue as distutils is part of Python's standard library.
Having the build tool as part of Python means that a ``setup.py`` has
no external dependency that a project maintainer needs to worry about
to build a distribution of their project. There was no need to specify
any dependency information as the only dependency is Python.

But when a project chooses to use setuptools, the use of an executable
file like ``setup.py`` becomes an issue. You can't execute a
``setup.py`` file without knowing its dependencies, but currently
there is no standard way to know what those dependencies are in an
automated fashion without executing the ``setup.py`` file where that
information is stored. It's a catch-22 of a file not being runnable
without knowing its own contents which can't be known programmatically
unless you run the file.

Setuptools tried to solve this with a ``setup_requires`` argument to
its ``setup()`` function [#setup_args]_. This solution has a number
of issues, such as:

* No tooling (besides setuptools itself) can access this information
  without executing the ``setup.py``, but ``setup.py`` can't be
  executed without having these items installed.
* While setuptools itself will install anything listed in this, they
  won't be installed until *during* the execution of the ``setup()``
  function, which means that the only way to actually use anything
  added here is through increasingly complex machinations that delay
  the import and usage of these modules until later on in the
  execution of the ``setup()`` function.
* This cannot include ``setuptools`` itself nor can it include a
  replacement to ``setuptools``, which means that projects such as
  ``numpy.distutils`` are largely incapable of utilizing it and
  projects cannot take advantage of newer setuptools features until
  their users naturally upgrade the version of setuptools to a newer
  one.
* The items listed in ``setup_requires`` get implicily installed
  whenever you execute the ``setup.py`` but one of the common ways
  that the ``setup.py`` is executed is via another tool, such as
  ``pip``, who is already managing dependencies. This means that
  a command like ``pip install spam`` might end up having both
  pip and setuptools downloading and installing packages and end
  users needing to configure *both* tools (and for ``setuptools``
  without being in control of the invocation) to change settings
  like which repository it installs from. It also means that users
  need to be aware of the discovery rules for both tools, as one
  may support different package formats or determine the latest
  version differently.

This has cumulated in a situation where use of ``setup_requires``
is rare, where projects tend to either simply copy and paste snippets
between ``setup.py`` files or they eschew it all together in favor
of simply documenting elsewhere what they expect the user to have
manually installed prior to 

Re: [Distutils] comparison of configuration languages

[Brett Cannon]

Just so everyone knows, I'm ignoring this thread as the PEP I'm drafting
with Donald and Nathaniel is nearly finished and thus has already settled
on the file format discussion and I haven't heard a new point made on any
file format proposal that has already been brought up previously.

On Tue, 10 May 2016 at 10:10 Paul Moore  wrote:

> On 10 May 2016 at 13:40, Wolfgang  wrote:
> > So why not use the ConfigParser available with Python and extend it to
> meet
> > the requirements. Custom getters can be written and for the complex
> > stuff ast.literal_eval() can be used to safely parse the complex list
> > of requirements with comments.
>
> Sadly, pip needs to work on Python 2.6+, so we can't use these new
> features of configparser.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 10 May 2016 at 13:40, Wolfgang  wrote:
> So why not use the ConfigParser available with Python and extend it to meet
> the requirements. Custom getters can be written and for the complex
> stuff ast.literal_eval() can be used to safely parse the complex list
> of requirements with comments.

Sadly, pip needs to work on Python 2.6+, so we can't use these new
features of configparser.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/10/2016 09:38 AM, Alex Grönholm wrote:

10.05.2016, 19:35, Ethan Furman kirjoitti:



It's too complicated and error-prone.  If we want buy-in from casual
packagers then our configuration language needs to be simple to
understand and simple to get right.  (The amount of leading whitespace
on a single line changes your data type? Really??  0644 and 0x644 both
map to 420? and 644 maps to 644?)

>

Do you actually expect to use these in your project's configuration?


I might.  A couple in-house projects have scripts that only root should run.


No?


Don't assume for me, and don't assume for the hundreds (thousands?  tens 
of thousands?) of others who will be using this.



Then why is this a problem for you in this case?


See above.

--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

10.05.2016, 19:35, Ethan Furman kirjoitti:

On 05/10/2016 08:41 AM, Alex Grönholm wrote:

10.05.2016, 18:26, Ethan Furman kirjoitti:



Please no.  I'd rather do xml than yaml.


Why do you hate it so much? I strongly prefer YAML to anything else I've
seen here.


It's too complicated and error-prone.  If we want buy-in from casual 
packagers then our configuration language needs to be simple to 
understand and simple to get right.  (The amount of leading whitespace 
on a single line changes your data type? Really??  0644 and 0x644 both 
map to 420? and 644 maps to 644?)
Do you actually expect to use these in your project's configuration? No? 
Then why is this a problem for you in this case?


https://docs.saltstack.com/en/latest/topics/troubleshooting/yaml_idiosyncrasies.html 



https://ciaranm.wordpress.com/2009/03/01/yaml-sucks-gems-sucks-syck-sucks/ 



While YAML is more easily readable than XML, with XML you already know 
you're in hell so you tread more carefully.  ;)


If we want to take the good ideas of YAML and make our own thing I'm 
okay with that -- but not YAML itself.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/10/2016 08:41 AM, Alex Grönholm wrote:

10.05.2016, 18:26, Ethan Furman kirjoitti:



Please no.  I'd rather do xml than yaml.


Why do you hate it so much? I strongly prefer YAML to anything else I've
seen here.


It's too complicated and error-prone.  If we want buy-in from casual 
packagers then our configuration language needs to be simple to 
understand and simple to get right.  (The amount of leading whitespace 
on a single line changes your data type?  Really??  0644 and 0x644 both 
map to 420? and 644 maps to 644?)


https://docs.saltstack.com/en/latest/topics/troubleshooting/yaml_idiosyncrasies.html

https://ciaranm.wordpress.com/2009/03/01/yaml-sucks-gems-sucks-syck-sucks/

While YAML is more easily readable than XML, with XML you already know 
you're in hell so you tread more carefully.  ;)


If we want to take the good ideas of YAML and make our own thing I'm 
okay with that -- but not YAML itself.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Wolfgang]

Hi,

have done a lot of configuration stuff. Also used the Python ConfigParser
(backported version from 3.x)
It can be improved, yes. ;-)

But the INI style syntax is known and there are tools and parsers available.
It is a simple format and this is good. Because it is still human readable
and simple configurations can be done with it. Configurations should be simple!

But if there are complex requirements the ConfigParser can be extended.
(done this for custom interpolation and config includes)
Also the allowed comment characters can customized and limited to '#'.

So why not use the ConfigParser available with Python and extend it to meet
the requirements. Custom getters can be written and for the complex
stuff ast.literal_eval() can be used to safely parse the complex list
of requirements with comments.
Like register new converter:

converters = {"literal": ast.literal_eval}

used then with:

config.getliteral("bootstrap", "requirements")

Config example:

[bootstrap]
reqirements = [ "setuptools >= 27",
"numpy >= 1.10",
"the-versionator == 0.13",
  ]


My opinion is, keep the configuration and used syntax for this simple.
YAML is not simple even the spec is complex.
TOML is not mature enough and even not known enough.

INI style is already used for setup.cfg. Use it in other places too.
Extend it where the need is to extend it and document this.
If extended with Python syntax, everyone should be familiar with it.
The tools are used for/with Python.


Regards,

Wolfgang

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Mark Niedzielski]

A new, important, use of --target is in creating installation bundles
for AWS Lambda functions using a Python runtime.  AWS expects a zip file
containing the primary source file and all dependent packages.  'pip
install -t' creates exactly the structure needed.  Further, because a
'pip install -t .' has the same behavior as a 'npm install' we can
easily use the same packaging code for Python and Node Lambda function
bundles.

Thanks.

-Mark

This e-mail message and all attachments transmitted with it may contain 
privileged and/or confidential information intended solely for the use of the 
addressee(s). If the reader of this message is not the intended recipient, you 
are hereby notified that any reading, dissemination, distribution, copying, 
forwarding or other use of this message or its attachments is strictly 
prohibited. If you have received this message in error, please notify the 
sender immediately and delete this message, all attachments and all copies and 
backups thereof.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 10 May 2016 at 16:00, Antoine Pitrou  wrote:
> On Tue, 10 May 2016 10:55:38 -0400
> Donald Stufft  wrote:
>>
>> I think TOML is more usable than ConfigParser and in particular I think that
>> the adhoc post processing step makes ConfigParser inherently less usable
>> because it forces a special syntax that is specific to this one file. It also
>> means that there's no "right" answer for when you have two different
>> implementations that interpret the same file differently.
>
> That's true. OTOH, the question is how much better it is for users
> that it's worthwhile bothering them with a syntax change that will
> require (at one point or another) migrating existing files. TOML doesn't
> seem that compelling to me in that regard (quite less than YAML, and I'm
> not a YAML fan).

The one aspect that's missing from this discussion (largely because
the PEP's still in the process of being written) is a clear statement
of what capabilities we *need* from a config file format. I suspect
we'll need:

  - Lists, for dependencies
  - Nesting (although this is more for "future expansion" by allowing
us to namespace the keys)

If we're looking to put package metadata in this file, we may also
need multi-line string capabilities (for things like descriptions).

Before we decide on a format based solely on tool support ("how easy
is it for us to write the code?") we should probably see examples of
the expected file config using the various different formats ("how
easy is the format for users to use?").

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

10.05.2016, 18:26, Ethan Furman kirjoitti:

On 05/10/2016 08:14 AM, Donald Stufft wrote:

On May 10, 2016, at 11:00 AM, Antoine Pitrou wrote:



(as an aside, if there's the question of forking an existing parser
implementation for better vendorability, forking a YAML parser may be
more useful to third-party folks than forking a TOML parser :-))


I’m seeing what I can come up with (https://github.com/dstufft/yaml) 
though
I don’t know that I feel like actually maintaining whatever it is I 
end up

figuring out there.


Please no.  I'd rather do xml than yaml.

Why do you hate it so much? I strongly prefer YAML to anything else I've 
seen here.

--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 10, 2016, at 10:16 AM, Antoine Pitrou  wrote:
> 
> On Tue, 10 May 2016 10:38:51 +0300
> Alex Grönholm  wrote:
>> TOML isn't much better than ConfigParser in terms of representing nested
>> structures.
> 
> Indeed, that seems to be a strong point against TOML.  If we don't care
> about nested structures that much, then ConfigParser should be more or
> less ok…
> 

TOML is infinitely better at nested structured that ConfigParser, given that
TOML actually *supports* nested structures beyond a level of 1. The only way
to get anything like:

[package.build]
dependencies = ["setuptools", "wheel"]

In ConfigParser is to add post-processing to the values, which then you're no
longer a "ConfigParser" file, you're a "ConfigParser + Whatever random one off
code you wrote to do post processing" file.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/10/2016 08:14 AM, Donald Stufft wrote:

On May 10, 2016, at 11:00 AM, Antoine Pitrou wrote:



(as an aside, if there's the question of forking an existing parser
implementation for better vendorability, forking a YAML parser may be
more useful to third-party folks than forking a TOML parser :-))


I’m seeing what I can come up with (https://github.com/dstufft/yaml) though
I don’t know that I feel like actually maintaining whatever it is I end up
figuring out there.


Please no.  I'd rather do xml than yaml.

--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

10.05.2016, 18:14, Donald Stufft kirjoitti:

On May 10, 2016, at 11:00 AM, Antoine Pitrou  wrote:

(as an aside, if there's the question of forking an existing parser
implementation for better vendorability, forking a YAML parser may be
more useful to third-party folks than forking a TOML parser :-))


I’m seeing what I can come up with (https://github.com/dstufft/yaml) though
I don’t know that I feel like actually maintaining whatever it is I end up
figuring out there.

What exactly are your goals here?


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 10, 2016, at 11:00 AM, Antoine Pitrou  wrote:
> 
> (as an aside, if there's the question of forking an existing parser
> implementation for better vendorability, forking a YAML parser may be
> more useful to third-party folks than forking a TOML parser :-))


I’m seeing what I can come up with (https://github.com/dstufft/yaml) though
I don’t know that I feel like actually maintaining whatever it is I end up
figuring out there.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

10.05.2016, 18:00, Antoine Pitrou kirjoitti:

On Tue, 10 May 2016 10:55:38 -0400
Donald Stufft  wrote:

I think TOML is more usable than ConfigParser and in particular I think that
the adhoc post processing step makes ConfigParser inherently less usable
because it forces a special syntax that is specific to this one file. It also
means that there's no "right" answer for when you have two different
implementations that interpret the same file differently.

That's true. OTOH, the question is how much better it is for users
that it's worthwhile bothering them with a syntax change that will
require (at one point or another) migrating existing files. TOML doesn't
seem that compelling to me in that regard (quite less than YAML, and I'm
not a YAML fan).

(as an aside, if there's the question of forking an existing parser
implementation for better vendorability, forking a YAML parser may be
more useful to third-party folks than forking a TOML parser :-))

Amen to that, and that's exactly what I'd like to do.
What should the parser be capable of to be accepted for this task? What 
are the requirements?

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Antoine Pitrou]

On Tue, 10 May 2016 10:55:38 -0400
Donald Stufft  wrote:
> 
> I think TOML is more usable than ConfigParser and in particular I think that
> the adhoc post processing step makes ConfigParser inherently less usable
> because it forces a special syntax that is specific to this one file. It also
> means that there's no "right" answer for when you have two different
> implementations that interpret the same file differently.

That's true. OTOH, the question is how much better it is for users
that it's worthwhile bothering them with a syntax change that will
require (at one point or another) migrating existing files. TOML doesn't
seem that compelling to me in that regard (quite less than YAML, and I'm
not a YAML fan).

(as an aside, if there's the question of forking an existing parser
implementation for better vendorability, forking a YAML parser may be
more useful to third-party folks than forking a TOML parser :-))

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 10, 2016, at 10:30 AM, Antoine Pitrou  wrote:
> 
> On Tue, 10 May 2016 10:24:10 -0400
> Donald Stufft  wrote:
>> 
>> TOML is infinitely better at nested structured that ConfigParser, given that
>> TOML actually *supports* nested structures beyond a level of 1. The only way
>> to get anything like:
>> 
>>[package.build]
>>dependencies = ["setuptools", "wheel"]
>> 
>> In ConfigParser is to add post-processing to the values, which then you're no
>> longer a "ConfigParser" file, you're a "ConfigParser + Whatever random one 
>> off
>> code you wrote to do post processing" file.
> 
> The post-processing doesn't seem difficult enough to make any fuss
> about it, IMHO.  The most important concern here should be how usable
> the format is for end users, not whether implementations need 20
> additional lines of code to work with it.

I think TOML is more usable than ConfigParser and in particular I think that
the adhoc post processing step makes ConfigParser inherently less usable
because it forces a special syntax that is specific to this one file. It also
means that there's no "right" answer for when you have two different
implementations that interpret the same file differently. There's no spec to
work off of so it ends up regressing to a smaller version of one of our current
problems with the toolchain-- Everything is implementation defined by whatever
the most popular tool at the time is.


> 
> (also, what is wrong with providing a pypa-specific library for parsing
> required configuration? are distlib / distil / pkgutil / the
> distutils-competitor-du-jour still alive?)


While we are providing unopinionated bindings like that for the PEPS, we don't
want to make that "special" any more than it's just a reference implementation.
Tools in languages other than Python will be expected to parse and read these
files.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Antoine Pitrou]

On Tue, 10 May 2016 10:24:10 -0400
Donald Stufft  wrote:
> 
> TOML is infinitely better at nested structured that ConfigParser, given that
> TOML actually *supports* nested structures beyond a level of 1. The only way
> to get anything like:
> 
> [package.build]
> dependencies = ["setuptools", "wheel"]
> 
> In ConfigParser is to add post-processing to the values, which then you're no
> longer a "ConfigParser" file, you're a "ConfigParser + Whatever random one off
> code you wrote to do post processing" file.

The post-processing doesn't seem difficult enough to make any fuss
about it, IMHO.  The most important concern here should be how usable
the format is for end users, not whether implementations need 20
additional lines of code to work with it.

(also, what is wrong with providing a pypa-specific library for parsing
required configuration? are distlib / distil / pkgutil / the
distutils-competitor-du-jour still alive?)

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/10/2016 01:54 AM, Paul Moore wrote:


Writing our own is simply a way to end up with additional maintenance
work, that we really don't have the resources for.


I like writing tools.

If the format is one I can get behind I'm happy to be the resource for 
it.  This rules out JSON and YAML, but leaves TOML in the running (as 
in: I'm happy to take over pytoml if its current author is agreeable).


I'm also happy to create one: The Sane/Simple/Super Config Language (or 
.scl for short).  It would be very similar to TOML, possibly a superset.


Tools written so far:

- dbf [https://pypi.python.org/pypi/dbf]
  project I learned Python with (so some rough edges, but very
  serviceable)

- scription [https://pypi.python.org/pypi/scription]
  opinionated command-line parser

- antipathy [https://pypi.python.org/pypi/antipathy]
  file system path library

- aenum [https://pypi.python.org/pypi/aneum]
  totally awesome Enum library ;)  (scaled-down version is the
  stdlib Enum)

- xaml [https://pypi.python.org/pypi/xaml]
  xml processor similar to Ruby's haml

PEPs (co)authored so for have also been tool/library oriented:

- PEP 409 [https://www.python.org/dev/peps/pep-0409]
  raise from None (dbf inspired)

- PEP 435 [https://www.python.org/dev/peps/pep-0435]
  Enum

- PEP 461 [https://www.python.org/dev/peps/pep-0461]
  %-interpolation for bytes & bytearrays

In other words:  this is a serious offer.  ;)

--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Antoine Pitrou]

On Tue, 10 May 2016 10:38:51 +0300
Alex Grönholm  wrote:
> TOML isn't much better than ConfigParser in terms of representing nested 
> structures.

Indeed, that seems to be a strong point against TOML.  If we don't care
about nested structures that much, then ConfigParser should be more or
less ok...

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Barry Warsaw]

On May 09, 2016, at 08:30 PM, Donald Stufft wrote:

>How hard is it to bundle it with pip by copying the source files into
>pip._vendor.*

Every time another package is vendored, a kitten falls off a unicorn. ;)

Cheers,
-Barry


pgpetwbrYXBeb.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Leonardo Rochael Almeida]

In all this talk about using a YAML subset, I'm surprised no one mentioned
YAMLish:

https://pypi.python.org/pypi/yamlish

It is a well defined subset of YAML and there are implementations in other
programming languages.

The problem with the 200+-lines-single-file library above is that it
depends on PyYAML itself so, vendoring it will be challenging.

Anyway, I think TOML 0.4.0 is good enough for our needs.

On Tue, 10 May 2016 10:03 Nick Coghlan,  wrote:

> On 10 May 2016 at 18:54, Paul Moore  wrote:
> > Well, IMO, the state of things in terms of config file formats (and
> > not just in Python) is itself pretty dreadful - every time I write an
> > application, I am astounded that there are no good options for
> > something as basic as a configuration file format.
>
> This is pretty normal for software - no good options, but a plethora
> of "good enough" ones. Hence https://xkcd.com/927/ :)
>
> We just have a particularly exacting use case here, since we want:
>
> - a format that's attractive for folks just learning to program in 2016
> - a format that's attractive for folks that have been programming for 50+
> years
> - a format that's easy to parse even in Python 2.6
> - a format that's version control friendly
> - a format that's text editor friendly (syntax highlighting, etc)
>
> For me, the two leading contenders out of the current discussion have been:
>
> - the poyo subset of YAML 1.1
> - TOML 0.4.0, as implemented by pytoml
>
> The "subset of PyYAML" approach turns this into a documentation
> exercise (which subset?), and also runs into the problem that poyo
> doesn't handle serialisation yet, only *reading* YAML files.
>
> For TOML, the main questions being asked are around how widely
> supported it is, and it's actually in a pretty good state on that
> front. 0.4.0 (rather than TOML in general) is stable by definition,
> and there are quite a few implementations of that across different
> languages:
> https://github.com/toml-lang/toml/blob/master/versions/en/toml-v0.4.0.md#implementations
>
> The other big thing someone might want is a schema validator, and
> there I think it may be possible to just use jsonschema and validate
> the result of parsing the file, rather than validating the serialised
> form directly.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nick Coghlan]

On 10 May 2016 at 18:54, Paul Moore  wrote:
> Well, IMO, the state of things in terms of config file formats (and
> not just in Python) is itself pretty dreadful - every time I write an
> application, I am astounded that there are no good options for
> something as basic as a configuration file format.

This is pretty normal for software - no good options, but a plethora
of "good enough" ones. Hence https://xkcd.com/927/ :)

We just have a particularly exacting use case here, since we want:

- a format that's attractive for folks just learning to program in 2016
- a format that's attractive for folks that have been programming for 50+ years
- a format that's easy to parse even in Python 2.6
- a format that's version control friendly
- a format that's text editor friendly (syntax highlighting, etc)

For me, the two leading contenders out of the current discussion have been:

- the poyo subset of YAML 1.1
- TOML 0.4.0, as implemented by pytoml

The "subset of PyYAML" approach turns this into a documentation
exercise (which subset?), and also runs into the problem that poyo
doesn't handle serialisation yet, only *reading* YAML files.

For TOML, the main questions being asked are around how widely
supported it is, and it's actually in a pretty good state on that
front. 0.4.0 (rather than TOML in general) is stable by definition,
and there are quite a few implementations of that across different
languages: 
https://github.com/toml-lang/toml/blob/master/versions/en/toml-v0.4.0.md#implementations

The other big thing someone might want is a schema validator, and
there I think it may be possible to just use jsonschema and validate
the result of parsing the file, rather than validating the serialised
form directly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Randy Syring]

For what it's worth, I've been following this thread, and I like the 
idea of using TOML for all the "pro" reasons posted so far.  It's 
newness or not reaching 1.0 yet don't bother me as I believe the plans 
to specify TOML 0.4 or optionally support the later versions if they 
don't cause problems makes a lot of sense.  The fact that the parser is 
300+ lines of code and can be easily vendored is also a big plus.  Given 
that rust is using TOML, if Python adopts it as well, that is big enough 
"market share" for me and people will get used to it soon enough.


*Randy Syring*
Husband | Father | Redeemed Sinner

/"For what does it profit a man to gain the whole world
and forfeit his soul?" (Mark 8:36 ESV)/

On 05/10/2016 03:38 AM, Alex Grönholm wrote:

A few facts:

  * YAML is good enough for Salt, Ansible and numerous other common tools
  * The YAML standard has been stable for many years, unlike TOML
which still hasn't even reached 1.0
  * YAML has widespread tooling support, unlike TOML

We all agree that JSON is not the solution. No comments, trailing 
commas etc.
TOML isn't much better than ConfigParser in terms of representing 
nested structures.
So far the ONLY objective problems with YAML seems to be the 
problematic implementation named PyYAML. If this is really the case, 
I'd gladly help build a better one just to prevent TOML from being 
chosen for this task. That we're even /considering/ building something 
as important as this on an unstable standard is pretty horrifying to 
me in itself.


10.05.2016, 06:37, Chris Barker kirjoitti:

Really?

writing Yet Another Markup Language (YAML :-) ) CAN'T be the 
simplest, best option.


> After further consideration, and pytoml's author's comment about 
the spec changing without a version increase, I think we might be 
better off rolling our own.



I like the general simplicity, and would stick with that, but I'd
be a lot more comfortable if we had our spec that was more
consistent.


If we're going to do that, then why not the 'simple part of yaml'.

or Python literals. (if I recall, the main reason not to do that was 
that no other language has a lib to read it -- rolling out own does 
not solve that!)


Or just go with JSON -- I'm annoyed by it at times, but it's not SO bad.

(and you can kinda-sorta simulate comments with useless keys :-)

{ "comment": "this is just something i wanted to say here",
...
}

or we could do "JSON with comments" -- not hard to write a tiny 
pre-processor before passing it off to the json lib.


Anyway -- let's avoid the temptation to role your own everything, and 
use something standard!


-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov 


___
Distutils-SIG maillist  -Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig




___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 10 May 2016 at 07:08, Chris Barker  wrote:
> But I started this whole line of conversation because it seemed that there
> was desire for:
>
> Ability to isolate the build environment.
> Ability to better handle/manage non-python dependencies

I don't care about the first one - between disposable VMs and Linux
containers, we're already spoiled for choice when it comes to
supporting isolated build environments, and every project still
gaining net new contributors gets a natural test of this whenever
someone tries to set up their own local build environment.

I do care about the second one - Tennessee Leeuwenburg's draft PEP for
that is something he put together at the PyCon Australia sprints, and
it's the cornerstone of eventually being able to publish to PyPI and
have RPMs, Debian packages, conda packages, homebrew packages, etc,
"just happen" without any need for human intervention, even if your
package has external binary dependencies.

The fact that people would potentially be able to do "pip wheel" more
easily (since they'd be presented with a clean "you need , but
don't have it" error message rather than a cryptic build failure) is a
nice bonus, but it's not the reason I personally care about the
feature - I care about making more efficient use of distro packager's
time, by only asking them to do things a computer couldn't be doing
instead. The more complete we're able to make upstream dependency
metadata, the less people will need to manually tweak the output of
pyp2rpm (and similar tools for other platforms).

>> (twice, actually, anchored on
>> /usr/bin/python and /usr/bin/python3), it should eventually be
>> feasible to have the upstream->conda pipeline fully automated as well.
>
> yeah -- there's been talk for ages of automatically building conda packages
> (on the fly, maybe) from PyPi packages. But currently on conda-forge we've
> decided to NOT try to do that -- it's turned out in practice that enough
> pypi packages end up needing some hand-tweaking to build. So teh planned
> workflow is now:
>
> Auto-build a conda build script for a PyPi package
> Test it
> Tweak it as required
> Add it to conda-forge.
>
> Then -- *maybe* write a tool that auto-updates the PyPi based packages in a
> chron job or whatever.
>
> So not quite a automated conda-PyPi bridge, but not a bad start.

Yep, this is around the same level the Linux distros are generally at
- a distro level config gets generated *once* (perhaps with a tool
like pyp2rpm), but the result of that process then needs to be hand
edited when new upstream releases come out, even if nothing
significant has changed except the application code itself (no new
build dependencies, no new build steps, etc).

Utilities like Fedora's rebase-helper can help automate those updates,
but I still consider the ideal arrangement to be for the upstream
metadata to be of a sufficient standard that post-generation manual
tweaking ceases to be necessary in most cases.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

This looks very close to what I'd like to have, but then we'd have the 
situation of an uncommon format with no tooling support, won't we? 
Assuming the actual config file is in xaml format.


10.05.2016, 02:56, Ethan Furman kirjoitti:

On 05/06/2016 07:59 PM, Nathaniel Smith wrote:


Here's that one-stop writeup/comparison of all the major configuration
languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f


Very nice work-up, thanks!


However, you didn't include XML -- which, while absolutely horrid, can 
be quite readable with the appropriate preprocessor, such as xaml [1] 
:


--- 8< whatever.xaml ---
!!! xml1.0
~base

~schema
// optional
~version: 1

~bootstrap
~requirements
// Temporarily commented out 2016-01-10
// magic-build-helper
~setuptools
~version: >= 27
// for the new frobnicate feature
~numpy
~version: >= 1.10
//Pinned until we get a fix for
// @https://github.com/cyberdyne/the-versionator/issues/123
~the-versionator
~version: 0.13

// The owner of pypi name "flit" decides what goes under the
// extension: flit:
// key
~extensions
~flit
~whatever: true
--- 8< -

which ends up as:

--- 8< whatever.xml 





1






= 27



= 1.10



0.13









true


--- 8< -

--
~Ethan~

[1] https://pypi.python.org/pypi/xaml
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

10.05.2016, 12:43, Ionel Cristian Mărieș kirjoitti:


On Tue, May 10, 2016 at 10:38 AM, Alex Grönholm 
> wrote:


So far the ONLY objective problems with YAML seems to be the
problematic implementation named PyYAML. If this is really the
case, I'd gladly help build a better one just to prevent TOML from
being chosen for this task. That we're even /considering/ building
something as important as this on an unstable standard is pretty
horrifying to me in itself.


​Just my two cents here: every time, but every every time,​ I have to 
google around about how to create a multi-line string in YAML. There 
are too many ways to write the same thing. And lets not forget those 
damn sexagesimal literals. The complexity of that language is beyond 
repair, it's not a *library* problem. Just look at insanities like 
this 
 
​or this ​ .



I have no problem with any of the examples you linked to.

Thanks,
-- IonelCristian Mărieș, http://blog.ionelmc.ro


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ionel Cristian Mărieș]

On Tue, May 10, 2016 at 10:38 AM, Alex Grönholm 
wrote:

> So far the ONLY objective problems with YAML seems to be the problematic
> implementation named PyYAML. If this is really the case, I'd gladly help
> build a better one just to prevent TOML from being chosen for this task.
> That we're even *considering* building something as important as this on
> an unstable standard is pretty horrifying to me in itself.
>

​Just my two cents here: every time, but every every time,​ I have to
google around about how to create a multi-line string in YAML. There are
too many ways to write the same thing. And lets not forget those damn
sexagesimal literals. The complexity of that language is beyond repair,
it's not a *library* problem. Just look at insanities like this

​or this ​.

Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 10 May 2016 at 08:38, Alex Grönholm  wrote:
> A few facts:
>
> YAML is good enough for Salt, Ansible and numerous other common tools
> The YAML standard has been stable for many years, unlike TOML which still
> hasn't even reached 1.0
> YAML has widespread tooling support, unlike TOML
>
> We all agree that JSON is not the solution. No comments, trailing commas
> etc.
> TOML isn't much better than ConfigParser in terms of representing nested
> structures.

Just as another data point, Cookiecutter started off using pyYAML,
moved to ruamel.yaml, and ended up with their own (separated out into
an independent project, but written specifically for Cookicutter)
implementation of "some of" YAML, called poyo. I believe the reason
they did this was repeated issues with the existing YAML libraries.
And while the YAML-subset provided by poyo *looks* reasonable, I don't
see any documentation of precisely *what* subset of YAML is
implemented (apart from some examples). The poyo library is also very
new, and I doubt it's seen much usage/testing outside of cookiecutter
so far.

I would love to use YAML. I really would. But for pip, we need a
robust, easy to vendor Python implementation (with no C dependency)
that is safe (see
http://community.embarcadero.com/blogs/entry/yaml-and-remote-code-execution-38738).
Writing our own is simply a way to end up with additional maintenance
work, that we really don't have the resources for.

> So far the ONLY objective problems with YAML seems to be the problematic
> implementation named PyYAML. If this is really the case, I'd gladly help
> build a better one just to prevent TOML from being chosen for this task.

If you can get a robust, stable YAML library written fast enough to be
an option for the PEP, then that would certainly be a possibility. But
given that poyo has been under development for 5 months, are you going
to be able to do better than that in a few weeks? (And see my comments
above on poyo).

> That we're even considering building something as important as this on an
> unstable standard is pretty horrifying to me in itself.

Well, IMO, the state of things in terms of config file formats (and
not just in Python) is itself pretty dreadful - every time I write an
application, I am astounded that there are no good options for
something as basic as a configuration file format.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

A few facts:

 * YAML is good enough for Salt, Ansible and numerous other common tools
 * The YAML standard has been stable for many years, unlike TOML which
   still hasn't even reached 1.0
 * YAML has widespread tooling support, unlike TOML

We all agree that JSON is not the solution. No comments, trailing commas 
etc.
TOML isn't much better than ConfigParser in terms of representing nested 
structures.
So far the ONLY objective problems with YAML seems to be the problematic 
implementation named PyYAML. If this is really the case, I'd gladly help 
build a better one just to prevent TOML from being chosen for this task. 
That we're even /considering/ building something as important as this on 
an unstable standard is pretty horrifying to me in itself.


10.05.2016, 06:37, Chris Barker kirjoitti:

Really?

writing Yet Another Markup Language (YAML :-) ) CAN'T be the simplest, 
best option.


> After further consideration, and pytoml's author's comment about the 
spec changing without a version increase, I think we might be better 
off rolling our own.



I like the general simplicity, and would stick with that, but I'd
be a lot more comfortable if we had our spec that was more consistent.


If we're going to do that, then why not the 'simple part of yaml'.

or Python literals. (if I recall, the main reason not to do that was 
that no other language has a lib to read it -- rolling out own does 
not solve that!)


Or just go with JSON -- I'm annoyed by it at times, but it's not SO bad.

(and you can kinda-sorta simulate comments with useless keys :-)

{ "comment": "this is just something i wanted to say here",
...
}

or we could do "JSON with comments" -- not hard to write a tiny 
pre-processor before passing it off to the json lib.


Anyway -- let's avoid the temptation to role your own everything, and 
use something standard!


-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov 


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig