Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 16 May 2016 at 16:41, Chris Barker - NOAA Federal
 wrote:
> As pointed out, it's a C lib. But as we all like writing tools, it wouldn't
> be very hard to write a Python parser for the format.

There is one (on PyPI - can't recall the name now, sorry).

> But it's a bit C-y for my taste, and yet another configure language? Really?

Also, the spec seems to imply that it doesn't support Unicode (no
Unicode escapes in strings).

Anyway, as noted this is very off-topic now, so that's all I'll say.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ian Cordasco]

The PEP has been accepted by Nick. Let's turn our thoughts to more
productive topics rather than talking about a moot point.

Cheers, Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Wes Turner]

1. again, ConfigObj is pure Python, supports nesting, and is read/write (so
there is no need to template injectable config files (e.g., with  \n, =,
#;etc. must be escaped))

| PyPI: https://pypi.python.org/pypi/configobj/5.0.6
| Src: https://github.com/DiffSK/configobj
| Docs: https://configobj.readthedocs.io/en/latest/

2. A grammar for configparser would make this more apparent
https://hg.python.org/cpython/file/tip/Lib/configparser.py

On Sunday, May 15, 2016, Robert Collins  wrote:

> On 15 May 2016 at 08:21, Ionel Cristian Mărieș  > wrote:
> >
> > On Fri, May 13, 2016 at 9:22 PM, Brett Cannon  > wrote:
> >>
> >> No need to think; the decision is made and it's TOML. I know Chris
> doesn't
> >> mean to stir up trouble, but at this point if someone wants to propose
> >> something other than TOML they are going to have to write their own PEP.
> >
> >
> > Not asking for any change but has anyone looked at libconfig? It looks
> quite
> > interesting: simple grammar and nesting support. What do you think of it?
>
> I hadn't, but its certainly irrelevant here, as vendoring it suitably
> for pip would be excruciating due to the C dependency.
>
> -Rob
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org 
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker - NOAA Federal]

​Not asking for any change but has anyone looked at libconfig
? ​It looks
quite interesting: simple grammar and nesting support. What do you think of
it


As pointed out, it's a C lib. But as we all like writing tools, it wouldn't
be very hard to write a Python parser for the format.

But it's a bit C-y for my taste, and yet another configure language? Really?

-CHB



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Robert Collins]

On 15 May 2016 at 08:21, Ionel Cristian Mărieș  wrote:
>
> On Fri, May 13, 2016 at 9:22 PM, Brett Cannon  wrote:
>>
>> No need to think; the decision is made and it's TOML. I know Chris doesn't
>> mean to stir up trouble, but at this point if someone wants to propose
>> something other than TOML they are going to have to write their own PEP.
>
>
> Not asking for any change but has anyone looked at libconfig? It looks quite
> interesting: simple grammar and nesting support. What do you think of it?

I hadn't, but its certainly irrelevant here, as vendoring it suitably
for pip would be excruciating due to the C dependency.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ionel Cristian Mărieș]

On Fri, May 13, 2016 at 9:22 PM, Brett Cannon  wrote:

> No need to think; the decision is made and it's TOML. I know Chris doesn't
> mean to stir up trouble, but at this point if someone wants to propose
> something other than TOML they are going to have to write their own PEP.


​Not asking for any change but has anyone looked at libconfig
? ​It looks
quite interesting: simple grammar and nesting support. What do you think of
it?



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Lele Gaifax]

Chris Barker  writes:

> Oh, and why not "JSON with comments and trailing commas" - it would be well
> defined and easy to implement.

And mostly done, even: https://bitbucket.org/intellimath/pyaxon

ciao, lele.
-- 
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
l...@metapensiero.it  | -- Fortunato Depero, 1929.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

On Fri, May 13, 2016 at 11:22 AM, Brett Cannon  wrote:


> No need to think; the decision is made and it's TOML. I know Chris doesn't
> mean to stir up trouble,
>

I got a bit out of sync with the conversation -- sorry for the noise.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Brett Cannon]

On Fri, 13 May 2016 at 10:47 Paul Moore  wrote:

> On 13 May 2016 at 18:15, Chris Barker  wrote:
> > I think we're freaking out way too much about what *could* go wrong.
>
> It's more that:pip would have to vendor pyyaml, and it's not small. I
> have no idea whether it's easy to vendor, either (does it have
> separate code paths for Python 2 and 3? I don't know, I've never
> looked). Also, we'd have to forego the C implementation - I have no
> idea how much of a performance hit that would give (of course,
> performance is hardly a key issue here) or how likely it is that bugs
> exist in the Python version that aren't normally noticed because
> people use the C implementation (which is definitely just speculation,
> conceded).
>
> But I think the decision is made, so it's not worth debating any more,
> honestly.
>

No need to think; the decision is made and it's TOML. I know Chris doesn't
mean to stir up trouble, but at this point if someone wants to propose
something other than TOML they are going to have to write their own PEP.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nathaniel Smith]

On Fri, May 13, 2016 at 10:15 AM, Chris Barker  wrote:
> On Tue, May 10, 2016 at 1:54 AM, Paul Moore  wrote:
>>
>> I would love to use YAML. I really would. But for pip, we need a
>> robust, easy to vendor Python implementation
>
>
> conda has been using yaml forever (with pyyaml) , and whatever problem is
> has (and there are many), I don't think I've ever seen a single issue cause
> by yaml or pyyaml. Might be worth asking the conda devs.

"vendor" here means "munge the library so that it becomes a
pure-python subpackage of pip named pip._vendor.whatever". This is a
specific technical challenge that conda doesn't attempt...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 13 May 2016 at 18:15, Chris Barker  wrote:
> I think we're freaking out way too much about what *could* go wrong.

It's more that:pip would have to vendor pyyaml, and it's not small. I
have no idea whether it's easy to vendor, either (does it have
separate code paths for Python 2 and 3? I don't know, I've never
looked). Also, we'd have to forego the C implementation - I have no
idea how much of a performance hit that would give (of course,
performance is hardly a key issue here) or how likely it is that bugs
exist in the Python version that aren't normally noticed because
people use the C implementation (which is definitely just speculation,
conceded).

But I think the decision is made, so it's not worth debating any more, honestly.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

On Tue, May 10, 2016 at 1:54 AM, Paul Moore  wrote:

> I would love to use YAML. I really would. But for pip, we need a
> robust, easy to vendor Python implementation


conda has been using yaml forever (with pyyaml) , and whatever problem is
has (and there are many), I don't think I've ever seen a single issue cause
by yaml or pyyaml. Might be worth asking the conda devs.

I think we're freaking out way too much about what *could* go wrong.

Oh, and why not "JSON with comments and trailing commas" - it would be well
defined and easy to implement.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 12, 2016, at 3:20 AM, Nick Coghlan  wrote:
> 
> On 11 May 2016 at 15:47, Greg Ewing  wrote:
>> Having looked over the TOML spec, I like the simplicity
>> of it (and I cringe from the complexity of YAML).
>> The only thing I don't like about TOML is the way it
>> cops out on nesting.
>> 
>> The only reason it does that as far as I can see is
>> because of a dislike for significant indentation.
> 
> This is a feature, not a bug.

It did occur to me yesterday an unambiguous way to reduce the repetition of 
nested tables without introducing significant whitespace. I opened an issue on 
the TOML Github repo [1] but the gist of it is similar to how Python uses . to 
do relative imports. They may or may not like it though, who knows :)

[1] https://github.com/toml-lang/toml/issues/413

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nick Coghlan]

On 11 May 2016 at 15:47, Greg Ewing  wrote:
> Having looked over the TOML spec, I like the simplicity
> of it (and I cringe from the complexity of YAML).
> The only thing I don't like about TOML is the way it
> cops out on nesting.
>
> The only reason it does that as far as I can see is
> because of a dislike for significant indentation.

This is a feature, not a bug. The trick is that TOML allows *implicit*
namespace creation when you only care about the innermost level:

[this.is.nested]
nice = "As a human reader & writer, you don't care about the nesting"

[this-is-not-nested]
nice = "As a human reader & writer, you don't care about the lack of nesting"

The only folks that need to care about whether the configuration is
actually nested or flat are those working with the config file
*programmatically* - from the point of view of the folks hand editing
the config file, it's always just the flat list of named key=value
sections that they're used to in traditional ini files.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 11, 2016, at 9:46 AM, Nicholas Chammas  
> wrote:
> 
> On Tue, May 10, 2016 at 11:15 AM Donald Stufft don...@stufft.io 
>  wrote:
> 
> 
> 
> 
> > On May 10, 2016, at 11:00 AM, Antoine Pitrou  > > wrote:
> >
> > (as an aside, if there's the question of forking an existing parser
> > implementation for better vendorability, forking a YAML parser may be
> > more useful to third-party folks than forking a TOML parser :-))
> 
> 
> I’m seeing what I can come up with (https://github.com/dstufft/yaml 
> ) though
> I don’t know that I feel like actually maintaining whatever it is I end up
> figuring out there.
> 
> 
> Did you intend to fork from https://github.com/yaml/pyyaml 
> ? I believe the latest PyYAML is actually 
> hosted on BitBucket:
> 
> https://bitbucket.org/xi/pyyaml/ 
> 

Yea, there was only a very tiny, mostly meaningless delta between Github and 
bitbucket and I’m lazy.


> Also, a better starting point than PyYAML may be this existing fork of the 
> library, ruamel.yaml:
> 
> https://bitbucket.org/ruamel/yaml 
> The fork makes several  
> changes 
> 
>  which may be relevant to you.
> 
> 

To be clear, my thing was a proof of concept to see how hard it would be to 
help inform the decision about what file format to use. The PEP has decided 
that TOML best represents the tradeoffs we want to make so my lib will likely 
just die on my GitHub :)

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nicholas Chammas]

On Tue, May 10, 2016 at 11:15 AM Donald Stufft don...@stufft.io
 wrote:


> > On May 10, 2016, at 11:00 AM, Antoine Pitrou 
> wrote:
> >
> > (as an aside, if there's the question of forking an existing parser
> > implementation for better vendorability, forking a YAML parser may be
> > more useful to third-party folks than forking a TOML parser :-))
>
>
> I’m seeing what I can come up with (https://github.com/dstufft/yaml)
> though
> I don’t know that I feel like actually maintaining whatever it is I end up
> figuring out there.
>
Did you intend to fork from https://github.com/yaml/pyyaml? I believe the
latest PyYAML is actually hosted on BitBucket:

https://bitbucket.org/xi/pyyaml/

Also, a better starting point than PyYAML may be this existing fork of the
library, ruamel.yaml:

https://bitbucket.org/ruamel/yaml

The fork makes several 
changes

which may be relevant to you.

Nick
​
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Greg Ewing]

Having looked over the TOML spec, I like the simplicity
of it (and I cringe from the complexity of YAML).
The only thing I don't like about TOML is the way it
cops out on nesting.

The only reason it does that as far as I can see is
because of a dislike for significant indentation.
But that doesn't apply to us -- we're Python folks,
we're not afraid of indentation! On the contrary,
for us it's the One Obvious Way to do nesting.

So I'd like a format that is very similar to TOML
except that you're allowed to represent nesting using
indented blocks.

--
Greg

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Brett Cannon]

Just so everyone knows, I'm ignoring this thread as the PEP I'm drafting
with Donald and Nathaniel is nearly finished and thus has already settled
on the file format discussion and I haven't heard a new point made on any
file format proposal that has already been brought up previously.

On Tue, 10 May 2016 at 10:10 Paul Moore  wrote:

> On 10 May 2016 at 13:40, Wolfgang  wrote:
> > So why not use the ConfigParser available with Python and extend it to
> meet
> > the requirements. Custom getters can be written and for the complex
> > stuff ast.literal_eval() can be used to safely parse the complex list
> > of requirements with comments.
>
> Sadly, pip needs to work on Python 2.6+, so we can't use these new
> features of configparser.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 10 May 2016 at 13:40, Wolfgang  wrote:
> So why not use the ConfigParser available with Python and extend it to meet
> the requirements. Custom getters can be written and for the complex
> stuff ast.literal_eval() can be used to safely parse the complex list
> of requirements with comments.

Sadly, pip needs to work on Python 2.6+, so we can't use these new
features of configparser.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/10/2016 09:38 AM, Alex Grönholm wrote:

10.05.2016, 19:35, Ethan Furman kirjoitti:



It's too complicated and error-prone.  If we want buy-in from casual
packagers then our configuration language needs to be simple to
understand and simple to get right.  (The amount of leading whitespace
on a single line changes your data type? Really??  0644 and 0x644 both
map to 420? and 644 maps to 644?)

>

Do you actually expect to use these in your project's configuration?


I might.  A couple in-house projects have scripts that only root should run.


No?


Don't assume for me, and don't assume for the hundreds (thousands?  tens 
of thousands?) of others who will be using this.



Then why is this a problem for you in this case?


See above.

--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

10.05.2016, 19:35, Ethan Furman kirjoitti:

On 05/10/2016 08:41 AM, Alex Grönholm wrote:

10.05.2016, 18:26, Ethan Furman kirjoitti:



Please no.  I'd rather do xml than yaml.


Why do you hate it so much? I strongly prefer YAML to anything else I've
seen here.


It's too complicated and error-prone.  If we want buy-in from casual 
packagers then our configuration language needs to be simple to 
understand and simple to get right.  (The amount of leading whitespace 
on a single line changes your data type? Really??  0644 and 0x644 both 
map to 420? and 644 maps to 644?)
Do you actually expect to use these in your project's configuration? No? 
Then why is this a problem for you in this case?


https://docs.saltstack.com/en/latest/topics/troubleshooting/yaml_idiosyncrasies.html 



https://ciaranm.wordpress.com/2009/03/01/yaml-sucks-gems-sucks-syck-sucks/ 



While YAML is more easily readable than XML, with XML you already know 
you're in hell so you tread more carefully.  ;)


If we want to take the good ideas of YAML and make our own thing I'm 
okay with that -- but not YAML itself.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/10/2016 08:41 AM, Alex Grönholm wrote:

10.05.2016, 18:26, Ethan Furman kirjoitti:



Please no.  I'd rather do xml than yaml.


Why do you hate it so much? I strongly prefer YAML to anything else I've
seen here.


It's too complicated and error-prone.  If we want buy-in from casual 
packagers then our configuration language needs to be simple to 
understand and simple to get right.  (The amount of leading whitespace 
on a single line changes your data type?  Really??  0644 and 0x644 both 
map to 420? and 644 maps to 644?)


https://docs.saltstack.com/en/latest/topics/troubleshooting/yaml_idiosyncrasies.html

https://ciaranm.wordpress.com/2009/03/01/yaml-sucks-gems-sucks-syck-sucks/

While YAML is more easily readable than XML, with XML you already know 
you're in hell so you tread more carefully.  ;)


If we want to take the good ideas of YAML and make our own thing I'm 
okay with that -- but not YAML itself.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Wolfgang]

Hi,

have done a lot of configuration stuff. Also used the Python ConfigParser
(backported version from 3.x)
It can be improved, yes. ;-)

But the INI style syntax is known and there are tools and parsers available.
It is a simple format and this is good. Because it is still human readable
and simple configurations can be done with it. Configurations should be simple!

But if there are complex requirements the ConfigParser can be extended.
(done this for custom interpolation and config includes)
Also the allowed comment characters can customized and limited to '#'.

So why not use the ConfigParser available with Python and extend it to meet
the requirements. Custom getters can be written and for the complex
stuff ast.literal_eval() can be used to safely parse the complex list
of requirements with comments.
Like register new converter:

converters = {"literal": ast.literal_eval}

used then with:

config.getliteral("bootstrap", "requirements")

Config example:

[bootstrap]
reqirements = [ "setuptools >= 27",
"numpy >= 1.10",
"the-versionator == 0.13",
  ]


My opinion is, keep the configuration and used syntax for this simple.
YAML is not simple even the spec is complex.
TOML is not mature enough and even not known enough.

INI style is already used for setup.cfg. Use it in other places too.
Extend it where the need is to extend it and document this.
If extended with Python syntax, everyone should be familiar with it.
The tools are used for/with Python.


Regards,

Wolfgang

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 10 May 2016 at 16:00, Antoine Pitrou  wrote:
> On Tue, 10 May 2016 10:55:38 -0400
> Donald Stufft  wrote:
>>
>> I think TOML is more usable than ConfigParser and in particular I think that
>> the adhoc post processing step makes ConfigParser inherently less usable
>> because it forces a special syntax that is specific to this one file. It also
>> means that there's no "right" answer for when you have two different
>> implementations that interpret the same file differently.
>
> That's true. OTOH, the question is how much better it is for users
> that it's worthwhile bothering them with a syntax change that will
> require (at one point or another) migrating existing files. TOML doesn't
> seem that compelling to me in that regard (quite less than YAML, and I'm
> not a YAML fan).

The one aspect that's missing from this discussion (largely because
the PEP's still in the process of being written) is a clear statement
of what capabilities we *need* from a config file format. I suspect
we'll need:

  - Lists, for dependencies
  - Nesting (although this is more for "future expansion" by allowing
us to namespace the keys)

If we're looking to put package metadata in this file, we may also
need multi-line string capabilities (for things like descriptions).

Before we decide on a format based solely on tool support ("how easy
is it for us to write the code?") we should probably see examples of
the expected file config using the various different formats ("how
easy is the format for users to use?").

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

10.05.2016, 18:26, Ethan Furman kirjoitti:

On 05/10/2016 08:14 AM, Donald Stufft wrote:

On May 10, 2016, at 11:00 AM, Antoine Pitrou wrote:



(as an aside, if there's the question of forking an existing parser
implementation for better vendorability, forking a YAML parser may be
more useful to third-party folks than forking a TOML parser :-))


I’m seeing what I can come up with (https://github.com/dstufft/yaml) 
though
I don’t know that I feel like actually maintaining whatever it is I 
end up

figuring out there.


Please no.  I'd rather do xml than yaml.

Why do you hate it so much? I strongly prefer YAML to anything else I've 
seen here.

--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 10, 2016, at 10:16 AM, Antoine Pitrou  wrote:
> 
> On Tue, 10 May 2016 10:38:51 +0300
> Alex Grönholm  wrote:
>> TOML isn't much better than ConfigParser in terms of representing nested
>> structures.
> 
> Indeed, that seems to be a strong point against TOML.  If we don't care
> about nested structures that much, then ConfigParser should be more or
> less ok…
> 

TOML is infinitely better at nested structured that ConfigParser, given that
TOML actually *supports* nested structures beyond a level of 1. The only way
to get anything like:

[package.build]
dependencies = ["setuptools", "wheel"]

In ConfigParser is to add post-processing to the values, which then you're no
longer a "ConfigParser" file, you're a "ConfigParser + Whatever random one off
code you wrote to do post processing" file.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/10/2016 08:14 AM, Donald Stufft wrote:

On May 10, 2016, at 11:00 AM, Antoine Pitrou wrote:



(as an aside, if there's the question of forking an existing parser
implementation for better vendorability, forking a YAML parser may be
more useful to third-party folks than forking a TOML parser :-))


I’m seeing what I can come up with (https://github.com/dstufft/yaml) though
I don’t know that I feel like actually maintaining whatever it is I end up
figuring out there.


Please no.  I'd rather do xml than yaml.

--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

10.05.2016, 18:14, Donald Stufft kirjoitti:

On May 10, 2016, at 11:00 AM, Antoine Pitrou  wrote:

(as an aside, if there's the question of forking an existing parser
implementation for better vendorability, forking a YAML parser may be
more useful to third-party folks than forking a TOML parser :-))


I’m seeing what I can come up with (https://github.com/dstufft/yaml) though
I don’t know that I feel like actually maintaining whatever it is I end up
figuring out there.

What exactly are your goals here?


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 10, 2016, at 11:00 AM, Antoine Pitrou  wrote:
> 
> (as an aside, if there's the question of forking an existing parser
> implementation for better vendorability, forking a YAML parser may be
> more useful to third-party folks than forking a TOML parser :-))


I’m seeing what I can come up with (https://github.com/dstufft/yaml) though
I don’t know that I feel like actually maintaining whatever it is I end up
figuring out there.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

10.05.2016, 18:00, Antoine Pitrou kirjoitti:

On Tue, 10 May 2016 10:55:38 -0400
Donald Stufft  wrote:

I think TOML is more usable than ConfigParser and in particular I think that
the adhoc post processing step makes ConfigParser inherently less usable
because it forces a special syntax that is specific to this one file. It also
means that there's no "right" answer for when you have two different
implementations that interpret the same file differently.

That's true. OTOH, the question is how much better it is for users
that it's worthwhile bothering them with a syntax change that will
require (at one point or another) migrating existing files. TOML doesn't
seem that compelling to me in that regard (quite less than YAML, and I'm
not a YAML fan).

(as an aside, if there's the question of forking an existing parser
implementation for better vendorability, forking a YAML parser may be
more useful to third-party folks than forking a TOML parser :-))

Amen to that, and that's exactly what I'd like to do.
What should the parser be capable of to be accepted for this task? What 
are the requirements?

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Antoine Pitrou]

On Tue, 10 May 2016 10:55:38 -0400
Donald Stufft  wrote:
> 
> I think TOML is more usable than ConfigParser and in particular I think that
> the adhoc post processing step makes ConfigParser inherently less usable
> because it forces a special syntax that is specific to this one file. It also
> means that there's no "right" answer for when you have two different
> implementations that interpret the same file differently.

That's true. OTOH, the question is how much better it is for users
that it's worthwhile bothering them with a syntax change that will
require (at one point or another) migrating existing files. TOML doesn't
seem that compelling to me in that regard (quite less than YAML, and I'm
not a YAML fan).

(as an aside, if there's the question of forking an existing parser
implementation for better vendorability, forking a YAML parser may be
more useful to third-party folks than forking a TOML parser :-))

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 10, 2016, at 10:30 AM, Antoine Pitrou  wrote:
> 
> On Tue, 10 May 2016 10:24:10 -0400
> Donald Stufft  wrote:
>> 
>> TOML is infinitely better at nested structured that ConfigParser, given that
>> TOML actually *supports* nested structures beyond a level of 1. The only way
>> to get anything like:
>> 
>>[package.build]
>>dependencies = ["setuptools", "wheel"]
>> 
>> In ConfigParser is to add post-processing to the values, which then you're no
>> longer a "ConfigParser" file, you're a "ConfigParser + Whatever random one 
>> off
>> code you wrote to do post processing" file.
> 
> The post-processing doesn't seem difficult enough to make any fuss
> about it, IMHO.  The most important concern here should be how usable
> the format is for end users, not whether implementations need 20
> additional lines of code to work with it.

I think TOML is more usable than ConfigParser and in particular I think that
the adhoc post processing step makes ConfigParser inherently less usable
because it forces a special syntax that is specific to this one file. It also
means that there's no "right" answer for when you have two different
implementations that interpret the same file differently. There's no spec to
work off of so it ends up regressing to a smaller version of one of our current
problems with the toolchain-- Everything is implementation defined by whatever
the most popular tool at the time is.


> 
> (also, what is wrong with providing a pypa-specific library for parsing
> required configuration? are distlib / distil / pkgutil / the
> distutils-competitor-du-jour still alive?)


While we are providing unopinionated bindings like that for the PEPS, we don't
want to make that "special" any more than it's just a reference implementation.
Tools in languages other than Python will be expected to parse and read these
files.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Antoine Pitrou]

On Tue, 10 May 2016 10:24:10 -0400
Donald Stufft  wrote:
> 
> TOML is infinitely better at nested structured that ConfigParser, given that
> TOML actually *supports* nested structures beyond a level of 1. The only way
> to get anything like:
> 
> [package.build]
> dependencies = ["setuptools", "wheel"]
> 
> In ConfigParser is to add post-processing to the values, which then you're no
> longer a "ConfigParser" file, you're a "ConfigParser + Whatever random one off
> code you wrote to do post processing" file.

The post-processing doesn't seem difficult enough to make any fuss
about it, IMHO.  The most important concern here should be how usable
the format is for end users, not whether implementations need 20
additional lines of code to work with it.

(also, what is wrong with providing a pypa-specific library for parsing
required configuration? are distlib / distil / pkgutil / the
distutils-competitor-du-jour still alive?)

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/10/2016 01:54 AM, Paul Moore wrote:


Writing our own is simply a way to end up with additional maintenance
work, that we really don't have the resources for.


I like writing tools.

If the format is one I can get behind I'm happy to be the resource for 
it.  This rules out JSON and YAML, but leaves TOML in the running (as 
in: I'm happy to take over pytoml if its current author is agreeable).


I'm also happy to create one: The Sane/Simple/Super Config Language (or 
.scl for short).  It would be very similar to TOML, possibly a superset.


Tools written so far:

- dbf [https://pypi.python.org/pypi/dbf]
  project I learned Python with (so some rough edges, but very
  serviceable)

- scription [https://pypi.python.org/pypi/scription]
  opinionated command-line parser

- antipathy [https://pypi.python.org/pypi/antipathy]
  file system path library

- aenum [https://pypi.python.org/pypi/aneum]
  totally awesome Enum library ;)  (scaled-down version is the
  stdlib Enum)

- xaml [https://pypi.python.org/pypi/xaml]
  xml processor similar to Ruby's haml

PEPs (co)authored so for have also been tool/library oriented:

- PEP 409 [https://www.python.org/dev/peps/pep-0409]
  raise from None (dbf inspired)

- PEP 435 [https://www.python.org/dev/peps/pep-0435]
  Enum

- PEP 461 [https://www.python.org/dev/peps/pep-0461]
  %-interpolation for bytes & bytearrays

In other words:  this is a serious offer.  ;)

--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Antoine Pitrou]

On Tue, 10 May 2016 10:38:51 +0300
Alex Grönholm  wrote:
> TOML isn't much better than ConfigParser in terms of representing nested 
> structures.

Indeed, that seems to be a strong point against TOML.  If we don't care
about nested structures that much, then ConfigParser should be more or
less ok...

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Barry Warsaw]

On May 09, 2016, at 08:30 PM, Donald Stufft wrote:

>How hard is it to bundle it with pip by copying the source files into
>pip._vendor.*

Every time another package is vendored, a kitten falls off a unicorn. ;)

Cheers,
-Barry


pgpetwbrYXBeb.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Leonardo Rochael Almeida]

In all this talk about using a YAML subset, I'm surprised no one mentioned
YAMLish:

https://pypi.python.org/pypi/yamlish

It is a well defined subset of YAML and there are implementations in other
programming languages.

The problem with the 200+-lines-single-file library above is that it
depends on PyYAML itself so, vendoring it will be challenging.

Anyway, I think TOML 0.4.0 is good enough for our needs.

On Tue, 10 May 2016 10:03 Nick Coghlan,  wrote:

> On 10 May 2016 at 18:54, Paul Moore  wrote:
> > Well, IMO, the state of things in terms of config file formats (and
> > not just in Python) is itself pretty dreadful - every time I write an
> > application, I am astounded that there are no good options for
> > something as basic as a configuration file format.
>
> This is pretty normal for software - no good options, but a plethora
> of "good enough" ones. Hence https://xkcd.com/927/ :)
>
> We just have a particularly exacting use case here, since we want:
>
> - a format that's attractive for folks just learning to program in 2016
> - a format that's attractive for folks that have been programming for 50+
> years
> - a format that's easy to parse even in Python 2.6
> - a format that's version control friendly
> - a format that's text editor friendly (syntax highlighting, etc)
>
> For me, the two leading contenders out of the current discussion have been:
>
> - the poyo subset of YAML 1.1
> - TOML 0.4.0, as implemented by pytoml
>
> The "subset of PyYAML" approach turns this into a documentation
> exercise (which subset?), and also runs into the problem that poyo
> doesn't handle serialisation yet, only *reading* YAML files.
>
> For TOML, the main questions being asked are around how widely
> supported it is, and it's actually in a pretty good state on that
> front. 0.4.0 (rather than TOML in general) is stable by definition,
> and there are quite a few implementations of that across different
> languages:
> https://github.com/toml-lang/toml/blob/master/versions/en/toml-v0.4.0.md#implementations
>
> The other big thing someone might want is a schema validator, and
> there I think it may be possible to just use jsonschema and validate
> the result of parsing the file, rather than validating the serialised
> form directly.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nick Coghlan]

On 10 May 2016 at 18:54, Paul Moore  wrote:
> Well, IMO, the state of things in terms of config file formats (and
> not just in Python) is itself pretty dreadful - every time I write an
> application, I am astounded that there are no good options for
> something as basic as a configuration file format.

This is pretty normal for software - no good options, but a plethora
of "good enough" ones. Hence https://xkcd.com/927/ :)

We just have a particularly exacting use case here, since we want:

- a format that's attractive for folks just learning to program in 2016
- a format that's attractive for folks that have been programming for 50+ years
- a format that's easy to parse even in Python 2.6
- a format that's version control friendly
- a format that's text editor friendly (syntax highlighting, etc)

For me, the two leading contenders out of the current discussion have been:

- the poyo subset of YAML 1.1
- TOML 0.4.0, as implemented by pytoml

The "subset of PyYAML" approach turns this into a documentation
exercise (which subset?), and also runs into the problem that poyo
doesn't handle serialisation yet, only *reading* YAML files.

For TOML, the main questions being asked are around how widely
supported it is, and it's actually in a pretty good state on that
front. 0.4.0 (rather than TOML in general) is stable by definition,
and there are quite a few implementations of that across different
languages: 
https://github.com/toml-lang/toml/blob/master/versions/en/toml-v0.4.0.md#implementations

The other big thing someone might want is a schema validator, and
there I think it may be possible to just use jsonschema and validate
the result of parsing the file, rather than validating the serialised
form directly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Randy Syring]

For what it's worth, I've been following this thread, and I like the 
idea of using TOML for all the "pro" reasons posted so far.  It's 
newness or not reaching 1.0 yet don't bother me as I believe the plans 
to specify TOML 0.4 or optionally support the later versions if they 
don't cause problems makes a lot of sense.  The fact that the parser is 
300+ lines of code and can be easily vendored is also a big plus.  Given 
that rust is using TOML, if Python adopts it as well, that is big enough 
"market share" for me and people will get used to it soon enough.


*Randy Syring*
Husband | Father | Redeemed Sinner

/"For what does it profit a man to gain the whole world
and forfeit his soul?" (Mark 8:36 ESV)/

On 05/10/2016 03:38 AM, Alex Grönholm wrote:

A few facts:

  * YAML is good enough for Salt, Ansible and numerous other common tools
  * The YAML standard has been stable for many years, unlike TOML
which still hasn't even reached 1.0
  * YAML has widespread tooling support, unlike TOML

We all agree that JSON is not the solution. No comments, trailing 
commas etc.
TOML isn't much better than ConfigParser in terms of representing 
nested structures.
So far the ONLY objective problems with YAML seems to be the 
problematic implementation named PyYAML. If this is really the case, 
I'd gladly help build a better one just to prevent TOML from being 
chosen for this task. That we're even /considering/ building something 
as important as this on an unstable standard is pretty horrifying to 
me in itself.


10.05.2016, 06:37, Chris Barker kirjoitti:

Really?

writing Yet Another Markup Language (YAML :-) ) CAN'T be the 
simplest, best option.


> After further consideration, and pytoml's author's comment about 
the spec changing without a version increase, I think we might be 
better off rolling our own.



I like the general simplicity, and would stick with that, but I'd
be a lot more comfortable if we had our spec that was more
consistent.


If we're going to do that, then why not the 'simple part of yaml'.

or Python literals. (if I recall, the main reason not to do that was 
that no other language has a lib to read it -- rolling out own does 
not solve that!)


Or just go with JSON -- I'm annoyed by it at times, but it's not SO bad.

(and you can kinda-sorta simulate comments with useless keys :-)

{ "comment": "this is just something i wanted to say here",
...
}

or we could do "JSON with comments" -- not hard to write a tiny 
pre-processor before passing it off to the json lib.


Anyway -- let's avoid the temptation to role your own everything, and 
use something standard!


-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov 


___
Distutils-SIG maillist  -Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig




___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

This looks very close to what I'd like to have, but then we'd have the 
situation of an uncommon format with no tooling support, won't we? 
Assuming the actual config file is in xaml format.


10.05.2016, 02:56, Ethan Furman kirjoitti:

On 05/06/2016 07:59 PM, Nathaniel Smith wrote:


Here's that one-stop writeup/comparison of all the major configuration
languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f


Very nice work-up, thanks!


However, you didn't include XML -- which, while absolutely horrid, can 
be quite readable with the appropriate preprocessor, such as xaml [1] 
:


--- 8< whatever.xaml ---
!!! xml1.0
~base

~schema
// optional
~version: 1

~bootstrap
~requirements
// Temporarily commented out 2016-01-10
// magic-build-helper
~setuptools
~version: >= 27
// for the new frobnicate feature
~numpy
~version: >= 1.10
//Pinned until we get a fix for
// @https://github.com/cyberdyne/the-versionator/issues/123
~the-versionator
~version: 0.13

// The owner of pypi name "flit" decides what goes under the
// extension: flit:
// key
~extensions
~flit
~whatever: true
--- 8< -

which ends up as:

--- 8< whatever.xml 





1






= 27



= 1.10



0.13









true


--- 8< -

--
~Ethan~

[1] https://pypi.python.org/pypi/xaml
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

10.05.2016, 12:43, Ionel Cristian Mărieș kirjoitti:


On Tue, May 10, 2016 at 10:38 AM, Alex Grönholm 
> wrote:


So far the ONLY objective problems with YAML seems to be the
problematic implementation named PyYAML. If this is really the
case, I'd gladly help build a better one just to prevent TOML from
being chosen for this task. That we're even /considering/ building
something as important as this on an unstable standard is pretty
horrifying to me in itself.


​Just my two cents here: every time, but every every time,​ I have to 
google around about how to create a multi-line string in YAML. There 
are too many ways to write the same thing. And lets not forget those 
damn sexagesimal literals. The complexity of that language is beyond 
repair, it's not a *library* problem. Just look at insanities like 
this 
 
​or this ​ .



I have no problem with any of the examples you linked to.

Thanks,
-- IonelCristian Mărieș, http://blog.ionelmc.ro


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ionel Cristian Mărieș]

On Tue, May 10, 2016 at 10:38 AM, Alex Grönholm 
wrote:

> So far the ONLY objective problems with YAML seems to be the problematic
> implementation named PyYAML. If this is really the case, I'd gladly help
> build a better one just to prevent TOML from being chosen for this task.
> That we're even *considering* building something as important as this on
> an unstable standard is pretty horrifying to me in itself.
>

​Just my two cents here: every time, but every every time,​ I have to
google around about how to create a multi-line string in YAML. There are
too many ways to write the same thing. And lets not forget those damn
sexagesimal literals. The complexity of that language is beyond repair,
it's not a *library* problem. Just look at insanities like this

​or this ​.

Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 10 May 2016 at 08:38, Alex Grönholm  wrote:
> A few facts:
>
> YAML is good enough for Salt, Ansible and numerous other common tools
> The YAML standard has been stable for many years, unlike TOML which still
> hasn't even reached 1.0
> YAML has widespread tooling support, unlike TOML
>
> We all agree that JSON is not the solution. No comments, trailing commas
> etc.
> TOML isn't much better than ConfigParser in terms of representing nested
> structures.

Just as another data point, Cookiecutter started off using pyYAML,
moved to ruamel.yaml, and ended up with their own (separated out into
an independent project, but written specifically for Cookicutter)
implementation of "some of" YAML, called poyo. I believe the reason
they did this was repeated issues with the existing YAML libraries.
And while the YAML-subset provided by poyo *looks* reasonable, I don't
see any documentation of precisely *what* subset of YAML is
implemented (apart from some examples). The poyo library is also very
new, and I doubt it's seen much usage/testing outside of cookiecutter
so far.

I would love to use YAML. I really would. But for pip, we need a
robust, easy to vendor Python implementation (with no C dependency)
that is safe (see
http://community.embarcadero.com/blogs/entry/yaml-and-remote-code-execution-38738).
Writing our own is simply a way to end up with additional maintenance
work, that we really don't have the resources for.

> So far the ONLY objective problems with YAML seems to be the problematic
> implementation named PyYAML. If this is really the case, I'd gladly help
> build a better one just to prevent TOML from being chosen for this task.

If you can get a robust, stable YAML library written fast enough to be
an option for the PEP, then that would certainly be a possibility. But
given that poyo has been under development for 5 months, are you going
to be able to do better than that in a few weeks? (And see my comments
above on poyo).

> That we're even considering building something as important as this on an
> unstable standard is pretty horrifying to me in itself.

Well, IMO, the state of things in terms of config file formats (and
not just in Python) is itself pretty dreadful - every time I write an
application, I am astounded that there are no good options for
something as basic as a configuration file format.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

A few facts:

 * YAML is good enough for Salt, Ansible and numerous other common tools
 * The YAML standard has been stable for many years, unlike TOML which
   still hasn't even reached 1.0
 * YAML has widespread tooling support, unlike TOML

We all agree that JSON is not the solution. No comments, trailing commas 
etc.
TOML isn't much better than ConfigParser in terms of representing nested 
structures.
So far the ONLY objective problems with YAML seems to be the problematic 
implementation named PyYAML. If this is really the case, I'd gladly help 
build a better one just to prevent TOML from being chosen for this task. 
That we're even /considering/ building something as important as this on 
an unstable standard is pretty horrifying to me in itself.


10.05.2016, 06:37, Chris Barker kirjoitti:

Really?

writing Yet Another Markup Language (YAML :-) ) CAN'T be the simplest, 
best option.


> After further consideration, and pytoml's author's comment about the 
spec changing without a version increase, I think we might be better 
off rolling our own.



I like the general simplicity, and would stick with that, but I'd
be a lot more comfortable if we had our spec that was more consistent.


If we're going to do that, then why not the 'simple part of yaml'.

or Python literals. (if I recall, the main reason not to do that was 
that no other language has a lib to read it -- rolling out own does 
not solve that!)


Or just go with JSON -- I'm annoyed by it at times, but it's not SO bad.

(and you can kinda-sorta simulate comments with useless keys :-)

{ "comment": "this is just something i wanted to say here",
...
}

or we could do "JSON with comments" -- not hard to write a tiny 
pre-processor before passing it off to the json lib.


Anyway -- let's avoid the temptation to role your own everything, and 
use something standard!


-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov 


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/09/2016 08:35 PM, Nathaniel Smith wrote:

On Mon, May 9, 2016 at 7:22 PM, Ethan Furman wrote:



After further consideration, and pytoml's author's comment about the spec
changing without a version increase, I think we might be better off
rolling our own.


He's a bit confused -- they didn't change 0.4.0; they made changes in
their dev branch working towards 1.0.0 (some cleanups related to the
date/time stuff I think?"). But of course when you go to github it shows
you the current dev version, and the dev version has a prominent link at
the top to the 0.4.0 tag, so if you're skimming it's easy to misread it
as saying that what you're looking at is 0.4.0.


Ah, okay.  Thanks for clarifying!

--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

Really?

writing Yet Another Markup Language (YAML :-) ) CAN'T be the simplest, best
option.

> After further consideration, and pytoml's author's comment about the spec
changing without a version increase, I think we might be better off rolling
our own.

>
> I like the general simplicity, and would stick with that, but I'd be a lot
> more comfortable if we had our spec that was more consistent.
>

If we're going to do that, then why not the 'simple part of yaml'.

or Python literals. (if I recall, the main reason not to do that was that
no other language has a lib to read it -- rolling out own does not solve
that!)

Or just go with JSON -- I'm annoyed by it at times, but it's not SO bad.

(and you can kinda-sorta simulate comments with useless keys :-)

{ "comment": "this is just something i wanted to say here",
...
}

or we could do "JSON with comments" -- not hard to write a tiny
pre-processor before passing it off to the json lib.

Anyway -- let's avoid the temptation to role your own everything, and use
something standard!

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nathaniel Smith]

On Mon, May 9, 2016 at 7:22 PM, Ethan Furman  wrote:
> On 05/09/2016 05:19 PM, Ethan Furman wrote:
>>
>> On 05/07/2016 09:32 AM, Brett Cannon wrote:
>
>
>>> I also checked pytoml at https://github.com/avakar/pytoml and it looks
>>> like it's pretty stable; no changes in the past 5 months except to
>>> support Python 3.5 and only 3 issues. And the format is simple enough
>>> that if someone had to fork the code like Nathaniel suggested or we did
>>> it from scratch it wouldn't be a huge burden.
>
>
> After further consideration, and pytoml's author's comment about the spec
> changing without a version increase, I think we might be better off
rolling
> our own.

He's a bit confused -- they didn't change 0.4.0; they made changes in their
dev branch working towards 1.0.0 (some cleanups related to the date/time
stuff I think?"). But of course when you go to github it shows you the
current dev version, and the dev version has a prominent link at the top to
the 0.4.0 tag, so if you're skimming it's easy to misread it as saying that
what you're looking at is 0.4.0.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Łukasz Langa]

Next thing you know we end up with a new setup.py, with imports, PYTHONPATH 
hacking et al ;-)

> On May 9, 2016, at 7:34 PM, Ethan Furman  wrote:
> 
> I just found this on StackOverflow:
> 
>  http://stackoverflow.com/a/648487/208880
> 
> tl;dr
> -
> 
> > Recently I was working upon a project and I realised that I wanted to
> > have conditionals inside my configuration file [...]
> >
> > I didn't want to write a mini-language, because unless I did it very
> > carefully I couldn't allow the flexibility that would be useful.
> >
> > Instead I decided that I'd have two forms:  If the file started with
> > "#!" and was executable I'd parse the result of running it; otherwise
> > I'd read it as-is
> 
> That approach seems like a win-win: the plain-vanilla static file can be 
> promoted as best-practice, yet we have a fall-back for the complicated and 
> edge cases.
> 
> --
> ~Ethan~
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

I just found this on StackOverflow:

  http://stackoverflow.com/a/648487/208880

tl;dr
-

> Recently I was working upon a project and I realised that I wanted to
> have conditionals inside my configuration file [...]
>
> I didn't want to write a mini-language, because unless I did it very
> carefully I couldn't allow the flexibility that would be useful.
>
> Instead I decided that I'd have two forms:  If the file started with
> "#!" and was executable I'd parse the result of running it; otherwise
> I'd read it as-is

That approach seems like a win-win: the plain-vanilla static file can be 
promoted as best-practice, yet we have a fall-back for the complicated 
and edge cases.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/09/2016 05:19 PM, Ethan Furman wrote:

On 05/07/2016 09:32 AM, Brett Cannon wrote:



I also checked pytoml at https://github.com/avakar/pytoml and it looks
like it's pretty stable; no changes in the past 5 months except to
support Python 3.5 and only 3 issues. And the format is simple enough
that if someone had to fork the code  like Nathaniel suggested or we did
it from scratch it wouldn't be a huge burden.


After further consideration, and pytoml's author's comment about the 
spec changing without a version increase, I think we might be better off 
rolling our own.


I like the general simplicity, and would stick with that, but I'd be a 
lot more comfortable if we had our spec that was more consistent.


If there is interest I'll write the PEP and the tool.

--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 9, 2016, at 8:21 PM, Ethan Furman  wrote:
> 
> Um, what does "vendorability" mean?

How hard is it to bundle it with pip by copying the source files into 
pip._vendor.*

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/09/2016 12:30 PM, Barry Warsaw wrote:


Also, I think it makes a lot of sense to go with YAML even if it isn't the
best most readable option.  It's much more common than TOML so the learning
curve will be lessened.


I'd rather learn some new syntax that is readable than be stuck with a 
pain in my eyes and in my brain. ;)


--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/07/2016 04:11 PM, Robert Collins wrote:


Actually, Nathaniel didn't test vendorability of the libraries, and pip
needs that. Pyyaml isn't in good shape there.


Um, what does "vendorability" mean?

--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/07/2016 09:32 AM, Brett Cannon wrote:


+1 for TOML from me as well. I know Paul brought up the lack of
familiarity, but the format is simple and the Rust community is already
fully dependent on it so at worst Rust + us could always just ignore
future format versions if necessary.

If TOML is the chosen format we could ask how long until a 1.0 release
to know if we waited a month or so to implement we could make sure we're
compliant with that version.

I also checked pytoml at https://github.com/avakar/pytoml and it looks
like it's pretty stable; no changes in the past 5 months except to
support Python 3.5 and only 3 issues. And the format is simple enough
that if someone had to fork the code  like Nathaniel suggested or we did
it from scratch it wouldn't be a huge burden.


I would prefer TOML over anything else mentioned so far, and taking over 
pytoml would probably be beneficial, given the author's comments.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Ethan Furman]

On 05/06/2016 07:59 PM, Nathaniel Smith wrote:


Here's that one-stop writeup/comparison of all the major configuration
languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f


Very nice work-up, thanks!


However, you didn't include XML -- which, while absolutely horrid, can 
be quite readable with the appropriate preprocessor, such as xaml [1] 
:


--- 8< whatever.xaml ---
!!! xml1.0
~base

~schema
// optional
~version: 1

~bootstrap
~requirements
// Temporarily commented out 2016-01-10
// magic-build-helper
~setuptools
~version: >= 27
// for the new frobnicate feature
~numpy
~version: >= 1.10
//Pinned until we get a fix for
// @https://github.com/cyberdyne/the-versionator/issues/123
~the-versionator
~version: 0.13

// The owner of pypi name "flit" decides what goes under the
// extension: flit:
// key
~extensions
~flit
~whatever: true
--- 8< -

which ends up as:

--- 8< whatever.xml 





1






= 27



= 1.10



0.13









true


--- 8< -

--
~Ethan~

[1] https://pypi.python.org/pypi/xaml
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Barry Warsaw]

On May 08, 2016, at 09:05 AM, Robert Collins wrote:

>E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
>as pybuild is a thing in the debian space, and this will confuse the
>heck out of folk). https://wiki.debian.org/Python/Pybuild

Yes, please don't call it pybuild ;)

Also, I think it makes a lot of sense to go with YAML even if it isn't the
best most readable option.  It's much more common than TOML so the learning
curve will be lessened.

Cheers,
-Barry


pgpcWJAr9wuKi.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Brett Cannon]

On Mon, 9 May 2016 at 11:21 Chris Barker  wrote:

> "pymeta" feels very "inessentially weird" to me [1].
>
>
> yeah...
>
> setup.toml
>
> ???
>

You can all stop guessing at file names. The PEP will have a recommendation
and you all can either agree or disagree at that point. Please don't give
me more names to list in the rejected section. :)

-Brett


>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

>
> "pymeta" feels very "inessentially weird" to me [1].


yeah...

setup.toml

???

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Brett Cannon]

On Mon, 9 May 2016 at 06:29 Nick Coghlan  wrote:

> On 9 May 2016 at 01:43, Donald Stufft  wrote:
> > Overall, my suggestion here would be to have a file called
> ``pymeta.toml`` (or
> > ``meta.toml``)
>
> pymeta.toml would be fine by me.
>
> I don't really buy the "collision with Debian build tool" argument
> against "pybuild" (if I did, I'd be objecting to "pymeta" colliding
> with an existing PyPI package), so it's mainly the fact the metadata
> in this file covers more than just building has soured me on it.
>
> > and have it look like::
> >
> > [dependencies]
> > build = [
> > "setuptools",
> > "wheel>=1.0",
> > ]
> >
> > If at some point we decide we need to add a bootstrap_requires (and then
> the
> > ability to add dynamic build requirements) we can do that by just saying
> that
> > if you plan on having dynamic build requirements, you need to omit the
> build
> > key under the [dependencies] section.
>
> Looking at my previous ideas for semantic dependencies in PEP 426,
> what if we start in the near term by defining development
> requirements?
>
> That can then be used to hold arbitrary development dependencies
> (metadata generation, sdist creation, test execution, wheel build,
> docs build, etc), everything that you need to work on, build, and test
> the software, but don't need if you just want to run the published
> releases.
>
> We may later decide that we want to break that down and separate out
> specific requirements for sdist creation and for wheel creation, but
> we can handle that by saying that if there's no more specific
> dependency definition for an operation, then the tools will fall back
> to pre-installing all the listed development dependencies.
>
> That is, someone might write:
>
>  [dependencies]
>  develop = [
>  "setuptools",
>  "wheel>=1.0",
>  "sphinx",
>  "pytest",
>]
>
> And it would be useful not only for running setup.py commands, but
> setting up their local dev environment in general.
>

I'm not going to touch the concept of development dependencies to keep the
PEP focused.

But one thing this discussion does bring up is everyone is assuming there
is simply going to be a [dependencies] section to the configuration file
where build, development, and/or installation dependencies will eventually
end up.

My current plan for the PEP is to not do that. Instead, I'm going to have a
[build] section that will have a `dependencies` field. My thinking is that
we don't need to have any end-users who choose to look at this file care
about anything related to building or developing a project. Plus if we
expose any details about the Future Awesome Build API (aka the FAB API ;)
then it should probably be kept near all of the other build-related
details, including the dependencies for building instead of inherently
spanning multiple sections (yes, tools will quite possibly have their own
section, but we can try to keep it to [build] and e.g. [tools.setuptools]
and not [dependencies], [build], and [tools.setuptools]).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nathaniel Smith]

On Mon, May 9, 2016 at 6:28 AM, Nick Coghlan  wrote:
> On 9 May 2016 at 01:43, Donald Stufft  wrote:
>> Overall, my suggestion here would be to have a file called ``pymeta.toml`` 
>> (or
>> ``meta.toml``)
>
> pymeta.toml would be fine by me.
>
> I don't really buy the "collision with Debian build tool" argument
> against "pybuild" (if I did, I'd be objecting to "pymeta" colliding
> with an existing PyPI package), so it's mainly the fact the metadata
> in this file covers more than just building has soured me on it.

Re: filename bikeshedding:

"pymeta" feels very "inessentially weird" to me [1]. This file is
going to front and center for newcomers, many of whom will never have
encountered the word "metadata" and especially not the hacker fetish
for the "meta" morpheme. I like meta-things in general! But I don't
like the image of trying to explain what a "pymeta" is over and over
and over again when teaching :-).

pymetadata would be better, but it seems like there must be something
less jargony available?

pypackage
pypackaging
pydevelop
pysource
pytools
pysettings
...?

Or if we're really daring and wasteful of characters, I guess we could
even go for something like python-tools.toml :-)

-n

(Tangent, but I'll write it down so I don't forget when we circle back
to the question of adding config for third-party tools into this
thing: [tool.flit], [tool.coverage] is probably a lot more obvious to
newcomers than [extension.flit], [extension.coverage]!)


[1] https://www.crummy.com/sumana/2014/08/10/1

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nathaniel Smith]

On Mon, May 9, 2016 at 6:52 AM, Nick Coghlan  wrote:
> On 9 May 2016 at 23:38, Donald Stufft  wrote:
>>
>>> On May 9, 2016, at 9:28 AM, Nick Coghlan  wrote:
>>>
>>> Looking at my previous ideas for semantic dependencies in PEP 426,
>>> what if we start in the near term by defining development
>>> requirements?
>>
>> I think the biggest reason not to do this, but instead do something like 
>> build
>> requirements is that development dependencies is already reasonably well
>> addressed in a way that something other than setuptools can access it using
>> extras. It's not as great as a dedicated key for it, but it works pretty OK.
>> The thing that is really painful is setup_requires and how it forces you to
>> delay importing until *during* the execution of the setup() function. We 
>> could
>> try and lump setup_requires and development dependencies together, but that
>> seems less than optimal to me. Unless someone's setup.py uses pytest, I'm not
>> sure I see a reason for pytest to be installed anytime pip builds that 
>> project.
>>
>> A more concrete example would be pyca/cryptography, which has a development
>> dependency that consists of 27MB of data which was purposely kept separate 
>> from
>> cryptography itself so as not to incur an additional 27MB of download just to
>> install cryptography.
>
> OK, that makes sense to me as a rationale for prioritising build
> dependencies over more general dev environment dependencies.
>
> To feel confident we haven't overlooked a near term need by doing
> that, I think the main question I'd like to see the PEP explicitly
> cover is "when will these new requirements be implicitly installed?"
>
> I currently believe the answers to that are:
>
> 1. When pip* is asked to install from an sdist
> 2. When pip* is asked to install from a VCS URL
> 3. When pip* is asked to install from a local directory
> 4. When pip* is asked to create a wheel file for any of those
>
> * - or another compatible installer
>
> If those answers are correct, then the metadata consumer will always
> be proceeding on to do a build in the use cases we currently care
> about, so it's OK that it may also be relying on those build
> requirements to generate metadata or create an sdist archive.

I think what we want is precisely, "what stuff do you need before you
can invoke setup.py", and in the future that will be replaced with
"what stuff do you need before you can invoke [our awesome
hypothetical build system abstraction interface TBD, which will
default to a wrapper around setup.py]".

We all agree that there will be some sort of "build system" concept,
and that invoking a build system is something that requires running
arbitrary code, some of which might come from PyPI.

It remains to be determined exactly what the boundary of that "build
system" concept is (e.g., it definitely will have 'build a wheel'; not
as obvious whether it will have 'build an sdist' or 'run tests').

But I think for now that's fine -- for now we can just say that
eventually we will fill in the details of this interface, and the
point of this part of the config file is that it lets you bootstrap
into whatever that ends up being.

(Notice that both the PEP 516/517 proposals actually distinguish
between bootstrap-requires, which are what you need before invoking
the build system, and build-requires, which is a dynamic hook for
querying the build system to ask it what else it needs before it can
build a wheel. Example of where the wheel-build requirements might
vary at runtime: when building an extension that needs a C library,
your build system might provide envvar that lets you pick between 'I'm
building an RPM, so please just link to the system-provided copy of
this library' versus 'I'm building a wheel, so please install the
relevant pynativelib package -- making it a build requirement -- and
then link to that'. I think we should focus narrowly on the
bootstrapping problem right now, because it's not yet clear whether
things like test dependencies actually make more sense as static
metadata or as something you query the build system for dynamically.
And a single static list of build-system-requirements is already
enough to bandage over most of the really egregious setup_requires
problems.)

So I'd suggest:

[build-system]
requires = [ ... ]
# Later we will add something like:
# entry-point = ...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nick Coghlan]

On 9 May 2016 at 23:38, Donald Stufft  wrote:
>
>> On May 9, 2016, at 9:28 AM, Nick Coghlan  wrote:
>>
>> Looking at my previous ideas for semantic dependencies in PEP 426,
>> what if we start in the near term by defining development
>> requirements?
>
> I think the biggest reason not to do this, but instead do something like build
> requirements is that development dependencies is already reasonably well
> addressed in a way that something other than setuptools can access it using
> extras. It's not as great as a dedicated key for it, but it works pretty OK.
> The thing that is really painful is setup_requires and how it forces you to
> delay importing until *during* the execution of the setup() function. We could
> try and lump setup_requires and development dependencies together, but that
> seems less than optimal to me. Unless someone's setup.py uses pytest, I'm not
> sure I see a reason for pytest to be installed anytime pip builds that 
> project.
>
> A more concrete example would be pyca/cryptography, which has a development
> dependency that consists of 27MB of data which was purposely kept separate 
> from
> cryptography itself so as not to incur an additional 27MB of download just to
> install cryptography.

OK, that makes sense to me as a rationale for prioritising build
dependencies over more general dev environment dependencies.

To feel confident we haven't overlooked a near term need by doing
that, I think the main question I'd like to see the PEP explicitly
cover is "when will these new requirements be implicitly installed?"

I currently believe the answers to that are:

1. When pip* is asked to install from an sdist
2. When pip* is asked to install from a VCS URL
3. When pip* is asked to install from a local directory
4. When pip* is asked to create a wheel file for any of those

* - or another compatible installer

If those answers are correct, then the metadata consumer will always
be proceeding on to do a build in the use cases we currently care
about, so it's OK that it may also be relying on those build
requirements to generate metadata or create an sdist archive.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 9, 2016, at 9:28 AM, Nick Coghlan  wrote:
> 
> Looking at my previous ideas for semantic dependencies in PEP 426,
> what if we start in the near term by defining development
> requirements?

I think the biggest reason not to do this, but instead do something like build
requirements is that development dependencies is already reasonably well
addressed in a way that something other than setuptools can access it using
extras. It's not as great as a dedicated key for it, but it works pretty OK.
The thing that is really painful is setup_requires and how it forces you to
delay importing until *during* the execution of the setup() function. We could
try and lump setup_requires and development dependencies together, but that
seems less than optimal to me. Unless someone's setup.py uses pytest, I'm not
sure I see a reason for pytest to be installed anytime pip builds that project.

A more concrete example would be pyca/cryptography, which has a development
dependency that consists of 27MB of data which was purposely kept separate from
cryptography itself so as not to incur an additional 27MB of download just to
install cryptography.

I think this *could* make sense, if we could reasonably assume that the 95%
case would always be handled by wheels, but I don't think that we can. A lot of
projects have compiled C code as part of them, and as soon as you do that you
end up where installation from sdist is one of your main supported methods (yes
Wheel will cover a lot of them, but only for the most popular platforms).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nick Coghlan]

On 9 May 2016 at 01:43, Donald Stufft  wrote:
> Overall, my suggestion here would be to have a file called ``pymeta.toml`` (or
> ``meta.toml``)

pymeta.toml would be fine by me.

I don't really buy the "collision with Debian build tool" argument
against "pybuild" (if I did, I'd be objecting to "pymeta" colliding
with an existing PyPI package), so it's mainly the fact the metadata
in this file covers more than just building has soured me on it.

> and have it look like::
>
> [dependencies]
> build = [
> "setuptools",
> "wheel>=1.0",
> ]
>
> If at some point we decide we need to add a bootstrap_requires (and then the
> ability to add dynamic build requirements) we can do that by just saying that
> if you plan on having dynamic build requirements, you need to omit the build
> key under the [dependencies] section.

Looking at my previous ideas for semantic dependencies in PEP 426,
what if we start in the near term by defining development
requirements?

That can then be used to hold arbitrary development dependencies
(metadata generation, sdist creation, test execution, wheel build,
docs build, etc), everything that you need to work on, build, and test
the software, but don't need if you just want to run the published
releases.

We may later decide that we want to break that down and separate out
specific requirements for sdist creation and for wheel creation, but
we can handle that by saying that if there's no more specific
dependency definition for an operation, then the tools will fall back
to pre-installing all the listed development dependencies.

That is, someone might write:

 [dependencies]
 develop = [
 "setuptools",
 "wheel>=1.0",
 "sphinx",
 "pytest",
   ]

And it would be useful not only for running setup.py commands, but
setting up their local dev environment in general.

(Having docs build and test dependencies listed would be entirely
acceptable for the RPM spec autogeneration case, since we just have
the BuildRequires/Requires split at the RPM level, and often need the
docs dependency to generate man pages, and the test dependencies to
run the unit tests in the %check scriptlet)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Brett Cannon]

Based on this email and Nathaniel's evaluation I've gone ahead and taken it
upon myself to start writing a PEP so we have something concrete to work
from. I'm hoping to have it done some time this week.

On Sun, 8 May 2016 at 08:43 Donald Stufft  wrote:

>
> > On May 8, 2016, at 9:13 AM, Nick Coghlan  wrote:
> >
> > So let's reduce our scope to: "We want *current users* of d2to1 and
> > pbr to be able to declare those dependencies to pip and other
> > installation tools in a way that avoids the implicit invocation of
> > easy_install by setuptools"
>
> I think it's less pbr and d2to1 that we're designing for here (although
> they'll
> benefit from it) but more like numpy.distutils. The reasoning is that pbr
> and
> such have already been able to architecture themselves so that they work
> within
> the current system. One of the goals here is to enable projects that
> haven't
> been able to (or can't reasonably) be written to work as a setuptools
> extension
> to be used here without requiring a manual pre-installation step.
>
> The other side of this is anytime you have dependencies that aren't
> installed
> by pip (such as setup_requires being installed by setuptools) you end up
> with
> a confusing situation where settings don't get passed down into the "inner"
> installer (like ``--index-url``) or when they have different resolution
> algorithms (like setuptools supporting pre-releases by default) or when
> they
> support different formats (like pip not supporting eggs, but setuptools not
> supporting wheels).
>
> However, while I think that a new format is more work just to make sure we
> don't back ourselves into a corner, I also don't think it's so much more
> work
> that it's really worth it to skip doing it now. Their isn't a whole lot of
> ways
> to solve the very narrow problem of "we need some dependencies installed
> prior
> to building". We need a field in a file that takes a list of PEP 508 style
> specifiers. We can skip any of the build system support right now and add
> it in
> later (and when we add it in, we can just make it so that a lack of a
> declaration is an implicit setuptool). So the only real things we need to
> decide on are:
>
> 1) What format to use? For this I think that it's been pretty clearly that
> TOML
>has had the greatest amount of support here and I think it represents
> the
>best set of trade offs for us.
>
> 2) What do we want to call this file? This one is pure bikeshed but also a
> bit
>important since this is one of the things that will be the hardest to
> change
>once we actually pick the name. Ideas I've seen so far (using the toml
>extension) are:
>
>* pypa.toml - This one is specific to us which is good, but it's also a
> bit
>  bad in that I don't think it's very widely known what the PyPA even
> is and
>  I think that the PyPA works better if it's just sort of the thing in
> the
>  background.
>
>* pybuild.toml - This one might be a bit too oriented towards building
>  rather than all of the possible uses of this file, but the bigger
> problem
>  with it I think is that hte name clashes with pybuild on Debian which
> is
>  their tool used for building Python packages.
>
>* pip.toml - Also specific to us which is good, but I think it's a bit
> too
>  overly specific maybe? One of our goals has been to make this stuff
> not
>  dependent on a specific implementation so that we can replace it if we
>  need to (as we more or less replaced distutils with setuptools, or
>  easy_install with pip) however this also isn't exactly depending on an
>  implementation-- just the name of a particular implementation. It does
>  have the benefit that for a lot of people they associate "pip" with
>  everything packaging related (e.g. I see people calling them
>  "pip packages" now).
>
>* pymeta.toml - This one might be reasonable, it's a bit generic but at
>  least it has the ``py`` prefix to tie it to us. Not much to say about
> it
>  otherwise.
>
>Overall from those names, I think I probably like pymeta.toml the best
> (or
>maybe just ``meta.toml`` if we don't like the py prefix) but maybe other
>people have ideas/opinions that I haven't seen yet.
>
> 3) How do we structure the file and what keys do we use for just this part
> of
>the overall feature we're hoping to eventually get (and what semantics
> do
>we give these keys). We could just call it ``setup_requires``, but I
> think
>that's a bad name for something we're not planning on getting rid of
> since
>one of the eventual goals is to make ``setup.py`` itself optional.
> Another
>option is ``build_requires`` but that isn't quite right because we don't
>*just* need these dependencies for the build step (``setup.py
> bdist_wheel``)
>but also for the metadata step (``setup.py egg_info``). Although
> perhaps it
>doesn't really matter. 

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 8 May 2016 at 22:38, Nicholas Chammas  wrote:
> Perhaps a more limited YAML library like Poyo would address some of the
> concerns we have about using PyYAML.

I'm -1 on using "a subset of" a standard format. It immediately
invites debate and confusion over *precisely* which subset is
supported.

I like (basic) YAML, but IMO it made the mistake of allowing itself to
get too complex.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 8, 2016, at 5:38 PM, Nicholas Chammas  
> wrote:
> 
> Poyo in particular may be too limited, but I wonder how much it would sway 
> people if we used YAML but not PyYAML.


If we found a reasonable library for parsing YAML it might be a more reasonable 
alternative. Though I think we really need something that supports most if not 
all of YAML otherwise I think it starts to get confusing because you have to 
figure out what the specific subset of YAML that the library that whatever 
packaging tool you’re using supports.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nicholas Chammas]

On Sat, May 7, 2016 at 6:23 AM Paul Moore p.f.mo...@gmail.com
 wrote:

- YAML ought to be wonderful, but it ended up over-engineered (yes, we
> can ignore the bits we don't care about). Also, pyYAML is a bit of an
> annoying dependency (big, reportedly slow unless you use the C
> version) - not something I'd want pip to have to vendor.
>
Perhaps a more limited YAML library like Poyo would address some of the
concerns we have about using PyYAML.

https://github.com/hackebrot/poyo#readme

Poyo in particular may be *too* limited, but I wonder how much it would
sway people if we used YAML but not PyYAML.

Anyway, here are some relevant snippets from Poyo’s README:

Please note that Poyo supports only a chosen subset of the YAML format.
It can only read but not write and is not compatible with JSON.
…
Poyo is 100% Python and does not require any additional libs.

Nick
​
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 8, 2016, at 9:13 AM, Nick Coghlan  wrote:
> 
> So let's reduce our scope to: "We want *current users* of d2to1 and
> pbr to be able to declare those dependencies to pip and other
> installation tools in a way that avoids the implicit invocation of
> easy_install by setuptools"

I think it's less pbr and d2to1 that we're designing for here (although they'll
benefit from it) but more like numpy.distutils. The reasoning is that pbr and
such have already been able to architecture themselves so that they work within
the current system. One of the goals here is to enable projects that haven't
been able to (or can't reasonably) be written to work as a setuptools extension
to be used here without requiring a manual pre-installation step.

The other side of this is anytime you have dependencies that aren't installed
by pip (such as setup_requires being installed by setuptools) you end up with
a confusing situation where settings don't get passed down into the "inner"
installer (like ``--index-url``) or when they have different resolution
algorithms (like setuptools supporting pre-releases by default) or when they
support different formats (like pip not supporting eggs, but setuptools not
supporting wheels).

However, while I think that a new format is more work just to make sure we
don't back ourselves into a corner, I also don't think it's so much more work
that it's really worth it to skip doing it now. Their isn't a whole lot of ways
to solve the very narrow problem of "we need some dependencies installed prior
to building". We need a field in a file that takes a list of PEP 508 style
specifiers. We can skip any of the build system support right now and add it in
later (and when we add it in, we can just make it so that a lack of a
declaration is an implicit setuptool). So the only real things we need to
decide on are:

1) What format to use? For this I think that it's been pretty clearly that TOML
   has had the greatest amount of support here and I think it represents the
   best set of trade offs for us.

2) What do we want to call this file? This one is pure bikeshed but also a bit
   important since this is one of the things that will be the hardest to change
   once we actually pick the name. Ideas I've seen so far (using the toml
   extension) are:

   * pypa.toml - This one is specific to us which is good, but it's also a bit
 bad in that I don't think it's very widely known what the PyPA even is and
 I think that the PyPA works better if it's just sort of the thing in the
 background.

   * pybuild.toml - This one might be a bit too oriented towards building
 rather than all of the possible uses of this file, but the bigger problem
 with it I think is that hte name clashes with pybuild on Debian which is
 their tool used for building Python packages.

   * pip.toml - Also specific to us which is good, but I think it's a bit too
 overly specific maybe? One of our goals has been to make this stuff not
 dependent on a specific implementation so that we can replace it if we
 need to (as we more or less replaced distutils with setuptools, or
 easy_install with pip) however this also isn't exactly depending on an
 implementation-- just the name of a particular implementation. It does
 have the benefit that for a lot of people they associate "pip" with
 everything packaging related (e.g. I see people calling them
 "pip packages" now).

   * pymeta.toml - This one might be reasonable, it's a bit generic but at
 least it has the ``py`` prefix to tie it to us. Not much to say about it
 otherwise.

   Overall from those names, I think I probably like pymeta.toml the best (or
   maybe just ``meta.toml`` if we don't like the py prefix) but maybe other
   people have ideas/opinions that I haven't seen yet.

3) How do we structure the file and what keys do we use for just this part of
   the overall feature we're hoping to eventually get (and what semantics do
   we give these keys). We could just call it ``setup_requires``, but I think
   that's a bad name for something we're not planning on getting rid of since
   one of the eventual goals is to make ``setup.py`` itself optional. Another
   option is ``build_requires`` but that isn't quite right because we don't
   *just* need these dependencies for the build step (``setup.py bdist_wheel``)
   but also for the metadata step (``setup.py egg_info``). Although perhaps it
   doesn't really matter. Another option is to call them ``bootstrap_requires``
   though it's a bit wonky to cram *build* requirements into that.

   I guess if I were deciding this I would just call it build requirements
   because the nature of ``setup.py`` is that pretty much anything you need to
   execute ``setup.py`` is going to be a requirement for all ways you invoke
   ``setup.py`` (metadata and build and the legacyish direct to install). We
   could then say that if a project is missing this new 

Re: [Distutils] comparison of configuration languages

[Nick Coghlan]

On 8 May 2016 at 08:46, Donald Stufft  wrote:
>
>> On May 7, 2016, at 5:05 PM, Robert Collins  wrote:
>>
>> Either we are defining the long term thing now, in which case that
>> huge pile of complexity lands on us, and we have to get everything
>> right.
>>
>> Or we are defining a thing which solves the present bug, and as long
>> as we make sure it does not bind us in future, we're not hamstrung.
>>
>> E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
>> as pybuild is a thing in the debian space, and this will confuse the
>> heck out of folk). https://wiki.debian.org/Python/Pybuild
>
> I think this is roughly true, we could either do the simplest thing and just
> add ``setup_requires`` to ``setup.cfg`` and teach pip how to understand them
> and then worry about a new format later, or we can do a new format now and add
> a bit of complexity to what we need to specify (though I don't think _too_ 
> much
> complexity, we don't have to define the build system stuff now, just make sure
> we don't back ourselves into a corner with that).
>
> I think either answer is OK, just the second one is a bit more work and we
> might either get the start of a better format _now_ or end up regretting what
> we pick when we add more things to it.

Hmm, are we perhaps forcing a false choice on ourselves here?

1. setup_requires in setup.py is the de facto baseline

2. d2to1 and pbr already use setup.cfg, so any arguments about the
readability of the format are moot for users of those projects

3. we're not sure we're comfortable mandating the use of an ini-style
format for all *future* Python projects

So let's reduce our scope to: "We want *current users* of d2to1 and
pbr to be able to declare those dependencies to pip and other
installation tools in a way that avoids the implicit invocation of
easy_install by setuptools"

For *that* design goal, the conclusions I'd draw from Nathaniel's
write-up would be those Łukasz suggested:

- use https://pypi.python.org/pypi/configparser in Python 2.7
- explicitly define how to handle end-of-line comments

We should perhaps also recommend that "#" be used for comments (even
if the parser didn't enforce that), as I haven't seen anyone using ';'
for comments since I stopped writing assembly code by hand.

Improving the status quo for d2to1 and pbr users doesn't lock us into
anything in terms of the "future of Python packaging as a whole".
Individual build system authors may *choose* to take advantage of the
setup.cfg mechanism, but they wouldn't be obliged to - they could keep
using setup.py based bootstrapping if they preferred to do so.

This approach would also provide the ability to iterate on a TOML
based dependency declaration system *independent of pip itself*:
setup.cfg could be used to bootstrap that system while it was still
experimental, letting the idiosyncrasies get worked out before we
commit to anything in pip itself.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Wes Turner]

On Sat, May 7, 2016 at 4:05 PM, Robert Collins 
wrote:

> Couple thoughts.
>
> Firstly, the human-editable bit: who in the last *decade* has been
> writing code using a non-syntax-aware/helping editor? Its a supremely
> uninteresting aspect IMO.
>
> On ConfigParser - yes, its horrid. OTOH we do get all the lines
> reliably, and setuptools will need to cover the unicode aspect itself
> in its own time. All we need to do is permit # inline as a comment -
> line a requirements.txt file for pip - and it becomes trivial to parse
> in all cases that we need *now*.
>

ConfigObj is read/write with full Unicode support (and type validation)


| Src: https://github.com/DiffSK/configobj
| PyPI: https://pypi.python.org/pypi/configobj/5.0.6
| Docs: https://configobj.readthedocs.io/en/latest/

*Having read/write support avoids the primary issue (injection) with
templating config files*

>From https://github.com/DiffSK/configobj/blob/master/setup.py

LONG_DESCRIPTION = """**ConfigObj** is a simple but powerful config
file reader and writer: an *ini
file round tripper*. Its main feature is that it is very easy to use, with a
straightforward programmer's interface and a simple syntax for config files.
It has lots of other features though :

* Nested sections (subsections), to any level
* List values
* Multiple line values
* Full Unicode support
* String interpolation (substitution)
* Integrated with a powerful validation system

- including automatic type checking/conversion
- and allowing default values
- repeated sections

* All comments in the file are preserved
* The order of keys/sections is preserved
* Powerful ``unrepr`` mode for storing/retrieving Python data-types



>
> Either we are defining the long term thing now, in which case that
> huge pile of complexity lands on us, and we have to get everything
> right.
>
> Or we are defining a thing which solves the present bug, and as long
> as we make sure it does not bind us in future, we're not hamstrung.
>
> E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
> as pybuild is a thing in the debian space, and this will confuse the
> heck out of folk). https://wiki.debian.org/Python/Pybuild
>
>
> -Rob
>
>
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

This is fine as long as developer convenience does not suffer. 
Underlying implementations can always be improved, but if we decide on a 
sucky format, we'll have to live with that for a long time.


08.05.2016, 08:07, Chris Barker kirjoitti:
On Sat, May 7, 2016 at 6:04 PM, Brett Cannon > wrote:


For both options I hear "pick a new format", which suggests we
might as well do it from the get-go for clear separation of the
new stuff and just bite the bullet instead of simply postponing a
decision; it isn't like our format options are going to
significantly change between now and later in the year.


Agreed. However, in another thread, I understood you to say that ALL 
we are talking about now is how to specify the build requirements. If 
that's the case, then we might a well as well just go with setup.cfg.


However, I'd rather we were setting the stage for grater things -- in 
which case, let's go with a new config file.


BTW, IIRC, there seemed to a consensus moving toward using a Python 
API, rather than a command line API for the mythical pluggable build 
systems


In which case, we can require python, and could use python literals 
for configuration. With the discussion of PyYaml, I"m thinking more 
and more that something that can be parsed with only the stdlib is a 
good idea.


-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov 


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

On Sat, May 7, 2016 at 6:04 PM, Brett Cannon  wrote:

For both options I hear "pick a new format", which suggests we might as
> well do it from the get-go for clear separation of the new stuff and just
> bite the bullet instead of simply postponing a decision; it isn't like our
> format options are going to significantly change between now and later in
> the year.
>

Agreed. However, in another thread, I understood you to say that ALL we are
talking about now is how to specify the build requirements. If that's the
case, then we might a well as well just go with setup.cfg.

However, I'd rather we were setting the stage for grater things -- in which
case, let's go with a new config file.

BTW, IIRC, there seemed to a consensus moving toward using a Python API,
rather than a command line API for the mythical pluggable build systems

In which case, we can require python, and could use python literals for
configuration. With the discussion of PyYaml, I"m thinking more and more
that something that can be parsed with only the stdlib is a good idea.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Brett Cannon]

On Sat, May 7, 2016, 15:47 Donald Stufft  wrote:

>
> > On May 7, 2016, at 5:05 PM, Robert Collins 
> wrote:
> >
> > Either we are defining the long term thing now, in which case that
> > huge pile of complexity lands on us, and we have to get everything
> > right.
> >
> > Or we are defining a thing which solves the present bug, and as long
> > as we make sure it does not bind us in future, we're not hamstrung.
> >
> > E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
> > as pybuild is a thing in the debian space, and this will confuse the
> > heck out of folk). https://wiki.debian.org/Python/Pybuild
>
> I think this is roughly true, we could either do the simplest thing and
> just
> add ``setup_requires`` to ``setup.cfg`` and teach pip how to understand
> them
> and then worry about a new format later, or we can do a new format now and
> add
> a bit of complexity to what we need to specify (though I don't think _too_
> much
> complexity, we don't have to define the build system stuff now, just make
> sure
> we don't back ourselves into a corner with that).
>
> I think either answer is OK, just the second one is a bit more work and we
> might either get the start of a better format _now_ or end up regretting
> what
> we pick when we add more things to it.
>

For both options I hear "pick a new format", which suggests we might as
well do it from the get-go for clear separation of the new stuff and just
bite the bullet instead of simply postponing a decision; it isn't like our
format options are going to significantly change between now and later in
the year.

-Brett


> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

08.05.2016, 02:08, Donald Stufft kirjoitti:


On May 7, 2016, at 7:05 PM, Alex Grönholm > wrote:


07.05.2016, 17:48, Nick Coghlan kirjoitti:



On 7 May 2016 13:00, "Nathaniel Smith"  wrote:
>
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
>https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thanks for that, and "yikes" on the comment handling variations in 
ConfigParser - you can tell I've never even tried to use end-of-line 
comments in INI files, and apparently neither has anyone I've worked 
with :)


For YAML, my main concern isn't quirkiness of the syntax, or code 
quality in PyYAML, it's the ease with which you can expose yourself 
to security problems (even if *pip* loads the config file safely, 
that doesn't mean every other tool will). Since we don't need the 
extra power, the easiest way to reduce the collective attack surface 
is to use a strictly less powerful (but still sufficient) format.


Sounds like a far-fetched hypothetical problem. You're concerned 
about the custom tags provided by PyYAML? Do you happen to know a 
tool that defaults to loading files in unsafe mode?


Yea, pyYAML itself does (yaml.load() does it unsafely, you have to use 
yaml.safe_load()).


I don’t think it’s that big of a deal though, we could easily add a 
thing to PyPI that rejects any YAML file that can’t be parsed in safe 
mode. The bigger deal to me is just that the library to work with it 
is a bit of a bear to use as a dependency.
Sounds like we'd need an alternate implementation of YAML then (I'd love 
to see a "yaml" module in the standard library too, but PyYAML isn't a 
good candidate for that, agreed).


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 
3372 DCFA




___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 7, 2016, at 7:05 PM, Alex Grönholm  wrote:
> 
> 07.05.2016, 17:48, Nick Coghlan kirjoitti:
>> 
>> On 7 May 2016 13:00, "Nathaniel Smith" < 
>> n...@pobox.com > wrote:
>> >
>> > Here's that one-stop writeup/comparison of all the major configuration
>> > languages that I mentioned:
>> >
>> > https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f 
>> > 
>> Thanks for that, and "yikes" on the comment handling variations in 
>> ConfigParser - you can tell I've never even tried to use end-of-line 
>> comments in INI files, and apparently neither has anyone I've worked with :)
>> 
>> For YAML, my main concern isn't quirkiness of the syntax, or code quality in 
>> PyYAML, it's the ease with which you can expose yourself to security 
>> problems (even if *pip* loads the config file safely, that doesn't mean 
>> every other tool will). Since we don't need the extra power, the easiest way 
>> to reduce the collective attack surface is to use a strictly less powerful 
>> (but still sufficient) format.
>> 
> Sounds like a far-fetched hypothetical problem. You're concerned about the 
> custom tags provided by PyYAML? Do you happen to know a tool that defaults to 
> loading files in unsafe mode?

Yea, pyYAML itself does (yaml.load() does it unsafely, you have to use 
yaml.safe_load()).

I don’t think it’s that big of a deal though, we could easily add a thing to 
PyPI that rejects any YAML file that can’t be parsed in safe mode. The bigger 
deal to me is just that the library to work with it is a bit of a bear to use 
as a dependency.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

07.05.2016, 17:48, Nick Coghlan kirjoitti:



On 7 May 2016 13:00, "Nathaniel Smith" > wrote:

>
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thanks for that, and "yikes" on the comment handling variations in 
ConfigParser - you can tell I've never even tried to use end-of-line 
comments in INI files, and apparently neither has anyone I've worked 
with :)


For YAML, my main concern isn't quirkiness of the syntax, or code 
quality in PyYAML, it's the ease with which you can expose yourself to 
security problems (even if *pip* loads the config file safely, that 
doesn't mean every other tool will). Since we don't need the extra 
power, the easiest way to reduce the collective attack surface is to 
use a strictly less powerful (but still sufficient) format.


Sounds like a far-fetched hypothetical problem. You're concerned about 
the custom tags provided by PyYAML? Do you happen to know a tool that 
defaults to loading files in unsafe mode?


For ast.literal_eval, we'd still need to come up with a way to do 
sections, key:value mappings and define rules for comments.


For completeness, I'll note that XML combines even more user 
unfriendly syntax than JSON with similar security risks to YAML.


So with the trade-offs laid out like that (and particularly the 
inconsistent comment and Unicode handling in ConfigParser), I'm 
prompted to favour following Rust in adopting TOML.


Cheers,
Nick.

P.S. I particularly like the idea of using extension sections to 
eventually consolidate other static config into a common file - that 
nicely addresses my concern with config file proliferation, since it 
opens the door to eventually subsuming other files like MANIFEST.in 
and setup.cfg as archiving and build systems are updated


>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  - Distutils-SIG@python.org 


> https://mail.python.org/mailman/listinfo/distutils-sig



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 7, 2016, at 5:05 PM, Robert Collins  wrote:
> 
> Either we are defining the long term thing now, in which case that
> huge pile of complexity lands on us, and we have to get everything
> right.
> 
> Or we are defining a thing which solves the present bug, and as long
> as we make sure it does not bind us in future, we're not hamstrung.
> 
> E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
> as pybuild is a thing in the debian space, and this will confuse the
> heck out of folk). https://wiki.debian.org/Python/Pybuild

I think this is roughly true, we could either do the simplest thing and just
add ``setup_requires`` to ``setup.cfg`` and teach pip how to understand them
and then worry about a new format later, or we can do a new format now and add
a bit of complexity to what we need to specify (though I don't think _too_ much
complexity, we don't have to define the build system stuff now, just make sure
we don't back ourselves into a corner with that).

I think either answer is OK, just the second one is a bit more work and we
might either get the start of a better format _now_ or end up regretting what
we pick when we add more things to it.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nathaniel Smith]

To further explore what would be involved if we did go down the TOML
route, I posted an issue to give the pytoml developer(s) a heads up
about this conversation:
  https://github.com/avakar/pytoml/issues/15

On Fri, May 6, 2016 at 7:59 PM, Nathaniel Smith  wrote:
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org



-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Łukasz Langa]

> On May 7, 2016, at 2:05 PM, Robert Collins  wrote:
> 
> Couple thoughts.
> 
> Firstly, the human-editable bit: who in the last *decade* has been
> writing code using a non-syntax-aware/helping editor? Its a supremely
> uninteresting aspect IMO.

Unless you’re faced with adding that syntax highlighting yourself because your 
favorite language du jour is not supported.

> On ConfigParser - yes, its horrid. OTOH we do get all the lines
> reliably, and setuptools will need to cover the unicode aspect itself
> in its own time. All we need to do is permit # inline as a comment -
> line a requirements.txt file for pip - and it becomes trivial to parse
> in all cases that we need *now*.

Yeah, the point about removing implicit inline comment handling was that it’s 
possible by the application to strip the inline comments itself if needed, 
whereas it’s impossible to add that information back if it wasn’t really 
supposed to be a comment. In other words, we can wrap a ConfigParser class 
around and add this ourselves.

--
Lukasz Langa | Facebook
Production Engineer | The Ministry of Silly Walks
(+1) 650-681-7811


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Robert Collins]

Couple thoughts.

Firstly, the human-editable bit: who in the last *decade* has been
writing code using a non-syntax-aware/helping editor? Its a supremely
uninteresting aspect IMO.

On ConfigParser - yes, its horrid. OTOH we do get all the lines
reliably, and setuptools will need to cover the unicode aspect itself
in its own time. All we need to do is permit # inline as a comment -
line a requirements.txt file for pip - and it becomes trivial to parse
in all cases that we need *now*.

Either we are defining the long term thing now, in which case that
huge pile of complexity lands on us, and we have to get everything
right.

Or we are defining a thing which solves the present bug, and as long
as we make sure it does not bind us in future, we're not hamstrung.

E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
as pybuild is a thing in the debian space, and this will confuse the
heck out of folk). https://wiki.debian.org/Python/Pybuild


-Rob



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Łukasz Langa]

> On May 7, 2016, at 9:32 AM, Brett Cannon  wrote:
> 
> On Sat, 7 May 2016 at 07:49 Nick Coghlan  > wrote:
> 
> On 7 May 2016 13:00, "Nathaniel Smith"  > wrote:
> >
> > Here's that one-stop writeup/comparison of all the major configuration
> > languages that I mentioned:
> >
> > https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f 
> > 
> Thanks for that, and "yikes" on the comment handling variations in 
> ConfigParser - you can tell I've never even tried to use end-of-line comments 
> in INI files, and apparently neither has anyone I've worked with :)
> 
> Yeah, that's pretty bad. :/ I checked when ConfigParser was added to Python 
> and it's late 1997: https://hg.python.org/cpython/rev/5b24cbb1f99b 
> , so rather old and predates 
> our stricter code quality rules for additions to the stdlib.

Some context: treating just semicolons as inline comments was broken anyway as 
there were legitimate cases where people wanted to use semicolons inside values 
and it didn’t work. There were more examples of similar features with lurking 
edge cases. We’ve plowed through them with Fred in 2010 and so configparser in 
3.2+ should have less surprising characteristics. Some things we sadly had to 
leave as they are due to unsolvable backwards compatibility issues.

We don’t have to consider using the legacy ConfigParser even for 2.7 compat at 
all as there is a backport of the new version which handles Unicode, etc. I 
know, it’s due a release, I’m on it. But what I’m saying is, for Python 2.7, we 
can and should just bundle the backport and be done with it. If there’s issues 
with it, let me know and yo I'll fix ‘em.

I was involved in standardizing TOML to some extent, seeing it as a clean way 
out from the undefined INI world. Unfortunately it doesn’t seem like it got 
very popular and the tooling around it is still lacking. This is why I’d still 
prefer YAML. Rust is depending on TOML already but that’s still very limited 
adoption and more cognitive churn for users. OTOH, us becoming serious TOML 
users might just change the landscape of its adoption…

--
Lukasz Langa | Facebook
Production Engineer | The Ministry of Silly Walks
(+1) 650-681-7811


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Brett Cannon]

On Sat, 7 May 2016 at 07:49 Nick Coghlan  wrote:

>
> On 7 May 2016 13:00, "Nathaniel Smith"  wrote:
> >
> > Here's that one-stop writeup/comparison of all the major configuration
> > languages that I mentioned:
> >
> > https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
>
> Thanks for that, and "yikes" on the comment handling variations in
> ConfigParser - you can tell I've never even tried to use end-of-line
> comments in INI files, and apparently neither has anyone I've worked with :)
>
Yeah, that's pretty bad. :/ I checked when ConfigParser was added to Python
and it's late 1997: https://hg.python.org/cpython/rev/5b24cbb1f99b, so
rather old and predates our stricter code quality rules for additions to
the stdlib.



> For YAML, my main concern isn't quirkiness of the syntax, or code quality
> in PyYAML, it's the ease with which you can expose yourself to security
> problems (even if *pip* loads the config file safely, that doesn't mean
> every other tool will). Since we don't need the extra power, the easiest
> way to reduce the collective attack surface is to use a strictly less
> powerful (but still sufficient) format.
>
> For ast.literal_eval, we'd still need to come up with a way to do
> sections, key:value mappings and define rules for comments.
>
> For completeness, I'll note that XML combines even more user unfriendly
> syntax than JSON with similar security risks to YAML.
>
> So with the trade-offs laid out like that (and particularly the
> inconsistent comment and Unicode handling in ConfigParser), I'm prompted to
> favour following Rust in adopting TOML.
>
+1 for TOML from me as well. I know Paul brought up the lack of
familiarity, but the format is simple and the Rust community is already
fully dependent on it so at worst Rust + us could always just ignore future
format versions if necessary.

If TOML is the chosen format we could ask how long until a 1.0 release to
know if we waited a month or so to implement we could make sure we're
compliant with that version.

I also checked pytoml at https://github.com/avakar/pytoml and it looks like
it's pretty stable; no changes in the past 5 months except to support
Python 3.5 and only 3 issues. And the format is simple enough that if
someone had to fork the code  like Nathaniel suggested or we did it from
scratch it wouldn't be a huge burden.

-Brett


> Cheers,
> Nick.
>
> P.S. I particularly like the idea of using extension sections to
> eventually consolidate other static config into a common file - that nicely
> addresses my concern with config file proliferation, since it opens the
> door to eventually subsuming other files like MANIFEST.in and setup.cfg as
> archiving and build systems are updated
>
> >
> > -n
> >
> > --
> > Nathaniel J. Smith -- https://vorpus.org
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nick Coghlan]

On 7 May 2016 13:00, "Nathaniel Smith"  wrote:
>
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thanks for that, and "yikes" on the comment handling variations in
ConfigParser - you can tell I've never even tried to use end-of-line
comments in INI files, and apparently neither has anyone I've worked with :)

For YAML, my main concern isn't quirkiness of the syntax, or code quality
in PyYAML, it's the ease with which you can expose yourself to security
problems (even if *pip* loads the config file safely, that doesn't mean
every other tool will). Since we don't need the extra power, the easiest
way to reduce the collective attack surface is to use a strictly less
powerful (but still sufficient) format.

For ast.literal_eval, we'd still need to come up with a way to do sections,
key:value mappings and define rules for comments.

For completeness, I'll note that XML combines even more user unfriendly
syntax than JSON with similar security risks to YAML.

So with the trade-offs laid out like that (and particularly the
inconsistent comment and Unicode handling in ConfigParser), I'm prompted to
favour following Rust in adopting TOML.

Cheers,
Nick.

P.S. I particularly like the idea of using extension sections to eventually
consolidate other static config into a common file - that nicely addresses
my concern with config file proliferation, since it opens the door to
eventually subsuming other files like MANIFEST.in and setup.cfg as
archiving and build systems are updated

>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Wes Turner]

TOML-LD might work for representing JSONLD, as well.

http://json-ld.org/#developers

* https://github.com/RDFLib/rdflib-jsonld
* https://github.com/digitalbazaar/pyld

JSON-LD as a target makes sense because we're describing nodes (with
attributes) and edges in a package graph.

On Sat, May 7, 2016 at 7:06 AM, Wes Turner  wrote:

> +1 for YAML
>
> YAML-LD (YAML & JSONLD) would make expressing the actual graphs for what
> could be "#PEP426JSONLD" much easier.
>
> https://github.com/pypa/interoperability-peps/issues/31
>
>
> On Saturday, May 7, 2016, Alex Grönholm  wrote:
>
>> +1. I don't think the pathological cases of YAML syntax are of any
>> concern in this context. Plus it has excellent tooling support, unlike TOML.
>>
>> 07.05.2016, 09:25, Fred Drake kirjoitti:
>>
>>> On May 6, 2016, at 10:59 PM, Nathaniel Smith  wrote:
>>>
 Here's that one-stop writeup/comparison of all the major configuration
 languages that I mentioned:

 https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

>>> Thank you for this!  A very nice summary.
>>>
>>> On Fri, May 6, 2016 at 11:14 PM, Donald Stufft  wrote:
>>>
 While I personally prefer YAML to any of the options on a purely syntax
 based
 level, when you weigh in all the other considerations for this I think
 that it
 makes sense to go with TOML for it.

>>> I expect either YAML or TOML would be acceptable, based on this.  I'll
>>> admit that I'd not heard of TOML before, but it warrants consideration
>>> (possibly for some of my projects as well).
>>>
>>> I've spent a fair bit of time using YAML with Ansible lately, as well
>>> as some time looking at RAML, and don't find myself worried about the
>>> syntax.  Every oddness I've run across has been handled with an error
>>> when the content couldn't be parsed correctly, rather than unexpected
>>> behavior resulting from misunderstanding how it would be parsed.  It's
>>> entirely possible I just haven't run across the particular problems
>>> Donald has run across, though.
>>>
>>> (The embedded Jinja2 in Ansible playbooks is another matter; let's not
>>> make that mistake.)
>>>
>>>
>>>-Fred
>>>
>>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Wayne Werner]

On May 6, 2016 10:14 PM, "Donald Stufft"  wrote:
>
> While I personally prefer YAML to any of the options on a purely syntax
based
> level, when you weigh in all the other considerations for this I think
that it
> makes sense to go with TOML for it.

I feel the same way. I use YAML fairly extensively with Salt, and while
most of the basic cases are usually fine, any time I try to do something
advanced, I find that it takes a few tries to get right.

Also note with YAML that whichever library we picked would *also* become a
pip upstream, FWIW.

> The only other option I think that could work is what Chris (I think?)
> suggested and just use a Python literal evaluated using
``ast.literal_eval()``
> this is safe to do but it would make it harder for other languages to
parse our
> files. It's similar to the approach taken by Lua Rocks for how their
packaging
> system works (although their uses variables instead of one big dictionary
which
> I think looks nicer) but Lua is much better suited for trying to execute
safely
> outside of ``ast.literal_eval()`` too.

I'd be interested to see what that option looked like, though I *suspect*
that the vagaries in unicode/non-unicode between 2.x and 3.x may produce
the same weakness that we saw in the ConfigParser option.

-W
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Wes Turner]

+1 for YAML

YAML-LD (YAML & JSONLD) would make expressing the actual graphs for what
could be "#PEP426JSONLD" much easier.

https://github.com/pypa/interoperability-peps/issues/31

On Saturday, May 7, 2016, Alex Grönholm  wrote:

> +1. I don't think the pathological cases of YAML syntax are of any concern
> in this context. Plus it has excellent tooling support, unlike TOML.
>
> 07.05.2016, 09:25, Fred Drake kirjoitti:
>
>> On May 6, 2016, at 10:59 PM, Nathaniel Smith  wrote:
>>
>>> Here's that one-stop writeup/comparison of all the major configuration
>>> languages that I mentioned:
>>>
>>> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
>>>
>> Thank you for this!  A very nice summary.
>>
>> On Fri, May 6, 2016 at 11:14 PM, Donald Stufft  wrote:
>>
>>> While I personally prefer YAML to any of the options on a purely syntax
>>> based
>>> level, when you weigh in all the other considerations for this I think
>>> that it
>>> makes sense to go with TOML for it.
>>>
>> I expect either YAML or TOML would be acceptable, based on this.  I'll
>> admit that I'd not heard of TOML before, but it warrants consideration
>> (possibly for some of my projects as well).
>>
>> I've spent a fair bit of time using YAML with Ansible lately, as well
>> as some time looking at RAML, and don't find myself worried about the
>> syntax.  Every oddness I've run across has been handled with an error
>> when the content couldn't be parsed correctly, rather than unexpected
>> behavior resulting from misunderstanding how it would be parsed.  It's
>> entirely possible I just haven't run across the particular problems
>> Donald has run across, though.
>>
>> (The embedded Jinja2 in Ansible playbooks is another matter; let's not
>> make that mistake.)
>>
>>
>>-Fred
>>
>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[David Cournapeau]

A missing dimension for comparison: round tripping support. It is quite
useful for formats when used as a configuration. The best I know in that
dimension is yaml (if using ruamel.yaml), which round trip comments.

OTOH, adding round tripping to something like toml should not be too hard
if the need arises.

David

On Sat, May 7, 2016 at 3:59 AM, Nathaniel Smith  wrote:

> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 7 May 2016 at 04:14, Donald Stufft  wrote:
> While I personally prefer YAML to any of the options on a purely syntax based
> level, when you weigh in all the other considerations for this I think that it
> makes sense to go with TOML for it.
>
> The only other option I think that could work is what Chris (I think?)
> suggested and just use a Python literal evaluated using ``ast.literal_eval()``
> this is safe to do but it would make it harder for other languages to parse 
> our
> files. It's similar to the approach taken by Lua Rocks for how their packaging
> system works (although their uses variables instead of one big dictionary 
> which
> I think looks nicer) but Lua is much better suited for trying to execute 
> safely
> outside of ``ast.literal_eval()`` too.
>
> All in all, I think TOML is the right answer (and that's why my partially
> written PEP used TOML).

FWIW (and because obsessing about config formats is a long-running
hobby of mine :-)) my view is:

- JSON sucks as a human-editable format, because it's too strict over
things like commas and has no comments. It's supported by the stdlib,
though, which is nice.
- YAML ought to be wonderful, but it ended up over-engineered (yes, we
can ignore the bits we don't care about). Also, pyYAML is a bit of an
annoying dependency (big, reportedly slow unless you use the C
version) - not something I'd want pip to have to vendor.
- INI (ConfigParser) format is too simple for many use cases. It has
stdlib support, though.
- Python literals are good, but they define values, they aren't a file
format. We'd need to write a parser for "the rest"
- TOML seems pretty good, but is immature (I remember when YAML seemed
like TOML does now...) Also, it's unfamiliar to people (I wasn't aware
of the use in Rust)
- Just for a laugh, can I mention XML? :-)

Overall, *any* choice feels like I'm choosing the "best of a bad job" :-(

With all that said, I'm inclined to agree that TOML looks good.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[tritium-list]

I am +1 to TOML; it's INI (a human editable format) with data-types (I think
it is even valid INI).  I find the format pleasant to work with both in the
available libraries and in the editor.


-Original Message-
From: Distutils-SIG
[mailto:distutils-sig-bounces+tritium-list=sdamon@python.org] On Behalf
Of Alex Grönholm
Sent: Saturday, May 7, 2016 4:29 AM
To: distutils-sig@python.org
Subject: Re: [Distutils] comparison of configuration languages

+1. I don't think the pathological cases of YAML syntax are of any 
concern in this context. Plus it has excellent tooling support, unlike TOML.

07.05.2016, 09:25, Fred Drake kirjoitti:
> On May 6, 2016, at 10:59 PM, Nathaniel Smith <n...@pobox.com> wrote:
>> Here's that one-stop writeup/comparison of all the major configuration
>> languages that I mentioned:
>>
>> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
> Thank you for this!  A very nice summary.
>
> On Fri, May 6, 2016 at 11:14 PM, Donald Stufft <don...@stufft.io> wrote:
>> While I personally prefer YAML to any of the options on a purely syntax
based
>> level, when you weigh in all the other considerations for this I think
that it
>> makes sense to go with TOML for it.
> I expect either YAML or TOML would be acceptable, based on this.  I'll
> admit that I'd not heard of TOML before, but it warrants consideration
> (possibly for some of my projects as well).
>
> I've spent a fair bit of time using YAML with Ansible lately, as well
> as some time looking at RAML, and don't find myself worried about the
> syntax.  Every oddness I've run across has been handled with an error
> when the content couldn't be parsed correctly, rather than unexpected
> behavior resulting from misunderstanding how it would be parsed.  It's
> entirely possible I just haven't run across the particular problems
> Donald has run across, though.
>
> (The embedded Jinja2 in Ansible playbooks is another matter; let's not
> make that mistake.)
>
>
>-Fred
>

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

+1. I don't think the pathological cases of YAML syntax are of any 
concern in this context. Plus it has excellent tooling support, unlike TOML.


07.05.2016, 09:25, Fred Drake kirjoitti:

On May 6, 2016, at 10:59 PM, Nathaniel Smith  wrote:

Here's that one-stop writeup/comparison of all the major configuration
languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thank you for this!  A very nice summary.

On Fri, May 6, 2016 at 11:14 PM, Donald Stufft  wrote:

While I personally prefer YAML to any of the options on a purely syntax based
level, when you weigh in all the other considerations for this I think that it
makes sense to go with TOML for it.

I expect either YAML or TOML would be acceptable, based on this.  I'll
admit that I'd not heard of TOML before, but it warrants consideration
(possibly for some of my projects as well).

I've spent a fair bit of time using YAML with Ansible lately, as well
as some time looking at RAML, and don't find myself worried about the
syntax.  Every oddness I've run across has been handled with an error
when the content couldn't be parsed correctly, rather than unexpected
behavior resulting from misunderstanding how it would be parsed.  It's
entirely possible I just haven't run across the particular problems
Donald has run across, though.

(The embedded Jinja2 in Ansible playbooks is another matter; let's not
make that mistake.)


   -Fred



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Fred Drake]

On May 6, 2016, at 10:59 PM, Nathaniel Smith  wrote:
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thank you for this!  A very nice summary.

On Fri, May 6, 2016 at 11:14 PM, Donald Stufft  wrote:
> While I personally prefer YAML to any of the options on a purely syntax based
> level, when you weigh in all the other considerations for this I think that it
> makes sense to go with TOML for it.

I expect either YAML or TOML would be acceptable, based on this.  I'll
admit that I'd not heard of TOML before, but it warrants consideration
(possibly for some of my projects as well).

I've spent a fair bit of time using YAML with Ansible lately, as well
as some time looking at RAML, and don't find myself worried about the
syntax.  Every oddness I've run across has been handled with an error
when the content couldn't be parsed correctly, rather than unexpected
behavior resulting from misunderstanding how it would be parsed.  It's
entirely possible I just haven't run across the particular problems
Donald has run across, though.

(The embedded Jinja2 in Ansible playbooks is another matter; let's not
make that mistake.)


  -Fred

-- 
Fred L. Drake, Jr.
"A storm broke loose in my mind."  --Albert Einstein
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nathaniel Smith]

On Fri, May 6, 2016 at 8:14 PM, Donald Stufft  wrote:
[...]
> The only other option I think that could work is what Chris (I think?)
> suggested and just use a Python literal evaluated using ``ast.literal_eval()``

Oh, good point, that should definitely be on the list of options
considered regardless of whether we go for it. I won't add it now
because I just switched to Windows to play games with my wife and my
emacs is on the other partition, but I'll add it later.

Does anyone know how literal_eval handles Unicode on py2?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 6, 2016, at 10:59 PM, Nathaniel Smith  wrote:
> 
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
> 
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
> 
> -n
> 
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


While I personally prefer YAML to any of the options on a purely syntax based
level, when you weigh in all the other considerations for this I think that it
makes sense to go with TOML for it.

The only other option I think that could work is what Chris (I think?)
suggested and just use a Python literal evaluated using ``ast.literal_eval()``
this is safe to do but it would make it harder for other languages to parse our
files. It's similar to the approach taken by Lua Rocks for how their packaging
system works (although their uses variables instead of one big dictionary which
I think looks nicer) but Lua is much better suited for trying to execute safely
outside of ``ast.literal_eval()`` too.

All in all, I think TOML is the right answer (and that's why my partially
written PEP used TOML).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig