Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

This is fine as long as developer convenience does not suffer. 
Underlying implementations can always be improved, but if we decide on a 
sucky format, we'll have to live with that for a long time.


08.05.2016, 08:07, Chris Barker kirjoitti:
On Sat, May 7, 2016 at 6:04 PM, Brett Cannon > wrote:


For both options I hear "pick a new format", which suggests we
might as well do it from the get-go for clear separation of the
new stuff and just bite the bullet instead of simply postponing a
decision; it isn't like our format options are going to
significantly change between now and later in the year.


Agreed. However, in another thread, I understood you to say that ALL 
we are talking about now is how to specify the build requirements. If 
that's the case, then we might a well as well just go with setup.cfg.


However, I'd rather we were setting the stage for grater things -- in 
which case, let's go with a new config file.


BTW, IIRC, there seemed to a consensus moving toward using a Python 
API, rather than a command line API for the mythical pluggable build 
systems


In which case, we can require python, and could use python literals 
for configuration. With the discussion of PyYaml, I"m thinking more 
and more that something that can be parsed with only the stdlib is a 
good idea.


-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov 


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

On Sat, May 7, 2016 at 6:04 PM, Brett Cannon  wrote:

For both options I hear "pick a new format", which suggests we might as
> well do it from the get-go for clear separation of the new stuff and just
> bite the bullet instead of simply postponing a decision; it isn't like our
> format options are going to significantly change between now and later in
> the year.
>

Agreed. However, in another thread, I understood you to say that ALL we are
talking about now is how to specify the build requirements. If that's the
case, then we might a well as well just go with setup.cfg.

However, I'd rather we were setting the stage for grater things -- in which
case, let's go with a new config file.

BTW, IIRC, there seemed to a consensus moving toward using a Python API,
rather than a command line API for the mythical pluggable build systems

In which case, we can require python, and could use python literals for
configuration. With the discussion of PyYaml, I"m thinking more and more
that something that can be parsed with only the stdlib is a good idea.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Brett Cannon]

On Sat, May 7, 2016, 15:47 Donald Stufft  wrote:

>
> > On May 7, 2016, at 5:05 PM, Robert Collins 
> wrote:
> >
> > Either we are defining the long term thing now, in which case that
> > huge pile of complexity lands on us, and we have to get everything
> > right.
> >
> > Or we are defining a thing which solves the present bug, and as long
> > as we make sure it does not bind us in future, we're not hamstrung.
> >
> > E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
> > as pybuild is a thing in the debian space, and this will confuse the
> > heck out of folk). https://wiki.debian.org/Python/Pybuild
>
> I think this is roughly true, we could either do the simplest thing and
> just
> add ``setup_requires`` to ``setup.cfg`` and teach pip how to understand
> them
> and then worry about a new format later, or we can do a new format now and
> add
> a bit of complexity to what we need to specify (though I don't think _too_
> much
> complexity, we don't have to define the build system stuff now, just make
> sure
> we don't back ourselves into a corner with that).
>
> I think either answer is OK, just the second one is a bit more work and we
> might either get the start of a better format _now_ or end up regretting
> what
> we pick when we add more things to it.
>

For both options I hear "pick a new format", which suggests we might as
well do it from the get-go for clear separation of the new stuff and just
bite the bullet instead of simply postponing a decision; it isn't like our
format options are going to significantly change between now and later in
the year.

-Brett


> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

08.05.2016, 02:08, Donald Stufft kirjoitti:


On May 7, 2016, at 7:05 PM, Alex Grönholm > wrote:


07.05.2016, 17:48, Nick Coghlan kirjoitti:



On 7 May 2016 13:00, "Nathaniel Smith"  wrote:
>
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
>https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thanks for that, and "yikes" on the comment handling variations in 
ConfigParser - you can tell I've never even tried to use end-of-line 
comments in INI files, and apparently neither has anyone I've worked 
with :)


For YAML, my main concern isn't quirkiness of the syntax, or code 
quality in PyYAML, it's the ease with which you can expose yourself 
to security problems (even if *pip* loads the config file safely, 
that doesn't mean every other tool will). Since we don't need the 
extra power, the easiest way to reduce the collective attack surface 
is to use a strictly less powerful (but still sufficient) format.


Sounds like a far-fetched hypothetical problem. You're concerned 
about the custom tags provided by PyYAML? Do you happen to know a 
tool that defaults to loading files in unsafe mode?


Yea, pyYAML itself does (yaml.load() does it unsafely, you have to use 
yaml.safe_load()).


I don’t think it’s that big of a deal though, we could easily add a 
thing to PyPI that rejects any YAML file that can’t be parsed in safe 
mode. The bigger deal to me is just that the library to work with it 
is a bit of a bear to use as a dependency.
Sounds like we'd need an alternate implementation of YAML then (I'd love 
to see a "yaml" module in the standard library too, but PyYAML isn't a 
good candidate for that, agreed).


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 
3372 DCFA




___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

[Brett Cannon]

On Sat, May 7, 2016, 12:16 Chris Barker  wrote:

> On Sat, May 7, 2016 at 11:18 AM, Brett Cannon  wrote:
>
>> What fields there will be and their semantics ...
>>
>>1. Format version (so just deciding on a name -- which also includes
>>   whether it should be top-level or in a subsection -- and initial value)
>>   2. The actual bootstrap field (so its name and what to do if a
>>   dependency is already installed but at a version that doesn't match 
>> what
>>   the bootstrap specification asks for)
>>
>> Am I missing anything?
>>
>
> So what is this new configuration file supposed to cover?
>

How to specify what has to be installed to simply build a project, e.g. is
setuptools needed to run setup.py, and if so what version?

All the package meta-data? i.e. everything that would be needed by a
> package manager to properly install the package?
>


> or the build meta-data: everything needed by the build system to build the
> package?
>


> both in one file?
>
> And am missing something?
>

You're missing that you're talking several PEPs down the road. :) Right now
all we are discussing is how to specify what build tools a project needs
(historically setuptools, but in the future it could be flit or something
else).

how is this about "bootstrapping" -- to me, bootstrapping is when you need
> X to build X. Isn't this just regular old configuration: you need x,y to
> build z?
>

Sure, if you don't like the term "bootstrap" then you can call it "build
requirements". We have not been calling it " configuration" in a general
sense as this doesn't cover how to invoke the build step (that will
probably be the next PEP), just what needs to be installed to even
potentially do a build.

-Brett



> -CHB
>
>
>
>
>
>
>
>> And since I keep pushing on this I'm willing to be a co-author on any PEP
>> if there's no hard deadline in getting the PEP written (i.e. I can help
>> write the prose, but I don't have the time to do the coding as this will be
>> the fourth PEP I have going in some state; got to catch up to Nick's 35
>> PEPs somehow ;).
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 7, 2016, at 7:05 PM, Alex Grönholm  wrote:
> 
> 07.05.2016, 17:48, Nick Coghlan kirjoitti:
>> 
>> On 7 May 2016 13:00, "Nathaniel Smith" < 
>> n...@pobox.com > wrote:
>> >
>> > Here's that one-stop writeup/comparison of all the major configuration
>> > languages that I mentioned:
>> >
>> > https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f 
>> > 
>> Thanks for that, and "yikes" on the comment handling variations in 
>> ConfigParser - you can tell I've never even tried to use end-of-line 
>> comments in INI files, and apparently neither has anyone I've worked with :)
>> 
>> For YAML, my main concern isn't quirkiness of the syntax, or code quality in 
>> PyYAML, it's the ease with which you can expose yourself to security 
>> problems (even if *pip* loads the config file safely, that doesn't mean 
>> every other tool will). Since we don't need the extra power, the easiest way 
>> to reduce the collective attack surface is to use a strictly less powerful 
>> (but still sufficient) format.
>> 
> Sounds like a far-fetched hypothetical problem. You're concerned about the 
> custom tags provided by PyYAML? Do you happen to know a tool that defaults to 
> loading files in unsafe mode?

Yea, pyYAML itself does (yaml.load() does it unsafely, you have to use 
yaml.safe_load()).

I don’t think it’s that big of a deal though, we could easily add a thing to 
PyPI that rejects any YAML file that can’t be parsed in safe mode. The bigger 
deal to me is just that the library to work with it is a bit of a bear to use 
as a dependency.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

07.05.2016, 17:48, Nick Coghlan kirjoitti:



On 7 May 2016 13:00, "Nathaniel Smith" > wrote:

>
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thanks for that, and "yikes" on the comment handling variations in 
ConfigParser - you can tell I've never even tried to use end-of-line 
comments in INI files, and apparently neither has anyone I've worked 
with :)


For YAML, my main concern isn't quirkiness of the syntax, or code 
quality in PyYAML, it's the ease with which you can expose yourself to 
security problems (even if *pip* loads the config file safely, that 
doesn't mean every other tool will). Since we don't need the extra 
power, the easiest way to reduce the collective attack surface is to 
use a strictly less powerful (but still sufficient) format.


Sounds like a far-fetched hypothetical problem. You're concerned about 
the custom tags provided by PyYAML? Do you happen to know a tool that 
defaults to loading files in unsafe mode?


For ast.literal_eval, we'd still need to come up with a way to do 
sections, key:value mappings and define rules for comments.


For completeness, I'll note that XML combines even more user 
unfriendly syntax than JSON with similar security risks to YAML.


So with the trade-offs laid out like that (and particularly the 
inconsistent comment and Unicode handling in ConfigParser), I'm 
prompted to favour following Rust in adopting TOML.


Cheers,
Nick.

P.S. I particularly like the idea of using extension sections to 
eventually consolidate other static config into a common file - that 
nicely addresses my concern with config file proliferation, since it 
opens the door to eventually subsuming other files like MANIFEST.in 
and setup.cfg as archiving and build systems are updated


>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  - Distutils-SIG@python.org 


> https://mail.python.org/mailman/listinfo/distutils-sig



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 7, 2016, at 5:05 PM, Robert Collins  wrote:
> 
> Either we are defining the long term thing now, in which case that
> huge pile of complexity lands on us, and we have to get everything
> right.
> 
> Or we are defining a thing which solves the present bug, and as long
> as we make sure it does not bind us in future, we're not hamstrung.
> 
> E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
> as pybuild is a thing in the debian space, and this will confuse the
> heck out of folk). https://wiki.debian.org/Python/Pybuild

I think this is roughly true, we could either do the simplest thing and just
add ``setup_requires`` to ``setup.cfg`` and teach pip how to understand them
and then worry about a new format later, or we can do a new format now and add
a bit of complexity to what we need to specify (though I don't think _too_ much
complexity, we don't have to define the build system stuff now, just make sure
we don't back ourselves into a corner with that).

I think either answer is OK, just the second one is a bit more work and we
might either get the start of a better format _now_ or end up regretting what
we pick when we add more things to it.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nathaniel Smith]

To further explore what would be involved if we did go down the TOML
route, I posted an issue to give the pytoml developer(s) a heads up
about this conversation:
  https://github.com/avakar/pytoml/issues/15

On Fri, May 6, 2016 at 7:59 PM, Nathaniel Smith  wrote:
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org



-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Łukasz Langa]

> On May 7, 2016, at 2:05 PM, Robert Collins  wrote:
> 
> Couple thoughts.
> 
> Firstly, the human-editable bit: who in the last *decade* has been
> writing code using a non-syntax-aware/helping editor? Its a supremely
> uninteresting aspect IMO.

Unless you’re faced with adding that syntax highlighting yourself because your 
favorite language du jour is not supported.

> On ConfigParser - yes, its horrid. OTOH we do get all the lines
> reliably, and setuptools will need to cover the unicode aspect itself
> in its own time. All we need to do is permit # inline as a comment -
> line a requirements.txt file for pip - and it becomes trivial to parse
> in all cases that we need *now*.

Yeah, the point about removing implicit inline comment handling was that it’s 
possible by the application to strip the inline comments itself if needed, 
whereas it’s impossible to add that information back if it wasn’t really 
supposed to be a comment. In other words, we can wrap a ConfigParser class 
around and add this ourselves.

--
Lukasz Langa | Facebook
Production Engineer | The Ministry of Silly Walks
(+1) 650-681-7811


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Robert Collins]

Couple thoughts.

Firstly, the human-editable bit: who in the last *decade* has been
writing code using a non-syntax-aware/helping editor? Its a supremely
uninteresting aspect IMO.

On ConfigParser - yes, its horrid. OTOH we do get all the lines
reliably, and setuptools will need to cover the unicode aspect itself
in its own time. All we need to do is permit # inline as a comment -
line a requirements.txt file for pip - and it becomes trivial to parse
in all cases that we need *now*.

Either we are defining the long term thing now, in which case that
huge pile of complexity lands on us, and we have to get everything
right.

Or we are defining a thing which solves the present bug, and as long
as we make sure it does not bind us in future, we're not hamstrung.

E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
as pybuild is a thing in the debian space, and this will confuse the
heck out of folk). https://wiki.debian.org/Python/Pybuild


-Rob



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Łukasz Langa]

> On May 7, 2016, at 9:32 AM, Brett Cannon  wrote:
> 
> On Sat, 7 May 2016 at 07:49 Nick Coghlan  > wrote:
> 
> On 7 May 2016 13:00, "Nathaniel Smith"  > wrote:
> >
> > Here's that one-stop writeup/comparison of all the major configuration
> > languages that I mentioned:
> >
> > https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f 
> > 
> Thanks for that, and "yikes" on the comment handling variations in 
> ConfigParser - you can tell I've never even tried to use end-of-line comments 
> in INI files, and apparently neither has anyone I've worked with :)
> 
> Yeah, that's pretty bad. :/ I checked when ConfigParser was added to Python 
> and it's late 1997: https://hg.python.org/cpython/rev/5b24cbb1f99b 
> , so rather old and predates 
> our stricter code quality rules for additions to the stdlib.

Some context: treating just semicolons as inline comments was broken anyway as 
there were legitimate cases where people wanted to use semicolons inside values 
and it didn’t work. There were more examples of similar features with lurking 
edge cases. We’ve plowed through them with Fred in 2010 and so configparser in 
3.2+ should have less surprising characteristics. Some things we sadly had to 
leave as they are due to unsolvable backwards compatibility issues.

We don’t have to consider using the legacy ConfigParser even for 2.7 compat at 
all as there is a backport of the new version which handles Unicode, etc. I 
know, it’s due a release, I’m on it. But what I’m saying is, for Python 2.7, we 
can and should just bundle the backport and be done with it. If there’s issues 
with it, let me know and yo I'll fix ‘em.

I was involved in standardizing TOML to some extent, seeing it as a clean way 
out from the undefined INI world. Unfortunately it doesn’t seem like it got 
very popular and the tooling around it is still lacking. This is why I’d still 
prefer YAML. Rust is depending on TOML already but that’s still very limited 
adoption and more cognitive churn for users. OTOH, us becoming serious TOML 
users might just change the landscape of its adoption…

--
Lukasz Langa | Facebook
Production Engineer | The Ministry of Silly Walks
(+1) 650-681-7811


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

[Chris Barker]

On Sat, May 7, 2016 at 11:18 AM, Brett Cannon  wrote:

> What fields there will be and their semantics ...
>
>1. Format version (so just deciding on a name -- which also includes
>   whether it should be top-level or in a subsection -- and initial value)
>   2. The actual bootstrap field (so its name and what to do if a
>   dependency is already installed but at a version that doesn't match what
>   the bootstrap specification asks for)
>
> Am I missing anything?
>

So what is this new configuration file supposed to cover? All the package
meta-data? i.e. everything that would be needed by a package manager to
properly install the package?

or the build meta-data: everything needed by the build system to build the
package?

both in one file?

And am missing something? how is this about "bootstrapping" -- to me,
bootstrapping is when you need X to build X. Isn't this just regular old
configuration: you need x,y to build z?

-CHB







> And since I keep pushing on this I'm willing to be a co-author on any PEP
> if there's no hard deadline in getting the PEP written (i.e. I can help
> write the prose, but I don't have the time to do the coding as this will be
> the fourth PEP I have going in some state; got to catch up to Nick's 35
> PEPs somehow ;).
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Chris Barker]

On Sat, May 7, 2016 at 6:17 AM, Greg Ewing 
wrote:

> Do you expect that
>
>> projects ... should (somehow) contain simplified instructions on how to
>> build the various C/Fortran extensions supplied in the bundle as
>> source code?
>>
>
> Essentially, yes. I'm not sure how achievable it would
> be, but ideally that's what I'd like.
>

I think we've all come to conclusion that that's simply not possible --
build configuration cannot be purely declarative in the general case -- if
you are building a package you are going to be running arbitrary code (and
really, if you're running a compiler, you're doing that anyway, so there
isn't an additional security issue here -- if you trust the code, you might
as well trust the build script)

> On a unix system, most of the time they would all be in

> well-known locations. If I install something in an unusual
> place or in an unusual way, I'm going to have to tell
> something about it anyway. I don't see how an executable
> setup file provided by the package author is going to
> magically figure out all the weird stuff I've done.
>

You can look at any number of packages -- essentially, they do what
configure scripts do with autotools -- various hacky ways to find the stuff
it's looking for.

and users really want this -- maybe on a "normal" *nix system there isn't
much to it, but on OS-X there sure is -- people may have hand installed the
dependencies, or they may have used fink, or macports, or homebrew, or...
and, except for the hand-install case, they may have no idea what the heck
they did and where it is.

I don't know if there are conventions for such things on
> Windows. I suspect not, in which case manual input is
> going to be needed one way or another.
>

Windows is even worse -- not only no conventions, but also no common
package manager, either (at least OS-X is limited to four :-) )

Not all of it, only the parts that strictly have to be
> performed as part of the build step of the install
> process, to use your terminology. That's a fairly
> restricted subset of the whole problem of compiling
> software.
>

I don't think it's possible (or desirable) to make a clear distinction
between "the build step of the install process" and building in general.

Could a purely declarative config file be flexible
> enough to handle this? I don't know. The distutils
> API is actually pretty declarative already if you
> use it in a straightforward way.
>

indeed -- and for the common cases, that works fine, but there's always
SOMETHING That some weird case is going to need.

You could, I suppose, separate out the configuration step from the build
step -- analogous to ./configure, vs make.

So the configure step would generate a purely declarative config file of
some sort, and then the build step would use that. In the simple case,
there might be no need for a configure step at all.

though I'm not sure even this is possible -- for instance, numpy used a
heavily enhanced distutils to do it's thing. Cython extends distutils to
understand the "build the C from the pyx" step, etc This is still used
decoratively, but it's third party code that is doing part of the build
step -- so in order to make the build system extendable, you need to it run
code

Anyway, I thought it was clear that we need to make the distinction between
building and installing/packaging, etc clear -- both form an API
perspective and a user perspective. So at this point, we need to establish
and API to the build system (probably compatible with what we have
(setup.py build) but we leave it up to the build system to figure out how
to do it's thing -- maybe someone will come up with a purely declarative
system -- who knows?

>  Running Pyrex to generate .c files

> from .pyx files is one that I've encountered.
> (I encouraged that one myself by including a distutils
> extension for it, which I later decided had been a
> mistake.)
>

I don't think it was a mistake :-) -- that's got to be done some time --
why add another layer???

That's nice, but it wouldn't help me when I encounter
> a package that *hadn't* been set up to use gregsbuild. :-(
>

sure -- but we can't have a one-build-system-to-rule-them-all until we can
first have a way to have ANY build system other than setuptools :-)

Key issue here:

Right now, in order to make things easier for users, and due to history,
the whole build/package/install processes are fully intermingled. I think
this is a BAD THING, for two reasons:

1) you can't replace any of the tools individually (except I suppose the
packaging part, as it's the last step)

2) user's don't know what's going on, or what's going to happen when they
try to "intsall" something:

It's great that you can "just run pip install" and it often "just works" --
but when it doesn't it's kind of ugly. And there are security concerns:
When a user runs:

pip install some_package

They don't know what's going to happen. pip goes and looks on PyPi to see
if 

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

[Brett Cannon]

On Fri, 6 May 2016 at 16:58 Nathaniel Smith  wrote:

> On Fri, May 6, 2016 at 11:14 AM, Brett Cannon  wrote:
> >
> >
> > On Fri, 6 May 2016 at 09:40 Donald Stufft  wrote:
> >>
> >>
> >> On May 6, 2016, at 12:36 PM, Brett Cannon  wrote:
> >>
> >> So who is the BDFL on this decision? It seems we need someone to stop
> the
> >> bikeshedding on the field name and what file is going to house this
> >> configuration data. And do we need someone to write a PEP for this
> proposal
> >> to have something to target?
> >>
> >>
> >> We need someone to write the PEP and someone to offer to be BDFL for it.
> >> For this particular change the default would be Nick for BDFL but if
> he’s
> >> busy someone else can take it over for this PEP. Though I think we need
> >> someone writing an actual PEP first.
> >
> >
> > OK, assuming the Nick will be pronouncing, who wants to write the PEP?
>

And Paul also stepped forward to pronounce if Nick didn't want to, so we
have the role of Great Decider covered one way or another.


>
> I've just been writing up a comparison of the different file formats,
> partly in case it's useful to others and partly just for my own use in
> looking at them against each other and seeing how much it actually
> matters. This might also be reusable for the
> rationale/rejected-alternatives section of a PEP, if anyone wants it,
> or I could go ahead and add a few paragraphs to turn it into a proper
> PEP.
>

 What does the PEP need to cover?

   1. The syntax of the file (which based on the replies to your great
   overview, Nathaniel, looks to be TOML).
   2. The name of the file (although I'm assuming it will be setup.* out of
   tradition, probably setup.toml if TOML wins the format battle).
   3. What fields there will be and their semantics ...
   1. Format version (so just deciding on a name -- which also includes
  whether it should be top-level or in a subsection -- and initial value)
  2. The actual bootstrap field (so its name and what to do if a
  dependency is already installed but at a version that doesn't match what
  the bootstrap specification asks for)

Am I missing anything? And since I keep pushing on this I'm willing to be a
co-author on any PEP if there's no hard deadline in getting the PEP written
(i.e. I can help write the prose, but I don't have the time to do the
coding as this will be the fourth PEP I have going in some state; got to
catch up to Nick's 35 PEPs somehow ;).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Sat, May 7, 2016 at 6:51 AM, Nick Coghlan  wrote:

> On 7 May 2016 01:55, "Chris Barker"  wrote:
> > So my point is about scope-creep -- if you want the PyPa tools to solve
> all these problems, then you are re-inventing conda -- better to simply
> adopt conda (or fork it and fix issues that I'm sure are there)
>
> conda doesn't solve these problems either - it solves the *end user*
> problem for data analysts (install the Python library they want to use),
>
I really need to make this clear -- there is NOTHING "data analyst"
specific about these problems -- they do come up more in the computational
programming domain, but there are any number of other application that have
the same problems (pyQT has come up in this conversation, yes?) -- and
we're finding conda to be a good solution for our web development needs,
too -- a conda environment  is kinda like a lighter-weight, platform
independent docker container. And of course, there is more an more data
analysis going on behind web services these days too -- any python
developer is going to run into these issues eventually...

> by ignoring the system integrator one (interoperate with the system
> integrator's existing platform level package management systems, of which
> we all tend to have our own with no plans for consolidation)
>
> That's by design, though - conda was specifically created as a language
> independent cross-platform platform, not as a cross-platform plugin
> management system for Python runtimes.
>
exactly!

> For a long time I was convinced the existence of conda and Linux
> containers as end user tools meant we wouldn't ever need to solve these
> problems at the language runtime layer, but it's since become clear to me
> that there's significant value in solving these problems in a way that
> doesn't care about how you obtained your Python runtime.
>
yup -- that would be the scope creep I'm referring too :-)

But now I'm confused about what problems we're talking about. From above:

"""
interoperate with the system integrator's existing platform level package
management systems
"""

you mean rpm, .deb, homebrew, etc?  distutils did have (does it still)
bdist_rpm (and, for that matter bdist_wininst) -- which I see as an attempt
to inter-operate with the system integrator's platform. But I think it
turns out that this is a hopeless task -- there are just too many different
systems to consider and maintain -- much better to let the external systems
handle that. And it got really ugly when you wanted to mingle virtualenv
with rpm, etc

The "solution" to that is to actually not do it -- use Docker or Conda if
you want an isolated environment.

What all this means to me is that  we really do need to keep the building
separate from the packaging -- so that the building tools can be used /
supported by the systems integrator. But people shouldn't be using rpm to
manage their system, and expect to install binary wheels...

The trick is that the pip/wheel system is really handy -- certainly for
Windows and OS-X that don't provide a standard system package manager. It
is nice to be able to "just pip install" various packages. But the problem
is that it is a half way solution -- it only works well for python-centered
packages -- pure python is trivial of course, but custom written C
extensions work great too -- but as soon as you get external dependencies,
it all goes to heck. We can improve that -- defining the manylinux platform
is a great step, but to really get there, there needs to be a way to handle
third party libs (which we could probably do with the existing tools by
establishing conventions), but then you get to non-python support tools:
Fortran compilers, other language run times, all sorts of stuff. Oh, and
then we want isolated environments, and they need to isolate these other
non-python tools, so 

I think it's WAY beyond the current PEP under discussion, but we really
should map out a "where we'd like to get" plan.

"significant value in solving these problems in a way that doesn't care
about how you obtained your Python runtime."

I don't get this -- I don't think it's possible to not care about the
python run-time -- we've only gotten this far because most folks have more
or less standardized on the python.org builds as the standard python build.
Even when you get your python elsewhere (Anaconda, for instance) they take
pains to keep it compatible.

I think we should count on python itself being managed by the same package
manager as the packages -- or at least conforming to a standard (which I
think we have with teh ABI tags, yes?)

BTW, I only recently learned that there are more "other" python builds out
there than I thought. Apparently the Visual Effect folks are on the habit
of building their own python2.7 on Windows, so they can use a newer MS
compiler, and then build their extensions to be compatible (all so you can
use modern C++ in the extension code). I don't 

Re: [Distutils] comparison of configuration languages

[Brett Cannon]

On Sat, 7 May 2016 at 07:49 Nick Coghlan  wrote:

>
> On 7 May 2016 13:00, "Nathaniel Smith"  wrote:
> >
> > Here's that one-stop writeup/comparison of all the major configuration
> > languages that I mentioned:
> >
> > https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
>
> Thanks for that, and "yikes" on the comment handling variations in
> ConfigParser - you can tell I've never even tried to use end-of-line
> comments in INI files, and apparently neither has anyone I've worked with :)
>
Yeah, that's pretty bad. :/ I checked when ConfigParser was added to Python
and it's late 1997: https://hg.python.org/cpython/rev/5b24cbb1f99b, so
rather old and predates our stricter code quality rules for additions to
the stdlib.



> For YAML, my main concern isn't quirkiness of the syntax, or code quality
> in PyYAML, it's the ease with which you can expose yourself to security
> problems (even if *pip* loads the config file safely, that doesn't mean
> every other tool will). Since we don't need the extra power, the easiest
> way to reduce the collective attack surface is to use a strictly less
> powerful (but still sufficient) format.
>
> For ast.literal_eval, we'd still need to come up with a way to do
> sections, key:value mappings and define rules for comments.
>
> For completeness, I'll note that XML combines even more user unfriendly
> syntax than JSON with similar security risks to YAML.
>
> So with the trade-offs laid out like that (and particularly the
> inconsistent comment and Unicode handling in ConfigParser), I'm prompted to
> favour following Rust in adopting TOML.
>
+1 for TOML from me as well. I know Paul brought up the lack of
familiarity, but the format is simple and the Rust community is already
fully dependent on it so at worst Rust + us could always just ignore future
format versions if necessary.

If TOML is the chosen format we could ask how long until a 1.0 release to
know if we waited a month or so to implement we could make sure we're
compliant with that version.

I also checked pytoml at https://github.com/avakar/pytoml and it looks like
it's pretty stable; no changes in the past 5 months except to support
Python 3.5 and only 3 issues. And the format is simple enough that if
someone had to fork the code  like Nathaniel suggested or we did it from
scratch it wouldn't be a huge burden.

-Brett


> Cheers,
> Nick.
>
> P.S. I particularly like the idea of using extension sections to
> eventually consolidate other static config into a common file - that nicely
> addresses my concern with config file proliferation, since it opens the
> door to eventually subsuming other files like MANIFEST.in and setup.cfg as
> archiving and build systems are updated
>
> >
> > -n
> >
> > --
> > Nathaniel J. Smith -- https://vorpus.org
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nick Coghlan]

On 7 May 2016 13:00, "Nathaniel Smith"  wrote:
>
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thanks for that, and "yikes" on the comment handling variations in
ConfigParser - you can tell I've never even tried to use end-of-line
comments in INI files, and apparently neither has anyone I've worked with :)

For YAML, my main concern isn't quirkiness of the syntax, or code quality
in PyYAML, it's the ease with which you can expose yourself to security
problems (even if *pip* loads the config file safely, that doesn't mean
every other tool will). Since we don't need the extra power, the easiest
way to reduce the collective attack surface is to use a strictly less
powerful (but still sufficient) format.

For ast.literal_eval, we'd still need to come up with a way to do sections,
key:value mappings and define rules for comments.

For completeness, I'll note that XML combines even more user unfriendly
syntax than JSON with similar security risks to YAML.

So with the trade-offs laid out like that (and particularly the
inconsistent comment and Unicode handling in ConfigParser), I'm prompted to
favour following Rust in adopting TOML.

Cheers,
Nick.

P.S. I particularly like the idea of using extension sections to eventually
consolidate other static config into a common file - that nicely addresses
my concern with config file proliferation, since it opens the door to
eventually subsuming other files like MANIFEST.in and setup.cfg as
archiving and build systems are updated

>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Wes Turner]

TOML-LD might work for representing JSONLD, as well.

http://json-ld.org/#developers

* https://github.com/RDFLib/rdflib-jsonld
* https://github.com/digitalbazaar/pyld

JSON-LD as a target makes sense because we're describing nodes (with
attributes) and edges in a package graph.

On Sat, May 7, 2016 at 7:06 AM, Wes Turner  wrote:

> +1 for YAML
>
> YAML-LD (YAML & JSONLD) would make expressing the actual graphs for what
> could be "#PEP426JSONLD" much easier.
>
> https://github.com/pypa/interoperability-peps/issues/31
>
>
> On Saturday, May 7, 2016, Alex Grönholm  wrote:
>
>> +1. I don't think the pathological cases of YAML syntax are of any
>> concern in this context. Plus it has excellent tooling support, unlike TOML.
>>
>> 07.05.2016, 09:25, Fred Drake kirjoitti:
>>
>>> On May 6, 2016, at 10:59 PM, Nathaniel Smith  wrote:
>>>
 Here's that one-stop writeup/comparison of all the major configuration
 languages that I mentioned:

 https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

>>> Thank you for this!  A very nice summary.
>>>
>>> On Fri, May 6, 2016 at 11:14 PM, Donald Stufft  wrote:
>>>
 While I personally prefer YAML to any of the options on a purely syntax
 based
 level, when you weigh in all the other considerations for this I think
 that it
 makes sense to go with TOML for it.

>>> I expect either YAML or TOML would be acceptable, based on this.  I'll
>>> admit that I'd not heard of TOML before, but it warrants consideration
>>> (possibly for some of my projects as well).
>>>
>>> I've spent a fair bit of time using YAML with Ansible lately, as well
>>> as some time looking at RAML, and don't find myself worried about the
>>> syntax.  Every oddness I've run across has been handled with an error
>>> when the content couldn't be parsed correctly, rather than unexpected
>>> behavior resulting from misunderstanding how it would be parsed.  It's
>>> entirely possible I just haven't run across the particular problems
>>> Donald has run across, though.
>>>
>>> (The embedded Jinja2 in Ansible playbooks is another matter; let's not
>>> make that mistake.)
>>>
>>>
>>>-Fred
>>>
>>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Wayne Werner]

On May 6, 2016 10:14 PM, "Donald Stufft"  wrote:
>
> While I personally prefer YAML to any of the options on a purely syntax
based
> level, when you weigh in all the other considerations for this I think
that it
> makes sense to go with TOML for it.

I feel the same way. I use YAML fairly extensively with Salt, and while
most of the basic cases are usually fine, any time I try to do something
advanced, I find that it takes a few tries to get right.

Also note with YAML that whichever library we picked would *also* become a
pip upstream, FWIW.

> The only other option I think that could work is what Chris (I think?)
> suggested and just use a Python literal evaluated using
``ast.literal_eval()``
> this is safe to do but it would make it harder for other languages to
parse our
> files. It's similar to the approach taken by Lua Rocks for how their
packaging
> system works (although their uses variables instead of one big dictionary
which
> I think looks nicer) but Lua is much better suited for trying to execute
safely
> outside of ``ast.literal_eval()`` too.

I'd be interested to see what that option looked like, though I *suspect*
that the vagaries in unicode/non-unicode between 2.x and 3.x may produce
the same weakness that we saw in the ConfigParser option.

-W
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Paul Moore]

On 7 May 2016 at 14:17, Greg Ewing  wrote:
> I don't know if there are conventions for such things on
> Windows. I suspect not, in which case manual input is
> going to be needed one way or another.

There aren't. You typically need to specify the exact locations of all
non-system libraries you use when you link. Judging by the various
complex systems in use (autoconf, pkg_config, ...) I don't think it's
as simple as you're suggesting on Unix either.

The complexities of configuring and building C libraries are a hard
problem, and one that people have been trying to solve for years. It's
not an area we (the Python packaging community) have any interest or
intention of getting involved in. Distutils went there and provided a
solution that did a fantastic job of solving the simpler parts of the
problem[1], but these days, the limitations of that approach are clear
- and we'd much rather enable specialists to build better (or more
specific) solutions and plug them into pip, than (as non-experts) try
to write those solutions for them.

> That's nice, but it wouldn't help me when I encounter
> a package that *hadn't* been set up to use gregsbuild. :-(

And ultimately that's the key issue that pip has to deal with - we
have to provide support for the many thousands of packages on PyPI and
elsewhere (including in closed-source in-house projects that we aren't
even theoretically capable of finding out about) that as you say
haven't been set up to use gregsbuild.

You suggest a declarative way of specifying compilation details. But
I'm not clear how *that* would help when I encounter a project that
hasn't been set up to use that new system, either.
Paul

[1] I was around before distutils, and it's incredibly easy these
days, with such a high awareness of the limitations of distutils, to
forget what a huge improvement it was over the previous situation. I
strongly believe that distutils - and the way it enabled all of
Python's subsequent packaging infrastructure - is one of the key
reasons why Python has become as popular as it is today. The debt we
owe to Greg Ward, who wrote distutils, is huge.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 7 May 2016 01:55, "Chris Barker"  wrote:
> So my point is about scope-creep -- if you want the PyPa tools to solve
all these problems, then you are re-inventing conda -- better to simply
adopt conda (or fork it and fix issues that I'm sure are there)

conda doesn't solve these problems either - it solves the *end user*
problem for data analysts (install the Python library they want to use), by
ignoring the system integrator one (interoperate with the system
integrator's existing platform level package management systems, of which
we all tend to have our own with no plans for consolidation)

That's by design, though - conda was specifically created as a language
independent cross-platform platform, not as a cross-platform plugin
management system for Python runtimes.

For a long time I was convinced the existence of conda and Linux containers
as end user tools meant we wouldn't ever need to solve these problems at
the language runtime layer, but it's since become clear to me that there's
significant value in solving these problems in a way that doesn't care
about how you obtained your Python runtime.

Cheers,
Nick.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Greg Ewing]

Paul Moore wrote:


Do you expect that
projects ... should (somehow) contain simplified instructions on how to
build the various C/Fortran extensions supplied in the bundle as
source code?


Essentially, yes. I'm not sure how achievable it would
be, but ideally that's what I'd like.


would the user need to put all the various details of his system into
a configuration file somewhere (and update that file whenever he
installs a new library, updates his compiler, or whatever)?


On a unix system, most of the time they would all be in
well-known locations. If I install something in an unusual
place or in an unusual way, I'm going to have to tell
something about it anyway. I don't see how an executable
setup file provided by the package author is going to
magically figure out all the weird stuff I've done.

I don't know if there are conventions for such things on
Windows. I suspect not, in which case manual input is
going to be needed one way or another.


How would
this cope with (for example) projects on Windows that *have* to be
compiled with mingw, and not with MSVC?


An option to specify which compiler to use would be a fairly
obvious thing that the config format should provide. As
would a way to include different option settings for
different platforms. Combine them in the obvious way.


This sounds to me more like an attempt to replace the "build" part of
distutils/setuptools with a more declarative system.


Not all of it, only the parts that strictly have to be
performed as part of the build step of the install
process, to use your terminology. That's a fairly
restricted subset of the whole problem of compiling
software.

(Ideally, I would make it *very* restricted, such as
only compiling C code (and possibly C++, not sure).
For anything else you would have to write a C wrapper
or use a tool that generates C. But that might be
a step too far.)

Could a purely declarative config file be flexible
enough to handle this? I don't know. The distutils
API is actually pretty declarative already if you
use it in a straightforward way.

The trouble is that there's nothing to prevent, or
even discourage, writing a setup.py that works in
non-straightforward ways. I've seen some pretty
convoluted setup.py code that worked great when
your system happened to match one of the possibilites
that the author had in mind, but was very difficult
to deal with otherwise. If I'd been able to just
set a few compile options and library paths directly,
it would have been a lot easier.

There's also nothing to prevent the setup.py from
doing things that don't strictly need to be done at
install time. Running Pyrex to generate .c files
from .pyx files is one that I've encountered.
(I encouraged that one myself by including a distutils
extension for it, which I later decided had been a
mistake.)


the proposals currently on the table
would allow you to ask pip to use that build system rather than
setuptools:

>

[buildsystem]
requires=gregsbuild
build_command=gregsbuild --make-wheel


That's nice, but it wouldn't help me when I encounter
a package that *hadn't* been set up to use gregsbuild. :-(

--
Greg
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Wes Turner]

+1 for YAML

YAML-LD (YAML & JSONLD) would make expressing the actual graphs for what
could be "#PEP426JSONLD" much easier.

https://github.com/pypa/interoperability-peps/issues/31

On Saturday, May 7, 2016, Alex Grönholm  wrote:

> +1. I don't think the pathological cases of YAML syntax are of any concern
> in this context. Plus it has excellent tooling support, unlike TOML.
>
> 07.05.2016, 09:25, Fred Drake kirjoitti:
>
>> On May 6, 2016, at 10:59 PM, Nathaniel Smith  wrote:
>>
>>> Here's that one-stop writeup/comparison of all the major configuration
>>> languages that I mentioned:
>>>
>>> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
>>>
>> Thank you for this!  A very nice summary.
>>
>> On Fri, May 6, 2016 at 11:14 PM, Donald Stufft  wrote:
>>
>>> While I personally prefer YAML to any of the options on a purely syntax
>>> based
>>> level, when you weigh in all the other considerations for this I think
>>> that it
>>> makes sense to go with TOML for it.
>>>
>> I expect either YAML or TOML would be acceptable, based on this.  I'll
>> admit that I'd not heard of TOML before, but it warrants consideration
>> (possibly for some of my projects as well).
>>
>> I've spent a fair bit of time using YAML with Ansible lately, as well
>> as some time looking at RAML, and don't find myself worried about the
>> syntax.  Every oddness I've run across has been handled with an error
>> when the content couldn't be parsed correctly, rather than unexpected
>> behavior resulting from misunderstanding how it would be parsed.  It's
>> entirely possible I just haven't run across the particular problems
>> Donald has run across, though.
>>
>> (The embedded Jinja2 in Ansible playbooks is another matter; let's not
>> make that mistake.)
>>
>>
>>-Fred
>>
>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[David Cournapeau]

A missing dimension for comparison: round tripping support. It is quite
useful for formats when used as a configuration. The best I know in that
dimension is yaml (if using ruamel.yaml), which round trip comments.

OTOH, adding round tripping to something like toml should not be too hard
if the need arises.

David

On Sat, May 7, 2016 at 3:59 AM, Nathaniel Smith  wrote:

> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 7 May 2016 at 04:14, Donald Stufft  wrote:
> While I personally prefer YAML to any of the options on a purely syntax based
> level, when you weigh in all the other considerations for this I think that it
> makes sense to go with TOML for it.
>
> The only other option I think that could work is what Chris (I think?)
> suggested and just use a Python literal evaluated using ``ast.literal_eval()``
> this is safe to do but it would make it harder for other languages to parse 
> our
> files. It's similar to the approach taken by Lua Rocks for how their packaging
> system works (although their uses variables instead of one big dictionary 
> which
> I think looks nicer) but Lua is much better suited for trying to execute 
> safely
> outside of ``ast.literal_eval()`` too.
>
> All in all, I think TOML is the right answer (and that's why my partially
> written PEP used TOML).

FWIW (and because obsessing about config formats is a long-running
hobby of mine :-)) my view is:

- JSON sucks as a human-editable format, because it's too strict over
things like commas and has no comments. It's supported by the stdlib,
though, which is nice.
- YAML ought to be wonderful, but it ended up over-engineered (yes, we
can ignore the bits we don't care about). Also, pyYAML is a bit of an
annoying dependency (big, reportedly slow unless you use the C
version) - not something I'd want pip to have to vendor.
- INI (ConfigParser) format is too simple for many use cases. It has
stdlib support, though.
- Python literals are good, but they define values, they aren't a file
format. We'd need to write a parser for "the rest"
- TOML seems pretty good, but is immature (I remember when YAML seemed
like TOML does now...) Also, it's unfamiliar to people (I wasn't aware
of the use in Rust)
- Just for a laugh, can I mention XML? :-)

Overall, *any* choice feels like I'm choosing the "best of a bad job" :-(

With all that said, I'm inclined to agree that TOML looks good.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Paul Moore]

tl;dr version

I think you're right that terminology can be confusing. I think the
definitions people typically work to are:

1. The "packaging" or "release" process - the process (run on a
developer's machine) of creating files that get published for users to
download and install.
2. The "install" process - the process (run on a user's machine) of
taking a published file and making it available in their environment.
This consists of separate steps:
  2a. Optional, and to be avoided wherever possible (by distribution
of wheels) - the "build" step that takes a published file and
configures (compiles) it for the user's environment
  2b. The "install" step (confusion alert! - the "install" step is
only one step of the "install" *process*) that puts the files in the
correct places on the user's machine.

We're not interested in trying to dictate the "packaging" process -
pip isn't involved in that process at all (see flit for a system that
lets projects build releases in a completely different way).

Sigh. Even the tl;dr version is too long :-)

On 7 May 2016 at 01:55, Greg Ewing  wrote:
> Chris Barker wrote:
>>
>> But I think there is consensus here that build systems need to be
>> customisable -- which means arbitrary code may have to be run.
>
> I think different people are using the word "build" in
> different ways here.
>
> To my mind, "building" is what the developer of a package
> does, and a "build system" is what he uses to do it. I
> don't care how much arbitrary code gets run during that
> process.

That is correct, and I agree with you that making a build process like
this declarative is not particularly useful. However...

> But when I do "python setup.py install" or "pip install"
> or whatever the recommended way is going to be, from my
> point of view I'm not "building" the package, I'm
> *installing* it.

Unfortunately, "python setup.py install" does not work that way - it
builds the project and then installs the files. So whether you want to
or not, you're building. That's basically why we're trying to make
"pip install foo" the canonical way of installing packages. So let's
ignore "setup.py install" for the rest of this discussion.

Now, for "pip install foo", *if the foo project provides a wheel
compatible with your system* then what you expect is what you get - a
pure install with no build step.

The problem lies with projects that don't supply wheels, only source.
Or unusual systems that we can't expect projects to have wheels for.
Or local checkouts ("pip install ."). In those cases, it's necessary
to do a build before you can install.

So while we're aiming for 80% or more of the time "pip install" to do
a pure install from a binary distribution, we can't avoid the fact
that occasionally the install will need to run an implicit build step.

> Confusion arises because the process of installation may
> require running a C compiler to generate extension modules.
> But figuring out how to do that shouldn't require
> running arbitrary code supplied by the developer. All the
> tricky stuff should have been done before the package
> was released.

I'm not sure I follow what you are suggesting here. Do you expect that
projects should be able to publish something (it's not exactly a
sdist, but it's not a wheel either as it doesn't contain everything
built) should (somehow) contain simplified instructions on how to
build the various C/Fortran extensions supplied in the bundle as
source code? That's an interesting idea, but how would it work in
practice? The bundles would need to be platform specific, I assume? Or
would the user need to put all the various details of his system into
a configuration file somewhere (and update that file whenever he
installs a new library, updates his compiler, or whatever)? How would
this cope with (for example) projects on Windows that *have* to be
compiled with mingw, and not with MSVC?

> If it's having trouble finding some library or header
> file or whatever on my system, I'd much rather have a
> nice, clear declarative config file that I can edit to
> tell it where to find them, than some overly clever
> piece of python code that's great when it works but
> a pain to unravel when it doesn't.

This sounds to me more like an attempt to replace the "build" part of
distutils/setuptools with a more declarative system. While that may be
a worthwhile goal (I genuinely have no opinion on that) it's largely
orthogonal to the current discussions. Except in the sense that if you
were to build such a system, the proposals currently on the table
would allow you to ask pip to use that build system rather than
setuptools:

# Invented syntax, because syntax is what we *haven't* agreed on yet :-)
[buildsystem]
requires=gregsbuild
build_command=gregsbuild --make-wheel

Then if a user on a system for which the project doesn't have a binary
wheel installed tries to install the project, your build system will
be downloaded, and the "gregsbuild 

Re: [Distutils] comparison of configuration languages

[tritium-list]

I am +1 to TOML; it's INI (a human editable format) with data-types (I think
it is even valid INI).  I find the format pleasant to work with both in the
available libraries and in the editor.


-Original Message-
From: Distutils-SIG
[mailto:distutils-sig-bounces+tritium-list=sdamon@python.org] On Behalf
Of Alex Grönholm
Sent: Saturday, May 7, 2016 4:29 AM
To: distutils-sig@python.org
Subject: Re: [Distutils] comparison of configuration languages

+1. I don't think the pathological cases of YAML syntax are of any 
concern in this context. Plus it has excellent tooling support, unlike TOML.

07.05.2016, 09:25, Fred Drake kirjoitti:
> On May 6, 2016, at 10:59 PM, Nathaniel Smith  wrote:
>> Here's that one-stop writeup/comparison of all the major configuration
>> languages that I mentioned:
>>
>> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
> Thank you for this!  A very nice summary.
>
> On Fri, May 6, 2016 at 11:14 PM, Donald Stufft  wrote:
>> While I personally prefer YAML to any of the options on a purely syntax
based
>> level, when you weigh in all the other considerations for this I think
that it
>> makes sense to go with TOML for it.
> I expect either YAML or TOML would be acceptable, based on this.  I'll
> admit that I'd not heard of TOML before, but it warrants consideration
> (possibly for some of my projects as well).
>
> I've spent a fair bit of time using YAML with Ansible lately, as well
> as some time looking at RAML, and don't find myself worried about the
> syntax.  Every oddness I've run across has been handled with an error
> when the content couldn't be parsed correctly, rather than unexpected
> behavior resulting from misunderstanding how it would be parsed.  It's
> entirely possible I just haven't run across the particular problems
> Donald has run across, though.
>
> (The embedded Jinja2 in Ansible playbooks is another matter; let's not
> make that mistake.)
>
>
>-Fred
>

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Alex Grönholm]

+1. I don't think the pathological cases of YAML syntax are of any 
concern in this context. Plus it has excellent tooling support, unlike TOML.


07.05.2016, 09:25, Fred Drake kirjoitti:

On May 6, 2016, at 10:59 PM, Nathaniel Smith  wrote:

Here's that one-stop writeup/comparison of all the major configuration
languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thank you for this!  A very nice summary.

On Fri, May 6, 2016 at 11:14 PM, Donald Stufft  wrote:

While I personally prefer YAML to any of the options on a purely syntax based
level, when you weigh in all the other considerations for this I think that it
makes sense to go with TOML for it.

I expect either YAML or TOML would be acceptable, based on this.  I'll
admit that I'd not heard of TOML before, but it warrants consideration
(possibly for some of my projects as well).

I've spent a fair bit of time using YAML with Ansible lately, as well
as some time looking at RAML, and don't find myself worried about the
syntax.  Every oddness I've run across has been handled with an error
when the content couldn't be parsed correctly, rather than unexpected
behavior resulting from misunderstanding how it would be parsed.  It's
entirely possible I just haven't run across the particular problems
Donald has run across, though.

(The embedded Jinja2 in Ansible playbooks is another matter; let's not
make that mistake.)


   -Fred



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Fred Drake]

On May 6, 2016, at 10:59 PM, Nathaniel Smith  wrote:
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thank you for this!  A very nice summary.

On Fri, May 6, 2016 at 11:14 PM, Donald Stufft  wrote:
> While I personally prefer YAML to any of the options on a purely syntax based
> level, when you weigh in all the other considerations for this I think that it
> makes sense to go with TOML for it.

I expect either YAML or TOML would be acceptable, based on this.  I'll
admit that I'd not heard of TOML before, but it warrants consideration
(possibly for some of my projects as well).

I've spent a fair bit of time using YAML with Ansible lately, as well
as some time looking at RAML, and don't find myself worried about the
syntax.  Every oddness I've run across has been handled with an error
when the content couldn't be parsed correctly, rather than unexpected
behavior resulting from misunderstanding how it would be parsed.  It's
entirely possible I just haven't run across the particular problems
Donald has run across, though.

(The embedded Jinja2 in Ansible playbooks is another matter; let's not
make that mistake.)


  -Fred

-- 
Fred L. Drake, Jr.
"A storm broke loose in my mind."  --Albert Einstein
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig