Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Ethan Furman 

On 05/09/2016 08:35 PM, Nathaniel Smith wrote:

On Mon, May 9, 2016 at 7:22 PM, Ethan Furman wrote:



After further consideration, and pytoml's author's comment about the spec
changing without a version increase, I think we might be better off
rolling our own.


He's a bit confused -- they didn't change 0.4.0; they made changes in
their dev branch working towards 1.0.0 (some cleanups related to the
date/time stuff I think?"). But of course when you go to github it shows
you the current dev version, and the dev version has a prominent link at
the top to the 0.4.0 tag, so if you're skimming it's easy to misread it
as saying that what you're looking at is 0.4.0.


Ah, okay.  Thanks for clarifying!

--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Chris Barker 
Really?

writing Yet Another Markup Language (YAML :-) ) CAN'T be the simplest, best
option.

> After further consideration, and pytoml's author's comment about the spec
changing without a version increase, I think we might be better off rolling
our own.

>
> I like the general simplicity, and would stick with that, but I'd be a lot
> more comfortable if we had our spec that was more consistent.
>

If we're going to do that, then why not the 'simple part of yaml'.

or Python literals. (if I recall, the main reason not to do that was that
no other language has a lib to read it -- rolling out own does not solve
that!)

Or just go with JSON -- I'm annoyed by it at times, but it's not SO bad.

(and you can kinda-sorta simulate comments with useless keys :-)

{ "comment": "this is just something i wanted to say here",
...
}

or we could do "JSON with comments" -- not hard to write a tiny
pre-processor before passing it off to the json lib.

Anyway -- let's avoid the temptation to role your own everything, and use
something standard!

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Nathaniel Smith 
On Mon, May 9, 2016 at 7:22 PM, Ethan Furman  wrote:
> On 05/09/2016 05:19 PM, Ethan Furman wrote:
>>
>> On 05/07/2016 09:32 AM, Brett Cannon wrote:
>
>
>>> I also checked pytoml at https://github.com/avakar/pytoml and it looks
>>> like it's pretty stable; no changes in the past 5 months except to
>>> support Python 3.5 and only 3 issues. And the format is simple enough
>>> that if someone had to fork the code like Nathaniel suggested or we did
>>> it from scratch it wouldn't be a huge burden.
>
>
> After further consideration, and pytoml's author's comment about the spec
> changing without a version increase, I think we might be better off
rolling
> our own.

He's a bit confused -- they didn't change 0.4.0; they made changes in their
dev branch working towards 1.0.0 (some cleanups related to the date/time
stuff I think?"). But of course when you go to github it shows you the
current dev version, and the dev version has a prominent link at the top to
the 0.4.0 tag, so if you're skimming it's easy to misread it as saying that
what you're looking at is 0.4.0.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Łukasz Langa 
Next thing you know we end up with a new setup.py, with imports, PYTHONPATH 
hacking et al ;-)

> On May 9, 2016, at 7:34 PM, Ethan Furman  wrote:
> 
> I just found this on StackOverflow:
> 
>  http://stackoverflow.com/a/648487/208880
> 
> tl;dr
> -
> 
> > Recently I was working upon a project and I realised that I wanted to
> > have conditionals inside my configuration file [...]
> >
> > I didn't want to write a mini-language, because unless I did it very
> > carefully I couldn't allow the flexibility that would be useful.
> >
> > Instead I decided that I'd have two forms:  If the file started with
> > "#!" and was executable I'd parse the result of running it; otherwise
> > I'd read it as-is
> 
> That approach seems like a win-win: the plain-vanilla static file can be 
> promoted as best-practice, yet we have a fall-back for the complicated and 
> edge cases.
> 
> --
> ~Ethan~
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Ethan Furman 

I just found this on StackOverflow:

  http://stackoverflow.com/a/648487/208880

tl;dr
-

> Recently I was working upon a project and I realised that I wanted to
> have conditionals inside my configuration file [...]
>
> I didn't want to write a mini-language, because unless I did it very
> carefully I couldn't allow the flexibility that would be useful.
>
> Instead I decided that I'd have two forms:  If the file started with
> "#!" and was executable I'd parse the result of running it; otherwise
> I'd read it as-is

That approach seems like a win-win: the plain-vanilla static file can be 
promoted as best-practice, yet we have a fall-back for the complicated 
and edge cases.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Ethan Furman 

On 05/09/2016 05:19 PM, Ethan Furman wrote:

On 05/07/2016 09:32 AM, Brett Cannon wrote:



I also checked pytoml at https://github.com/avakar/pytoml and it looks
like it's pretty stable; no changes in the past 5 months except to
support Python 3.5 and only 3 issues. And the format is simple enough
that if someone had to fork the code  like Nathaniel suggested or we did
it from scratch it wouldn't be a huge burden.


After further consideration, and pytoml's author's comment about the 
spec changing without a version increase, I think we might be better off 
rolling our own.


I like the general simplicity, and would stick with that, but I'd be a 
lot more comfortable if we had our spec that was more consistent.


If there is interest I'll write the PEP and the tool.

--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Donald Stufft 

> On May 9, 2016, at 8:21 PM, Ethan Furman  wrote:
> 
> Um, what does "vendorability" mean?

How hard is it to bundle it with pip by copying the source files into 
pip._vendor.*

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Ethan Furman 

On 05/09/2016 12:30 PM, Barry Warsaw wrote:


Also, I think it makes a lot of sense to go with YAML even if it isn't the
best most readable option.  It's much more common than TOML so the learning
curve will be lessened.


I'd rather learn some new syntax that is readable than be stuck with a 
pain in my eyes and in my brain. ;)


--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Ethan Furman 

On 05/07/2016 04:11 PM, Robert Collins wrote:


Actually, Nathaniel didn't test vendorability of the libraries, and pip
needs that. Pyyaml isn't in good shape there.


Um, what does "vendorability" mean?

--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Ethan Furman 

On 05/07/2016 09:32 AM, Brett Cannon wrote:


+1 for TOML from me as well. I know Paul brought up the lack of
familiarity, but the format is simple and the Rust community is already
fully dependent on it so at worst Rust + us could always just ignore
future format versions if necessary.

If TOML is the chosen format we could ask how long until a 1.0 release
to know if we waited a month or so to implement we could make sure we're
compliant with that version.

I also checked pytoml at https://github.com/avakar/pytoml and it looks
like it's pretty stable; no changes in the past 5 months except to
support Python 3.5 and only 3 issues. And the format is simple enough
that if someone had to fork the code  like Nathaniel suggested or we did
it from scratch it wouldn't be a huge burden.


I would prefer TOML over anything else mentioned so far, and taking over 
pytoml would probably be beneficial, given the author's comments.


--
~Ethan~
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Ethan Furman 

On 05/06/2016 07:59 PM, Nathaniel Smith wrote:


Here's that one-stop writeup/comparison of all the major configuration
languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f


Very nice work-up, thanks!


However, you didn't include XML -- which, while absolutely horrid, can 
be quite readable with the appropriate preprocessor, such as xaml [1] 
:


--- 8< whatever.xaml ---
!!! xml1.0
~base

~schema
// optional
~version: 1

~bootstrap
~requirements
// Temporarily commented out 2016-01-10
// magic-build-helper
~setuptools
~version: >= 27
// for the new frobnicate feature
~numpy
~version: >= 1.10
//Pinned until we get a fix for
// @https://github.com/cyberdyne/the-versionator/issues/123
~the-versionator
~version: 0.13

// The owner of pypi name "flit" decides what goes under the
// extension: flit:
// key
~extensions
~flit
~whatever: true
--- 8< -

which ends up as:

--- 8< whatever.xml 





1






= 27



= 1.10



0.13









true


--- 8< -

--
~Ethan~

[1] https://pypi.python.org/pypi/xaml
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setuptools >= 20.2 may break applications using pyparsing

2016-05-09 Thread Sebastian Krause 
On 09.05.2016, at 15:17, Donald Stufft  wrote:
> This sounds like something you should open as a bug with setuptools, the 
> problem lies with pkg_resources.extern:VendorImporter. Probably it should 
> stop trying to be as tricky and do something more like what pip does here.

Done: https://github.com/pypa/setuptools/issues/580

Sebastian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

2016-05-09 Thread Chris Barker 
On Sun, May 8, 2016 at 5:31 AM, Nick Coghlan  wrote:

> > any python developer is going to
> > run into these issues eventually...
>
> Aye, I know - conda was one of the systems I evaluated as a possible
> general purpose tool for user level package management in Fedora and
> derivatives (see
>
> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement#Directions_to_be_Explored
> for details).
>
> Hence my comment about conda not currently solving *system integrator*
> problems in the general case - it's fine as a platform for running
> software *on top of* Fedora and for DIY integration within an
> organisation, but as a tool for helping to *build Fedora*, it turned
> out to introduce a whole slew of new problems


Sure -- I don't think it was ever intended to support rpm et. all -- it's
to be used INSTEAD of rpm et. all -- for the most part, rpm has solved the
problems that conda is trying to solve -- but only for rpm-based Linux
systems. And I'm going to guess that Continuum didn't want to:

build packages for rpm systems (which ones?)
build packages for deb-based systems (which ones?)
build packages for gentoo
build packages for arch..
.
build packages for homebrew

build packages for cygwin
build packages for Windows.. OOPS, there IS not package manager for
Windows!!

And, AFAICT, none of those package management systems support
"environments", either.

Clearly -- a tool like conda was required to meet Continuum's goals -- and
those goals are very much the same as PyPa's goals, actually. (except the
curated collection of packages part, but conda itself is not about the
Curation...)

However, the kinds of enhancements we're now considering upstream in
> pip should improve things for conda users as well - just as some folks
> in Fedora are exploring the feasibility of automatically rebuilding
> the whole of PyPI as RPMs


yes -- that's a good analogy -- for python packages, conda relies entirely
on distutils/setuptools/pip -- so yes, making those tools better and more
flexible is great.

but I'm still confused about what "the kinds of enhancements we're now
considering upstream in pip" are.

Here are a few:

More flexibility about the build system used
Easier to get metadata without actually initiating a build

These are great!

But I started this whole line of conversation because it seemed that there
was desire for:

Ability to isolate the build environment.
Ability to better handle/manage non-python dependencies

These are what I was referring to as mission-creep, and that overlap with
conda (and other tools).


> (twice, actually, anchored on
> /usr/bin/python and /usr/bin/python3), it should eventually be
> feasible to have the upstream->conda pipeline fully automated as well.


yeah -- there's been talk for ages of automatically building conda packages
(on the fly, maybe) from PyPi packages. But currently on conda-forge we've
decided to NOT try to do that -- it's turned out in practice that enough
pypi packages end up needing some hand-tweaking to build. So teh planned
workflow is now:

Auto-build a conda build script for a PyPi package
Test it
Tweak it as required
Add it to conda-forge.

Then -- *maybe* write a tool that auto-updates the PyPi based packages in a
chron job or whatever.

So not quite a automated conda-PyPi bridge, but not a bad start.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Barry Warsaw 
On May 08, 2016, at 09:05 AM, Robert Collins wrote:

>E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
>as pybuild is a thing in the debian space, and this will confuse the
>heck out of folk). https://wiki.debian.org/Python/Pybuild

Yes, please don't call it pybuild ;)

Also, I think it makes a lot of sense to go with YAML even if it isn't the
best most readable option.  It's much more common than TOML so the learning
curve will be lessened.

Cheers,
-Barry


pgpcWJAr9wuKi.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Brett Cannon 
On Mon, 9 May 2016 at 11:21 Chris Barker  wrote:

> "pymeta" feels very "inessentially weird" to me [1].
>
>
> yeah...
>
> setup.toml
>
> ???
>

You can all stop guessing at file names. The PEP will have a recommendation
and you all can either agree or disagree at that point. Please don't give
me more names to list in the rejected section. :)

-Brett


>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Chris Barker 
>
> "pymeta" feels very "inessentially weird" to me [1].


yeah...

setup.toml

???

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

2016-05-09 Thread Chris Barker 
On Sun, May 8, 2016 at 5:49 AM, Nick Coghlan  wrote:

> The reason I think "bootstrap" is a better name at this point


I *really* don't want to add to the bike-shedding of the name at this point
-- I really don't care.

I was just trying to see if I was misunderstanding something, as it didn't
seem to be bootstrapping anything to me, and I see I was kinda missing some
of the scope and complexity -- thanks.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Brett Cannon 
On Mon, 9 May 2016 at 06:29 Nick Coghlan  wrote:

> On 9 May 2016 at 01:43, Donald Stufft  wrote:
> > Overall, my suggestion here would be to have a file called
> ``pymeta.toml`` (or
> > ``meta.toml``)
>
> pymeta.toml would be fine by me.
>
> I don't really buy the "collision with Debian build tool" argument
> against "pybuild" (if I did, I'd be objecting to "pymeta" colliding
> with an existing PyPI package), so it's mainly the fact the metadata
> in this file covers more than just building has soured me on it.
>
> > and have it look like::
> >
> > [dependencies]
> > build = [
> > "setuptools",
> > "wheel>=1.0",
> > ]
> >
> > If at some point we decide we need to add a bootstrap_requires (and then
> the
> > ability to add dynamic build requirements) we can do that by just saying
> that
> > if you plan on having dynamic build requirements, you need to omit the
> build
> > key under the [dependencies] section.
>
> Looking at my previous ideas for semantic dependencies in PEP 426,
> what if we start in the near term by defining development
> requirements?
>
> That can then be used to hold arbitrary development dependencies
> (metadata generation, sdist creation, test execution, wheel build,
> docs build, etc), everything that you need to work on, build, and test
> the software, but don't need if you just want to run the published
> releases.
>
> We may later decide that we want to break that down and separate out
> specific requirements for sdist creation and for wheel creation, but
> we can handle that by saying that if there's no more specific
> dependency definition for an operation, then the tools will fall back
> to pre-installing all the listed development dependencies.
>
> That is, someone might write:
>
>  [dependencies]
>  develop = [
>  "setuptools",
>  "wheel>=1.0",
>  "sphinx",
>  "pytest",
>]
>
> And it would be useful not only for running setup.py commands, but
> setting up their local dev environment in general.
>

I'm not going to touch the concept of development dependencies to keep the
PEP focused.

But one thing this discussion does bring up is everyone is assuming there
is simply going to be a [dependencies] section to the configuration file
where build, development, and/or installation dependencies will eventually
end up.

My current plan for the PEP is to not do that. Instead, I'm going to have a
[build] section that will have a `dependencies` field. My thinking is that
we don't need to have any end-users who choose to look at this file care
about anything related to building or developing a project. Plus if we
expose any details about the Future Awesome Build API (aka the FAB API ;)
then it should probably be kept near all of the other build-related
details, including the dependencies for building instead of inherently
spanning multiple sections (yes, tools will quite possibly have their own
section, but we can try to keep it to [build] and e.g. [tools.setuptools]
and not [dependencies], [build], and [tools.setuptools]).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

2016-05-09 Thread Brett Cannon 
On Mon, 9 May 2016 at 06:42 Nick Coghlan  wrote:

> On 7 May 2016 at 08:21, Paul Moore  wrote:
> > On 6 May 2016 at 19:14, Brett Cannon  wrote:
> >> OK, assuming the Nick will be pronouncing, who wants to write the PEP?
> >
> > ... and if Nick doesn't want to pronounce, I'm willing to offer to be
> > BDFL for this one. But a PEP is the first thing. (And IMO the key
> > point of the PEP is to be very clear on what is in scope and what
> > isn't - the discussions have covered a *lot* of ground and being clear
> > on what's excluded will be at least as important as stating what's in
> > scope).
>
> Answering this specifically: I'm happy to be the arbiter-of-consensus
> for this one, as my schedule's pretty clear right now (at least until
> I head to PyCon US on the 27th).
>

OK, I'll list Nick as the BDFL Delegate in the PEP then. Hoping to have a
draft to send to the list today or tomorrow.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Nathaniel Smith 
On Mon, May 9, 2016 at 6:28 AM, Nick Coghlan  wrote:
> On 9 May 2016 at 01:43, Donald Stufft  wrote:
>> Overall, my suggestion here would be to have a file called ``pymeta.toml`` 
>> (or
>> ``meta.toml``)
>
> pymeta.toml would be fine by me.
>
> I don't really buy the "collision with Debian build tool" argument
> against "pybuild" (if I did, I'd be objecting to "pymeta" colliding
> with an existing PyPI package), so it's mainly the fact the metadata
> in this file covers more than just building has soured me on it.

Re: filename bikeshedding:

"pymeta" feels very "inessentially weird" to me [1]. This file is
going to front and center for newcomers, many of whom will never have
encountered the word "metadata" and especially not the hacker fetish
for the "meta" morpheme. I like meta-things in general! But I don't
like the image of trying to explain what a "pymeta" is over and over
and over again when teaching :-).

pymetadata would be better, but it seems like there must be something
less jargony available?

pypackage
pypackaging
pydevelop
pysource
pytools
pysettings
...?

Or if we're really daring and wasteful of characters, I guess we could
even go for something like python-tools.toml :-)

-n

(Tangent, but I'll write it down so I don't forget when we circle back
to the question of adding config for third-party tools into this
thing: [tool.flit], [tool.coverage] is probably a lot more obvious to
newcomers than [extension.flit], [extension.coverage]!)


[1] https://www.crummy.com/sumana/2014/08/10/1

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Nathaniel Smith 
On Mon, May 9, 2016 at 6:52 AM, Nick Coghlan  wrote:
> On 9 May 2016 at 23:38, Donald Stufft  wrote:
>>
>>> On May 9, 2016, at 9:28 AM, Nick Coghlan  wrote:
>>>
>>> Looking at my previous ideas for semantic dependencies in PEP 426,
>>> what if we start in the near term by defining development
>>> requirements?
>>
>> I think the biggest reason not to do this, but instead do something like 
>> build
>> requirements is that development dependencies is already reasonably well
>> addressed in a way that something other than setuptools can access it using
>> extras. It's not as great as a dedicated key for it, but it works pretty OK.
>> The thing that is really painful is setup_requires and how it forces you to
>> delay importing until *during* the execution of the setup() function. We 
>> could
>> try and lump setup_requires and development dependencies together, but that
>> seems less than optimal to me. Unless someone's setup.py uses pytest, I'm not
>> sure I see a reason for pytest to be installed anytime pip builds that 
>> project.
>>
>> A more concrete example would be pyca/cryptography, which has a development
>> dependency that consists of 27MB of data which was purposely kept separate 
>> from
>> cryptography itself so as not to incur an additional 27MB of download just to
>> install cryptography.
>
> OK, that makes sense to me as a rationale for prioritising build
> dependencies over more general dev environment dependencies.
>
> To feel confident we haven't overlooked a near term need by doing
> that, I think the main question I'd like to see the PEP explicitly
> cover is "when will these new requirements be implicitly installed?"
>
> I currently believe the answers to that are:
>
> 1. When pip* is asked to install from an sdist
> 2. When pip* is asked to install from a VCS URL
> 3. When pip* is asked to install from a local directory
> 4. When pip* is asked to create a wheel file for any of those
>
> * - or another compatible installer
>
> If those answers are correct, then the metadata consumer will always
> be proceeding on to do a build in the use cases we currently care
> about, so it's OK that it may also be relying on those build
> requirements to generate metadata or create an sdist archive.

I think what we want is precisely, "what stuff do you need before you
can invoke setup.py", and in the future that will be replaced with
"what stuff do you need before you can invoke [our awesome
hypothetical build system abstraction interface TBD, which will
default to a wrapper around setup.py]".

We all agree that there will be some sort of "build system" concept,
and that invoking a build system is something that requires running
arbitrary code, some of which might come from PyPI.

It remains to be determined exactly what the boundary of that "build
system" concept is (e.g., it definitely will have 'build a wheel'; not
as obvious whether it will have 'build an sdist' or 'run tests').

But I think for now that's fine -- for now we can just say that
eventually we will fill in the details of this interface, and the
point of this part of the config file is that it lets you bootstrap
into whatever that ends up being.

(Notice that both the PEP 516/517 proposals actually distinguish
between bootstrap-requires, which are what you need before invoking
the build system, and build-requires, which is a dynamic hook for
querying the build system to ask it what else it needs before it can
build a wheel. Example of where the wheel-build requirements might
vary at runtime: when building an extension that needs a C library,
your build system might provide envvar that lets you pick between 'I'm
building an RPM, so please just link to the system-provided copy of
this library' versus 'I'm building a wheel, so please install the
relevant pynativelib package -- making it a build requirement -- and
then link to that'. I think we should focus narrowly on the
bootstrapping problem right now, because it's not yet clear whether
things like test dependencies actually make more sense as static
metadata or as something you query the build system for dynamically.
And a single static list of build-system-requirements is already
enough to bandage over most of the really egregious setup_requires
problems.)

So I'd suggest:

[build-system]
requires = [ ... ]
# Later we will add something like:
# entry-point = ...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Nick Coghlan 
On 9 May 2016 at 23:38, Donald Stufft  wrote:
>
>> On May 9, 2016, at 9:28 AM, Nick Coghlan  wrote:
>>
>> Looking at my previous ideas for semantic dependencies in PEP 426,
>> what if we start in the near term by defining development
>> requirements?
>
> I think the biggest reason not to do this, but instead do something like build
> requirements is that development dependencies is already reasonably well
> addressed in a way that something other than setuptools can access it using
> extras. It's not as great as a dedicated key for it, but it works pretty OK.
> The thing that is really painful is setup_requires and how it forces you to
> delay importing until *during* the execution of the setup() function. We could
> try and lump setup_requires and development dependencies together, but that
> seems less than optimal to me. Unless someone's setup.py uses pytest, I'm not
> sure I see a reason for pytest to be installed anytime pip builds that 
> project.
>
> A more concrete example would be pyca/cryptography, which has a development
> dependency that consists of 27MB of data which was purposely kept separate 
> from
> cryptography itself so as not to incur an additional 27MB of download just to
> install cryptography.

OK, that makes sense to me as a rationale for prioritising build
dependencies over more general dev environment dependencies.

To feel confident we haven't overlooked a near term need by doing
that, I think the main question I'd like to see the PEP explicitly
cover is "when will these new requirements be implicitly installed?"

I currently believe the answers to that are:

1. When pip* is asked to install from an sdist
2. When pip* is asked to install from a VCS URL
3. When pip* is asked to install from a local directory
4. When pip* is asked to create a wheel file for any of those

* - or another compatible installer

If those answers are correct, then the metadata consumer will always
be proceeding on to do a build in the use cases we currently care
about, so it's OK that it may also be relying on those build
requirements to generate metadata or create an sdist archive.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

2016-05-09 Thread Nick Coghlan 
On 7 May 2016 at 08:21, Paul Moore  wrote:
> On 6 May 2016 at 19:14, Brett Cannon  wrote:
>> OK, assuming the Nick will be pronouncing, who wants to write the PEP?
>
> ... and if Nick doesn't want to pronounce, I'm willing to offer to be
> BDFL for this one. But a PEP is the first thing. (And IMO the key
> point of the PEP is to be very clear on what is in scope and what
> isn't - the discussions have covered a *lot* of ground and being clear
> on what's excluded will be at least as important as stating what's in
> scope).

Answering this specifically: I'm happy to be the arbiter-of-consensus
for this one, as my schedule's pretty clear right now (at least until
I head to PyCon US on the 27th).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Donald Stufft 

> On May 9, 2016, at 9:28 AM, Nick Coghlan  wrote:
> 
> Looking at my previous ideas for semantic dependencies in PEP 426,
> what if we start in the near term by defining development
> requirements?

I think the biggest reason not to do this, but instead do something like build
requirements is that development dependencies is already reasonably well
addressed in a way that something other than setuptools can access it using
extras. It's not as great as a dedicated key for it, but it works pretty OK.
The thing that is really painful is setup_requires and how it forces you to
delay importing until *during* the execution of the setup() function. We could
try and lump setup_requires and development dependencies together, but that
seems less than optimal to me. Unless someone's setup.py uses pytest, I'm not
sure I see a reason for pytest to be installed anytime pip builds that project.

A more concrete example would be pyca/cryptography, which has a development
dependency that consists of 27MB of data which was purposely kept separate from
cryptography itself so as not to incur an additional 27MB of download just to
install cryptography.

I think this *could* make sense, if we could reasonably assume that the 95%
case would always be handled by wheels, but I don't think that we can. A lot of
projects have compiled C code as part of them, and as soon as you do that you
end up where installation from sdist is one of your main supported methods (yes
Wheel will cover a lot of them, but only for the most popular platforms).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

2016-05-09 Thread Nick Coghlan 
On 9 May 2016 at 01:43, Donald Stufft  wrote:
> Overall, my suggestion here would be to have a file called ``pymeta.toml`` (or
> ``meta.toml``)

pymeta.toml would be fine by me.

I don't really buy the "collision with Debian build tool" argument
against "pybuild" (if I did, I'd be objecting to "pymeta" colliding
with an existing PyPI package), so it's mainly the fact the metadata
in this file covers more than just building has soured me on it.

> and have it look like::
>
> [dependencies]
> build = [
> "setuptools",
> "wheel>=1.0",
> ]
>
> If at some point we decide we need to add a bootstrap_requires (and then the
> ability to add dynamic build requirements) we can do that by just saying that
> if you plan on having dynamic build requirements, you need to omit the build
> key under the [dependencies] section.

Looking at my previous ideas for semantic dependencies in PEP 426,
what if we start in the near term by defining development
requirements?

That can then be used to hold arbitrary development dependencies
(metadata generation, sdist creation, test execution, wheel build,
docs build, etc), everything that you need to work on, build, and test
the software, but don't need if you just want to run the published
releases.

We may later decide that we want to break that down and separate out
specific requirements for sdist creation and for wheel creation, but
we can handle that by saying that if there's no more specific
dependency definition for an operation, then the tools will fall back
to pre-installing all the listed development dependencies.

That is, someone might write:

 [dependencies]
 develop = [
 "setuptools",
 "wheel>=1.0",
 "sphinx",
 "pytest",
   ]

And it would be useful not only for running setup.py commands, but
setting up their local dev environment in general.

(Having docs build and test dependencies listed would be entirely
acceptable for the RPM spec autogeneration case, since we just have
the BuildRequires/Requires split at the RPM level, and often need the
docs dependency to generate man pages, and the test dependencies to
run the unit tests in the %check scriptlet)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setuptools >= 20.2 may break applications using pyparsing

2016-05-09 Thread Donald Stufft 

> On May 9, 2016, at 8:37 AM, Sebastian Krause  wrote:
> 
> Hi,
> 
> I'm developing an application that uses pyparsing and after upgrading 
> setuptools to the newest version I noticed some tests failing. In my main 
> parser module I define an alias for the ParseBaseException which I then use 
> in other parts of the application to catch exceptions:
> 
> # definition of the ParseException
> ParseException = pyparsing.ParseBaseException
> 
> # importing this alias in another module
> from ...filterreader.parser import ParseException
> 
> Now my tests were failing because the ParseException was never actually 
> caught. Some investigation by comparing the id() of the objects showed that 
> the ParseException alias was no longer the same object as 
> pyparsing.ParseBaseException. This was because the module "pyparsing" at the 
> time of the alias definition was not the same "pyparsing" module which is 
> later used for parsing. Looking at sys.module I can see that I have two 
> pyparsing modules:
> 
> pyparsing:  'D:\\Project\\python34.zip\\pyparsing.pyc'>
> pkg_resources.extern.pyparsing:  'D:\\Project\\python34.zip\\pyparsing.pyc'>
> 
> At the time of the alias definition id(pyparsing) is equal to the id() of 
> pkg_resources.extern.pyparsing. When I later import pyparsing I get the other 
> module. This whole problem only happens when I use the application packaged 
> by cx_Freeze, so maybe some kind of race condition happens when importing 
> from a ZIP file. I'm using 64 bit Python 3.4.4 on Windows.
> 
> The first version of setuptools where I can see this problem is 20.2, until 
> 20.1 everything is fine. Looking at the source I can see that starting with 
> 20.2 setuptools also includes its own pyparsing copy, so most likely that 
> change is related to my problem.
> 
> Is there a simple way in which I can guarantee that there will only ever be a 
> single "pyparsing" module in my application? Of course I could just stop 
> using the alias and use the pyparsing exceptions directly, but I feel a bit 
> uneasy when a module just changes its identity at some point between imports.
> 
> Sebastian
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


This sounds like something you should open as a bug with setuptools, the 
problem lies with pkg_resources.extern:VendorImporter. Probably it should stop 
trying to be as tricky and do something more like what pip does here.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] setuptools >= 20.2 may break applications using pyparsing

2016-05-09 Thread Sebastian Krause 

Hi,

I'm developing an application that uses pyparsing and after upgrading 
setuptools to the newest version I noticed some tests failing. In my 
main parser module I define an alias for the ParseBaseException which I 
then use in other parts of the application to catch exceptions:


# definition of the ParseException
ParseException = pyparsing.ParseBaseException

# importing this alias in another module
from ...filterreader.parser import ParseException

Now my tests were failing because the ParseException was never actually 
caught. Some investigation by comparing the id() of the objects showed 
that the ParseException alias was no longer the same object as 
pyparsing.ParseBaseException. This was because the module "pyparsing" at 
the time of the alias definition was not the same "pyparsing" module 
which is later used for parsing. Looking at sys.module I can see that I 
have two pyparsing modules:


pyparsing: 'D:\\Project\\python34.zip\\pyparsing.pyc'>
pkg_resources.extern.pyparsing: 'D:\\Project\\python34.zip\\pyparsing.pyc'>


At the time of the alias definition id(pyparsing) is equal to the id() 
of pkg_resources.extern.pyparsing. When I later import pyparsing I get 
the other module. This whole problem only happens when I use the 
application packaged by cx_Freeze, so maybe some kind of race condition 
happens when importing from a ZIP file. I'm using 64 bit Python 3.4.4 on 
Windows.


The first version of setuptools where I can see this problem is 20.2, 
until 20.1 everything is fine. Looking at the source I can see that 
starting with 20.2 setuptools also includes its own pyparsing copy, so 
most likely that change is related to my problem.


Is there a simple way in which I can guarantee that there will only ever 
be a single "pyparsing" module in my application? Of course I could just 
stop using the alias and use the pyparsing exceptions directly, but I 
feel a bit uneasy when a module just changes its identity at some point 
between imports.


Sebastian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] ez_setup.py can not get setuptools

2016-05-09 Thread Benedek Zoltan via Distutils-SIG 
Hi,
Thanks for all the comments. I tried the get-pip.py script and installs 
successfully setuptools and pip.
Usually I installed virtualenv and setuptools from the install scripts: 
virtualenv.py, ez_setup.py and pip from pip-*.*.*.tar.gz.I liked this approach, 
because in this way I didn't affect the system by any of those packages, 
everything remained in the created virtualenv.
Now I have to change only my install script in order to use get-pip.py
Best regardsZoltan


  From: Donald Stufft 
 To: tritium-l...@sdamon.com 
Cc: Chris Barker ; Benedek Zoltan ; 
distutils-sig@python.org
 Sent: Friday, May 6, 2016 7:43 PM
 Subject: Re: [Distutils] ez_setup.py can not get setuptools
   


On May 6, 2016, at 12:31 PM, tritium-l...@sdamon.com wrote:
If you don’t have setuptools, you don’t have pip.

Not true anymore, pip is perfectly capable of running and installing things 
without setuptools now days. The only time you *need* setuptools installed is 
if you’re installing from a sdist (and setuptools has wheels, so you can 
install setuptools with pip withouth setuptools already being installed).
-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


  ___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig