Re: [Distutils] moving things forward

[Greg Ewing]

Chris Barker wrote:
But I think there is consensus here that build systems need to be 
customisable -- which means arbitrary code may have to be run.


I think different people are using the word "build" in
different ways here.

To my mind, "building" is what the developer of a package
does, and a "build system" is what he uses to do it. I
don't care how much arbitrary code gets run during that
process.

But when I do "python setup.py install" or "pip install"
or whatever the recommended way is going to be, from my
point of view I'm not "building" the package, I'm
*installing* it.

Confusion arises because the process of installation may
require running a C compiler to generate extension modules.
But figuring out how to do that shouldn't require
running arbitrary code supplied by the developer. All the
tricky stuff should have been done before the package
was released.

If it's having trouble finding some library or header
file or whatever on my system, I'd much rather have a
nice, clear declarative config file that I can edit to
tell it where to find them, than some overly clever
piece of python code that's great when it works but
a pain to unravel when it doesn't.

--
Greg
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nathaniel Smith]

On Fri, May 6, 2016 at 8:14 PM, Donald Stufft  wrote:
[...]
> The only other option I think that could work is what Chris (I think?)
> suggested and just use a Python literal evaluated using ``ast.literal_eval()``

Oh, good point, that should definitely be on the list of options
considered regardless of whether we go for it. I won't add it now
because I just switched to Windows to play games with my wife and my
emacs is on the other partition, but I'll add it later.

Does anyone know how literal_eval handles Unicode on py2?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Donald Stufft]

> On May 6, 2016, at 10:59 PM, Nathaniel Smith  wrote:
> 
> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
> 
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
> 
> -n
> 
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


While I personally prefer YAML to any of the options on a purely syntax based
level, when you weigh in all the other considerations for this I think that it
makes sense to go with TOML for it.

The only other option I think that could work is what Chris (I think?)
suggested and just use a Python literal evaluated using ``ast.literal_eval()``
this is safe to do but it would make it harder for other languages to parse our
files. It's similar to the approach taken by Lua Rocks for how their packaging
system works (although their uses variables instead of one big dictionary which
I think looks nicer) but Lua is much better suited for trying to execute safely
outside of ``ast.literal_eval()`` too.

All in all, I think TOML is the right answer (and that's why my partially
written PEP used TOML).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] comparison of configuration languages

[Nathaniel Smith]

Here's that one-stop writeup/comparison of all the major configuration
languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

[Nathaniel Smith]

On Fri, May 6, 2016 at 11:14 AM, Brett Cannon  wrote:
>
>
> On Fri, 6 May 2016 at 09:40 Donald Stufft  wrote:
>>
>>
>> On May 6, 2016, at 12:36 PM, Brett Cannon  wrote:
>>
>> So who is the BDFL on this decision? It seems we need someone to stop the
>> bikeshedding on the field name and what file is going to house this
>> configuration data. And do we need someone to write a PEP for this proposal
>> to have something to target?
>>
>>
>> We need someone to write the PEP and someone to offer to be BDFL for it.
>> For this particular change the default would be Nick for BDFL but if he’s
>> busy someone else can take it over for this PEP. Though I think we need
>> someone writing an actual PEP first.
>
>
> OK, assuming the Nick will be pronouncing, who wants to write the PEP?

I've just been writing up a comparison of the different file formats,
partly in case it's useful to others and partly just for my own use in
looking at them against each other and seeing how much it actually
matters. This might also be reusable for the
rationale/rejected-alternatives section of a PEP, if anyone wants it,
or I could go ahead and add a few paragraphs to turn it into a proper
PEP.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

[Paul Moore]

On 6 May 2016 at 19:14, Brett Cannon  wrote:
> On Fri, 6 May 2016 at 09:40 Donald Stufft  wrote:
>>
>>
>> On May 6, 2016, at 12:36 PM, Brett Cannon  wrote:
>>
>> So who is the BDFL on this decision? It seems we need someone to stop the
>> bikeshedding on the field name and what file is going to house this
>> configuration data. And do we need someone to write a PEP for this proposal
>> to have something to target?
>>
>>
>> We need someone to write the PEP and someone to offer to be BDFL for it.
>> For this particular change the default would be Nick for BDFL but if he’s
>> busy someone else can take it over for this PEP. Though I think we need
>> someone writing an actual PEP first.
>
>
> OK, assuming the Nick will be pronouncing, who wants to write the PEP?

... and if Nick doesn't want to pronounce, I'm willing to offer to be
BDFL for this one. But a PEP is the first thing. (And IMO the key
point of the PEP is to be very clear on what is in scope and what
isn't - the discussions have covered a *lot* of ground and being clear
on what's excluded will be at least as important as stating what's in
scope).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Fri, May 6, 2016 at 9:39 AM, Donald Stufft  wrote:

> On May 6, 2016, at 11:54 AM, Chris Barker  wrote:
>
> So my point is about scope-creep -- if you want the PyPa tools to solve
> all these problems, then you are re-inventing conda -- better to simply
> adopt conda (or fork it and fix issues that I'm sure are there….)
>
>
> Adopting Conda is unlikely to ever happen for the defaulting tooling.
>

I didn't put the emphasis right in that sentence -- I was not actually
suggesting that conda be adopted -- what I was suggesting was the we DON'T
expand the scope of pip et al to be able to do everything that conda does.

That being said, I think there are some misunderstanding here that may be
relevant to this discussion:


> The problems that the default tooling attempt to solve are significantly
> harder than the problems that Conda attempts to solve
>

kinda-sorta -- conda is a packaging system. period, end of story (actually
it is an isolated environment system, too -- is that packaging? )-- it is
NOT a build system. Right now, you can invoke pip to:

* Find and install a binary package (wheel) -- this conda does
* Find, download, build and install install a source package -- nope, conda
doesn't do anything like that.
* Build and install a local package from source
* Install a local package from source in "develop mode" (editable mde) --
actually conda does have that -- though I'm not sure who well it works, of
it it's python specific.

But the fact that this all is (apparently) done by one tool is actually a
problem -- we have a tangled mess of binary and source and building and
installing, and it isn't obvious to the user what they are getting ) often
they don't care -- as long as it works :-) ) And, under the hood, it's all
driven by setuptools, which also has duplicate and slightly different
functionality that can be accidentally invoked (fun with easy_install!).

So I thought the goal now was to clean up an untangle this mess, yes?


and switching to it would be a regression.
>

only in the sense the switching to pip (and getting rid of setuptools)
would be a regression


> The primary benefit of Conda’s packages over Wheels are that they have a
> curated repository with people who are ensuring that things build correctly
> and since they don’t rely on authors to do it correctly, they don’t have to
> wait for, or convince authors to do this fresh new thing.
>

That simply is not true. Let's not mix social issues from technical ones.
Continuum set out to create a curated set of packages, yes -- but they
didn't put the enormous amount of work into conda for no reason -- they did
it because pip + wheel didn't meet their needs (and it doesn't meet a lot
of our needs, either). And, as I understand it, they came to the Python
community, and said something along the lines of "the packaging tooling
doesn't meet the needs of the scientific community, can we contribute
changes, improvements" -- and were told something along the lines of: "We
don't want the standard tooling to solve those problems -- go make what you
need, and then maybe we'll talk"

(I can dig out the message from Travis Oliphant about this if you like)

And they did go out and make something else, and they have released it as
an open source project, and a number of folks are adopting it to solve
their problems, too.


> The problem is, none of those benefits are something that would apply if
> we decided to throw away the 588,074 files that are currently able to be
> installed on PyPI.
>

conda aside, I think there is a real hang-up here. yes, of course you don't
want to throw away all that -- we absolutely need to maintain the current
packaging, building structure to support existing projects. But that
doesn't mean something new and better can't be built to be used in parallel
-- as folks move forward with pacakge development, they can adopt the new
system or they can keep using teh same old setup.py they have. MAybe
decades later something will get depreciated. But this approach is what got
us in the ugly mess to begin with.


Us deciding that Conda is the thing to use isn’t going to magically create
> an army of volunteers to go through and take all 80,000 packages on PyPI
> and ensure that we get a correctly compiled package on every platform for
> each version. If we could do that we could just convince everyone to go out
> and build binary packages, we could just do that with Wheels without
> requiring forklifting an entire ecosystem.
>

Again, conda packages and wheels are not the same thing -- there are
technical differences, and those differences really are why continuum uses
it for Anaconda, and why a lot of folks are adopting it for community led
efforts, like conda-forge. And there are a LOT of packages on conda-forge
despite it being very new and only a small handful of people contributing.

https://conda-forge.github.io/feedstocks.html


> While wheels are optimized for the pure 

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

[Brett Cannon]

On Fri, 6 May 2016 at 09:40 Donald Stufft  wrote:

>
> On May 6, 2016, at 12:36 PM, Brett Cannon  wrote:
>
> So who is the BDFL on this decision? It seems we need someone to stop the
> bikeshedding on the field name and what file is going to house this
> configuration data. And do we need someone to write a PEP for this proposal
> to have something to target?
>
>
> We need someone to write the PEP and someone to offer to be BDFL for it.
> For this particular change the default would be Nick for BDFL but if he’s
> busy someone else can take it over for this PEP. Though I think we need
> someone writing an actual PEP first.
>

OK, assuming the Nick will be pronouncing, who wants to write the PEP?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Ethan Furman]

On 05/06/2016 09:48 AM, Leonardo Rochael Almeida wrote:

On 6 May 2016 at 13:15, Chris Barker wrote:



"python literals" is perfectly well defined -- both by the language
reference, and by "can be parsed by ast.literal_eval" and it addresses

>> the limitations of JSON and is fully declarative.


There is actually prior art for that kind of use. Odoo uses such a
language for its addons system, including package dependencies. See
example file here:

https://github.com/OCA/maintainer-tools/blob/master/template/module/__openerp__.py

Notice the `depends` key, that lists other addons, and the
`external_dependencies` key that can list both python distribution
dependencies as well as external program dependencies.


This is one of the very few things Odoo got right.  Let's not look to 
them for other examples of good coding practices.


--
~Ethan~

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Leonardo Rochael Almeida]

On 6 May 2016 at 13:15, Chris Barker  wrote:

> On Fri, May 6, 2016 at 5:41 AM, Nick Coghlan  wrote:
>
>> [...]
>
>
>
>> So rather than saying "the bootstrapping dependency declaration file
>> is Python-but-not-really", it's easier to say "it's an ini-file format
>> that can be parsed with the configparser module" or "it's JSON" (I'm
>> ruling out any options that don't have a stdlib parser in Python 2.7)
>>
>
> Last time, I promise :-)
>
> "python literals" is perfectly well defined -- both by the language
> reference, and by "can be parsed by ast.literal_eval"
>

> and it addresses the limitations of JSON and is fully declarative.
>

There is actually prior art for that kind of use. Odoo uses such a language
for its addons system, including package dependencies. See example file
here:

https://github.com/OCA/maintainer-tools/blob/master/template/module/__openerp__.py

Notice the `depends` key, that lists other addons, and the
`external_dependencies` key that can list both python distribution
dependencies as well as external program dependencies.

ISTM the reluctance in adopting this idea comes from the desire of using
other programming languages to parse the bootstraping information (imagine
an sdist -> deb converter implemented in go. It will have to exec python
eventually, during the build process, but not before it has prepared the
build environment).

And even though, unlike JSON, the ConfigParser format of setup.cfg is not
officially defined anywhere, there are sufficiently compatible ini parsers
in other languages that the idea still has merit.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] ez_setup.py can not get setuptools

[Donald Stufft]

> On May 6, 2016, at 12:31 PM, tritium-l...@sdamon.com wrote:
> 
> If you don’t have setuptools, you don’t have pip.

Not true anymore, pip is perfectly capable of running and installing things 
without setuptools now days. The only time you *need* setuptools installed is 
if you’re installing from a sdist (and setuptools has wheels, so you can 
install setuptools with pip withouth setuptools already being installed).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

[Donald Stufft]

> On May 6, 2016, at 12:36 PM, Brett Cannon  wrote:
> 
> So who is the BDFL on this decision? It seems we need someone to stop the 
> bikeshedding on the field name and what file is going to house this 
> configuration data. And do we need someone to write a PEP for this proposal 
> to have something to target?


We need someone to write the PEP and someone to offer to be BDFL for it. For 
this particular change the default would be Nick for BDFL but if he’s busy 
someone else can take it over for this PEP. Though I think we need someone 
writing an actual PEP first.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Donald Stufft]

> On May 6, 2016, at 11:54 AM, Chris Barker  wrote:
> 
> So my point is about scope-creep -- if you want the PyPa tools to solve all 
> these problems, then you are re-inventing conda -- better to simply adopt 
> conda (or fork it and fix issues that I'm sure are there….)


Adopting Conda is unlikely to ever happen for the defaulting tooling. The 
problems that the default tooling attempt to solve are significantly harder 
than the problems that Conda attempts to solve and switching to it would be a 
regression. The primary benefit of Conda’s packages over Wheels are that they 
have a curated repository with people who are ensuring that things build 
correctly and since they don’t rely on authors to do it correctly, they don’t 
have to wait for, or convince authors to do this fresh new thing.

The problem is, none of those benefits are something that would apply if we 
decided to throw away the 588,074 files that are currently able to be installed 
on PyPI. Us deciding that Conda is the thing to use isn’t going to magically 
create an army of volunteers to go through and take all 80,000 packages on PyPI 
and ensure that we get a correctly compiled package on every platform for each 
version. If we could do that we could just convince everyone to go out and 
build binary packages, we could just do that with Wheels without requiring 
forklifting an entire ecosystem.

While wheels are optimized for the pure Python case, there is nothing 
preventing them from being used to install anything else (Erlang or R or even 
Python itself). The pynativelib stuff is proof of the ability to do just that— 
and in distutils-sig land we tend to care a lot more about how these things 
will affect our downstream consumers like Debian and such than Conda needs to 
(or should need to!).

Now, this isn’t to say that Conda is bad or anything, but it’s use as a 
replacement for the current ecosystem is about as interesting as suggesting we 
all adopt RPM, or apt-get, or Choclatey, or Ports, or Pacman or whatever flavor 
of downstream you wish to insert here.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

[Brett Cannon]

The emails seem to have reached an equilibrium point of bikeshedding on the
(bootstrap|setup)_requires issue that is being discussed (as Daniel points
out below, this has nothing to do with how building works and instead is
only about statically declaring what tools need to be installed to simply
run your builder to do whatever the heck it wants; this is the first baby
step to build decoupling/customization).

So who is the BDFL on this decision? It seems we need someone to stop the
bikeshedding on the field name and what file is going to house this
configuration data. And do we need someone to write a PEP for this proposal
to have something to target?

On Thu, 5 May 2016 at 13:54 Daniel Holth  wrote:

> This is a recurring point of confusion. setup.py is not ever going away.
> In general it is necessary for you to be able to write software to build
> your software, and there is no intention to take that feature away.
>
> People repeatedly come to the conclusion that static metadata means the
> entire build is static. It's only the dependencies that need to be static
> to enable better dependency resolution in pip. The build does not need to
> be static.
>
> The proposed feature means you will be able to have a simpler setup.py or
> no setup.py it by using something like flit or pbr that are configured with
> .cfg or .ini. setup.py is not going away.
>
> Static metadata means the list of dependencies, author name, trove
> classifiers is static. Not the build itself.
>
> Enforced staticness of the build script is right out.
>
> On Thu, May 5, 2016 at 4:34 PM Alex Grönholm 
> wrote:
>
>> I think it would be best to gather a few extreme examples of setup.py
>> files from real world projects and figure out if they can be implemented in
>> a declarative fashion. That at least would help us identify the pain points.
>>
>> For starters, gevent's setup.py looks like it needs a fair bit of custom
>> logic:
>> https://github.com/gevent/gevent/blob/master/setup.py
>>
>> 05.05.2016, 23:30, Chris Barker kirjoitti:
>>
>> On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan  wrote:
>>
>>
>>> This configuration vs customisation distinction is probably worth
>>> spelling out for folks without a formal software engineering or
>>> computer science background, so:
>>>
>>
>> fair enough -- good to be clear on the terms.
>>
>>
>>> Configuration is different: you're choosing amongst a set of
>>> possibilities that have been constrained in some way, and those
>>> constraints are structurally enforced.
>>
>>
>> That's a key point here -- I guess I'm skeptical that we can have the
>> flexibility we need with a purely configuration-based system -- we probably
>> don't WANT to constrain the options completely. If you think about it,
>> while distutils has it's many, many flaws, what made it possible for it to
>> be as useful as it is, and last as long as it has because is CAN be
>> customized -- users are NOT constrained to the built-in functionality.
>>
>> I suspect the idea of this thread is to keep the API to a build system
>> constrained -- and let the build systems themselves be as customizable as
>> the want to be. And I haven't thought it out carefully, but I have a
>> feeling that we're going to hit a wall that way .. but maybe not.
>>
>>
>>> Usually that enforcement is
>>> handled by making the configuration declarative - it's in some passive
>>> format like an ini file or JSON, and if it gets too repetitive then
>>> you introduce a config generator, rather than making the format itself
>>> more sophisticated.
>>>
>>
>> OK -- that's more or less my thought -- if it's  python that gets run,
>> then you've got your config generator built in -- why not?
>>
>>
>>
>>> The big advantage of configuration over customisation is that you
>>> substantially increase the degrees of freedom in how *consumers* of
>>> that configuration are implemented - no longer do you need a full
>>> Python runtime (or whatever), you just need an ini file parser, or a
>>> JSON decoder, and then you can look at just the bits you care about
>>> for your particular use case and ignore the rest.
>>>
>>
>> Sure -- but do we care? this is about python packaging -- is it too big a
>> burden to say you need python to read the configuration?
>>
>> -CHB
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR(206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115   (206) 526-6317   main reception
>>
>> chris.bar...@noaa.gov
>>
>>
>>
>> ___
>> Distutils-SIG maillist  -  
>> Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
> 

Re: [Distutils] ez_setup.py can not get setuptools

[tritium-list]

If you are using ez_setup in your setup.py, presumably you have guarded against 
the presence of setuptools in the target environment.  If you don’t have 
setuptools, you don’t have pip.

 

From: Distutils-SIG 
[mailto:distutils-sig-bounces+tritium-list=sdamon@python.org] On Behalf Of 
Chris Barker
Sent: Friday, May 6, 2016 12:17 PM
To: Benedek Zoltan 
Cc: distutils-sig@python.org
Subject: Re: [Distutils] ez_setup.py can not get setuptools

 

ez_setup.py is pretty darn old.

 

Any reason you can't:

 

python -m pip install setuptools

 

?

 

-CHB

 

 

On Fri, May 6, 2016 at 12:11 AM, Benedek Zoltan via Distutils-SIG 
 > wrote:

Hi,

 

I don't know what happened recently. Usually I install setuptools by a script 
using the ez_setup.py script.

 

Recently I get an error:

 

Downloading 
https://pypi.python.org/packages/source/s/setuptools/setuptools-21.0.0.zip

Traceback (most recent call last):

  File "downloads/ez_setup.py", line 415, in 

sys.exit(main())

  File "downloads/ez_setup.py", line 411, in main

archive = download_setuptools(**_download_args(options))

  File "downloads/ez_setup.py", line 336, in download_setuptools

downloader(url, saveto)

  File "downloads/ez_setup.py", line 287, in download_file_insecure

src = urlopen(url)

  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen

return opener.open(url, data, timeout)

  File "/usr/lib/python3.4/urllib/request.py", line 469, in open

response = meth(req, response)

  File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response

'http', request, response, code, msg, hdrs)

  File "/usr/lib/python3.4/urllib/request.py", line 507, in error

return self._call_chain(*args)

  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain

result = func(*args)

  File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default

raise HTTPError(req.full_url, code, msg, hdrs, fp)

urllib.error.HTTPError: HTTP Error 404: Not Found

 

For now I can copy the package from an old virtualenv, but I'd appreciate a 
better solution/advise.

 

Thanks

Zoltan


___
Distutils-SIG maillist  -  Distutils-SIG@python.org 
 
https://mail.python.org/mailman/listinfo/distutils-sig





 

-- 


Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov  

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Fri, May 6, 2016 at 5:41 AM, Nick Coghlan  wrote:

> The "Python-with-imports" case is the status quo with setup.py, and we
> already know that's a pain because you need to set up an environment
> that already has the right dependencies installed to enable your
> module level imports in order to run the script and find out what
> dependencies you need to install to let you run the script in the
> first place.
>

good point -- this is really key.

The "Python-without-imports" approach would just be confusing -


I agree -- I never suggested that -- it's full python or fully declarative.


> So rather than saying "the bootstrapping dependency declaration file
> is Python-but-not-really", it's easier to say "it's an ini-file format
> that can be parsed with the configparser module" or "it's JSON" (I'm
> ruling out any options that don't have a stdlib parser in Python 2.7)
>

Last time, I promise :-)

"python literals" is perfectly well defined -- both by the language
reference, and by "can be parsed by ast.literal_eval"

and it addresses the limitations of JSON and is fully declarative.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Fri, May 6, 2016 at 5:55 AM, Nick Coghlan  wrote:

> > And maybe it's good to keep "new style" configuration clearly separate.
>
> Part of my motivation for suggesting re-using setup.cfg is the
> proliferation of packaging related config sprawl in project root
> directories - setup.py won't be going anywhere any time soon for most
> projects, some folks already have a setup.cfg (e.g. to specify
> universal wheel creation), and there's also MANIFEST.in to control
> sdist generation.


yeah -- ugly, but will  one more file make a difference? now that I think
about it -- IIUC the goals here, we want to separate packaging from
building -- I'd argue that setuup.cfg is about building -- we really should
keep the package configuration  separate. Some day, some how, we hoping to
have a new (or multiple build systems) they won't use setup.cfg to
configure the build. So we probably shouldn't marry ourselves to setup.cfg
for package configuration.

The other issue is social -- if we stick with setup.cfg, we are implying
that this is all about tweaking distutils/setuptools, not doing something
different -- seemingly keeping the mingling of building and packaging
forevermore

But this isn't that big a deal -- enough bike-shedding, time to make a
declaration.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Thu, May 5, 2016 at 6:37 PM, Robert Collins 
wrote:

>
> Thats good. It occurs to me that scientific builds may be univerally
> broken because folk want to avoid easy-install and the high cost of
> double builds of things. E.g. adding bootstrap_requires will let folk
> fix things?
>

scientific builds are universally "broken" because their dependencies are
often non-python (and oftne complex and hard to build) external libs.

that is simply a problem that setuptools and pip were never designed to
address.


> But the main question is : why are these packages staying inaccurate?
> Even in the absence of a systematic solution I'd expect bug reports
> and pull requests to converge on good dependencies.
>

the problem is not that people haven't declared the proper dependencies in
their packages -- it's because it's impossible to declare the proper
dependencies in their packages...

The solution is to do one of three things:

1) punt -- and let folks use Canopy, or conda, or third party collections
of wheels like the Gohlke repository, or let folks build it themselves (a
lot of packages have all kinds of custom code to go look in macports, or
homebrew, or fink locations for libs on OS-X for instance.

2) build binary wheels that statically link in everything needed (or
include the dlls in the package)

3) make binary wheels out of C libs, so that other packages can depend on
them --  this is kind of a kludgy hack (abuse?) of pip/wheel but should be
do-able.

Some folks are making a valiant effort to do a combination of (2) and (3)
for Windows, OS-X, and now many linux -- but it is too bad that the tools
don't make it easy. And so far, all we really have are the core scipy stack
(and pyQT5???)

In short -- building fully supported binary packages is possible, but
requires a fair bit of expertise and a valiant effort on the part of
someone...

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Thu, May 5, 2016 at 4:32 PM, Nathaniel Smith  wrote:

> > You do know that we're on our way to re-writing conda, now, don't you?
> :-)
> >
> > I think we need to be careful of scope-creep...
>
> conda did not invent the idea of creating separate python environments
> for different tasks :-)


I'm not suggesting conda invented anything -- I'm suggesting it has
implemented many of the things being talked about here -- the truth is
conda was designed to solve exactly the problems that scientific python
packages had that pip+wheel+setuptools do not currently solve.

So my point is about scope-creep -- if you want the PyPa tools to solve all
these problems, then you are re-inventing conda -- better to simply adopt
conda (or fork it and fix issues that I'm sure are there)

On the other hand, improving the PyPa tools while maintaining their current
scope is a lovely idea -- but that means leaving isolation of build
environments etc, to external tools: Docker, VMs, conda..

Actually, conda has a good lesson here: conda is about packaging, NOT
building -- I was quite disappointed that conda provided NO support for
cross platform building at all -- but after using it a bit I realized that
that was a great decision -- if you want to support a wide variety of
packages, you really need to let the package authors use whatever the heck
build system they want -- you really don't want to have to implement (or
even debug) everyone else's builds.

And IIUC, that is the direction we are trying to go with pip now -- making
pip+wheel build-system independent -- good goal.

Which actually give me an idea: it seems we are very bogged down in
struggles with keeping backward compatibility, and coming up with a API for
an arbitrary build system. Maybe we can take a lesson from conda and
essentially punt:

conda works by reading a meta.yaml -- a purely declarative package
configuration. The actual building is done by calling a build script --
conda itself does not need to know or care what the heck is in that build
script -- all it needs to know is that it can be run. Why not keep the API
that simple for pip?

we could do total backward compatible, and simply say: pip will call
"python setup.py install".

And that's it, end of story -- individual packages could do ANYTHING in
that setup.py script -- current packages would be using setuptools, but pip
wouldn't need to care at all what was in there -- only know that it would
install the package.

And, I suppose, a setup.py bdist_wheel for building wheels, and setup.py
sdist for creating source distributions), though I'm not sure that those
need to be a standardized API -- or at least not the SAME standardized API.


-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Chris Barker]

On Thu, May 5, 2016 at 10:45 PM, Greg Ewing 
wrote:


> Even if python is available, you might not want to run
> arbitrary code just to install a package.
>
> If a config file can contain executable code, then it
> can contain bugs. Debugging is something the developer of
> a package should have to do, not the user. In my experience,
> fixing someone else's buggy setup.py is about as much fun
> as pulling one's own teeth out with a blunt screwdriver.


well, in my experience, debugging my OWN buggy setup.py is about equally
pleasant!

But I think there is consensus here that build systems need to be
customisable -- which means arbitrary code may have to be run. So I don't
think this is going to help anyone avoiding dealing with buggy builds :-(

or are we talking about a config file that would be delivered with a binary
wheel? IN which case, yes, it should not have any executable code in it.

And anyway, it seems folks want to go with static config anyway -- clearly
separating configuration from customization, so I'll stop now.

I'd still like to be able to use Python literals, though :-)

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Things that are not pip-installable (was Re: moving things forward) shouldn't)

[Cosimo Lupo]

On 5 May 2016 at 09:00, Marius Gedminas  wrote:

> pip install pyqt5



You need Python 3.5, and you also need to ensure you are calling the `pip`
command for Python 3.5, and not the default `pip` which may be linked to a
different Python version.
Try this for example:

python3.5 -m pip install --user pyqt5
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] ez_setup.py can not get setuptools

[Benedek Zoltan via Distutils-SIG]

Hi,
I don't know what happened recently. Usually I install setuptools by a script 
using the ez_setup.py script.
Recently I get an error:
Downloading 
https://pypi.python.org/packages/source/s/setuptools/setuptools-21.0.0.zipTraceback
 (most recent call last):  File "downloads/ez_setup.py", line 415, in   
  sys.exit(main())  File "downloads/ez_setup.py", line 411, in main    archive 
= download_setuptools(**_download_args(options))  File "downloads/ez_setup.py", 
line 336, in download_setuptools    downloader(url, saveto)  File 
"downloads/ez_setup.py", line 287, in download_file_insecure    src = 
urlopen(url)  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen 
   return opener.open(url, data, timeout)  File 
"/usr/lib/python3.4/urllib/request.py", line 469, in open    response = 
meth(req, response)  File "/usr/lib/python3.4/urllib/request.py", line 579, in 
http_response    'http', request, response, code, msg, hdrs)  File 
"/usr/lib/python3.4/urllib/request.py", line 507, in error    return 
self._call_chain(*args)  File "/usr/lib/python3.4/urllib/request.py", line 441, 
in _call_chain    result = func(*args)  File 
"/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default    
raise HTTPError(req.full_url, code, msg, hdrs, fp)urllib.error.HTTPError: HTTP 
Error 404: Not Found
For now I can copy the package from an old virtualenv, but I'd appreciate a 
better solution/advise.
ThanksZoltan___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 6 May 2016 at 06:41, Chris Barker  wrote:
> On Wed, May 4, 2016 at 8:09 PM, Nick Coghlan  wrote:
>> 3. The ongoing popularity of setup.cfg shows that while ini-style may
>> not be perfect for this use case, it clearly makes it over the
>> threshold of "good enough"
>
> it's only popular because it's what's there -- if we're using that standard,
> we could make the same argument about setuptools ;-)

That's exactly the course change we made following the release of
Python 3.3 - rather than trying to replace setuptools directly, the
scope of our ambitions was narrowed to eliminating the need to run
"./setup.py install", while keeping setuptools as the default build
system used to enable things like generating wheel files ("./setup.py
bdist_wheel" doesn't work if you're importing setup from
distutils.core - you have to be importing it from setuptools. If you
have a setup.py file that imports from distutils.core, you have to use
"pip wheel" if you want to get a wheel file)

>> So when I ask myself now "What's the *simplest* thing we could do that
>> will make things better than the status quo?", then the answer I come
>> up with today is your original idea: bless setup.cfg (or at least a
>> subset of it) as a standardised interface.
>
> IIUC, we would be changin, or at least adding to the current setup.dfg spec.
> So this is a change, no matter how you slice it, saying "configuration will
> be specified in setup.something, in some other format, is the least
> significant part of all this change.
>
> And maybe it's good to keep "new style" configuration clearly separate.

Part of my motivation for suggesting re-using setup.cfg is the
proliferation of packaging related config sprawl in project root
directories - setup.py won't be going anywhere any time soon for most
projects, some folks already have a setup.cfg (e.g. to specify
universal wheel creation), and there's also MANIFEST.in to control
sdist generation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 6 May 2016 at 06:30, Chris Barker  wrote:
> On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan  wrote:
>> Usually that enforcement is
>> handled by making the configuration declarative - it's in some passive
>> format like an ini file or JSON, and if it gets too repetitive then
>> you introduce a config generator, rather than making the format itself
>> more sophisticated.
>
>
> OK -- that's more or less my thought -- if it's  python that gets run, then
> you've got your config generator built in -- why not?

The immediate reason is because Python allows imports, and if imports
are permitted in the config script, people will use them, and if
they're not permitted, they'll complain about their absence.

The "Python-with-imports" case is the status quo with setup.py, and we
already know that's a pain because you need to set up an environment
that already has the right dependencies installed to enable your
module level imports in order to run the script and find out what
dependencies you need to install to let you run the script in the
first place.

The "Python-without-imports" approach would just be confusing - while
it would avoid the dependency bootstrapping problem, it would only be
kinda-sorta-Python rather than actual Python.

So rather than saying "the bootstrapping dependency declaration file
is Python-but-not-really", it's easier to say "it's an ini-file format
that can be parsed with the configparser module" or "it's JSON" (I'm
ruling out any options that don't have a stdlib parser in Python 2.7)

The "future benefit" reason is that it's a lot easier to be confident
that reading a config file from an uploaded artifact isn't going to
compromise a web service, so a future version of PyPI can readily pull
information out of the config file and republish it via an API. Once
you have that kind of information available via an API, you can
resolve it before downloading *anything* (which is especially useful
when your goal is dependency graph analysis rather than downloading
the whole of PyPI and running a Python script from every package you
downloaded).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig