Re: [Distutils] comparison of configuration languages

[Chris Barker]

On Fri, May 13, 2016 at 11:22 AM, Brett Cannon  wrote:


> No need to think; the decision is made and it's TOML. I know Chris doesn't
> mean to stir up trouble,
>

I got a bit out of sync with the conversation -- sorry for the noise.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Chris Barker]

On Fri, May 13, 2016 at 1:09 PM, Nathaniel Smith  wrote:

> But, the plan *is* to make wheels the standard way to build packages --
> that will be in the next pep :-). I'm not sure I'd call it "lock down",
> because there's nothing that will stop you running setup.py bdist_rpm or
> whatever. But our goal is to reach the point where package authors get a
> choice of what build system to use, and there's no guarantee that every
> build system will implement bdist_rpm.
>
> hmm -- this really feels like mingling packaging and building.

Does making sure everything builds a wheel help systems like rpm and the
like? Honestly I have no idea.

I do know that conda is very is very much designed to not care at all how a
package is build or installed, as long as it can be installed -- so if a
wheel is built and then that wheel is installed, that all the same to conda.

But is that the case for everything else?

I absolutely agree that we shouldn't expect a bdist_rpm and the like -- in
fact, those should all be deprecated.

but maybe a "install" that goes from source to installed package, without
passing through a wheel?

or maybe not -- I really don't know rpm or deb or anything else well enough
to know.

> So, the plan is to require all build systems to be able to output wheels,
> and then debian or conda-build or whoever will convert the wheel into
> whatever final package format they want.
>
easy for conda -- not sure about the others

hmm -- homebrew builds from source, so as long as you have a way to install
the wheel you built, it'll be fine (much like conda, but without ever
making a package)

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Nathaniel Smith]

On May 13, 2016 11:34 AM, "Chris Barker"  wrote:
>
> One other question:
>
> Is it just examples or is "build" being defined as "build a wheel"?
>
> i.e. there are times one might want to build a package without building a
wheel -- just it install it yourself, or to put it in another package
format -- conda, rpm, what have you.
>
> In conda's case, building a wheel, and then installing it would work
fine, but I'm not sure we want to lock that down as the only way to build a
package.

As Brett already clarified, this pep is just about how you get to the point
of being able to start the build system; it doesn't care what the build
system actually outputs.

But, the plan *is* to make wheels the standard way to build packages --
that will be in the next pep :-). I'm not sure I'd call it "lock down",
because there's nothing that will stop you running setup.py bdist_rpm or
whatever. But our goal is to reach the point where package authors get a
choice of what build system to use, and there's no guarantee that every
build system will implement bdist_rpm.

So, the plan is to require all build systems to be able to output wheels,
and then debian or conda-build or whoever will convert the wheel into
whatever final package format they want. This is way more scalable than
requiring N different build systems to each be able to output M different
formats for N*M code paths. And if wheels aren't sufficient, well, we can
add stuff to the spec :-)

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Brett Cannon]

On Fri, 13 May 2016 at 11:34 Chris Barker  wrote:

> One other question:
>
> Is it just examples or is "build" being defined as "build a wheel"?
>

Just an example (clarified previously).

-Brett


>
> i.e. there are times one might want to build a package without building a
> wheel -- just it install it yourself, or to put it in another package
> format -- conda, rpm, what have you.
>
> In conda's case, building a wheel, and then installing it would work fine,
> but I'm not sure we want to lock that down as the only way to build a
> package.
>
> Granted, if all it means is that someone will download an unnecessary
> dependency, big deal.
>
> I'm also a bit confused about whether we're trying to specify the
> dependencies required simply to run the build tool itself, or the
> dependencies required to actually do the build -- or the latter being saved
> for another day?
>

The minimal requirements to execute the build system. Providing some way to
specify more dependencies in some dynamic fashion and the like is for
another PEP.

-Brett


>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Chris Barker]

One other question:

Is it just examples or is "build" being defined as "build a wheel"?

i.e. there are times one might want to build a package without building a
wheel -- just it install it yourself, or to put it in another package
format -- conda, rpm, what have you.

In conda's case, building a wheel, and then installing it would work fine,
but I'm not sure we want to lock that down as the only way to build a
package.

Granted, if all it means is that someone will download an unnecessary
dependency, big deal.

I'm also a bit confused about whether we're trying to specify the
dependencies required simply to run the build tool itself, or the
dependencies required to actually do the build -- or the latter being saved
for another day?

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Brett Cannon]

On Fri, 13 May 2016 at 10:47 Paul Moore  wrote:

> On 13 May 2016 at 18:15, Chris Barker  wrote:
> > I think we're freaking out way too much about what *could* go wrong.
>
> It's more that:pip would have to vendor pyyaml, and it's not small. I
> have no idea whether it's easy to vendor, either (does it have
> separate code paths for Python 2 and 3? I don't know, I've never
> looked). Also, we'd have to forego the C implementation - I have no
> idea how much of a performance hit that would give (of course,
> performance is hardly a key issue here) or how likely it is that bugs
> exist in the Python version that aren't normally noticed because
> people use the C implementation (which is definitely just speculation,
> conceded).
>
> But I think the decision is made, so it's not worth debating any more,
> honestly.
>

No need to think; the decision is made and it's TOML. I know Chris doesn't
mean to stir up trouble, but at this point if someone wants to propose
something other than TOML they are going to have to write their own PEP.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Nathaniel Smith]

On Fri, May 13, 2016 at 10:15 AM, Chris Barker  wrote:
> On Tue, May 10, 2016 at 1:54 AM, Paul Moore  wrote:
>>
>> I would love to use YAML. I really would. But for pip, we need a
>> robust, easy to vendor Python implementation
>
>
> conda has been using yaml forever (with pyyaml) , and whatever problem is
> has (and there are many), I don't think I've ever seen a single issue cause
> by yaml or pyyaml. Might be worth asking the conda devs.

"vendor" here means "munge the library so that it becomes a
pure-python subpackage of pip named pip._vendor.whatever". This is a
specific technical challenge that conda doesn't attempt...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Paul Moore]

On 13 May 2016 at 18:15, Chris Barker  wrote:
> I think we're freaking out way too much about what *could* go wrong.

It's more that:pip would have to vendor pyyaml, and it's not small. I
have no idea whether it's easy to vendor, either (does it have
separate code paths for Python 2 and 3? I don't know, I've never
looked). Also, we'd have to forego the C implementation - I have no
idea how much of a performance hit that would give (of course,
performance is hardly a key issue here) or how likely it is that bugs
exist in the Python version that aren't normally noticed because
people use the C implementation (which is definitely just speculation,
conceded).

But I think the decision is made, so it's not worth debating any more, honestly.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

On Tue, May 10, 2016 at 1:54 AM, Paul Moore  wrote:

> I would love to use YAML. I really would. But for pip, we need a
> robust, easy to vendor Python implementation


conda has been using yaml forever (with pyyaml) , and whatever problem is
has (and there are many), I don't think I've ever seen a single issue cause
by yaml or pyyaml. Might be worth asking the conda devs.

I think we're freaking out way too much about what *could* go wrong.

Oh, and why not "JSON with comments and trailing commas" - it would be well
defined and easy to implement.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] devpi-server-4.0: fixing the pip-8.1.2 issue / pep 503 compliance

[holger krekel]

devpi-server-4.0: fixing the pip-8.1.2 problem / PEP503 compliance


We've made available critically important releases of the devpi private 
packaging
available.  If you are not using "devpi" yet then you can may just read
http://doc.devpi.net  and forget about the rest of this announcement.

This is for the many who experienced the "pip doesn't install packages
anymore with devpi" problem.  First of all, you may temporarily pin "pip" 
to avoid the problem on the client side:
 
pip install pip==8.1.1

This is obviously a crutch but gives you some time to perform the
export/import cycle required for hosting private packages via
devpi-server-4.0 and being compatible with pip-8.1.2.

If you are using devpi-server as a pure pypi.python.org cache you don't 
need to perform export/import and can just delete your server directory 
($HOME/.devpi/server by default) before you install and start up 
devpi-server-4.0.

If you are hosting private packages on devpi you will need to perform an
export/import cycle of your server state in order to run devpi-server-4.0.
The "4.0" in this case only signals this export/import need -- no other
big changes are coming with 4.0.  At the end of this announcement we explain 
some details of why we needed to go for a 4.0 and not just a micro bugfix 
release.


To export from devpi-server-3.X


upgrade to the new devpi-server-3.1.2 before you export, like this:

pip install "devpi-server<4.0" 

Now stop your server and run:

devpi-server --export EXPORTDIR --serverdir SERVERDIR

where EXPORTDIR should be a fresh new directory and SERVERDIR
should be the server state directory ($HOME/.devpi/server by default).


To export from devpi-server-2.X


Upgrade to the latest devpi-server-2.X release:

pip install "devpi-server<3.0" devpi-common>=2.0.10

Here we force the devpi-common dependency to not accidentally
be "devpi-common==2.0.9" which could lead to problems.

Now stop your server and run:

devpi-server --export EXPORTDIR --serverdir SERVERDIR

where EXPORTDIR should be a fresh new directory and SERVERDIR
should be the server state directory ($HOME/.devpi/server by default).


to import state into devpi-server-4.0


Upgrade to the latest devpi-server-4.X release:

pip install "devpi-server<5.0" devpi-web

If you don't use "devpi-web" you can leave it out from the pip command.
Check you have the right version:

devpi-server --version

Now import from your previously created EXPORTDIR:

devpi-server --serverdir SERVERDIR_NEW --import EXPORTDIR

This will take a while if you have many indexes or lots of documentation --
devpi-web will create a search index over all of it during import.

You are now good to go -- pip works again!


devpi-client also has a 2.6.3


We also published a minor bugfix "devpi-client-2.6.3" release which
should work with both devpi-server-2.6 and devpi-server-4.0 as we
are generally trying to keep devpi-client forward/backward compatible.
You only need to install devpi-client-2.6.3 if you also install
devpi-server into the same virtual environment.  Otherwise using
devpi-client-2.6.2 with both devpi-server-2.6 and devpi-server-4.0
probably works fine as well.

background on the pip/devpi bug for the curious
---

Besides devpi, also artifactory and other private index servers
have experienced failures with pip-8.1.2.  The change from 8.1.1
was that pip now asks for PEP503-normalized names when requesting
the simple page from an index.  Previously "-" and "." would be
allowed but with the new normalization "." is substituted with "-".
Now "pip install zope.interface" triggers a request to 
"+simple/zope-interface" and devpi in turns asks 
pypi.python.org/simple/zope-interface and gets an answer
with lots of "zope.interface-*.tar.gz" release links. But those
are not matched because without PEP503 "zope.interface" and "zope-interface"
are different things.  Moreover, pypi.python.org used to redirect 
to the "true" name but does not do this anymore which contributed
to the eventual problem.

We decided to go for 4.0 because since 3.0 we base database
keys on normalized project names -- and this normalization is
used in like 20-30 code places across the devpi system and plugins.
Trying to be clever and avoid the export/import and trick "pip-8.1.2"
into working looked like a can of worms.  Now with devpi-server-4.0
we are using proper PEP503 specified normalization so should be safe.

best,
holger and florian

P.S.: we offer support contracts btw and thank in particular
Dolby Laboratories, YouGov Inc and BlueYonder GmbH who funded a lot of
the last year's devpi work and now agreed to be named in public - and
no, we didn't get around to make a flashy web site yet.  For now,
just mail 

Re: [Distutils] PEP for specifying build dependencies

[Brett Cannon]

On Thu, 12 May 2016 at 20:52 Nick Coghlan  wrote:

> On 13 May 2016 at 02:33, Brett Cannon  wrote:
> > Both Donald and Nathaniel say to drop it, and since I put it in just to
> be
> > overly cautious I'm fine with dropping it. So unless Nick says
> > "semantics-version or death!", I agree w/ my co-authors and would update
> the
> > PEP to say:
> >
> > 1. no semantics-version
> > 2. [build-system] instead of [package.build-system]
> >
>
> I think both of those changes are improvements - having to change the
> filename is a reasonable disincentive against making breaking changes,
> and with just the two sections initially being defined ([build-system]
> and [tool.*]) it makes sense to have them both at the top level.
>

Seems everyone is in agreement then. I'll update the PEP and send out a new
draft later today.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Does pip Honour "Obsoletes"?

[Ian Cordasco]

On Thu, May 12, 2016 at 5:40 PM, Phil Thompson
 wrote:
> I may be doing something wrong, but...
>
> The METADATA in my wheel uses Obsoletes but pip does not remove the obsoleted 
> package on an install of my wheel.
>
> Is it supposed to?

No.

Unlike some other package managers, pip doesn't keep track of every
package that depends on another, so it will not remove the obsoleted
package unless you specifically do `pip uninstall obsoleted-package`.
This prevents silent/unintentional breakage.

Cheers,
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig