Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-21 Thread Chris Barker
On Wed, May 20, 2015 at 5:43 PM, Nick Coghlan ncogh...@gmail.com wrote: One of my hopes for the metadata extension system in PEP 426 is that we'll be able to define extensions like fedora.repackage, debian.repackage or conda.repackage which include whatever additional info is needed to

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-20 Thread Nick Coghlan
On 20 May 2015 at 15:38, David Cournapeau courn...@gmail.com wrote: On Wed, May 20, 2015 at 2:25 PM, Nick Coghlan ncogh...@gmail.com wrote: On 18 May 2015 at 14:32, David Cournapeau courn...@gmail.com wrote: On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan ncogh...@gmail.com wrote:

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-20 Thread Daniel Holth
The conda package specification is published at http://conda.pydata.org/docs/spec.html The file format is nice and simple. A conda package is a bzipped tar archive (.tar.bz2) which contains metadata under the info/ directory, and a collection of files which are installed directly into an install

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-20 Thread Paul Moore
On 20 May 2015 at 15:53, David Mertz dme...@continuum.io wrote: It's not *only* the 'setup.py install', but it's not *that* much mystery either. wxPython I can't seem to find, not sure what I'm missing. Yeah, I had been under the impression that there was a lot of knowledge on how to build the

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-20 Thread Chris Barker
On Wed, May 20, 2015 at 6:30 AM, Daniel Holth dho...@gmail.com wrote: The package includes its build recipe in info/recipe very cool -- I hadn't seen that -- I'll go take a look at some packages and see what I can find. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-20 Thread Chris Barker
The package includes its build recipe in info/recipe very cool -- I hadn't seen that -- I'll go take a look at some packages and see what I can find. Darn -- the recipe is not there in most (all?) of the packages that came from Anaconda -- probably due to the legacy issues David referred

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-20 Thread Wes Turner
On Wed, May 20, 2015 at 12:13 PM, Chris Barker chris.bar...@noaa.gov wrote: The package includes its build recipe in info/recipe very cool -- I hadn't seen that -- I'll go take a look at some packages and see what I can find. Darn -- the recipe is not there in most (all?) of the packages

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-20 Thread Nick Coghlan
On 21 May 2015 at 05:05, Wes Turner wes.tur...@gmail.com wrote: On Wed, May 20, 2015 at 12:13 PM, Chris Barker chris.bar...@noaa.gov wrote: The package includes its build recipe in info/recipe very cool -- I hadn't seen that -- I'll go take a look at some packages and see what I can

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-20 Thread Nick Coghlan
On 20 May 2015 at 23:30, Daniel Holth dho...@gmail.com wrote: It occurs to me that the setuptools packaging in general is more like a shared library format .so or .dll, aka libraries searched for along a path, than an OS level package manager. Yep, that was what PJE was after for Chandler, so

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-20 Thread Wes Turner
On May 20, 2015 7:43 PM, Nick Coghlan ncogh...@gmail.com wrote: On 21 May 2015 at 05:05, Wes Turner wes.tur...@gmail.com wrote: On Wed, May 20, 2015 at 12:13 PM, Chris Barker chris.bar...@noaa.gov wrote: The package includes its build recipe in info/recipe very cool -- I hadn't

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-20 Thread Nick Coghlan
On 21 May 2015 at 10:52, Wes Turner wes.tur...@gmail.com wrote: On May 20, 2015 7:43 PM, Nick Coghlan ncogh...@gmail.com wrote: One of my hopes for the metadata extension system in PEP 426 is that we'll be able to define extensions like fedora.repackage, debian.repackage or conda.repackage

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-20 Thread David Mertz
On Tue, May 19, 2015 at 4:11 PM, Chris Barker chris.bar...@noaa.gov wrote: On Tue, May 19, 2015 at 3:09 PM, Paul Moore p.f.mo...@gmail.com wrote: conda -- a fully open source package management system Anaconda -- a python and other stuff distribution produced by Continuum. How Continuum does

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Paul Moore
On 19 May 2015 at 00:25, Chris Barker chris.bar...@noaa.gov wrote: Pretty much, yes. conda provides a way to package up and manage arbitrary stuff -- in this case, that would be non-python dependencies -- i.e. shared libs. So you can say that my_python_package depends on this_c_lib, and as

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Oscar Benjamin
On 19 May 2015 at 10:55, Paul Moore p.f.mo...@gmail.com wrote: But python, setuptools, pip, wheel, etc. don't have a way to handle that shared lib as a dependency -- no standard way where to put it, no way to package it as a wheel, etc. So the way to deal with this with wheels is to

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Nick Coghlan
On 18 May 2015 at 14:32, David Cournapeau courn...@gmail.com wrote: On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan ncogh...@gmail.com wrote: Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread David Cournapeau
On Wed, May 20, 2015 at 2:25 PM, Nick Coghlan ncogh...@gmail.com wrote: On 18 May 2015 at 14:32, David Cournapeau courn...@gmail.com wrote: On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan ncogh...@gmail.com wrote: Indirection via pip injects the usage of setuptools even for plain distutils

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Wes Turner
On May 19, 2015 4:55 AM, Paul Moore p.f.mo...@gmail.com wrote: On 19 May 2015 at 00:25, Chris Barker chris.bar...@noaa.gov wrote: Pretty much, yes. conda provides a way to package up and manage arbitrary stuff -- in this case, that would be non-python dependencies -- i.e. shared libs.

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Chris Barker - NOAA Federal
Maybe I wasn't very clear -- I was addressing what conda might provide in the context of using conda packages with pip/pipy. A conda environment provides a great deal more, yes. system-site-packages may very well be for a different version of the python interpreter and stdlib. Isn't that

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Chris Barker
On Tue, May 19, 2015 at 4:27 AM, Oscar Benjamin oscar.j.benja...@gmail.com wrote: Surely the best way to manage non-Python shared libs is by exposing them as extension modules which can be packaged up on PyPI. Then you have dependency resolution for pip, you don't need to worry about the

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Chris Barker
On Tue, May 19, 2015 at 9:15 AM, David Cournapeau courn...@gmail.com wrote: Honestly, I still haven't seen a solid explanation of why (at least on Windows) static linking isn't a viable option. well - it does get us pretty far Because some libraries simply don't work as static

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread David Cournapeau
On Wed, May 20, 2015 at 12:46 AM, Paul Moore p.f.mo...@gmail.com wrote: On 19 May 2015 at 16:22, Chris Barker chris.bar...@noaa.gov wrote: The other issue is social: this would really only be a benefit if a wide variety of packages shared the same libs -- but each of those packages is

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Chris Barker
On Tue, May 19, 2015 at 3:09 PM, Paul Moore p.f.mo...@gmail.com wrote: So, for example the process for building the pyyaml package available via conda is private? well, I haven't been able to find them... I don't know if continuum keeps them private on purpose or, just haven't happened to

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Paul Moore
On 19 May 2015 at 20:26, Chris Barker chris.bar...@noaa.gov wrote: This entire conversation is about when the build dependencies are NOT simple :-). And while it may be project specific, commonly used libs are not project specific, and they are where the time and pain are. So some shared

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread David Mertz
It is certainly not our intention at Continuum to keep build recipes private. I have just come on board at the company, but I'll add it to my TODO list to work on making sure that those are better updated and maintained at https://github.com/conda/conda-recipes. On Tue, May 19, 2015 at 3:09 PM,

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Chris Barker
On Tue, May 19, 2015 at 1:27 PM, Paul Moore p.f.mo...@gmail.com wrote: And maybe all that needs to be is a gitHub project with build scripts. But I had little luck in getting any traction that way. That is, until we had Anaconda, conda and binstar --- an infrastructure that provides a way

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Paul Moore
On 19 May 2015 at 22:29, Chris Barker chris.bar...@noaa.gov wrote: As far as I can tell, Continuum does not publish the build scripts used to build all the stuff in Anaconda. So, for example the process for building the pyyaml package available via conda is private? (I want to say proprietary,

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread David Mertz
I will note that most recipes seem to consist of either 'python setup.py install' or './configure; make; make install'. So there is quite likely actually little significant work that has failed to have been published. But I'm not sure of pyyaml off the top of my head, and how that is built. On

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Paul Moore
On 19 May 2015 at 16:22, Chris Barker chris.bar...@noaa.gov wrote: The other issue is social: this would really only be a benefit if a wide variety of packages shared the same libs -- but each of those packages is maintained by different individuals and communities. So it's had to know if it

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Paul Moore
On 19 May 2015 at 17:15, David Cournapeau courn...@gmail.com wrote: On Wed, May 20, 2015 at 12:46 AM, Paul Moore p.f.mo...@gmail.com wrote: On 19 May 2015 at 16:22, Chris Barker chris.bar...@noaa.gov wrote: The other issue is social: this would really only be a benefit if a wide variety of

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-19 Thread Chris Barker
On Tue, May 19, 2015 at 11:10 AM, Paul Moore p.f.mo...@gmail.com wrote: to be the same. I suppose I could deliver the static libs themselves, along with the headers, etc, but that does get ugly. Hmm, that seems to me to be something of a non-goal. If you publish wheels, 99.999% of people

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-18 Thread Chris Barker
A member of the conda dev team could answer this better than I, but I've used enough to _think_ I understand the basics: On Mon, May 18, 2015 at 3:30 AM, Paul Moore p.f.mo...@gmail.com wrote: One way forward in terms of building wheels is to use any build process you like to do an isolated

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-18 Thread Paul Moore
On 18 May 2015 at 05:32, David Cournapeau courn...@gmail.com wrote: Note that some packages will push hard against injecting setuptools, at least until it does not offer a way to prevent from installing as an egg directory. Most of the core scientific packages avoid setuptools because of this.

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-17 Thread Robert Collins
On 17 May 2015 5:05 pm, Nick Coghlan ncogh...@gmail.com wrote: On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov wrote: On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-17 Thread Chris Barker
On Sun, May 17, 2015 at 5:12 PM, Robert Collins robe...@robertcollins.net wrote: But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ??? I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-17 Thread David Cournapeau
On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan ncogh...@gmail.com wrote: On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov wrote: On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-17 Thread Donald Stufft
On May 17, 2015, at 8:12 PM, Robert Collins robe...@robertcollins.net wrote: On 17 May 2015 5:05 pm, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov mailto:chris.bar...@noaa.gov wrote: On Sun, May 17,

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-17 Thread Nick Coghlan
On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov wrote: On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-17 Thread Chris Barker
On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com wrote: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip install some_fedora_rpm to be made to work. I agree here -- I

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-17 Thread Nick Coghlan
On 17 May 2015 at 05:04, David Mertz dme...@continuum.io wrote: What would be better as a user experience would be to let users do this: % pip install --upgrade pip % pip install some_conda_package This gets the respective role of the two tools reversed - it's like my asking for pip

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-17 Thread Robert Collins
On 17 May 2015 at 19:05, Nick Coghlan ncogh...@gmail.com wrote: On 17 May 2015 at 05:04, David Mertz dme...@continuum.io wrote: What would be better as a user experience would be to let users do this: % pip install --upgrade pip % pip install some_conda_package This gets the respective

[Distutils] Making pip and PyPI work with conda packages

2015-05-16 Thread David Mertz
I've just started monitoring this SIG to get a sense of the issues and status of things. I've also just started working for Continuum Analytics. Continuum has a great desire to make 'pip' work with conda packages. Obviously, we love for users to choose the Anaconda Python distribution but many

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-16 Thread Paul Moore
On 16 May 2015 at 20:04, David Mertz dme...@continuum.io wrote: What would be better as a user experience would be to let users do this: % pip install --upgrade pip % pip install some_conda_package Whether that second command ultimately downloads code from pyip.python.org or from

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-16 Thread Chris Barker
On Sat, May 16, 2015 at 12:04 PM, David Mertz dme...@continuum.io wrote: Continuum has a great desire to make 'pip' work with conda packages. Obviously, we love for users to choose the Anaconda Python distribution but many will not for a variety of reasons (many good reasons). Hmm -- this

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-16 Thread Donald Stufft
On May 16, 2015, at 3:04 PM, David Mertz dme...@continuum.io wrote: I've just started monitoring this SIG to get a sense of the issues and status of things. I've also just started working for Continuum Analytics. Continuum has a great desire to make 'pip' work with conda packages.

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-16 Thread Chris Barker
On Sat, May 16, 2015 at 4:16 PM, Donald Stufft don...@stufft.io wrote: On Sat, May 16, 2015 at 3:03 PM, Donald Stufft don...@stufft.io wrote: There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just things that conda has that pip is

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-16 Thread Donald Stufft
On May 16, 2015, at 8:50 PM, Chris Barker chris.bar...@noaa.gov wrote: On Sat, May 16, 2015 at 4:16 PM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: On Sat, May 16, 2015 at 3:03 PM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: There are a few other

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-16 Thread Donald Stufft
On May 16, 2015, at 7:09 PM, Chris Barker chris.bar...@noaa.gov wrote: On Sat, May 16, 2015 at 3:03 PM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just

Re: [Distutils] Making pip and PyPI work with conda packages

2015-05-16 Thread Chris Barker
On Sat, May 16, 2015 at 3:03 PM, Donald Stufft don...@stufft.io wrote: There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just things that conda has that pip is planning on getting, Huh? I'm confused -- didn't we just have a big