Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-21 Thread Vincent Povirk
On Thu, May 21, 2015 at 12:33 PM, Paul Moore p.f.mo...@gmail.com wrote: Doing this without getting sucked into trying to solve problems that the wheel format is *not* intended to cover (packaging and distribution of non-Python code) is hard - particularly where we need to express dependencies

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-21 Thread Chris Barker
On Wed, May 20, 2015 at 3:46 PM, Nick Coghlan ncogh...@gmail.com wrote: On 21 May 2015 at 03:37, Chris Barker chris.bar...@noaa.gov wrote: As such, it _could_ play the role that pip+wheel (secondarily pypi) play in the python ecosystem. In practice, it can't, as conda is entirely

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-21 Thread Chris Barker
On Wed, May 20, 2015 at 5:20 PM, Nick Coghlan ncogh...@gmail.com wrote: Coping with this problem is also why injecting setuptools when running vanilla distutils projects is one of the secrets of pip's success: Ahh! THAT is the role pip plays in building. It's the way that you get setuptools

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-21 Thread Donald Stufft
On May 21, 2015, at 11:33 AM, Chris Barker chris.bar...@noaa.gov wrote: On Wed, May 20, 2015 at 5:20 PM, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: Coping with this problem is also why injecting setuptools when running vanilla distutils projects is one of the secrets

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-21 Thread Paul Moore
On 21 May 2015 at 16:37, Donald Stufft don...@stufft.io wrote: Also, one of the goals a few of us has in the PyPA is that we move to a future where the build systems are pluggable. So one package could be building using setuptools, another building using some SciPy specific build tool,

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-20 Thread Paul Moore
On 19 May 2015 at 23:32, Chris Barker chris.bar...@noaa.gov wrote: lost track of where in the thred this was, but here's a conda recipe I found on gitHub: https://github.com/menpo/conda-recipes/tree/master/libxml2 don't know anything about it. OK, I'm still misunderstanding something, I

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-20 Thread Paul Moore
On 20 May 2015 at 00:04, Chris Barker chris.bar...@noaa.gov wrote: yup. which makes me think -- maybe not that hard to do a wininst to wheel converter for wxPython -- that would be nice. We also need it for the Mac, and that would be harder -- he's got some trickery in placing the libs in that

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-20 Thread Nick Coghlan
On 19 May 2015 at 09:43, Chris Barker chris.bar...@noaa.gov wrote: On Mon, May 18, 2015 at 11:21 AM, Paul Moore p.f.mo...@gmail.com wrote: My honest view is that unless conda is intending to replace pip and wheel totally, you cannot assume that people will be happy to use conda alongside pip

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-20 Thread Chris Barker
On Wed, May 20, 2015 at 1:04 AM, Paul Moore p.f.mo...@gmail.com wrote: https://github.com/menpo/conda-recipes/tree/master/libxml2 don't know anything about it. OK, I'm still misunderstanding something, I think. As far as I can see, all that does is copy a published binary and repack

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-20 Thread Chris Barker
On Wed, May 20, 2015 at 12:57 AM, Nick Coghlan ncogh...@gmail.com wrote: This is why I'm such a big fan of richer upstream metadata with automated conversion to downstream formats as my preferred long term solution - this isn't a pip vs conda story, it's pip vs conda vs yum vs apt vs MSI vs

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-20 Thread Nick Coghlan
On 21 May 2015 at 03:37, Chris Barker chris.bar...@noaa.gov wrote: As such, it _could_ play the role that pip+wheel (secondarily pypi) play in the python ecosystem. In practice, it can't, as conda is entirely inappropriate as an input format for yum/apt/enstaller/zc.buildout/pypm/MSI/etc. In

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-20 Thread Nick Coghlan
On 21 May 2015 at 08:46, Nick Coghlan ncogh...@gmail.com wrote: On 21 May 2015 at 03:37, Chris Barker chris.bar...@noaa.gov wrote: As such, it _could_ play the role that pip+wheel (secondarily pypi) play in the python ecosystem. In practice, it can't, as conda is entirely inappropriate as an

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Wes Turner
On Tue, May 19, 2015 at 6:04 PM, Chris Barker chris.bar...@noaa.gov wrote: On Tue, May 19, 2015 at 3:04 PM, Paul Moore p.f.mo...@gmail.com wrote: Yeah, I'm trying to never build anything for myself, just consume binaries. Having all binaries built by the conda people is a bottleneck. it

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Paul Moore
On 19 May 2015 at 00:41, Chris Barker chris.bar...@noaa.gov wrote: On Mon, May 18, 2015 at 3:17 AM, Paul Moore p.f.mo...@gmail.com wrote: Agreed. My personal use case is as a general programmer (mostly sysadmin and automation type of work) with some strong interest in business data analysis

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Chris Barker
On Mon, May 18, 2015 at 8:24 PM, Vincent Povirk madewokh...@gmail.com wrote: But maybe all you really need to do is teach conda to understand pip meta-data, and/or make sure that conda write pip-compatible meta data. Forgive me, I'm trying to follow as someone who is working with PyPI but

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Leonardo Rochael Almeida
On 19 May 2015 at 13:28, Chris Barker chris.bar...@noaa.gov wrote: [...] So is it possible for PyPA to grow the features to mange all the python bits, and then have things like conda use pip inside of Anaconda, maybe? or SOME transition where you can add conda if and only if you need its

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Paul Moore
On 19 May 2015 at 21:19, Wes Turner wes.tur...@gmail.com wrote: Would this be different than: # miniconda conda install pip conda install scientific_stuff pip install django_widget Having tried that in the past, I can say that I *very* rapidly got completely lost as to which

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Chris Barker
On Tue, May 19, 2015 at 10:18 AM, Leonardo Rochael Almeida leoroch...@gmail.com wrote: What if instead of focusing on pip being able to install more than just python packages, we made sure that a virtualenv was as strict subset of, say, a conda environment? This way, going from virtualenv to,

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Wes Turner
On Mon, May 18, 2015 at 12:50 PM, David Mertz dme...@continuum.io wrote: This pertains more to the other thread I started, but I'm sort of becoming convinced--especially by Paul Moore's suggestion there--that the better approach is to grow conda (the tool) rather than shoehorn conda packages

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Wes Turner
On Tue, May 19, 2015 at 2:34 PM, Chris Barker chris.bar...@noaa.gov wrote: On Tue, May 19, 2015 at 10:18 AM, Leonardo Rochael Almeida leoroch...@gmail.com wrote: What if instead of focusing on pip being able to install more than just python packages, we made sure that a virtualenv was as

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Chris Barker
On Tue, May 19, 2015 at 1:19 PM, Wes Turner wes.tur...@gmail.com wrote: So it might make sense to actually allow folks to push conda to budding web developers, if conda allowed installation (and environment management) of sdist packages on PyPI. So perhaps it would be good if *this* worked:

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Paul Moore
Sigh. I really am going to try to stop monopolising this thread - but you keep making good points I feel I have to respond to :-) I'll try to keep to essentials. On 19 May 2015 at 22:11, Chris Barker chris.bar...@noaa.gov wrote: On Tue, May 19, 2015 at 10:58 AM, Paul Moore p.f.mo...@gmail.com

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Chris Barker
On Tue, May 19, 2015 at 3:04 PM, Paul Moore p.f.mo...@gmail.com wrote: Yeah, I'm trying to never build anything for myself, just consume binaries. Having all binaries built by the conda people is a bottleneck. it is -- though the group of conda people is growing... Having pip auto-build

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Chris Barker
On Tue, May 19, 2015 at 10:58 AM, Paul Moore p.f.mo...@gmail.com wrote: Sure. Doesn't have to be the same way, but the user experience has to be the same. absolutely. But maybe that's not going to cut it -- in a way, we are headed there now, with a contingent of people porting pypi

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Wes Turner
On Tue, May 19, 2015 at 4:17 PM, Chris Barker chris.bar...@noaa.gov wrote: On Tue, May 19, 2015 at 1:19 PM, Wes Turner wes.tur...@gmail.com wrote: So it might make sense to actually allow folks to push conda to budding web developers, if conda allowed installation (and environment management)

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Chris Barker
lost track of where in the thred this was, but here's a conda recipe I found on gitHub: https://github.com/menpo/conda-recipes/tree/master/libxml2 don't know anything about it. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206)

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Chris Barker
On Tue, May 19, 2015 at 5:21 AM, Paul Moore p.f.mo...@gmail.com wrote: If conda did everything pip did (and that includes consuming wheels from PyPI, not just sdists, and it includes caching of downloads, autobuilding of wheels etc, etc.) hmm...what about half-way -- conda does everything

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-19 Thread Paul Moore
On 19 May 2015 at 17:28, Chris Barker chris.bar...@noaa.gov wrote: On Tue, May 19, 2015 at 5:21 AM, Paul Moore p.f.mo...@gmail.com wrote: If conda did everything pip did (and that includes consuming wheels from PyPI, not just sdists, and it includes caching of downloads, autobuilding of

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-18 Thread David Mertz
This pertains more to the other thread I started, but I'm sort of becoming convinced--especially by Paul Moore's suggestion there--that the better approach is to grow conda (the tool) rather than shoehorn conda packages into pip. Getting pip to recognize the archive format of conda would be easy

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-18 Thread Paul Moore
On 18 May 2015 at 18:50, David Mertz dme...@continuum.io wrote: % pip install conda % conda install scientific_stuff % conda install --sdist django_widget # we know to look on PyPI But that doesn't give (Windows, mainly) users a solution for things that need a C compiler, but aren't

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-18 Thread David Mertz
I don't really see any reason conda couldn't support bdist wheels also. But yes, basically the idea is that we'd like users to be able to rely entirely on conda as their packaging (and environment configuration) system if they choose to. It may be impolitic to say so, but I think conda can and

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-18 Thread Chris Barker
On Mon, May 18, 2015 at 11:21 AM, Paul Moore p.f.mo...@gmail.com wrote: On 18 May 2015 at 18:50, David Mertz dme...@continuum.io wrote: % pip install conda % conda install scientific_stuff % conda install --sdist django_widget # we know to look on PyPI But that doesn't give

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-18 Thread Vincent Povirk
But it gets messy when you have two systems trying to handle dependencies -- pip may not realize that conda has already installed something, and vice versa. So it's really nicer to have one package manager. But maybe all you really need to do is teach conda to understand pip meta-data,

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-18 Thread Chris Barker
On Mon, May 18, 2015 at 10:50 AM, David Mertz dme...@continuum.io wrote: This pertains more to the other thread I started, but I'm sort of becoming convinced--especially by Paul Moore's suggestion there--that the better approach is to grow conda (the tool) rather than shoehorn conda packages

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-18 Thread Paul Moore
On 17 May 2015 at 23:50, Chris Barker chris.bar...@noaa.gov wrote: I guess the key thing here for me is that I don't see pushing conda to budding web developers -- but what if web developers have the need for a bit of the scipy stack? or??? We really don't have a good solution for those

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-17 Thread Chris Barker
Trying to keep this brief, because the odds of my finding time to do much with this are slim.. I'm not proposing that we drop it -- just that we push pip and wheel a bit farther to broaden the supported user-base. I can't stop you working on something I consider a deep rabbithole, no -- but I

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-17 Thread Paul Moore
On 17 May 2015 at 04:48, Nick Coghlan ncogh...@gmail.com wrote: A large part of what *defines* a platform is making decisions about the ABI to publish target. Linux distros, nix, conda do that for everything they redistribute. I assume chocolatey does as well I'm picking on this because it

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-16 Thread Paul Moore
On 16 May 2015 at 20:04, Chris Barker chris.bar...@noaa.gov wrote: I was referring to the SetDllDirectory API. I don't think that gets picked up by other processes. from: https://msdn.microsoft.com/en-us/library/windows/desktop/ms686203%28v=vs.85%29.aspx It looks like you can add a path,

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-16 Thread Chris Barker
On Fri, May 15, 2015 at 11:35 PM, David Cournapeau courn...@gmail.com wrote: On Sat, May 16, 2015 at 4:56 AM, Chris Barker chris.bar...@noaa.gov wrote: But in short -- I'm pretty sure there is a way, on all systems, to have a standard way to build extension modules, combined with a standard

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-16 Thread Chris Barker
On Sat, May 16, 2015 at 4:13 AM, Paul Moore p.f.mo...@gmail.com wrote: Though it's a lot harder to provide a build environment than just the lib to link too .. Im going to have to think more about that... It seems to me that the end user doesn't really have a problem here (pip install

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-16 Thread Chris Barker
On Sat, May 16, 2015 at 10:12 AM, Nick Coghlan ncogh...@gmail.com wrote: Maybe, but it's a problem to be solved, and the Linux distros more or less solve it for us, but OS-X and Windows have no such system built in (OS-X does have Brew and macports) Windows 10 has Chocalatey and

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-16 Thread Paul Moore
On 16 May 2015 at 19:40, Chris Barker chris.bar...@noaa.gov wrote: With 2., you still have the issue of DLL hell, could you clarify a bit -- I thought that this could, at least, put a dir on the search path that was specific to that python context. So it would require cooperation among all

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-16 Thread Chris Barker
On Sat, May 16, 2015 at 11:54 AM, Paul Moore p.f.mo...@gmail.com wrote: could you clarify a bit -- I thought that this could, at least, put a dir on the search path that was specific to that python context. So it would require cooperation among all the packages being used at once, but not

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-16 Thread Nick Coghlan
On 15 May 2015 at 04:01, Chris Barker chris.bar...@noaa.gov wrote: I'm confused -- you don't want a system to be able to install ONE version of a lib that various python packages can all link to? That's really the key use-case for me Are we talking about Python libraries accessed

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-16 Thread Nick Coghlan
On 17 May 2015 06:19, Chris Barker chris.bar...@noaa.gov wrote: indeed -- but it does have a bunch of python-specific featuresit was built around the need to combine python with other systems. That makes it an interesting alternative to pip on the package *consumption* side for data

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-16 Thread David Cournapeau
On Sat, May 16, 2015 at 4:56 AM, Chris Barker chris.bar...@noaa.gov wrote: On Fri, May 15, 2015 at 1:49 AM, Paul Moore p.f.mo...@gmail.com wrote: On 14 May 2015 at 19:01, Chris Barker chris.bar...@noaa.gov wrote: Ah -- here is the issue -- but I think we HAVE pretty much got what we need

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-16 Thread Paul Moore
On 16 May 2015 at 07:35, David Cournapeau courn...@gmail.com wrote: But in short -- I'm pretty sure there is a way, on all systems, to have a standard way to build extension modules, combined with a standard way to install shared libs, so that a lib can be shared among multiple packages. So

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-15 Thread Paul Moore
On 14 May 2015 at 19:01, Chris Barker chris.bar...@noaa.gov wrote: Ah -- here is the issue -- but I think we HAVE pretty much got what we need here -- at least for Windows and OS-X. It depends what you mean by curated, but it seems we have a (defacto?) policy for PyPi: binary wheels should be

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-15 Thread Chris Barker
On Fri, May 15, 2015 at 1:44 PM, Paul Moore p.f.mo...@gmail.com wrote: Is there any point? or is the current approach of statically linking all third party libs the way to go? If someone can make it work, that would be good. But (a) nobody is actually offering to develop and maintain such

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-15 Thread Chris Barker
On Fri, May 15, 2015 at 1:49 AM, Paul Moore p.f.mo...@gmail.com wrote: On 14 May 2015 at 19:01, Chris Barker chris.bar...@noaa.gov wrote: Ah -- here is the issue -- but I think we HAVE pretty much got what we need here -- at least for Windows and OS-X. It depends what you mean by curated,

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-15 Thread Paul Moore
On 15 May 2015 at 20:56, Chris Barker chris.bar...@noaa.gov wrote: But in short -- I'm pretty sure there is a way, on all systems, to have a standard way to build extension modules, combined with a standard way to install shared libs, so that a lib can be shared among multiple packages. So the

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-14 Thread Chris Barker
On Thu, May 14, 2015 at 4:41 PM, Robert Collins robe...@robertcollins.net wrote: anyway) you now have three copies of the same lib (but maybe different versions) all linked into your executable. Maybe there is no downside to that (I haven't had a problem yet), but it seems like a bad way to

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-14 Thread Chris Barker
I'm confused -- you don't want a system to be able to install ONE version of a lib that various python packages can all link to? That's really the key use-case for me Are we talking about Python libraries accessed via Python APIs, or linking to external dependencies not written in

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-14 Thread Robert Collins
On 15 May 2015 at 06:01, Chris Barker chris.bar...@noaa.gov wrote: I'm confused -- you don't want a system to be able to install ONE version of a lib that various python packages can all link to? That's really the key use-case for me Are we talking about Python libraries accessed

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-13 Thread Nick Coghlan
On 13 May 2015 at 16:19, Ben Finney ben+pyt...@benfinney.id.au wrote: Chris Barker chris.bar...@noaa.gov writes: On Tue, Apr 14, 2015 at 8:41 AM, Nick Coghlan ncogh...@gmail.com wrote: The point where I draw the line is supporting *dynamic* linking between modules - I'm confused -- you

[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

2015-05-13 Thread Ben Finney
Chris Barker chris.bar...@noaa.gov writes: On Tue, Apr 14, 2015 at 8:41 AM, Nick Coghlan ncogh...@gmail.com wrote: The point where I draw the line is supporting *dynamic* linking between modules - I'm confused -- you don't want a system to be able to install ONE version of a lib that