On 3 December 2013 11:54, Nick Coghlan <ncogh...@gmail.com> wrote: > On 3 December 2013 21:22, Oscar Benjamin <oscar.j.benja...@gmail.com> wrote: >> AFAICT conda/binstar are alternatives for pip/PyPI that happen to host >> binaries for some packages that don't have binaries on PyPI. (conda >> also provides a different - incompatible - take on virtualenvs but >> that's not relevant to this proposal). > > It sounds like I may have been confusing two presentations at the > packaging mini-summit, as I would have sworn conda used hashes to > guarantee a consistent set of packages. I know I have mixed up > features between hashdist and conda in the past (and there have been > some NixOS features mixed in there as well), so it wouldn't be the > first time that has happened - the downside of mining different > distribution systems for ideas is that sometimes I forget where I > encountered particular features :)
I had the same confusion with hashdist at the start of this thread when I said that conda was targeted at HPC. So if we both make the same mistake I guess it's forgiveable :) > If conda doesn't offer such an internal consistency guarantee for > published package sets, then I agree with the criticism that it's just > an alternative to running a private PyPI index server hosting wheel > files pre-built with particular options, and thus it becomes > substantially less interesting to me :( Perhaps Travis who is still CC'ed here could comment on this since it is apparent that no one here really understands what conda is and he apparently works for Continuum Analytics so should (hopefully) know a little more... > Under that model, what conda is doing is *already covered* in the > draft metadata 2.0 spec (as of the changes I posted about the other > day), since that now includes an "integrator suffix" (to indicate when > a downstream rebuilder has patched the software), as well as a > "python.integrator" metadata extension to give details of the rebuild. > The namespacing in the wheel case is handled by not allowing rebuilds > to be published on PyPI - they have to be published on a separate > index server, and thus can be controlled based on where you tell pip > to look. Do you mean to say that PyPI can (should) only host a binary-compatible set of wheels and that other index servers should do the same? I still think that there needs to be some kind of compatibility tags either way. > So, I apologise for starting the thread based on what appears to be a > fundamentally false premise, although I think it has still been useful > despite that error on my part (as the user confusion is real, even > though my specific proposal no longer seems as useful as I first > thought). > > I believe helping the conda devs to get it to play nice with virtual > environments is still a worthwhile exercise though (even if just by > pointing out areas where it *doesn't* currently interoperate well, as > we've been doing in the last day or so), and if the conda > bootstrapping issue is fixed by publishing wheels (or vendoring > dependencies), then "try conda if there's no wheel" may still be a > reasonable fallback recommendation. Well for a start conda (at least according to my failed build) over-writes the virtualenv activate scripts with its own scripts that do something completely different and can't even be called with the same signature. So it looks to me as if there is no intention of virtualenv compatibility. As for "try conda if there's no wheel" according to what I've read that seems to be what people who currently use conda do. I thought about another thing during the course of this thread. To what extent can Provides/Requires help out with the binary incompatibility problems? For example numpy really does provide multiple interfaces: 1) An importable Python module that can be used from Python code. 2) A C-API that can be used by compiled C-extensions. 3) BLAS/LAPACK libraries with a particular Fortran ABI to any other libraries in the same process. Perhaps the solution is that a build of a numpy wheel should clarify explicitly what it Provides at each level e.g.: Provides: numpy Provides: numpy-capi-v1 Provides: numpy-openblas-g77 Then a built wheel for scipy can Require the same things. Cristoph Gohlke could provide a numpy wheel with: Provides: numpy Provides: numpy-capi-v1 Provides: numpy-intelmkl And his scipy wheel can require the same. This would mean that pip would understand the binary dependency problems during dependency resolution and could reject an incompatible wheel at install time as well as being able to find a compatible wheel automatically if one exists in the server. Unlike the hash-based dependencies we can see that it is possible to depend on the numpy C-API without necessarily depending on any particular BLAS/LAPACK library and Fortran compiler combination. The confusing part would be that then a built wheel doesn't Provide the same thing as the corresponding sdist. How would anyone know what would be Provided by an sdist without first building it into a wheel? Would there need to be a way for pip to tell the sdist what pip wants it to Provide when building it? Oscar _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
