[Distutils] Re: Environment markers for GPU/CUDA availibility

[David Cournapeau]

On Tue, Sep 4, 2018 at 7:32 PM Nathaniel Smith  wrote:

> On Tue, Sep 4, 2018 at 3:10 AM, Paul Moore  wrote:
> > There's very much an 80-20 question here, we need to avoid letting the
> > needs of the 20% of projects with unusual needs, complicate usage for
> > the 80%. On the other hand, of course, leaving the specialist cases
> > with no viable solution also isn't reasonable, so even if tags aren't
> > practical here, finding a solution that allows projects to ship
> > specialised binaries some other way would be good. Just as a
> > completely un-thought through suggestion, maybe we could have a
> > mechanism where a small "generic" wheel can include pointers to
> > specialised extra code that gets downloaded at install time?
> >
> > Package X -> x-1.0-cp37_cp37m_win_amd64.whl (includes generic code)
> > Metadata - Implementation links:
> > If we have a GPU ->  > the install>
> > If we don't have a GPU ->  archive>
> >
> > There's obviously a lot of unanswered questions here, but maybe
> > something like this would be better than forcing everything into the
> > wheel tags?
>
> I think you've reinvented Requires-Dist and PEP 508 markers :-). (The
> ones that look like '; python_version < "3.6"'.) Which IIUC was also
> Dustin's original suggestion: make it possible to write requirements
> like
>
>   tensorflow; not has_gpu
>   tensorflow-gpu; has_gpu
>
> But... do we actually know enough to define a "has_gpu" marker? It
> isn't literally "this system has a gpu", right, it's something more
> like "this system has an NVIDIA-brand GPU of a certain generation or
> later with their proprietary libraries installed"? Or something like
> that? There are actually lots of packages on PyPI with foo/foo-gpu
> pairs, e.g. strawberryfields, paddlepaddle, magenta, cntk, deepspeech,
> ... Do these -gpu packages all have the same environmental
> requirements, or is it different from package to package?
>
> It would help if we had folks in the conversation who actually work on
> these packages :-/. Anyone have contacts on the Tensorflow team? (It'd
> also be good to talk to them about platform specifiers... the
> tensorflow "manylinux1" wheels are really ubuntu-only, but they
> intentionally lie about that b/c there is no ubuntu tag; maybe they're
> interested in fixing that...?)
>
> Anyway, I don't see how we could add an environment marker without
> having a precise definition, and one that's useful for multiple
> packages. Which may or may not be possible here...
>
> One thing that would help would be if tensorflow-gpu could say
> "Provides-Dist: tensorflow", so that downstream packages can say
> "Requires-Dist: tensorflow" and pip won't freak out if the user has
> manually installed tensorflow-gpu instead. E.g. in the proposal at
> [1], you could have 'tensorflow' as one wheel and 'tensorflow[gpu]' as
> a second wheel that 'Provides-Dist: tensorflow'. Conflicts-Dist would
> also be useful, though might require a real resolver first.
>

I don't know the situation of pip w.r.t. resolver algo, but having a
provide-like mechanism would completely solve the issue of tensorflow vs
tensorflow-gpu for the usecases I have seen at my current company, heavy
user of tensorflow. It would solve the two main issues of not being able to
specify tensorflow as a dependency, and avoiding accidental clobbering of
tf vs tf-gpu.

At least in the usecases I am familiar with, I don't find the use of tag
and gpu detection that useful. And it would be really hard to get it right
because as you notice, it is not about GPU being available, but about
"nvidia GPU with proprietary nvidia driver and the appropriate version of
cuda version". I also don't necessarily want to install tensorflow-gpu
because I have a GPU.

David


> Another wacky idea, maybe worth thinking about: should we let packages
> specify their own auto-detection code that pip should run? E.g. you
> could have a PEP 508 requirement like "somepkg;
> extension[otherpackage.key] = ..." and that means "install
> otherpackage inside the target Python environment, look up
> otherpackage.key, and use its value to decide whether to install
> somepkg". Maybe that's too messy to be worth it, but if "gpu
> detection" isn't a well-defined problem then maybe it's the best
> approach? Though basically that's what sdists do right now, and IIUC
> how tensorflow-gpu-detect works. Maybe tensorflow-gpu-detect should
> become the standard tensorflow library, with an sdist only, and at
> install time it could decide whether to pull in 'tensorflow-gpu' or
> 'tensorflow-nogpu'...
>
> -n
>
> [1]
> https://mail.python.org/pipermail/distutils-sig/2015-October/027364.html
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> 

[Distutils] packaging guide and private packages index

[David Cournapeau]

Hi,

I see that there is almost no mention of private packages index in the
packaging guide, and no recommendation on what to use.

Currently googling for private packages mostly return obsolete (and not
very practical) recommendations based on dependency links.

In 2018, what would be the recommended practices for teams that develop
private packages that depend on each other ?

thank you,

David
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/QLDBOAJZOZACT3FGHFC664DLHOHMBLDT/

[Distutils] PyCon Colombia 2017 keynote on packaging

[David Cournapeau]

Hi,

I was invited to give a talk at PyCon Colombia 2017, and I did it on
packaging. I thought people here would be interested to know about it.

I insisted on the need for packaging to get software into as many hands as
possible, gave a history of the packaging ecosystem, advised people to use
packaging.python.org suggestions, and mentioned the manylinux effort. I
tried to be as objective as possible there and mention the key people
involved.

I also talked a bit about what can still be improved, and focused on 3
aspects, none of which are new nor particularly insightful for people here:
infrastructure for automatic wheel building, better decoupling of packaging
and build, and maybe more controversially, the need for tools to remove
python from the equation.

https://speakerdeck.com/cournape/python-packaging-in-2017

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] GSoC 2017 - Working on pip

[David Cournapeau]

On Fri, Feb 10, 2017 at 4:28 PM, Justin Cappos  wrote:

> So, there aren't "heuristics" to tweak here.  The algorithm just encodes
> the rules for trying package combinations (usually, latest version first)
> and then backtracks to a previous point when an unresolvable conflict is
> found.
>


> This is quite different from something like a SAT solver where it does use
> heuristics to come up with a matching scenario quickly.
>
> I don't think developers need to tweak heuristics in either case.  You
> just pick your SAT solver and it has reasonable heuristics built in, right?
>

Right, so there are 2 set of heuristics: the heuristics to make SAT solvers
more efficient, and heuristics to make it more useful as a dependency
resolution algorithm. I am only interested in the 2nd set of heuristics
here.

So for SAT solvers at least, you need heuristics to tweak the search space
toward something more likely solutions (from a dependency POV). E.g.
composer will favor already installed packages if they match the problem.
That's also why it is rather hard to use a SAT solver as a black box and
then wrap it to resolve dependencies, and you instead want to have access
to the SAT solver "internals".

Don't you need the same kind of heuristics to make backtracking actually
useful ?

I agree comparing on actual problems is the best way to move this
discussion forward, to compare speed, solution quality, feasibility in
pip's/pypi context. If you have access to "scenarios", I would be happy to
run our own SAT solver on it to compare solver's output.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] GSoC 2017 - Working on pip

[David Cournapeau]

On Fri, Feb 10, 2017 at 3:52 PM, David Cournapeau <courn...@gmail.com>
wrote:

>
>
> On Fri, Feb 10, 2017 at 2:33 PM, Justin Cappos <jcap...@nyu.edu> wrote:
>
>> Yes, don't use a SAT solver.  It requires all metadata from all packages
>> (~30MB uncompressed) and gives hard to predict results in some cases.
>>
>
> I doubt there exists an algorithm where this is not the case.
>
>   Also the lack of fixed dependencies is a substantial problem for a SAT
>> solver.  Overall, we think it makes more sense to use a simple backtracking
>> dependency resolution algorithm.
>>
>
> As soon as you want to deal with version ranges and ensure consistency of
> the installed packages, backtracking stops being simple rather quickly.
>
> I agree lack of fixed dependencies is an issue, but I doubt it is specific
> to a SAT solver. SAT solvers have been used successfully in many cases now:
> composer (php), dnf (Red Hat/Fedora), conda or our own packages manager at
> Enthought in python, 0install.
>
> I would certainly be interested in seeing a proper comparison with other
> algorithms.
>

I don't have experience implementing non SAT dependency solvers, but I
suspect that whatever algorithm you end up using, the "core" is the simple
part, and tweaking heuristics will be the hard, developer-time consuming
part.

David

>
> David
>
>
>> Sebastien Awwad (CCed) has been looking at a bunch of data around the
>> speed and other tradeoffs of the different algos.  Sebastien:  Sometime
>> next week, can you write it up in a way that is suitable for sharing?
>>
>> Justin
>>
>> On Fri, Feb 10, 2017 at 1:59 PM, Wes Turner <wes.tur...@gmail.com> wrote:
>>
>>> From the discussion on https://github.com/pypa/pip/is
>>> sues/988#issuecomment-279033079:
>>>
>>>
>>>- https://github.com/ContinuumIO/pycosat (picosat)
>>>   - https://github.com/ContinuumIO/pycosat/blob/master/pycosat.c (C)
>>>   - https://github.com/ContinuumIO/pycosat/blob/master/picosat.c
>>>   - https://github.com/ContinuumIO/pycosat/tree/master/examples
>>>- https://github.com/enthought/sat-solver (MiniSat)
>>>   - https://github.com/enthought/sat-solver/tree/master/simplesa
>>>   t/tests
>>>   - https://github.com/enthought/sat-solver/blob/master/requirem
>>>   ents.txt (PyYAML, enum34)
>>>
>>>
>>> Is there a better way than SAT?
>>>
>>> On Fri, Feb 10, 2017 at 12:20 PM, Pradyun Gedam <pradyu...@gmail.com>
>>> wrote:
>>>
>>>> Yay! Thank you so much for a prompt and positive response! I'm pretty
>>>> excited and looking forward to this.
>>>>
>>>> On Thu, Feb 9, 2017, 20:23 Donald Stufft <don...@stufft.io> wrote:
>>>>
>>>> I’ve never done it before, but I’m happy to provide mentoring on this.
>>>>
>>>> On Feb 8, 2017, at 9:15 PM, Pradyun Gedam <pradyu...@gmail.com> wrote:
>>>>
>>>> Hello Everyone!
>>>>
>>>> Ralf Gommers suggested that I put this proposal here on this list, for
>>>> feedback and for seeing if anyone would be willing to mentor me. So, here
>>>> it is.
>>>>
>>>> -
>>>>
>>>> My name is Pradyun Gedam. I'm currently a first year student VIT
>>>> University in India.
>>>>
>>>> I would like to apply for GSoC 2017 under PSF.
>>>>
>>>> I currently have a project in mind - the "pip needs a dependency
>>>> resolver" issue [1]. I would like to take on this specific project but am
>>>> willing to do some other project as well.
>>>>
>>>> For some background, around mid 2016, I started contributing to pip.
>>>> The first issue I tackled was #59 [2] - a request for upgrade command and
>>>> an upgrade-all command that has been open for over 5.5 years. Over the
>>>> months following that, I've have had the opportunity to work with and
>>>> understand multiple parts of pip's codebase while working on this issue and
>>>> a few others. This search on GitHub issues [3] also provides a good summary
>>>> of what work I've done on pip.
>>>>
>>>> [2]: https://github.com/pypa/pip/issues/988
>>>> [2]: https://github.com/pypa/pip/issues/59
>>>> [3]: https://github.com/pypa/pip/issues?q=author%3Apradyunsg
>>>>
>>>> Eagerly-waiting-for-a-response-ly,
>>>> Pradyun Gedam
>>>>
>>

Re: [Distutils] GSoC 2017 - Working on pip

[David Cournapeau]

On Fri, Feb 10, 2017 at 2:33 PM, Justin Cappos  wrote:

> Yes, don't use a SAT solver.  It requires all metadata from all packages
> (~30MB uncompressed) and gives hard to predict results in some cases.
>

I doubt there exists an algorithm where this is not the case.

  Also the lack of fixed dependencies is a substantial problem for a SAT
> solver.  Overall, we think it makes more sense to use a simple backtracking
> dependency resolution algorithm.
>

As soon as you want to deal with version ranges and ensure consistency of
the installed packages, backtracking stops being simple rather quickly.

I agree lack of fixed dependencies is an issue, but I doubt it is specific
to a SAT solver. SAT solvers have been used successfully in many cases now:
composer (php), dnf (Red Hat/Fedora), conda or our own packages manager at
Enthought in python, 0install.

I would certainly be interested in seeing a proper comparison with other
algorithms.

David


> Sebastien Awwad (CCed) has been looking at a bunch of data around the
> speed and other tradeoffs of the different algos.  Sebastien:  Sometime
> next week, can you write it up in a way that is suitable for sharing?
>
> Justin
>
> On Fri, Feb 10, 2017 at 1:59 PM, Wes Turner  wrote:
>
>> From the discussion on https://github.com/pypa/pip/is
>> sues/988#issuecomment-279033079:
>>
>>
>>- https://github.com/ContinuumIO/pycosat (picosat)
>>   - https://github.com/ContinuumIO/pycosat/blob/master/pycosat.c (C)
>>   - https://github.com/ContinuumIO/pycosat/blob/master/picosat.c
>>   - https://github.com/ContinuumIO/pycosat/tree/master/examples
>>- https://github.com/enthought/sat-solver (MiniSat)
>>   - https://github.com/enthought/sat-solver/tree/master/simplesa
>>   t/tests
>>   - https://github.com/enthought/sat-solver/blob/master/requirem
>>   ents.txt (PyYAML, enum34)
>>
>>
>> Is there a better way than SAT?
>>
>> On Fri, Feb 10, 2017 at 12:20 PM, Pradyun Gedam 
>> wrote:
>>
>>> Yay! Thank you so much for a prompt and positive response! I'm pretty
>>> excited and looking forward to this.
>>>
>>> On Thu, Feb 9, 2017, 20:23 Donald Stufft  wrote:
>>>
>>> I’ve never done it before, but I’m happy to provide mentoring on this.
>>>
>>> On Feb 8, 2017, at 9:15 PM, Pradyun Gedam  wrote:
>>>
>>> Hello Everyone!
>>>
>>> Ralf Gommers suggested that I put this proposal here on this list, for
>>> feedback and for seeing if anyone would be willing to mentor me. So, here
>>> it is.
>>>
>>> -
>>>
>>> My name is Pradyun Gedam. I'm currently a first year student VIT
>>> University in India.
>>>
>>> I would like to apply for GSoC 2017 under PSF.
>>>
>>> I currently have a project in mind - the "pip needs a dependency
>>> resolver" issue [1]. I would like to take on this specific project but am
>>> willing to do some other project as well.
>>>
>>> For some background, around mid 2016, I started contributing to pip. The
>>> first issue I tackled was #59 [2] - a request for upgrade command and an
>>> upgrade-all command that has been open for over 5.5 years. Over the months
>>> following that, I've have had the opportunity to work with and understand
>>> multiple parts of pip's codebase while working on this issue and a few
>>> others. This search on GitHub issues [3] also provides a good summary of
>>> what work I've done on pip.
>>>
>>> [2]: https://github.com/pypa/pip/issues/988
>>> [2]: https://github.com/pypa/pip/issues/59
>>> [3]: https://github.com/pypa/pip/issues?q=author%3Apradyunsg
>>>
>>> Eagerly-waiting-for-a-response-ly,
>>> Pradyun Gedam
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>>
>>> —
>>>
>>> Donald Stufft
>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

[David Cournapeau]

On Sat, Aug 20, 2016 at 6:30 PM, Nick Coghlan  wrote:

> On 20 August 2016 at 02:46, Chris Barker  wrote:
> > Maybe the alternative to setuptools_lite is to set up one of these other
> > tools to work well (easily) for building python packages. If it can do
> > everything that setuptools can do (that we want setuptools to do), and
> just
> > as easily, maybe that's the next step forward.
> >
> > but if that isn't happening soon, then a setuptools_lite would be a
> useful
> > step forward.
>
> Enabling folks to more easily use existing build systems like Scons,
> Waf, Meson, CMake, et al is one of the main goals of pyproject.toml.
>
> Daniel Holth has a working Scons proof of concept at
> https://pypi.python.org/pypi/enscons that actually works independently
> of pyproject.toml support in pip by using a boilerplate setup.py to do
> the bootstrapping and invocation of the build system.
>
> It's why we don't particularly want to replace distutils/setuptools
> with any build system in particular - the world actually has a surfeit
> of cross-platform build systems, and many of them are written in
> Python and are hence quite friendly to being bootstrapped for use in
> building Python extension modules.
>

Indeed. Having been involved w/ the mess that is numpy.distutils, this was
(and still is) the major issue w/ distutils for the scientific/pydata crowd.

Conflating packaging and building is an impediment to deploy software to
different platforms (build is generally better handled by developers,
packaging by platform-specific packagers).

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[David Cournapeau]

A missing dimension for comparison: round tripping support. It is quite
useful for formats when used as a configuration. The best I know in that
dimension is yaml (if using ruamel.yaml), which round trip comments.

OTOH, adding round tripping to something like toml should not be too hard
if the need arises.

David

On Sat, May 7, 2016 at 3:59 AM, Nathaniel Smith  wrote:

> Here's that one-stop writeup/comparison of all the major configuration
> languages that I mentioned:
>
> https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 425 vs pip

[David Cournapeau]

On Fri, Mar 11, 2016 at 3:26 PM, Nick Coghlan <ncogh...@gmail.com> wrote:

> On 11 March 2016 at 19:41, David Cournapeau <courn...@gmail.com> wrote:
> > Hi,
> >
> > At Enthought we have been trying to use PEP 425 tags to include its
> metadata
> > in our packages. It has been harder than expected because of missing
> > information in PEP 425, and maybe inconsistencies with what pip actually
> > does.
> >
> > I think PEP 425 should be updated to take into account pip actual
> > implementation:
> >
> > 1. python tag and pypy: the PEP says "the version is py_version_nodot .
> > CPython gets away with no dot, but if one is needed the underscore _ is
> used
> > instead. PyPy should probably use its own versions here pp18 , pp19 ".
> This
> > is not what pip does as of version 8.1.0 for pypy. Instead, the version
> is
> > python major + sys.pypy_version_info.major + sys.pypy_version_info.minor.
> >
> > 2. It would be nice to know how to deal with other alternative
> > implementations as well. Would the scheme in 1. work for jython, etc... ?
> >
> > 3. The abi tag should at least be specified for cpython IMO. E.g. the
> > meaning of `m`, `u` and `d` should be clearly specified.
> >
> > While the PEP mentions each implementation community is welcome to use
> the
> > abi tag as they wish, given that in practice it is defined in pip, I
> think
> > that should be specified as well.
> >
> > Do the above make sense ? If so, I could spend some time to update it (or
> > create a new amending PEP ?)
>
> Proposed errrata & minor admentments for PEP 425 can be submitted as a
> PyPUG issue with a PR against
>
> https://packaging.python.org/en/latest/specifications/#platform-compatibility-tags
>
> (the actual technical details of the proposed changes should still be
> discussed here)
>

Up: would it help doing the PR first as a basic for discussion ?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP 425 vs pip

[David Cournapeau]

Hi,

At Enthought we have been trying to use PEP 425 tags to include its
metadata in our packages. It has been harder than expected because of
missing information in PEP 425, and maybe inconsistencies with what pip
actually does.

I think PEP 425 should be updated to take into account pip actual
implementation:

1. python tag and pypy: the PEP says "the version is py_version_nodot .
CPython gets away with no dot, but if one is needed the underscore _ is
used instead. PyPy should probably use its own versions here pp18 , pp19 ".
This is not what pip does as of version 8.1.0 for pypy. Instead, the
version is python major + sys.pypy_version_info.major +
sys.pypy_version_info.minor.

2. It would be nice to know how to deal with other alternative
implementations as well. Would the scheme in 1. work for jython, etc... ?

3. The abi tag should at least be specified for cpython IMO. E.g. the
meaning of `m`, `u` and `d` should be clearly specified.

While the PEP mentions each implementation community is welcome to use the
abi tag as they wish, given that in practice it is defined in pip, I think
that should be specified as well.

Do the above make sense ? If so, I could spend some time to update it (or
create a new amending PEP ?)

thanks,
David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[David Cournapeau]

On Wed, Feb 17, 2016 at 8:55 AM, Antoine Pitrou  wrote:

> On Tue, 16 Feb 2016 16:10:34 -0800
> Glyph Lefkowitz  wrote:
> >
> > I am 100% on board with telling people "don't use `sudo pip install´".
> Frankly I have been telling the pip developers to just break this for years
> (see https://pip2014.com, which, much to my chagrin, still exists); `sudo
> pip install´ should just exit immediately with an error; to the extent that
> packagers need it, the only invocation that should work should be `sudo pip
> install --i-am-building-an-operating-system´.
>
> This is frankly ridiculous. The problem is not the use of "sudo" or the
> invocation under root, it's to install into a system Python.
>

Sure, but the people I tend to see using `sudo pip` are not the kind of
users where that distinction is very useful. If there were a different
simple, reliable way to avoid installing in system python, I would be happy
to change my own recommendations during sprints, talks, etc...

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[David Cournapeau]

On Tue, Feb 16, 2016 at 11:05 AM, Matthias Klose  wrote:

> On 02.02.2016 02:35, Glyph Lefkowitz wrote:
>
>>
>> On Feb 1, 2016, at 3:37 PM, Matthias Klose  wrote:
>>>
>>> On 30.01.2016 00:29, Nathaniel Smith wrote:
>>>
 Hi all,

 I think this is ready for pronouncement now -- thanks to everyone for
 all their feedback over the last few weeks!

>>>
>>> I don't think so.  I am biased because I'm the maintainer for Python in
>>> Debian/Ubuntu.  So I would like to have some feedback from maintainers of
>>> Python in other Linux distributions (Nick, no, you're not one of these).
>>>
>>
>> Possibly, but it would be very helpful for such maintainers to limit
>> their critique to "in what scenarios will this fail for users" and not have
>> the whole peanut gallery chiming in with "well on _my_ platform we would
>> have done it _this_ way".
>>
>> I respect what you've done for Debian and Ubuntu, Matthias, and I use the
>> heck out of that work, but honestly this whole message just comes across as
>> sour grapes that someone didn't pick a super-old Debian instead of a
>> super-old Red Hat.  I don't think it's promoting any progress.
>>
>
> You may call this sour grapes, but in the light of people installing
> these wheels to replace/upgrade system installed eggs, it becomes an
> issue. It's fine to use such wheels in a virtual environment, however
> people tell users to use these wheels to replace system installed packages,
> distros will have a problem identifying issues.
>

FWIW, I often point out when people put "sudo" and "pip" in the same
sentence.

What about adding some language around this to the PEP ?


In the future, more specific and featureful distro tags sound like a good
>> idea.  But could we please stop making the default position on
>> distutils-sig "this doesn't cater to my one specific environment in the
>> most optimal possible way, so let's give up on progress entirely"?  This is
>> a good proposal that addresses environment portability and gives Python a
>> substantially better build-artifact story than it currently has, in the
>> environment most desperately needing one (server-side linux).  Could it be
>> better?  Of course.  It could be lots better.  There are lots of use-cases
>> for dynamically linked wheels and fancy new platform library features in
>> newer linuxes.  But that can all come later, and none of it needs to have
>> an impact on this specific proposal, right now.
>>
>
> I'm unsure how more specific and featureful distro tags will help, unless
> you start building more than one binary version of a wheel.  From a distro
> point of view I only can discourage using such wheels outside a virtual
> environment, and I would like to see a possibility to easily identify such
> wheels, something like loading a binary kernel module is tainting the
> kernel. This way distros can point users to the provider of such wheels.
>

This sounds like a good idea to me. Do you have a specific idea on how you
would like to see the feature work ?

David

>
> This is not a "this doesn't cater to my one specific environment"
> position. Of course you probably get less push back from other
> distributions closer to the specific environment specified in the pep.  But
> please acknowledge that there might be issues that you and me don't yet
> even see, and provide a possibility to identify such issues.
>
> At least Debian/Ubuntu took a long ride to avoid "accidental" interaction
> with local Python installations and local installations into the system
> installed Python.  For me this PEP feels like a step back, promising too
> much (manylinux), not pointing out possible issues, and giving no help in
> identifying possible issues with these wheels.
>
> Matthias
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[David Cournapeau]

On Thu, Feb 11, 2016 at 2:32 PM, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 11 February 2016 at 14:02, David Cournapeau <courn...@gmail.com> wrote:
> >>> On Wed, Feb 10, 2016 at 1:52 PM, Paul Moore <p.f.mo...@gmail.com>
> wrote:
> >>>>
> >>>> We should probably also check with the flit people that the proposed
> >>>> approach works for them. (Are there any other alternative build
> >>>> systems apart from flit that exist at present?)
> >>>
> >>> I am not working on it ATM, but bento was fairly complete and could
> >>> interoperate w/ pip (a few years ago at least):
> >>> https://cournape.github.io/Bento/
> >>
> >> I plan to test with Bento (I'm still using it almost daily to work on
> >> Scipy) when an implementation is proposed for pip. The interface in the
> PEP
> >> is straightforward though, I don't see any fundamental reason why it
> >> wouldn't work for Bento if it works for flit.
> >
> > It should indeed work, I was just pointing at an alternative build
> system ;)
>
> Yes, I knew there was something other than flit, thanks for the reminder.
>
> > I am a bit worried about making a PEP for interfacing before we have a
> few
> > decent alternative implementations. Having the official interface is not
> > necessary to actually interoperate, even if it is ugly.
>
> Well, a lot of people have complained that setuptools is a problem.
>

You won't hear me defending setuptools/distutils, but the main pain points
of setuptools are not related to interoperability with `setup.py`.


> We've only really seen bento and now flit appear as alternatives, the
> only conclusion we've been able to draw is that the barrier to
> creating alternative build systems is the need to emulate setuptools.
> This PEP (hopefully!) removes that barrier, but I agree we need some
> validation that people who want to create alternative build systems
> (or have done so) can work with the interface in the PEP.
>

If this is indeed the main argument for the PEP, that it is IMO misguided.
Making new buildsystems for python is hard work (once you go beyond the
trivial packages), but interoperability w/ setup.py is not difficult

There is some value to the PEP even if it doesn't enable new build
> tools (we can fix the problem of install_requires triggering
> easy_install) but the key has to be the evolution of (one or more)
> replacements for setuptools.
>
> You suggest getting more alternative build systems before implementing
> the PEP. That would be nice, but how would we get that to happen?
> People have been wanting alternatives to setuptools for years, but
> no-one has delivered anything (as far as I know) except for you and
> the flit guys. So my preference is to implement the PEP (and remove
> the "behave like setuptools" pain point), and then wait to see what
> level of adoption flit/bento achieve with the simpler interface and
> automated use of the declared build tool. At that stage, all of flit,
> bento and setuptools, as well as any newly developed tools, will be
> competing on an equal footing, and either one will emerge as the
> victor, or people will be free to choose their preferred tool without
> concern about whether it is going to work with pip.
>

My main worry is about designing an interface before seeing multiple
implementations. This often causes trouble. If we think the issue is
allowing people to start working on the more "interesting" parts while
keeping compat w/ pip, then I would suggest working on an example
buildsystem with interoperability that people could steal for their own.

I played w/ that idea there: https://github.com/cournape/toydist. The main
point of that project was to progressively bootstrap itself by
re-implementing the basic features required by pip. E.g. right now, only
`python setup.py egg_info` is "distutils clean", but then you just need to
implement develop to get `pip install -e .` working. To get `pip install .`
to work, you only need to add install. Etc.

David


> I'm not sure what we gain by waiting (incremental improvements to the
> spec can me made over time - as long as the basic form of the PEP is
> acceptable to *current* tool developers, I think that's sufficient).
>
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[David Cournapeau]

On Thu, Feb 11, 2016 at 7:43 AM, Ralf Gommers <ralf.gomm...@gmail.com>
wrote:

>
>
> On Wed, Feb 10, 2016 at 3:30 PM, David Cournapeau <courn...@gmail.com>
> wrote:
>
>>
>>
>>
>> On Wed, Feb 10, 2016 at 1:52 PM, Paul Moore <p.f.mo...@gmail.com> wrote:
>>
>
>>> We should probably also check with the flit people that the proposed
>>> approach works for them. (Are there any other alternative build
>>> systems apart from flit that exist at present?)
>>>
>>
>> I am not working on it ATM, but bento was fairly complete and could
>> interoperate w/ pip (a few years ago at least):
>> https://cournape.github.io/Bento/
>>
>
> I plan to test with Bento (I'm still using it almost daily to work on
> Scipy) when an implementation is proposed for pip. The interface in the PEP
> is straightforward though, I don't see any fundamental reason why it
> wouldn't work for Bento if it works for flit.
>

It should indeed work, I was just pointing at an alternative build system ;)

I am a bit worried about making a PEP for interfacing before we have a few
decent alternative implementations. Having the official interface is not
necessary to actually interoperate, even if it is ugly.

David


> Ralf
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP: Build system abstraction for pip/conda etc

[David Cournapeau]

On Wed, Feb 10, 2016 at 1:52 PM, Paul Moore  wrote:

> On 10 February 2016 at 13:43, Paul Moore  wrote:
> >> In this case, the build system abstraction PEP should propose some
> >> additional text for
> >>
> https://packaging.python.org/en/latest/specifications/#source-distribution-format
> >> defining how to publish source archives containing a pypa.json file
> >> and the setup.py shim. At that point, it will effectively become the
> >> spec for sdist 1.0, since that's never previously been officially
> >> defined.
> >>
> >> The key difference from setuptools is that the setup.py shim will be a
> >> standard one that flit (and other source archive creation tools) can
> >> inject when building the sdist, rather than needing to be a custom
> >> file stored in each project's source repository.
> >
> > Neat! That sounds like a sensible approach, and if the build system
> > abstraction PEP adds this, then that addresses my remaining
> > objections.
>
> We should probably also check with the flit people that the proposed
> approach works for them. (Are there any other alternative build
> systems apart from flit that exist at present?)
>

I am not working on it ATM, but bento was fairly complete and could
interoperate w/ pip (a few years ago at least):
https://cournape.github.io/Bento/


>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Status update on the NumPy & SciPy vs SSE problem?

[David Cournapeau]

On Thu, Feb 4, 2016 at 11:42 AM, Antoine Pitrou  wrote:

> On Thu, 4 Feb 2016 21:22:32 +1000
> Nick Coghlan  wrote:
> >
> > I figured that was independent of the manylinux PEP (since it affects
> > Windows as well), but I'm also curious as to the current status (I
> > found a couple of apparently relevant threads on the NumPy list, but
> > figured it made more sense to just ask for an update rather than
> > trusting my Google-fu)
>
> While I'm not a Numpy maintainer, I don't think you can go much further
> than SSE2 (which is standard under the x86-64 definition).
>
> One factor is support by the kernel. The CentOS 5 kernel doesn't
> seem to support AVX, so you can't use AVX there even if your processor
> supports it (as the registers aren't preserved accross context
> switches). And one design point of manylinux is to support old Linux
> setups... (*)
>

I don't have precise numbers, but I can confirm we get from times to times
some customer reports related to avx not being supported (because of CPU or
OS).


>
> There are intermediate ISA additions between SSE2 and AVX (additions
> that don't require OS support), but I'm not sure they help much on
> compiler-vectorized code as opposed to hand-written assembly.  Numpy's
> pre-compiled loops are typically quite straightforward as far as I've
> seen.
>
> One mitigation is to delegate some operations to an optimized library
> implementing the appropriate runtime switches: for example linear
> algebra is delegated by Numpy and Scipy to optimized BLAS and LINPACK
> libraries (which exist in various implementations such as OpenBLAS or
> Intel's MKL).
>
> (*) (this is an issue a JIT compiler helps circumvent: it generates
> optimal code for the current CPU ;-))
>
> Regards
>
> Antoine.
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[David Cournapeau]

On Sat, Jan 30, 2016 at 8:37 AM, Nathaniel Smith  wrote:

> On Fri, Jan 29, 2016 at 11:52 PM, Nick Coghlan  wrote:
> > On 30 January 2016 at 09:29, Nathaniel Smith  wrote:
> >> Hi all,
> >>
> >> I think this is ready for pronouncement now -- thanks to everyone for
> >> all their feedback over the last few weeks!
> >>
> >> The only change relative to the last posting is that we rewrote the
> >> section on "Platform detection for installers", to switch to letting
> >> distributors explicitly control manylinux1 compatibility by means of a
> >> _manylinux module.
> >
> > In terms of the proposal itself, I think this version is excellent :)
> >
> > However, I realised that there's an implicit assumption we've been
> > making that really should be spelled out explicitly: manylinux1 wheels
> > targeting CPython 3.2 and earlier need to be compiled against a
> > CPython built in wide Unicode mode, and in those cases, the detection
> > of manylinux1 compatibility at the platform level should include
> > checking for "sys.maxunicode > 0x".
>
> Doh, excellent catch!
>
> I've just pushed the obvious update to handle this directly to the
> copy of the PEP in the manylinux repository.
>
> Diff:
> https://github.com/manylinux/manylinux/commit/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc
>
> New text in full:
>
> https://raw.githubusercontent.com/manylinux/manylinux/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc/pep-513.rst
>
> I haven't sent to the PEP editors, because they already have another
> diff from me sitting in their inboxes and I'm not sure how to do this
> in a way that doesn't confuse things :-)
>
> > The main reason we need to spell this out explicitly is that while
> > distros (and I believe other redistributors) build CPython-for-Linux
> > in wide mode as a matter of course, a Linux checkout of CPython 2.7
> > will build in narrow mode by default.
>
> I can confirm that Debian and Anaconda builds of CPython 2.7 both have
> sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
> 0x. Hmm. I guess they should fix that.
>

Yes, they should :) I am not sure why it was built this way (before my
time), it is unfortunately not easy to fix when you have a large existing
customer base.

David


> Also the manylinux docker image currently has sys.maxunicode ==
> 0x, so we should definitely fix that :-).
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[David Cournapeau]

On Wed, Jan 27, 2016 at 4:37 PM, Chris Barker <chris.bar...@noaa.gov> wrote:

>
>
> On Wed, Jan 27, 2016 at 1:37 AM, David Cournapeau <courn...@gmail.com>
> wrote:
>
>>
>> I will make sure to let the manylinux effort know when we decide to move
>> to Centos6 as the base system.
>>
>
> Thanks -- do you have any idea how many of your customers are running
> systems that old?
>
> i.e. have you stuck with CentOS5 because of actual customer demand as
> opposed to  uncertainty or inertia?
>

As mentioned by others before, Centos5 is a good way to ensure we link
against an old glibc (and few other key libraries, mostly X11-related).
That's really the main thing, as in general, you want to depend on the
system as little as possible when deploying binaries on Linux. Centos 6
uses glibc 2.12, which is newer than debian 6 and ubuntu 10.04 versions.
Even if debian 6 is still old, we see it on systems, and ubuntu 10.04 LTS
is definitely still out there in companies, even if officially unsupported.
And unsupported old versions of OS are used much more often than you may
think in enterprise (can't give names but companies anyone has heard of
still rely a lot on windows XP).

So now, one could argue that it is not the community's job to tackle old
OS, and they would be right, but:

1. Updating to e.g. 6 does not help that much, as the basic components
(compiler toolchain) are still old.
2. Updating the toolchain even on centos 5 is quite easy thanks to the
devtoolset effort.

The main argument against using centos 5 is GUI-related components, as the
old fontconfig/glib (the GTK one, not Gnu libc) are a problem. But those
are a tiny minority of what people do with python nowadays, and they
require a lot of work to get right.

David


>
> -CHB
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[David Cournapeau]

On Wed, Jan 27, 2016 at 5:18 PM, Donald Stufft <don...@stufft.io> wrote:

>
> On Jan 27, 2016, at 12:00 PM, David Cournapeau <courn...@gmail.com> wrote:
>
> So now, one could argue that it is not the community's job to tackle old
> OS, and they would be right, but:
>
>
> We can make a data driven decision here.
>

I like data-driven decision as well :)
Can you give an approximate of total download to convert this in % ?

David

>
> Here is the top 100 *nix OSs that are downloading files from PyPI using a
> version of pip >= 6.0. Any version of pip older than that and we don’t have
> insight into what they are (we can get kernel version if they’re using pip
> >= 1.4) and any installer other than that we don’t get any insight into
> either.
>
> One problem of course is deciding how representative only people who are
> using pip >= 6.0 is, though since we can’t get manylinux support into
> already released versions of pip it may be pretty representative of people
> who will use this feature (unless this feature causes people to upgrade
> their pip when they wouldn’t otherwise).
>
> Anyways, here’s the data:
> https://gist.github.com/dstufft/e1b1fbebb3482362198f
>
> It doesn’t matter to me if we use CentOS5 or CentOS6 as our base, but
> having some information to inform our choices is never a bad thing!
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[David Cournapeau]

On Wed, Jan 27, 2016 at 5:43 PM, Nathaniel Smith <n...@pobox.com> wrote:

> On Jan 27, 2016 09:00, "David Cournapeau" <courn...@gmail.com> wrote:
> >
> [...]
> > The main argument against using centos 5 is GUI-related components, as
> the old fontconfig/glib (the GTK one, not Gnu libc) are a problem. But
> those are a tiny minority of what people do with python nowadays, and they
> require a lot of work to get right.
>
> This is the part that intrigued me :-). Can you elaborate at all on what
> kind of problems you've encountered with fontconfig and glib?
>
> (Well, I can guess at one piece of excitement maybe: that glib is not
> really vendorable because a process can have only one main loop, and that
> lives in glib these days for both gtk and Qt, so any packages doing any GUI
> stuff are required to agree to use the same glib version.)
>

So vendoring glib is possible (we actually do it ATM, though we may revert
that). The problem is that now when you load say PyQt w/ Qt linked against
your vendored glib, you get into issues if that glib is higher than the
glib used in the system (that happens through pango IIRC). So if you want
to stay compatible, you need to build an old glib, which is what you were
trying to avoid in the first place. There is no good solution, really

David

> -n
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[David Cournapeau]

On Wed, Jan 27, 2016 at 12:02 AM, Chris Barker 
wrote:

> On Tue, Jan 26, 2016 at 3:56 PM, Nathaniel Smith  wrote:
>>
>> Well, the people who know what they're doing are still recommending
>> CentOS 5 today, and we don't know what we're doing :-).
>>
>
> well, yes, there is that. I sure don't. But the threshold for changing is
> higher than for starting fresh.
>
> Transitioning to a CentOS6-based manylinux2 shouldn't be a huge
>> problem --
>
>
> would CentOS5-based wheels run just fine on a centOS-6 based system ?
> Alongside CEntOS6-based wheels?
>
> If so, then I guess it's no biggie -- the safe bet is better.
>

I will make sure to let the manylinux effort know when we decide to move to
Centos6 as the base system.

David


>
> -CHB
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[David Cournapeau]

On Tue, Jan 26, 2016 at 11:44 AM, Antoine Pitrou 
wrote:

> On Tue, 26 Jan 2016 20:36:26 +1000
> Nick Coghlan  wrote:
> >
> > If I understand the problem correctly, the CentOS 5 gcc toolchain is
> > old enough that it simply doesn't emit the info libabigail needs in
> > order to work.
>
> If you build on CentOS 5, you certainly want to use the RH developer
> toolset 2 which gives you a modern-ish toolchain (gcc 4.8.2, IIRC).
>

Indeed, C++11 is the main reason why I added devtoolsset 2 in the
`Dockerfile` that was part of what originated the manylinux effort.

David


> Regards
>
> Antoine.
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[David Cournapeau]

On Thu, Jan 21, 2016 at 8:18 PM, Matthias Klose  wrote:

> On 21.01.2016 04:55, Nathaniel Smith wrote:
>
> the choice of compiler is questionable.  Just a pick into a release
> series. Not even the last released version on this release series.  Is this
> a good choice? Maybe for x86_64 and i386, but not for anything else.
>
> The permitted external shared libraries are: ::
>>
>>  libpanelw.so.5
>>  libncursesw.so.5
>>
>
> this doesn't include libtinfo, dependency of libncursesw.
>
>  libgcc_s.so.1
>>  libstdc++.so.6
>>
>
> if you insist on a version built from before GCC 5, you are in trouble
> with the libstdc++ ABI. So maybe link that one statically.
>
>  libgobject-2.0.so.0
>>  libgthread-2.0.so.0
>>  libglib-2.0.so.0
>>
>
> while glib2.0 is somehow frozen, what will you do if these change the
> soname?
>
> libgfortran is missing from this list while later down you mention
> gfortran. This will include libquadmath too.
>
> Any reason to not list libz?
>
> Compilation and Tooling
>> ===
>>
>
> so how are people supposed to build these wheels?  will you provide a
> development distribution, or do you "trust" people building such wheels?
>
> Platform Detection for Installers
>> =
>>
>> Because the ``manylinux1`` profile is already known to work for the many
>> thousands of users of popular commercial Python distributions, we suggest
>> that
>> installation tools like ``pip`` should error on the side of assuming that
>> a
>> system *is* compatible, unless there is specific reason to think
>> otherwise.
>>
>> We know of three main sources of potential incompatibility that are
>> likely to
>> arise in practice:
>>
>> * A linux distribution that is too old (e.g. RHEL 4)
>> * A linux distribution that does not use glibc (e.g. Alpine Linux, which
>> is
>>based on musl libc, or Android)
>> * Eventually, in the future, there may exist distributions that break
>>compatibility with this profile
>>
>
> add: "A Linux distribution that is built with clang", e.g. Mageia (libc++
> instead of libstdc++).
>
> Security Implications
>> =
>>
>> One of the advantages of dependencies on centralized libraries in Linux is
>> that bugfixes and security updates can be deployed system-wide, and
>> applications which depend on on these libraries will automatically feel
>> the
>> effects of these patches when the underlying libraries are updated. This
>> can
>> be particularly important for security updates in packages communication
>> across the network or cryptography.
>>
>> ``manylinux1`` wheels distributed through PyPI that bundle
>> security-critical
>> libraries like OpenSSL will thus assume responsibility for prompt updates
>> in
>> response disclosed vulnerabilities and patches. This closely parallels the
>> security implications of the distribution of binary wheels on Windows
>> that,
>> because the platform lacks a system package manager, generally bundle
>> their
>> dependencies. In particular, because its lacks a stable ABI, OpenSSL
>> cannot be
>> included in the ``manylinux1`` profile.
>>
>
> so you rely on the python build to provide this and access OpenSSL through
> the standard library?
>
>
> From my point of view this draft is too much influenced by Anaconda and
> their needs.  It doesn't talk at all about interaction with other system
> libraries, or interaction with extensions provided by distributions.


FWIW, the list of libraries and the dockerfile was originally built from
communications I had w/ Nathaniel and Matthew Brett, and I work for a
continuum's competitor, so you can be fairly confident that there is no
hidden agenda.

David



>
> Matthias
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[David Cournapeau]

On Thu, Jan 21, 2016 at 10:04 PM, David Cournapeau <courn...@gmail.com>
wrote:

>
>
> On Thu, Jan 21, 2016 at 7:42 PM, Chris Barker <chris.bar...@noaa.gov>
> wrote:
>
>> nice, idea, but
>>
>> libX11.so.6
>> libXext.so.6
>> libXrender.so.1
>> libGL.so.1
>>
>> These are all X11, yes? pretty much any workstation will have these, but
>> in general, servers won't.
>>
>
> Those would be required by GUI packages. People who know how to install
> headless systems would know how to handle errors because of missing sonames.
>

I would also mention that at Enthought, we have a fairly comprehensive
testing policy for our packages (e.g. running the test suite of a package
at every build as much as possible). We do the testing under headless
environments on every platform as much as possible, and it works for maybe
95 % of the packages (including things like matplotlib, etc...). E.g. on
Linux, most packages work well with a framebuffer if you only care about
offline rendering or testing.

David

I can talk from experience that this problem does not happen often.
>
> Note that this set of libraries was built from both what anaconda and
> canopy do
>
>>
>> Someone on this thread suggested that that's OK -- don't expect a GUI
>> package to work on a linux box without a GUI. But some of these libs might
>> get used for stuff like back-end rendering, etc, so would be expected to
>> work on a headless box. I think Anaconda an Canopy have gotten away with
>> this because both of their user bases are primarily desktop data analysis
>> type of stuff -- not web services, web servers, etc.
>>
>
> This is not true for us at Enthought, and I would be surprised if it were
> for anaconda.
>
>
>> So without a plan to provide all that stuff -- I"m not sure of the
>> utility of this -- how are you gong to get PIL/Pillow to work? statically
>> link up the ying-yang? Not sure the linux world will take to that.
>>
>
> We can explain how things work in details for some packages, but the main
> rationale for the PEP list is that this is a list that works in practice.
> It has worked well for us at Enthought for many years, and it has for
> (Ana)conda as well. Between both distributions, we are talking about
> millions of installs over the year, on many different systems.
>
> David
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[David Cournapeau]

On Thu, Jan 21, 2016 at 7:42 PM, Chris Barker  wrote:

> nice, idea, but
>
> libX11.so.6
> libXext.so.6
> libXrender.so.1
> libGL.so.1
>
> These are all X11, yes? pretty much any workstation will have these, but
> in general, servers won't.
>

Those would be required by GUI packages. People who know how to install
headless systems would know how to handle errors because of missing sonames.

I can talk from experience that this problem does not happen often.

Note that this set of libraries was built from both what anaconda and
canopy do

>
> Someone on this thread suggested that that's OK -- don't expect a GUI
> package to work on a linux box without a GUI. But some of these libs might
> get used for stuff like back-end rendering, etc, so would be expected to
> work on a headless box. I think Anaconda an Canopy have gotten away with
> this because both of their user bases are primarily desktop data analysis
> type of stuff -- not web services, web servers, etc.
>

This is not true for us at Enthought, and I would be surprised if it were
for anaconda.


> So without a plan to provide all that stuff -- I"m not sure of the utility
> of this -- how are you gong to get PIL/Pillow to work? statically link up
> the ying-yang? Not sure the linux world will take to that.
>

We can explain how things work in details for some packages, but the main
rationale for the PEP list is that this is a list that works in practice.
It has worked well for us at Enthought for many years, and it has for
(Ana)conda as well. Between both distributions, we are talking about
millions of installs over the year, on many different systems.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The Update Framework, integrate into PyPI?

[David Cournapeau]

On Wed, Nov 4, 2015 at 8:00 PM, Thomas Güttler  wrote:

> I read the RoadMap (Thank you Marcus Smith) and came across this:
>
> > An effort to integrate PyPI with the “The Update Framework” (TUF). This
> is specified in PEP458
>
> I see a trend to immutable systems everywhere.


Not everywhere. Keep in mind that there are a *lot* of different usecases
for packaging/deployment. Not just web app, not just CLI tools, etc... For
example, it is common for modern end user applications to use an
auto-update feature (e.g. chrome).

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Time for a setuptools_lite??

[David Cournapeau]

On Wed, Oct 28, 2015 at 9:04 AM, Nick Coghlan  wrote:

>
> On 27 Oct 2015 13:30, "Daniel Holth"  wrote:
> >
> > Mr. grumpy pants,
> >
> > Now you're trolling us by describing flit.
>
> I'd managed to miss Flit's creation, so I simply wasn't aware of it.
>
> Now that I've had a chance to remedy that oversight, yes, flit sounds
> exactly like what I meant, so it could be worth recommending it ahead of a
> full distutils/setuptools based setup.py for simple projects.
>

IMO, flit should be officially recommended until it at least supports
building sdist. Not having this feature is a significant pain point for
downstream distributors,

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wacky idea about reifying extras

[David Cournapeau]

On Mon, Oct 26, 2015 at 11:41 PM, Nathaniel Smith  wrote:

> On Mon, Oct 26, 2015 at 4:41 AM, Donald Stufft  wrote:
> > On October 26, 2015 at 3:36:47 AM, Nathaniel Smith (n...@pobox.com)
> wrote:
> >> > TL;DR
> >> -
> >>
> >> If we:
> >>
> >> - implement a real resolver, and
> >> - add a notion of a per-project namespace of distribution names,
> >> that
> >> are collected under the same PyPI registration and come from
> >> the same
> >> sdist, and
> >> - add Conflicts:, and Provides:,
> >>
> >> then we can elegantly solve a collection of important and difficult
> >> problems, and we can retroactively pun the old extras system
> >> onto the
> >> new system in a way that preserves 100% compatibility with all
> >> existing packages.
> >>
> >> I think?
> >>
> >> What do you think?
> >
> > My initial reaction when I started reading your idea was that I didn't
> see a
> > point in having something like foo[bar] be a "real" package when you
> could just
> > as easily have foo-bar. However, as I continued to read through the idea
> it
> > started to grow on me. I think I need to let it percolate in my brain a
> little
> > bit, but there may be a non-crazy (or at least, crazy in a good way)
> idea here
> > that could push things forward in a nice way.
>
> Oh good, at least I'm not the only one :-).
>
> I'd particularly like to hear Robert's thoughts when he has time,
> since the details depend strongly on some assumptions about how a real
> resolver would work.
>
> > Some random thoughts:
> >
> > * Reusing the extra syntax is nice because it doesn't require end users
> to
> >   learn any new concepts, however we shouldn't take a new syntax off the
> table
> >   either if it makes the feature easier to implement with regards to
> backwards
> >   compatability. Something like numpy{mkl,some-other-thing} could work
> just as
> >   well too. We'll need to make sure that whatever symbols we choose can
> be
> >   represented on all the major FS we care about and that they are
> ideally non
> >   ugly in an URL too. Of course, the filename and user interface symbols
> don't
> >   *need* to match. It could just as easily example numpy[mkl] out to
> numpy#mkl
> >   or whatever which should make it easier to come up with a nice scheme.
>
> Right -- obviously it would be *nice* to keep the number of concepts
> down, and to avoid skew between filenames and user interface (because
> users do see filenames), but if these turn out to be impossible then
> there are still options that would let us save the
> per-project-package-namespace idea.
>
> > * Provides is a bit of an odd duck, I think in my head I've mostly come
> to
> >   terms with allowing unrestricted Provides when you've already
> installed the
> >   package doing the Providing but completely ignoring the field when
> pulling
> >   data from a repository. Our threat model assumes that once you've
> selected to
> >   install something then it's generally safe to trust (though we still
> do try
> >   to limit that). The problem with Provides mostly comes into play when
> you
> >   will respect the Provides: field for any random package on PyPI (or
> any other
> >   repo).
>
> Yeah, I'm actually not too worried about malicious use either in
> practice, for the reason you say. But even so I can think of two good
> reasons we might want to be careful about stating exactly when
> "Provides:" can be trusted:
>
> 1) if you have neither scipy nor numpy installed, and you do 'pip
> install scipy', and scipy depends on the pure virtual package
> 'numpy[abi-2]' which is only available as a Provides: on the concrete
> package 'numpy', then in this case the resolver has to take Provides:
> into account when pulling data from the repo -- if it doesn't, then
> it'll ignore the Provides: on 'numpy' and say that scipy's
> dependencies can't be satisfied. So for this use case to work, we
> actually do need to be able to sometimes trust Provides: fields.
>
> 2) the basic idea of a resolver is that it considers a whole bunch of
> possible configurations for your environment, and picks the
> configuration that seems best. But if we pay attention to different
> metadata when installing as compared to after installation, then this
> skew makes it possible for the algorithm to pick a configuration that
> looks good a priori but is broken after installation. E.g. for a
> simple case:
>
>   Name: a
>   Conflicts: some-virtual-package
>
>   Name: b
>   Provides: some-virtual-package
>
> 'pip install a b' will work, because the resolver ignores the
> Provides: and treats the packages as non-conflicting -- but then once
> installed we have a broken system. This is obviously an artificial
> example, but creating the possibility of such messes just seems like
> the kind of headache we don't need. So I think whatever we do with
> Provides:, we should do the same thing both before and after
> installation.
>

Another simple solution for this particular case is to 

Re: [Distutils] Second draft of a plan for a new source tree / sdist format

[David Cournapeau]

On Tue, Oct 27, 2015 at 1:12 PM, Daniel Holth  wrote:

> The drawback of .zip is file size since it compresses each file
> individually rather than giving the compression algorithm a larger input,
> it's a great format otherwise. Ubiquitous including Apple iOS packages,
> Java, word processor file formats. And most Python packages are small.
>

I don't really buy the indexing advantages, especially w/ the current
implementation of zipfile in python (e.g. loading the whole set of archives
at creation time)

A common way to solve the fast metadata access from archive is to archive
separately the metadata data and data (e.g. a zipfile containing 2
zipfiles, one being the original sdist, the other one containing the
metadata).

David


> We must do the hard work to support Unicode file names, and spaces and
> accent marks in home directory names (historically a problem on Windows),
> in our packaging system. It is the right thing to do. It is not the
> publisher's fault that your system has broken Unicode.
>
> On Tue, Oct 27, 2015 at 6:43 AM Paul Moore  wrote:
>
>> On 26 October 2015 at 06:04, Nathaniel Smith  wrote:
>> > Here's a second round of text towards making a build-system
>> > independent interface between pip and source trees/sdists. My idea
>> > this time is to take a divide-and-conquer approach: this text tries to
>> > summarize all the stuff that it seemed like we had mostly reached
>> > consensus on in the previous thread + call, with blank chunks marked
>> > "TBD" where there are specific points that still need To Be
>> > Determined. So my hope is that everyone will read what's here and
>> > agree that it's great as far as it goes, and then we can go through
>> > and fill in each missing piece one at a time.
>>
>> I'll comment on what's here, but ignore the TBD items - I'd rather (as
>> you suggest) leave discussion of those details till the basic idea is
>> agreed.
>>
>> > Abstract
>> > 
>> >
>> > Distutils delenda est.
>>
>> While this makes a nice tagline, I'd rather something less negative.
>> Distutils does not "need" to be destroyed. It's perfectly adequate
>> (although hardly user friendly) for a lot of cases - I'd be willing to
>> suggest *most* users can work just fine with distutils.
>>
>> I'm not a fan of distutils, but I'd prefer it if we kept the rhetoric
>> limited - as Nick pointed out this whole area is as much a political
>> issue as a technical one.
>>
>> > Extended abstract
>> > =
>> >
>> > While ``distutils`` / ``setuptools`` have taken us a long way, they
>> > suffer from three serious problems: (a) they're missing important
>> > features like autoconfiguration and usable build-time dependency
>> > declaration, (b) extending them is quirky, complicated, and fragile,
>> > (c) it's very difficult to use anything else, because they provide the
>> > standard interface for installing python packages expected by both
>> > users and installation tools like ``pip``.
>>
>> Again, this is overstated. You very nearly lost me right here - people
>> won't read the details of the proposal if they disagree with the
>> abstract(s). Specifically:
>>
>> * The features in (a) are only important to *some* parts of the
>> community. The scientific community is the major one, and is a huge
>> influence over the direction we want to go in, but again, not crucial
>> to many people. And even where they might be useful (e.g., Windows
>> users building pyyaml, lxml, pillow, ...) the description implies
>> "working out what's there" rather than "allowing users to easily
>> manage non-Python dependencies", which gives the wrong impression.
>>
>> * The features in (b) are highly specialised. Very few people extend
>> setuptools/distutils. And those who do, have often invested a lot of
>> effort in doing so. Sure, they'd rather not have needed to, but now
>> that they have, a replacement system simply means that work is lost.
>> Arguably, fixing (b) is only useful for people (like the scientific
>> community) who have needed to extend setuptools and have been unable
>> to achieve their goals that way. That's an even smaller part of the
>> community.
>>
>> > Previous efforts (e.g. distutils2 or setuptools itself) have attempted
>> > to solve problems (a) and/or (b). We propose to solve (c).
>>
>> Agreed - this is a good approach. But it's at odds with your abstract,
>> which says distutils must die. Here you're saying you want to allow
>> people to keep using distutils but allow people with specialised needs
>> to choose an alternative. Or are you offering an alternative to people
>> who use distutils?
>>
>> The whole of the above is confusing on the face of it. The details
>> below clarify a lot, as does knowing how the previous discussions have
>> gone. But it would help a lot if the introduction to this PEP were
>> clearer.
>>
>> > The goal of this PEP is get distutils-sig out of the business of being
>> > a gatekeeper for 

Re: [Distutils] Monkey patching distutils was: Remove distutils, was: red, green, refactor ...

[David Cournapeau]

On Thu, Oct 22, 2015 at 5:04 PM, Thomas Güttler <
guettl...@thomas-guettler.de> wrote:

> Am 21.10.2015 um 17:15 schrieb Antoine Pitrou:
> > On Wed, 21 Oct 2015 17:05:29 +0200
> > Nick Coghlan <ncogh...@gmail.com> wrote:
> >> On 21 October 2015 at 14:55, David Cournapeau <courn...@gmail.com>
> wrote:
> >>>
> >>> On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler
> >>> <guettl...@thomas-guettler.de> wrote:
> >>>> ok, at the moment setuptools uses distutils.
> >>>>
> >>>> Why not melt them together into **one** underwear-pants-module?
> >>>
> >>>
> >>> What do you hope getting from that ? distutils is in the stdlib, so
> cannot
> >>> change easily, and even if putting setuptools in the stdlib were
> possible,
> >>> you would now need to handle different versions of setuptools for
> different
> >>> versions of python.
> >>
> >> It's more useful to go the other direction and vendor a modern version
> >> of distutils inside setuptools:
> >
> > It seems it would only add a bit more craziness to the current
> > landscape. What happens to projects which have a need to monkeypatch
> > distutils? How does it interact with the vendored version? etc.
>
> What are the needs to monkeypatch distutils?
>
>
That's the only way to extend distutils in a meaningful way, because of the
broken command concept (e.g. the command dependency graph is defined inside
the classes instead of being defined by an external scheduler).

As anybody who has dealt with that codebase will tell you, It is impossible
to fix distutils without breaking most non trivial uses of it.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Time for a setuptools_lite??

[David Cournapeau]

On Wed, Oct 21, 2015 at 6:44 PM, Wayne Werner <waynejwer...@gmail.com>
wrote:

>
> On Wed, Oct 21, 2015 at 12:32 PM, David Cournapeau <courn...@gmail.com>
> wrote:
>
>>
>>
>> On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt <
>> opensou...@ronnypfannschmidt.de> wrote:
>>
>>> why does that have to be in setuptools ?!
>>>
>>> if we want a new light system to begin with, shouldn't it be far more
>>> sustainable to use just implementations of the new standards rather than
>>> leaving all of setuptools
>>>
>>> there is no viable way in setuptools to get rid of the legacy ina sane
>>> and fast manner, it would drag out over years
>>>
>>
>> agreed. I have never met a person who had to deal substantially with
>> distutils code and enjoyed the experience.
>>
>> The whole architecture is fundamentally flawed. I wrote this a long time
>> ago, but I still stand by most arguments:
>> https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/
>>
>
> I've (luckily?) never had to deal with distutils code... I am definitely
> digging the post. First time I've read it, but I am definitely more
> pro-standard-interface-and-let-tools-do-with-it-what-they-will than I was a
> few minutes ago.
>
> Would pip's freeze format work a la the cabal file, or is it missing too
> much information?
>

the main cabal file is its own DSL that describes the package metadata (so
more a replacement of setup.py).

I used a similar idea when I wrote bento (https://cournape.github.io/Bento/),
which ended up being a mistake. If I were to write a similar tool today, I
would just use templated yaml. A big difference compared to 6-7 years ago
is the progress made on pip, wheels, etc... A lot of the current efforts in
python package (metadata formalization, etc...) are about enabling tools
like bento, flit, etc... to coexist:  as long as they produced
wheels/(wheel)sdist/etc... that pip can consume, you have a clear separate
of concern, and an healthy competition.

David

>
> -Wayne
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Time for a setuptools_lite??

[David Cournapeau]

On Wed, Oct 21, 2015 at 10:25 PM, Antoine Pitrou 
wrote:

> On Wed, 21 Oct 2015 21:41:35 +0100
> Paul Moore  wrote:
> > On 21 October 2015 at 20:41, Chris Barker  wrote:
> > > As I understand it, the current "API" that pip knows about is:
> > >
> > > "some subset of what setuptools provides"
> > >
> > > So my proposal is here is to provide a way for users to easily use
> jsut that
> > > subset.
> >
> >
> https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface
> >
> > All you need to do is write a setup.py, which can do *anything you
> > like*,
>
> Reading this, the CLI options which have to be implemented are
> completely tied to setuptools' own view of the world.
> `--single-version-externally-managed`? `--install-headers`? Why should
> a random build system care about that gunk? What should it do with it?
>

I agree it is not ideal, but it is not *that* hard. Regarding
`--single-version-externally-managed`, if you don't care about setuptools
interoperability, I believe it is safe to just ignore the flag (or more
exactly do like what distutils does, that is installed directly in
site-packages, and make `--single-version-externally-managed` a no-op).
install_headers, I am not sure how many projects depend on that. The only
projects I know that install headers (numpy, etc...) are not using the
setuptools mechanism.

If that's deemed useful, I could write a simple implementation that does
not use distutils/setuptools but support that interface as an example (I
have something similar enough times that I unfortunately know the ropes :)
).


> I think Nathaniel's PEP, for all its shortcomings, looked much saner
> than that piece of ad-hoc specification (a.k.a. "here's the random set
> of things we're currently doing, let's make a spec out of it"). This is
> like the Microsoft OOXML of Python packages distribution.
>
> Regards
>
> Antoine.
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Time for a setuptools_lite??

[David Cournapeau]

On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt <
opensou...@ronnypfannschmidt.de> wrote:

> why does that have to be in setuptools ?!
>
> if we want a new light system to begin with, shouldn't it be far more
> sustainable to use just implementations of the new standards rather than
> leaving all of setuptools
>
> there is no viable way in setuptools to get rid of the legacy ina sane and
> fast manner, it would drag out over years
>

agreed. I have never met a person who had to deal substantially with
distutils code and enjoyed the experience.

The whole architecture is fundamentally flawed. I wrote this a long time
ago, but I still stand by most arguments:
https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/

David

>
> and both distutils and setuptools are very very sub-par from the
> maintenance perspective
>
>
> On 10/21/2015 06:42 PM, Chris Barker wrote:
>
> This is kind-of related to the othe thread:
>
> "Remove distutils, was: ..."
>
> But more specific, so I thought I'd start a new one.
>
> Here are my thoughts:
>
> We had distutils -- there was a lot it didn't do that the "Masses" needed,
> so setuptools was born. It proved to be useful to a lot of people, and grew
> a large userbase
>
> But a lot was "wrong" with setuptools -- most prominently (in my mind
> anyway) that it put too many kind-sorta orthogonal stuff into one package:
> building, installing, distributing, managing version, managing
> dependencies, managing non-python resources, (and others??). And we didn't
> like how easy-install installed things :-)
>
> So distribute, and pip, and wheel, and now a new backward compatible
> setuptools was born.
>
> But we still have a bunch of should be orthogonal stuff tangled up
> together. In particular, I find that I often find easy-install getting
> invoked when I don't want ot to, and I get those darn eggs scattered all
> over the place, and easy_install.pth, and 
>
> I think if I am really careful about what I invoke when, this could be
> avoided, but the reality is that I've been dealing with this for years, and
> am trying desperately to do things the "right, modern" way, and I still get
> ugliness. I seriously doubt that I am alone.
>
> So -- here's my thought:
>
> I think we have it pretty well mapped out what functionality belongs where:
>
> one system for building packages (setuptools?)
> one system for installing packages and managing dependencies (pip)
> one system (really standard) for metadata and distributing packages (wheel)
>
> [I'm just whipping this out off the top of my head, I'm sure we'd need to
> more clearly define what belongs where]
>
> So why not have a setuptools-lite that only does the building? We need to
> keep the full over-burdened setuptools around, because lot sof folks are
> using those features. But for those of us that are doing something fairly
> new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd
> love to have a setuptools-lite that only did what I really need, and was
> guaranteed NOT to accidentally introduce easy_install, etc...
>
> This seems to me to be a way to go forward -- as it is we'll have people
> using setuptools features that they "shouldn't" forever, and never be able
> to move to a cleaner system.
>
> Or maybe a flag:
>
> import setuptools
> setuptools.use_only_modern()
>
> That would make the dependencies easier -- i.e. pip depends on some of
> setuptools being there -- hard to say that it needs either setuptools OR
> setuptools_lite.
>
> Maybe I'm missing something here, but if the goal is for there to be one
> way to do things, let's have a tool chain that only does things one way.
>
> -Chris
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
>
> ___
> Distutils-SIG maillist  -  
> Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Remove distutils, was: red, green, refactor ...

[David Cournapeau]

On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler <
guettl...@thomas-guettler.de> wrote:

> Am 21.10.2015 um 13:28 schrieb Ionel Cristian Mărieș:
>
>>
>> On Wed, Oct 21, 2015 at 1:42 PM, Thomas Güttler <
>> guettl...@thomas-guettler.de >
>> wrote:
>>
>>
>> Why not deprecate and remove distutils?
>>
>>
>> ​Should we or can we? There was an attempt at that, called distutils2.
>> I'd love to hear what people have learnt from that.
>>
>> And if now-days walking around with pants instead of only underwear is
>> the way to go, should we still have underwear
>> under the pants? I think yes :-) In other words, setuptools uses
>> distutils.
>>
>
> ok, at the moment setuptools uses distutils.
>
> Why not melt them together into **one** underwear-pants-module?
>

What do you hope getting from that ? distutils is in the stdlib, so cannot
change easily, and even if putting setuptools in the stdlib were possible,
you would now need to handle different versions of setuptools for different
versions of python.

On top of this, the goal of lots of efforts around packaging is to allow
people to move away from distutils/setuptools, as the underlying design is
fundamentally difficult to extend.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[David Cournapeau]

On Wed, Oct 14, 2015 at 8:52 PM, Chris Barker  wrote:

>
>
> On Wed, Oct 14, 2015 at 9:56 AM, Dave Forgac 
> wrote:
>
>> This was discussed recently here:
>> https://github.com/pypa/packaging-problems/issues/74
>>
>
> and on this list at other times. Though the above issue was pretty focused
> on restoring a deleted file without any changes -- which seems like a
> no-brainer to me, as long as someone wants to put the effort into the
> infrastructure.
>
> (the soft delete option seems like a good idea to me).
>
> But I'm talking about the cases of "whoops! I really wish I hadn't
> uploaded that one". We can improve the tooling (some discussion on this in
> this thread right now...), but people are people and some of us are stupid
> and/or careless. So this WILL happen.
>
> And it just seems pedantic to say: hey -- you've already put that one
> there -- maybe even two minutes ago, so there is NO WAY to fix your
> mistake. If it happens quickly, then no one has downloaded it, it hasn't
> made its way to the mirrors, etc...
>

It is not pendantic, for reasons mentioned by Donald.

I have not done numpy releases for half a decade now, but it was already
automated enough that putting a new version was not very costly then. And
you did not have travis-ci, appveyor, tox, ubiquitous AWS 5 years ago ...

I am sure there are things we can do to improve numpy's release process to
avoid this in the future.

David


> And again -- we are all adults here: if you as the package maintainer want
> to do somethign that is sonfusing to your users, is it up to PyPi to never
> let you do that? (I think it IS up to PyPi to strongly recommend that you
> don't -- i.e. make it hard to do, and impossible to do thoughtlessly)
>
> On Wed, Oct 14, 2015 at 10:00 AM, Jeremy Stanley 
>  wrote:
>
>> > You should have to jump through all sorts of hoops, and make it
>> > really clear that it is a BAD IDEA in the general case, but it'd
>> > be good to have it be possible.
>> [...]
>>
>> It used to be possible.
>
>
> Was it also easy to do without very careful consideration? Or were the
> hoops I mentioned in place?
>
> I can't find it right now, but I think someone in this thread suggested a
> "staging area", so we could essentially do a trail run: upload to PyPi,
> tell a few trusted friends about it, have them test it, then, and only
> then, push it to the official channel. Maybe the infrastructure for that
> would be more work than it's worth, but I like it.
>
> This would fit into the basic principle that you should always be able to
> test something in as close the final environment as possible, before you
> commit to it.
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[David Cournapeau]

On Wed, Oct 14, 2015 at 11:23 PM, Chris Barker <chris.bar...@noaa.gov>
wrote:

> On Wed, Oct 14, 2015 at 1:59 PM, David Cournapeau <courn...@gmail.com>
> wrote:
>
>> But I'm talking about the cases of "whoops! I really wish I hadn't
>>> uploaded that one". We can improve the tooling (some discussion on this in
>>> this thread right now...), but people are people and some of us are stupid
>>> and/or careless. So this WILL happen.
>>>
>>
>
>> I have not done numpy releases for half a decade now, but it was already
>>> automated enough that putting a new version was not very costly then.
>>>
>>
> yeah, I suppose releases are cheap -- though part of the problem is that
> your users are likley to think that you actually fixed a bug or something.
> And maybe wonder why you went from 1.2.3 to 1.3.5 seemingly all at once...
>
> another note-- conda has teh concetp of a "build" that's tacked on teh
> release for conda pacakges.
>

Note that build is a feature for binaries: different builds refer to the
same upstream source. Most linux distributions have the notion of
downstream version which is a generalization of that.

David


> So if I updated somethign about how teh packge is buitl, but am using teh
> same underllying version of teh package, I update teh build number, get a
> new "version" of the package, but it's clear that the pacakge itself is the
> same version.
>
> for instance, I'm messing around right now with building libgd for conda,
> and the latest version I have up on anaconda.org is:
>
> libgd-2.1.1
>
> but the actual file is:
>
> libgd-2.1.1-1.tar.bz2
>
> (and there is a libgd-2.1.1-0.tar.bz2 there too...)
>
> Maybe this would be helpful for PyPi, too?
>
> I am sure there are things we can do to improve numpy's release process to
>> avoid this in the future.
>>
>
> numpy was just an example -- we are all likely to make mistakes in the
> future - it's human nature.
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[David Cournapeau]

On Tue, Oct 6, 2015 at 9:30 AM, Ionel Cristian Mărieș 
wrote:

>
> On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou 
> wrote:
>
>> They should be inside the module. That way, you can check an installed
>> module is ok by running e.g. "python -m mypackage.tests". Any other
>> choice makes testing installed modules more cumbersome.
>>
>
> ​Does that really make sense? I haven't heard of any user actually running
> tests​
> that way. To be honest I haven't ever ran Python's own tests suite as part
> of a user installation.
>

It makes a lot of sense for downstream packagers. Allowing testing
installed packages is also the simplest way to enable testing on target
machines which are different from the build machines.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[David Cournapeau]

On Tue, Oct 6, 2015 at 10:54 AM, Donald Stufft  wrote:

> On October 6, 2015 at 3:21:04 AM, Glyph Lefkowitz (gl...@twistedmatrix.com)
> wrote:
> > > Inside the package.
> >
> > If you put your tests outside your package, then you can't install
> > the tests for two packages simultaneously, because everyone's
> > tests are just in the top-level package "tests". This tends to
> > infest the whole package, since then tests import things from
> > each other using 'from tests import ...'. This is recommended
> > by the hitchhiker's guide, and seconded by .
>
> I dislike putting tests inside the package.
>
> The supposed benefit is that anyone can run the tests at anytime, but I
> don't
> find that actually true because it means (as someone else pointed out)
> that you
> either have to depend on all your test dependencies or that there is
> already an
> additional step to install them. If you're going to have to locate and
> install
> the test dependencies, then you might as well fetch the tarball with tests
> as
> well.
>
> Someone suggested setuptools test_requires, but that only functions when
> you
> have a setup.py available and you execute ``setup.py test``. It does not
> help
> you at all once the package is installed and the sdist is gone.
>
> I also don't think people actually run the tests when they are installed
> in any
> significant number, at least I've never once in my life done it or even
> had a
> desire to do it.
>

The significant number is not so relevant if you buy the argument that it
is useful to downstream packagers: it may be a few users, but those are
crucial.

I also forgot to mention that  the ability to test something without
building is crucial when you want to distribute binaries.

David


> Some projects have test suites which are significantly large too. The PyCA
> cryptography project for instance, has to ship it's vectors as an
> additional
> package to reduce the size of the final build product.
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[David Cournapeau]

On Tue, Oct 6, 2015 at 11:21 AM, Donald Stufft <don...@stufft.io> wrote:

> On October 6, 2015 at 6:18:32 AM, David Cournapeau (courn...@gmail.com)
> wrote:
> > > The significant number is not so relevant if you buy the argument
> > that it is useful to downstream packagers: it may be a few users,
> > but those are crucial.
> >
> > I also forgot to mention that the ability to test something without
> > building is crucial when you want to distribute binaries.
>
> Is it actually useful to them? None of the Linux downstreams I know of have
> ever mentioned a preference for it. As far as I know, the only preference
> they've ever expressed to me is that the tests are included in the sdist.
>

It is at least useful to me, and I am packaging quite a few binaries.


>
> FreeBSD relies on ``python setup.py test`` as it's preferred test
> invocation,
> so it apparently doesn't find it useful either.
>

I would like to hear their rationale before guessing. It is hard for me to
imagine they would not rather test the binaries rather than from sources.
Something as simple as making sure you have not forgotten runtime
dependencies becomes much easier this way.

David


>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[David Cournapeau]

On Tue, Oct 6, 2015 at 1:57 PM, Donald Stufft  wrote:

>
>
> On October 6, 2015 at 8:51:30 AM, Antoine Pitrou (solip...@pitrou.net)
> wrote:
> > On Tue, 6 Oct 2015 15:34:38 +0300
> > Ionel Cristian Mărieș wrote:
> > >
> > > Very few
> > > test runners change the current working directory by default [1], so
> it's
> > > better to just get a better project layout. pyca/cryptography
> > > is a good example.
> >
> > The "src" convention is actually terrible when working with Python
> > code, since suddenly you can't experiment easily on a VCS checkout, you
> > have to do extra steps and/or write helper scripts for it.
>
> Without doing it, you have very little assurances you’re actually testing
> against the installed project and not the project that's sitting in curdir.
> This is why pyca/cryptography does it, attempting to run the copy in .
> won't
> do anything but raise an exception since the .so won't be built.
>
> It doesn't really make experimenting in a VCS any harder, since all you
> need to
> do first is run ``pip install -e .`` and it will do a development install
> and
> add the src/ directory to sys.path.
>
> >
> > The fact that few Python projects, including amongst the most popular
> > projects, use that convention mean it's really not considered a good
> > practice, nor convenient.
> >
> >
>
> Of course, the same argument can be made for installing tests, since it's
> not
> very common.
>

So we can actually get some data here :)

At Enthought, we support around 400 packages. More than half of them are
python packages, and we can make the reasonable assumption than the vast
majority of those packages are fairly popular.

More precisely, if I install all our supported packages on Linux:
 - I count ~ 308 packages by making the assumption that one directory in
site-packages is a package (wrong but decent approximation)
 - I count as `with tests` a package with at least one directory called
test in say package

With those assumptions, I count 46 % packages with tests installed. So it
is not "not very common". Granted, we are biased toward scientific packages
which to include tests in the package, but we also package popular
web/etc... packages.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[David Cournapeau]

On Fri, Oct 2, 2015 at 12:58 PM, Donald Stufft  wrote:

> On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (n...@pobox.com) wrote:
> > > We realized that actually as far as we could tell, it wouldn't
> > be that
> > hard at this point to clean up how sdists work so that it would be
> > possible to migrate away from distutils. So we wrote up a little
> > draft
> > proposal.
> >
> > The main question is, does this approach seem sound?
>
> I've just read over your proposal, but I've also just woken up so I might
> be
> a little slow still! After reading what you have, I don't think that this
> proposal is the right way to go about improving sdists.
>
> The first thing that immediately stood out to me, is that it's recommending
> that downstream redistributors like Debian, Fedora, etc utilize Wheels
> instead
> of the sdist to build their packages from. However, that is not really
> going to
> fly with most (all?) of the downstream redistributors. Debian for instance
> has
> policy that requires the use of building all of it's packages from Source,
> not
> from anything else and Wheels are not a source package. While it can
> theoretically work for pure python packages, it quickly devolves into a
> mess
> when you factor in packages that have any C code what so ever.
>
> Overall, this feels more like a sidegrade than an upgrade. One major theme
> throughout of the PEP is that we're going to push to rely heavily on
> wheels as
> the primary format of installation. While that works well for things like
> Debian, I don't think it's going to work as wheel for us. If we were only
> distributing pure python packages, then yes absolutely, however given that
> we
> are not, we have to worry about ABI issues. Given that there is so many
> different environments that a particular package might be installed into,
> all
> with different ABIs we have to assume that installing from source is still
> going to be a primary path for end users to install and that we are never
> going
> to have a world where we can assume a Wheel in a repository.
>
> One of the problems with the current system, is that we have no mechanism
> by
> which to determine dependencies of a source distribution without
> downloading
> the file and executing some potentially untrusted code. This makes
> dependency
> resolution harder and much much slower than if we could read that
> information
> statically from a source distribution. This PEP doesn't offer anything in
> the
> way of solving this problem.
>
> To a similar tune, this PEP also doesn't make it possible to really get at
> any other metadata without executing software. This makes it pratically
> impossible to safely inspect an unknown or untrusted package to determine
> what
> it is and to get information about it. Right now PyPI relies on the
> uploading
> tool to send that information alongside of the file it is uploading, but
> honestly what it should be doing is extracting that information from
> within the
> file. This is sort of possible right now since distutils and setuptools
> both
> create a static metadata file within the source distribution, but we don't
> rely
> on that within PyPI because that information may or may not be accurate
> and may
> or may not exist. However the twine uploading tool *does* rely on that, and
> this PEP would break the ability for twine to upload a package without
> executing arbitrary code.
>
> Overall, I don't think that this really solves most of the foundational
> problems with the current format. Largely it feels that what it achieves is
> shuffling around some logic (you need to create a hook that you reference
> from
> within a .cfg file instead of creating a setuptools extension or so) but
> without fixing most of the problems. The largest benefit I see to
> switching to
> this right now is that it would enable us to have build time dependencies
> that
> were controlled by pip rather than installed implicitly via the execution
> of
> the setup.py. That doesn't feel like a big enough benefit to me to do a
> mass
> shakeup of what we recommend and tell people to do. Having people adjust
> and
> change and do something new requires effort, and we need something to
> justify
> that effort to other people and I don't think that this PEP has something
> we
> can really use to justify that effort.
>
> I *do* think that there is a core of some ideas here that are valuable,
> and in
> fact are similar to some ideas I've had. The main flaw I see here is that
> it
> doesn't really fix sdists, it takes a solution that would work for VCS
> checkouts and then reuses it for sdists. In my mind, the supported flow for
> package installation would be:
>
> VCS/Bare Directory -> Source Distribution -> Wheel
>
> This would (eventually) be the only path that was supported for
> installation
> but you could "enter" the path at any stage. For example, if there is a
> Wheel
> already available, then you jump right on at the end and just install
> 

Re: [Distutils] PEP for dependencies on libraries like BLAS (was: Re: Working toward Linux wheel support)

[David Cournapeau]

On Thu, Aug 13, 2015 at 7:08 AM, Nathaniel Smith n...@pobox.com wrote:

 On Wed, Aug 12, 2015 at 8:10 PM, Robert Collins
 robe...@robertcollins.net wrote:
  On 13 August 2015 at 12:51, Nathaniel Smith n...@pobox.com wrote:
  On Aug 12, 2015 16:49, Robert Collins robe...@robertcollins.net
 wrote:
 
  I'm not sure what will be needed to get the PR accepted; At PyCon AU
  Tennessee Leuwenberg started drafting a PEP for the expression of
  dependencies on e.g. BLAS - its been given number 497, and is in the
  packaging-peps repo; I'm working on updating it now.
 
  I wanted to take a look at this PEP, but I can't seem to find it. PEP
 497:
https://www.python.org/dev/peps/pep-0497/
  appears to be something else entirely?
 
  I'm a bit surprised to hear that such a PEP is needed. We (= numpy devs)
  have actively been making plans to ship a BLAS wheel on windows, and
 AFAICT
  this is totally doable now -- the blocker is windows toolchain issues,
 not
  pypa-related infrastructure.
 
  Specifically the idea is to have a wheel that contains the shared
 library as
  a regular old data file, plus a stub python package that knows how to
 find
  this data file and how to make it accessible to the linker. So
  numpy/__init__.py would start by calling:
 
  import pyopenblas1
  # on Linux modifies LD_LIBRARY_PATH,
  # on Windows uses ctypes to preload... whatever
  pyopenblas1.enable()
 
  and then get on with things, or the build system might do:
 
  import pyopenblas1
  pyopenblas1.get_header_directories()
  pyopenblas1.get_linker_directories()
 

 Thanks to James for sending on the link!

 Two main thoughts, now that I've read it over:

 1) The motivating example is somewhat confused -- the draft says:

 + The example provided in the abstract is a
 + hypothetical package which needs versions of numpy and scipy, both of
 which
 + must have been compiled to be aware of the ATLAS compiled set of
 linear algebra
 + libraries (for performance reasons). This sounds esoteric but is, in
 fact, a
 + routinely encountered situation which drives people towards using the
 + alternative packaging for scientific python environments.

 Numpy and scipy actually work hard to export a consistent, append-only
 ABI regardless of what libraries are used underneath. (This is
 actually by far our biggest issue with wheels -- that there's still no
 way to tag the numpy ABI as part of the ABI string, so in practice
 it's just impossible to ever have a smooth migration to a new ABI and
 we have no choice but to forever maintain compatibility with numpy
 0.1. But that's not what this proposal addresses.) Possibly part of
 the confusion here is that Christoph Gohlke's popular numpy+scipy
 builds use a hack where instead of making the wheels self-contained
 via statically linking or something like that, then he ships the
 actual libBLAS.dll inside the numpy wheel, and then the scipy wheel
 has some code added that magically knows that there is this special
 numpy wheel that it can find libBLAS.dll inside and use it directly
 from scipy's own extensions. But this coupling is pretty much just
 broken, and it directly motivates the blas-in-its-own-wheel design I
 sketched out above.

 (I guess the one exception is that if you have a numpy or scipy build
 that dynamically links to a library like BLAS, and then another
 extension that links to a different BLAS with an incompatible ABI, and
 the two BLAS libraries have symbol name collisions, then that could be
 a problem because ELF is frustrating like that. But the obvious
 solution here is to be careful about how you do your builds -- either
 by using static linking, or making sure that incompatible ABIs get
 different symbol names.)

 Anyway, this doesn't particularly undermine the PEP, but it would be
 good to use a more realistic motivating example.

 2) AFAICT, the basic goal of this PEP is to provide machinery to let
 one reliably build a wheel for some specific version of some specific
 distribution, while depending on vendor-provided libraries for various
 external dependencies, and providing a nice user experience (e.g.,
 telling users explicitly which vendor-provided libraries they need to
 install). I say this because strings like libblas1.so or kernel.h
 do not define any fixed ABI or APIs, unless you are implicitly scoping
 to some particular distribution with at least some minimum version
 constraint.

 It seems like a reasonable effort at solving this problem, and I guess
 there are probably some people somewhere that have this problem, but
 my concern is that I don't actually know any of those people. The
 developers I know instead have the problem of, they want to be able to
 provide a small finite number of binaries (ideally six binaries per
 Python version: {32 bit, 64 bit} * {windows, osx, linux}) that
 together will Just Work on 99% of end-user systems. And that's the
 problem that Enthought, Continuum, etc., have been solving for years,
 and which wheels already mostly 

Re: [Distutils] Working toward Linux wheel support

[David Cournapeau]

On Thu, Aug 13, 2015 at 2:05 AM, Nathaniel Smith n...@pobox.com wrote:

 On Aug 12, 2015 13:57, Nate Coraor n...@bx.psu.edu wrote:
 
  Hello all,
 
  I've implemented the wheel side of Nick's suggestion from very early in
 this thread to support a vendor-providable binary-compatibility.cfg.
 
https://bitbucket.org/pypa/wheel/pull-request/54/
 
  If this is acceptable, I'll add support for it to the pip side. What
 else should be implemented at this stage to get the PR accepted?

 From my reading of what the Enthought and Continuum folks were saying
 about how they are successfully distributing binaries across different
 distributions, it sounds like the additional piece that would take this
 from a interesting experiment to basically-immediately-usable would be to
 teach pip that if no binary-compatibility.cfg is provided, then it should
 assume by default that the compatible systems whose wheels should be
 installed are: (1) the current system's exact tag, (2) the special
 hard-coded tag centos5. (That's what everyone actually uses in practice,
 right?)

 To make this *really* slick, it would be cool if, say, David C. could make
 a formal list of exactly which system libraries are important to depend on
 (xlib, etc.), and we could hard-code two compatibility profiles
 centos5-minimal (= just glibc and the C++ runtime) and centos5 (= that
 plus the core too-hard-to-ship libraries), and possibly teach pip how to
 check whether that hard-coded core set is available.


So this is a basic list I got w/ a few minutes of scripting, by installing
our 200 most used packages on centos 5, ldd'ing all of the .so, and
filtering out a few things/bugs of some of our own packages):

/usr/lib64/libatk-1.0.so.0
/usr/lib64/libcairo.so.2
/usr/lib64/libdrm.so.2
/usr/lib64/libfontconfig.so.1
/usr/lib64/libGL.so.1
/usr/lib64/libGLU.so.1
/usr/lib64/libstdc++.so.6
/usr/lib64/libX11.so.6
/usr/lib64/libXau.so.6
/usr/lib64/libXcursor.so.1
/usr/lib64/libXdmcp.so.6
/usr/lib64/libXext.so.6
/usr/lib64/libXfixes.so.3
/usr/lib64/libXft.so.2
/usr/lib64/libXinerama.so.1
/usr/lib64/libXi.so.6
/usr/lib64/libXrandr.so.2
/usr/lib64/libXrender.so.1
/usr/lib64/libXt.so.6
/usr/lib64/libXv.so.1
/usr/lib64/libXxf86vm.so.1
/usr/lib64/libz.so.1

This list should only be taken as a first idea, I can work on a more
precise list including the versions if that's deemed useful.

One significant issue is SSL: in theory, we (as a downstream distributor)
really want to avoid distributing such a key piece of infrastructure, but
in practice, there are so many versions which are incompatible across
distributions that it is not an option.

David

 Compare with osx, where there are actually a ton of different ABIs but in
 practice everyone distributing wheels basically sat down and picked one and
 wrote some ad hoc tools to make it work, and it does:
 https://github.com/MacPython/wiki/wiki/Spinning-wheels

 -n

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[David Cournapeau]

On Fri, Jul 10, 2015 at 12:16 AM, Ionel Cristian Mărieș cont...@ionelmc.ro
wrote:

 Would be quite useful to see some references and details about the vague
 issues being mentioned in the thread. It would help a lot the less versed
 engineers (like me) understand the issues at hand (and hopefully reduce the
 amount of disagreement overall).

 For example, for me it's not clear what's wrong with Antoine's proposal
 (compile on Centos 5) - it seemed quite sensible approach to produce a
 reasonably compatible binary.

 Some issues with kernel ABI have been mentioned - can anyone point me to
 some resources describing the possible problems? Is it correct to assume
 that it's about using vendor-specific kernel api?


No, it is about some python packages depending directly or indirectly on
kernel features not available in the kernel on centos 5. For example, you
can't build subprocess32 (https://pypi.python.org/pypi/subprocess32/) on
centos 5 kernels.



 Also, what does Conda do to solve the binary compatibility issues and
 distutils or pip could never ever do (or implement)?


They do what almost everybody distributing large applications on Linux do :
they ship the world. Any large binary python distribution provider does
the same here: except for low level X11/glibc libraries, everything else is
bundled as part of the distribution.

So no magic, just lots of maintenance work.

David


 Thanks,
 -- Ionel Cristian Mărieș

 On Thu, Jul 9, 2015 at 4:50 PM, Nick Coghlan ncogh...@gmail.com wrote:

 On 9 July 2015 at 05:06, Antoine Pitrou solip...@pitrou.net wrote:
  On Wed, 08 Jul 2015 13:05:45 -0400
  Tres Seaver tsea...@palladion.com wrote:
  -BEGIN PGP SIGNED MESSAGE-
  Hash: SHA1
 
  On 07/08/2015 07:10 AM, Antoine Pitrou wrote:
 
   Seriously, how this is even supposed to be relevant? The whole point
   is to produce best-effort packages that work on still-supported
   mainstream distros, not any arbitrary Linux setup.
 
  I'm arguing that allowing PyPI uploads of binary wheels for Linux will
 be
  actively harmful.
 
  There is no point in reinstating an argument that has already been made
  and discussed in the other subthread (of course, you would have to read
  it first to know that).

 Steady on folks - prebuilt binary software distribution is *really*,
 *really*, hard, and we're not going to magically solve problems in a
 couple of days that have eluded Linux distribution vendors for over a
 decade. Yes, it's annoying, yes, it's frustrating, but sniping at each
 other when we point out the many and varied reasons it's hard won't
 help us to improve the experience for Python users.

 The key is remembering that now matter how broken you think prebuilt
 binary software distribution might be, it's actually worse. And
 channeling Hofstadter's Law: this principle remains true, even when
 you attempt to take this principle into account :)

 If you look at various prebuilt binary ecosystems to date, there's
 either a central authority defining the ABI to link against:

 - CPython on Windows
 - CPython on Mac OS X
 - Linux distributions with centralised package review and build systems
 - conda
 - nix
 - MS Visual Studio
 - XCode
 - Google Play
 - Apple App Store

 Or else a relatively tightly controlled isolation layer between the
 application code and the host system:

 - JVM
 - .NET CLR

 (even Linux containers can still hit the kernel ABI compatibility
 issues mentioned elsewhere in the thread)

 As Donald notes, I think we're now in a good position to start making
 progress here, but the first step is going to be finding a way to
 ensure that *by default*, pip on Linux ignores wheel files published
 on PyPI, and requires that they be *whitelisted* in some fashion
 (whether individually or categorically). That way, we know we're not
 going to make the default user experience on Linux *worse* than the
 status quo while we're still experimenting with how we want the
 publication side of things to work. Debugging build time API
 compatibility errors can be hard enough, debugging runtime A*B*I
 compatibility errors is a nightmare even for seasoned support
 engineers.

 It seems to me that one possible way to do that might be to change
 PyPI from whitelisting Windows and Mac OS X (as I believe it does now)
 to instead blacklisting all the other currently possible results from
 distutils.util.get_platform().

 Regards,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig



 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[David Cournapeau]

On Fri, Jul 10, 2015 at 1:53 PM, Olivier Grisel olivier.gri...@ensta.org
wrote:

 I just checked and indeed the python exec installed by miniconda does
 not work on Alpine linux (launch via docker from the gliderlabs/alpine
 image):

 # ldd /root/miniconda3/pkgs/python-3.4.3-0/bin/python3.4
 /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000)
 libpython3.4m.so.1.0 =
 /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0
 (0x7f26bd153000)
 libpthread.so.0 = /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000)
 libdl.so.2 = /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000)
 libutil.so.1 = /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000)
 libm.so.6 = /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000)
 libc.so.6 = /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000)
 Error relocating
 /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0:
 __finite: symbol not found
 Error relocating
 /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0:
 __rawmemchr: symbol not found
 Error relocating
 /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0:
 __isinff: symbol not found
 Error relocating
 /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0:
 __isnan: symbol not found
 Error relocating
 /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0:
 __isinf: symbol not found

 We could still have a platform or ABI tag for linux that would include
 some libc information to ensure that it points to a compatible glibc
 and provide a reference docker image to build such wheels.

 We could assume that wheel binary packages should not link to any .so
 file from the system besides the libc.


This is too restrictive if you want plotting-related packages (which I
suspsect you are interested in ;) ).

The libraries we at Enthought depend on for our packages are:

* glibc (IMO if you use a system w/o glibc, you are expected to be on your
own to build packages from sources)
* X11/fontconfig
* libstdc++

Those are the ones you really do not want to ship.

I don't know the proportion of packages that would work from pypi if you
could assume the system has those available through some kind of
ABI/Platform specifier following pep425.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[David Cournapeau]

On Thu, Jul 9, 2015 at 3:50 PM, Antoine Pitrou solip...@pitrou.net wrote:

 On Thu, 9 Jul 2015 23:50:30 +1000
 Nick Coghlan ncogh...@gmail.com wrote:
 
  As Donald notes, I think we're now in a good position to start making
  progress here, but the first step is going to be finding a way to
  ensure that *by default*, pip on Linux ignores wheel files published
  on PyPI, and requires that they be *whitelisted* in some fashion
  (whether individually or categorically). That way, we know we're not
  going to make the default user experience on Linux *worse* than the
  status quo while we're still experimenting with how we want the
  publication side of things to work. Debugging build time API
  compatibility errors can be hard enough, debugging runtime A*B*I
  compatibility errors is a nightmare even for seasoned support
  engineers.

 By the way, I think there's another possibility if the Python packaging
 authority doesn't want to tackle this (admittedly delicate) problem:
 issue a public statement that Anaconda is the preferred way of
 installing Linux binary packages if they aren't provided (or the
 version is too old) by their Linux distribution of choice.

 It would then give more authority to software developers if they want
 to tell their users don't use pip to install our code under Linux, use
 conda.


I don't think it is reasonable for pypa to recommend one solution when
multiple are available (though it is certainly fair to mention them).

ActiveState, Enthought (my own employer) also provide linux binaries,

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[David Cournapeau]

On Tue, Jul 7, 2015 at 7:02 PM, Donald Stufft don...@stufft.io wrote:

 On July 7, 2015 at 10:22:55 AM, Nick Coghlan (ncogh...@gmail.com) wrote:
  On 8 July 2015 at 00:07, Antoine Pitrou wrote:
   On Tue, 7 Jul 2015 23:53:59 +1000
   Nick Coghlan wrote:
   Unfortunately, the compatibility tagging for Linux wheels is currently
   so thoroughly inadequate that even in tightly controlled environments
   having a wheel file escape from its intended target platforms can
   cause hard to debug problems.
  
   I'm not sure what you're pointing to, could you elaborate a bit?
 
  That was a reference to a case of someone building for Debian (I
  think), and then having one of their wheel files end up installed on a
  CentOS system and wondering why things weren't working.
 
   For the record, building against a well-known, old glibc + gcc has
   served the Anaconda platform well.
 
  The key problem is that there's no straightforward way for us to
  verify that folks are actually building against a suitably limited set
  of platform APIs that all Linux distros in widespread use provide.
 
  And when it inevitably fails (which it will, Python and PyPI are too
  popular for it not to), the folks running into the problem are
  unlikely to be able to diagnose what has happened, and even once we
  figure out what has gone wrong, we'd be left having to explain how to
  blacklist wheel files, and the UX would just generally be terrible
  (and the burden of dealing with that would fall on the pip and PyPI
  maintainers, not on the individual projects publishing insufficiently
  conservative Linux wheel files).
 
  That's why the platform override tags are such a nice idea, as it
  becomes possible to start iterating on possible solutions to the
  problem without affecting the default installation UX in the near
  term.
 
 

 pip 7+ actually has the UI for blacklisting binary packages now, primarily
 to
 ask no don't build a wheel for X, but it also functions to ask *not* to
 accept wheels for a particular project from PyPI.

 In my mind, the biggest reason to not just open up the ability to upload
 even
 generic linux wheels right now is the lack of a safe-ish default. I think
 if
 we added a few things:

 * Default to per platform tags (e.g. ubuntu_14_04), but allow this to be
   customized and also accept Generic Linux wheels as well.
 * Put the libc into the file name as well since it's reasonable to build a
   generic linux wheel that statically links all dependencies (afaik),
 however
   it does not really work to statically link glibc. This means that even if
   you build against an old version of glibc if you're running on a Linux
 that
   *doesnt* use glibc (like Alpine which uses MUSL) you'll run into
 problems.

 I think that it is entirely possible to build a generic linux wheel that
 will
 work on any Linux built the same libc* of the same or newer version,
 however I
 think that you have to be careful if you do it. You have to ensure all your
 dependencies are statically linked (if you have any) and you have to ensure
 that you build against an old enough Linux (likely some form of CentOS).

 * Side question, since I don't actually know how a computer works: Is it
 even
   possible to have a CPython extension link against a different libc than
   CPython itself is linked against? What if static linking is involved
 since
   there are non glibc libcs which actually do support static linking?


You can use versioned symbols to manage some of those issues (
https://www.kernel.org/pub/software/libs/glibc/hjl/compat/). Some softwares
even include their own libc, but this is getting hair fast.

The common solution is to do what Antoine mentioned: build on the lowest
common denominator, and reduce the dependencies on the system as much as
possible. To be honest, glibc is rarely your problem: the kernel is
actually more problematic (some common python packages don't build on
2.6.18 anymore), and C++ even more so. E.g. llvm 3.6 will not build on gcc
4.1 (the version of centos 5), so you need a new g++ which means a new
libtstdc++.

I am biased, but that's the kind of things where you may want to work with
professional providers with people paid to work on those boring but
critical issues.

David



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Making pip and PyPI work with conda packages

[David Cournapeau]

On Wed, May 20, 2015 at 2:25 PM, Nick Coghlan ncogh...@gmail.com wrote:

 On 18 May 2015 at 14:32, David Cournapeau courn...@gmail.com wrote:
  On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan ncogh...@gmail.com
 wrote:
  Indirection via pip injects the usage of setuptools even for plain
  distutils projects, and generates
 https://www.python.org/dev/peps/pep-0376/
  compliant metadata by default.
 
 
  Note that some packages will push hard against injecting setuptools, at
  least until it does not offer a way to prevent from installing as an egg
  directory. Most of the core scientific packages avoid setuptools because
 of
  this.

 pip changes the default behaviour of setuptools to be more in line
 with the behaviour of plain distutils (e.g. forcing
 --single-version-externally-managed).


Yes, but that cannot be the only way to install the package.

This is why I would like to see a way forward for
https://bitbucket.org/pypa/setuptools/issue/371/setuptools-and-state-of-pep-376,
to start pushing packages using setuptools in their setup.py in the
scientific stack. Without this, it will be hard to move forward politically
speaking. Lots of key contributors have a strong aversion to setuptools way
of doing things (partly historical reasons, but it is hard to change minds).

If every system (pip, setup.py install, conda, enstaller) were to at
least write the {dist/egg}-info directories correctly, it would be already
a significant step toward interoperation.

David


This means that pip install X consistently gives setuptools levels
 of capabilities in terms of dependency management, without defaulting
 to installing things as egg archives or directories (modulo the rough
 edges around setup-requires).

 As Donald notes, pip install also abstracts away any future
 invocation of a metadata based pluggable build system, while
 ./setup.py install assumes the distutils-based status quo will
 remain in place forever.

 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Making pip and PyPI work with conda packages

[David Cournapeau]

On Wed, May 20, 2015 at 12:46 AM, Paul Moore p.f.mo...@gmail.com wrote:

 On 19 May 2015 at 16:22, Chris Barker chris.bar...@noaa.gov wrote:
  The other issue is social: this would really only be a benefit if a wide
  variety of packages shared the same libs -- but each of those packages is
  maintained by different individuals and communities. So it's had to know
 if
  it would get used. I could put up a libpng wheel, for instance, and who
  knows if the Pillow folks would have any interest in using it? or the
  matplotlib folks, or, ... And this would be particularly difficult when
 the
  solution was hacked together...

 Honestly, I still haven't seen a solid explanation of why (at least on
 Windows) static linking isn't a viable option.


Because some libraries simply don't work as static libraries, or are too
big (MKL comes to mind). Also, we have been historically using static libs
for our eggs at Enthought on windows, and it has been a nightmare to
support. It just does not scale when you have 100s of packages.

But really, once wheels support arbitrary file locations, this becomes
fairly easy at the packaging level: the remaining issue is one of ABI /
standards, but that's mostly a non technical issue.

Gholke has 100s of packages using wheels, and we ourselves at Enthought
have close to 500 packages for windows, all packaged as eggs, maybe 30-40 %
of which are not python but libs, C/C++ programs, etc... It is more about
agreeing about a common way of doing things than a real technical
limitation.

David

If someone were to
 create and publish a Python compatible static .lib file for the
 various hard-to-build dependencies, extensions could specify that you
 link with it in setup.py, and all the person building the wheel has to
 do is download the needed libraries for the build.

 If there's a technical reason why dynamic linking at runtime is better
 than static linking (sufficiently better that it justifies all the
 effort needed to resolve the issues involved), then I've yet to see a
 good explanation of it. The only things I've seen are disk space, or
 maintenance (where this usually means it's easier to release a new
 DLL with a security fix than get all the various statically linked
 packages updated - that's a valid point, but given how hard it is to
 get a working dynamic linking solution in this environment, I have to
 wonder whether that argument still holds).

 All of this applies to Windows only, of course - dynamic linking and
 system management of shared libraries is very much a per-platform
 issue, and I don't pretend to know the trade-offs on OSX or Linux.

 Paul
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Making pip and PyPI work with conda packages

[David Cournapeau]

On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan ncogh...@gmail.com wrote:


 On 18 May 2015 07:32, Chris Barker chris.bar...@noaa.gov wrote:
 
  On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan ncogh...@gmail.com
 wrote:
 
 % pip install --upgrade pip
 % pip install some_conda_package
 
  This gets the respective role of the two tools reversed - it's like my
  asking for pip install some_fedora_rpm to be made to work.
 
 
  I agree here -- I was thinking there was some promise in a
 conda_package_to_wheel converter though. It would, of course, only work in
 a subset of conda packages, but would be nice.
 
  The trick is that conda packages for the hard-to-build python packages
 (the ones we care about) often (always?) depend on conda packages for
 dynamic libs, and pip+wheel have no support for that.
 
  And this is a trick, because while I have some ideas for supporting
 just-for-python dynamic libs, conda's are not just-for-python -- so that
 might be hard to mash together.
 
  Continuum has a bunch of smart people, though.
 
  However, having conda use pip install in its build scripts so that
  it reliably generates pip compatible installation metadata would be a
  possibility worth discussing - that's what we've started doing in
  Fedora, so that runtime utilities like pkg_resources can work
  correctly.
 
 
  Hmm -- that's something ot look into -- you can put essentially anything
 into a conda bulid script --  so this would be a matter of convention,
 rather than tooling. (of course the conventions used by Continuum for the
 offical conda packages is the standard).
 
  But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ???
 
  I see pip has handling the dependency resolution, and finding and
 downloading of packages part of the problem -- conda does those already.
 
  So what would using pip inside a conda build script buy you that using
 setuptools does not?

 Indirection via pip injects the usage of setuptools even for plain
 distutils projects, and generates
 https://www.python.org/dev/peps/pep-0376/ compliant metadata by default.


Note that some packages will push hard against injecting setuptools, at
least until it does not offer a way to prevent from installing as an egg
directory. Most of the core scientific packages avoid setuptools because of
this.

David


However, looking at the current packaging policy, I think I misremembered
 the situation - it looks like we *discussed* recommending indirection via
 pip  attaining PEP 376 compliance, but haven't actually moved forward with
 the idea yet. That makes sense, since pursuing it would have been gated on
 ensurepip, and the Python 3 migration has been higher priority recently.

 Cheers,
 Nick.

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI is a sick sick hoarder

[David Cournapeau]

On Sat, May 16, 2015 at 11:36 PM, Justin Cappos jcap...@nyu.edu wrote:

 I am no expert, but I don't understand why backtracking algorithms would
 to be faster than SAT, since they both potentially need to walk over the
 full set of possible solutions. It is hard to reason about the cost because
 the worst case is in theory growing exponentially in both cases.


 This is talked about a bit in this thread:
 https://github.com/pypa/pip/issues/988

 Each algorithm could be computationally more efficient.  Basically, *if
 there are no conflicts* backtracking will certainly win.  If there are a
 huge number of conflicts a SAT solver will certainly win.  It's not clear
 where the tipping point is between the two schemes.

 However, a better question is does the computational difference matter?
 If one is a microsecond faster than the other, I don't think anyone cares.
 However, from the OPIUM paper (listed off of that thread), it is clear that
 SAT solver resolution can be slow without optimizations to make them work
 more like backtracking resolvers.  From my experience backtracking
 resolvers are also slow when the conflict rate is high.


Pure SAT is fast enough in practice in my experience (concretely: solving
thousand of rules takes  1 sec). It becomes more complicated as you need
to optimize the solution, especially when you have already installed
packages. This is unfortunately not as well discussed in the literature.
Pseudo-boolean SAT for optimization was argued to be too slow by the 0
install people, but OTOH, this seems to be what's used in conda, so who
knows :)

If you SAT solver is in pure python, you can choose a direction of the
search which is more meaningful. I believe this is what 0install does from
reading http://0install.net/solver.html, and what we have in our own SAT
solver code. I unfortunately cannot look at the 0install code myself as it
is under the GPL and am working on a BSD solver implementation. I also do
not know how they handle updates and already installed packages.


 This only considers computation cost though.  Other factors can become
 more expensive than computation.  For example, SAT solvers need all the
 rules to consider.  So a SAT solution needs to effectively download the
 full dependency graph before starting.  A backtracking dependency resolver
 can just download packages or dependency information as it considers them.
 The bandwidth cost for SAT solvers should be higher.


With a reasonable representation, I think you can make it small enough. To
give an idea, our index @ Enthought containing around 20k packages takes
~340 kb compressed w/ bz2 if you only keep the data required for dependency
handling (name, version and runtime dependencies), and that's using json,
an inefficient encoding, so I suspect encoding all of pypi may be a few MB
only fetch, which is generally faster that doing tens of http requests.

The libsvolv people worked on a binary representation that may also be
worth looking at.


 P.S.  If you'd like to talk off list, possibly over Skype, I'd be happy to
 talk more with you and/or Robert about minutiae that others may not care
 about.


Sure, I would be happy too. As I mentioned before, we have some code around
a SAT-based solver, but it is not ready yet, which is why we kept it
private (https://github.com/enthought/sat-solver). It handles well (== both
speed and quality-wise) the case where nothing is installed, but behaves
poorly when packages are already installed, and does not handle the update
case yet. The code is also very prototype-ish, but is not too complicated
to experimente with it.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI is a sick sick hoarder

[David Cournapeau]

On Sun, May 17, 2015 at 12:40 AM, Daniel Holth dho...@gmail.com wrote:


 On May 16, 2015 11:22 AM, David Cournapeau courn...@gmail.com wrote:
 
 
 
  On Sat, May 16, 2015 at 11:36 PM, Justin Cappos jcap...@nyu.edu wrote:
 
  I am no expert, but I don't understand why backtracking algorithms
 would to be faster than SAT, since they both potentially need to walk over
 the full set of possible solutions. It is hard to reason about the cost
 because the worst case is in theory growing exponentially in both cases.
 
 
  This is talked about a bit in this thread:
 https://github.com/pypa/pip/issues/988
 
  Each algorithm could be computationally more efficient.  Basically, *if
 there are no conflicts* backtracking will certainly win.  If there are a
 huge number of conflicts a SAT solver will certainly win.  It's not clear
 where the tipping point is between the two schemes.
 
  However, a better question is does the computational difference
 matter?  If one is a microsecond faster than the other, I don't think
 anyone cares.  However, from the OPIUM paper (listed off of that thread),
 it is clear that SAT solver resolution can be slow without optimizations to
 make them work more like backtracking resolvers.  From my experience
 backtracking resolvers are also slow when the conflict rate is high.
 
 
  Pure SAT is fast enough in practice in my experience (concretely:
 solving thousand of rules takes  1 sec). It becomes more complicated as
 you need to optimize the solution, especially when you have already
 installed packages. This is unfortunately not as well discussed in the
 literature. Pseudo-boolean SAT for optimization was argued to be too slow
 by the 0 install people, but OTOH, this seems to be what's used in conda,
 so who knows :)

 Where optimizing means something like find a solution with the newest
 possible releases of the required packages, not execution speed.


Indeed, it was not obvious in this context :) Though in theory,
optimization is more general. It could be optimizing w.r.t. a cost function
taking into account #packages, download size, minimal number of changes,
etc... This is where you want a pseudo-boolean SAT, which is what conda
uses I think.

0install, composer and I believe libsolv took a different route, and use
heuristics to find a reasonably good solution by picking the next
candidate. This requires access to the internals of the SAT solver though
(not a problem if you have a python implementation).

David

  If you SAT solver is in pure python, you can choose a direction of the
 search which is more meaningful. I believe this is what 0install does from
 reading http://0install.net/solver.html, and what we have in our own SAT
 solver code. I unfortunately cannot look at the 0install code myself as it
 is under the GPL and am working on a BSD solver implementation. I also do
 not know how they handle updates and already installed packages.
 
 
  This only considers computation cost though.  Other factors can become
 more expensive than computation.  For example, SAT solvers need all the
 rules to consider.  So a SAT solution needs to effectively download the
 full dependency graph before starting.  A backtracking dependency resolver
 can just download packages or dependency information as it considers them.
 The bandwidth cost for SAT solvers should be higher.
 
 
  With a reasonable representation, I think you can make it small enough.
 To give an idea, our index @ Enthought containing around 20k packages takes
 ~340 kb compressed w/ bz2 if you only keep the data required for dependency
 handling (name, version and runtime dependencies), and that's using json,
 an inefficient encoding, so I suspect encoding all of pypi may be a few MB
 only fetch, which is generally faster that doing tens of http requests.
 
  The libsvolv people worked on a binary representation that may also be
 worth looking at.
 
 
  P.S.  If you'd like to talk off list, possibly over Skype, I'd be happy
 to talk more with you and/or Robert about minutiae that others may not care
 about.
 
 
  Sure, I would be happy too. As I mentioned before, we have some code
 around a SAT-based solver, but it is not ready yet, which is why we kept it
 private (https://github.com/enthought/sat-solver). It handles well (==
 both speed and quality-wise) the case where nothing is installed, but
 behaves poorly when packages are already installed, and does not handle the
 update case yet. The code is also very prototype-ish, but is not too
 complicated to experimente with it.
 
  David
 
 
  ___
  Distutils-SIG maillist  -  Distutils-SIG@python.org
  https://mail.python.org/mailman/listinfo/distutils-sig
 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[David Cournapeau]

On Sat, May 16, 2015 at 4:56 AM, Chris Barker chris.bar...@noaa.gov wrote:

 On Fri, May 15, 2015 at 1:49 AM, Paul Moore p.f.mo...@gmail.com wrote:

 On 14 May 2015 at 19:01, Chris Barker chris.bar...@noaa.gov wrote:
  Ah -- here is the issue -- but I think we HAVE pretty much got what we
 need
  here -- at least for Windows and OS-X. It depends what you mean by
  curated, but it seems we have a (defacto?) policy for PyPi: binary
 wheels
  should be compatible with the python.org builds. So while each package
 wheel
  is supplied by the package maintainer one way or another, rather than
 by a
  central entity, it is more or less curated -- or at least standardized.
 And
  if you are going to put a binary wheel up, you need to make sure it
 matches
  -- and that is less than trivial for packages that require a third party
  dependency -- but building the lib statically and then linking it in is
 not
  inherently easier than doing a dynamic link.

 I think the issue is that, if we have 5 different packages that depend
 on (say) libpng, and we're using dynamic builds, then how do those
 packages declare that they need access to libpng.dll?


 this is the missing link -- it is a binary build dependency, not a package
 dependency -- so not such much that matplotlib-1.4.3 depends on libpng.x.y,
 but that:



 matplotlib-1.4.3-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl

 depends on:

 libpng-x.y

 (all those binary parts will come from the platform)

 That's what's missing now.

 And on Windows,
 where does the user put libpng.dll so that it gets picked up?


 Well, here is the rub -- Windows dll hell really  is hell -- but I think
 it goes into the python dll searchpath (sorry, not on a
 Windows box where I can really check this out right now), it can work -- I
 know have an in-house product that has multiple python modules sharing a
 single dll somehow



 And how
 does a non-expert user do this (put it in $DIRECTORY, update your
 PATH, blah blah blah doesn't work for the average user)?


 That's why we may need to update the tooling to handle this -- Im not
 totally sure if the current wheel format can support this on Windows --
 though it can on OS-X.

 In particular, on Windows, note that the shared DLL must either be in
 the directory where the executable is located (which is fun when you
 have virtualenvs, embedded interpreters, etc), or on PATH (which has
 other implications - suppose I have an incompatible version of
 libpng.dll, from mingw, say, somewhere earlier on PATH).


 that would be dll hell, yes.


 The problem isn't so much defining a standard ABI that shared DLLs
 need - as you say, that's a more or less solved problem on Windows -
 it's managing how those shared DLLs are made available to Python
 extensions. And *that* is what Unix package managers do for you, and
 Windows doesn't have a good solution for (other than bundle all the
 dependent DLLs with the app, or suffer DLL hell).


 exactly -- but if we consider the python install to be the app, rather
 than an individual python bundle, then we _may_ be OK.

 PS For a fun exercise, it might be interesting to try breaking conda -


 Windows really is simply broken [1] in this regard -- so I'm quite sure
 you could break conda -- but it does seem to do a pretty good job of not
 being broken easily by common uses -- I can't say I know enough about
 Windows dll finding or conda to know how...

 Oh, and conda is actually broken in this regard on OS-X at this point --
 if you compile your own extension in an anaconda environment, it will find
 a shared lib at compile time that it won't find at run time. -- the conda
 install process fixes these, but that's a pain when under development --
 i.e. you don't want to have to actually install the package with conda to
 run a test each time you re-build the dll.. (or even change a bit of python
 code...)

 But in short -- I'm pretty sure there is a way, on all systems, to have a
 standard way to build extension modules, combined with a standard way to
 install shared libs, so that a lib can be shared among multiple packages.
 So the question remains:


There is actually no way to do that on windows without modifying the
interpreter somehow. This was somehow discussed a bit at PyCon when talking
about windows packaging:

 1. the simple way to share DLLs across extensions is to put them in the
%PATH%, but that's horrible.
 2. there are ways to put DLLs in a shared directory *not* in the %PATH%
since at least windows XP SP2 and above, through the SetDllDirectory API.

With 2., you still have the issue of DLL hell, which may be resolved
through naming and activation contexts. I had a brief chat with Steve where
he mentioned that this may be a solution, but he was not 100 % sure IIRC.
The main drawback of this solution is that it won't work when inheriting
virtual environments (as you can only set a single directory).

FWIW, we are 

Re: [Distutils] PyPI is a sick sick hoarder

[David Cournapeau]

On Sat, May 16, 2015 at 10:52 AM, Robert Collins robe...@robertcollins.net
wrote:

 On 16 May 2015 at 13:45, Donald Stufft don...@stufft.io wrote:
 
  On May 15, 2015, at 9:22 PM, Robert Collins robe...@robertcollins.net
 wrote:
 
  On 16 May 2015 at 11:08, Marcus Smith qwc...@gmail.com wrote:
  Why not start with pip at least being a simple fail-on-conflict
 resolver
  (vs the 1st found wins resolver it is now)...
 
  You'd backtrack for the sake of re-walking when new constraints are
 found,
  but not for the purpose of solving conflicts.
 
  I know you're motivated to solve Openstack build issues, but many of
 the
  issues I've seen in the pip tracker, I think would be solved without
 the
  backtracking resolver you're trying to build.
 
  Well, I'm scratching the itch I have. If its too hard to get something
  decent, sure I might back off in my goals, but I see no point aiming
  for something less than all the other language specific packaging
  systems out there have.
 
 
  So what makes the other language specific packaging systems different?
 As far
  as I know all of them have complete archives (e.g. they are like PyPI
 where they
  have a lot of versions, not like Linux Distros). What can we learn from
 how they
  solved this?

 NB; I have by no means finished low hanging heuristics and space
 trimming stuff :). I have some simple things in mind and am sure I'll
 end up with something 'good enough' for day to day use. The thing I'm
 worried about is the long term health of the approach.

 Good questions. Some of it is structural I suspect. A quick rundown.
 cabal (haskell) has a backtracking solver that accepts various
 parameters to tell it to try harder.
 javascript effectively vendors every dep ever, so you end up with many
 copies of the same library at different versions in the same process.
 rust's cargo system currently solves everything in a single project
 only - it has no binary packaging, only vendor-into-a-binary-build
 packaging.
 The gem behaviour I'm not yet familiar with.
 perl I used to know but time has eroded it :/.


FWIW, php uses a SAT-based solver in composer, which started as a port of
libsolv (the SAT solver used by openSUSE and soon Fedora).

I am no expert, but I don't understand why backtracking algorithms would to
be faster than SAT, since they both potentially need to walk over the full
set of possible solutions. It is hard to reason about the cost because the
worst case is in theory growing exponentially in both cases.

With a SAT-based algorithm for dependency resolution, it is relatively
simple to apply heuristics which massively prune the search space. For
example, when considering package A with say 10 potential versions A_1,
etc..., in theory, you need to generate the rules:

# - means not install, + means install
- A_1 | -  A_2
- A_1 | - A_3
...

and those constitute most of the rules in common cases. But it is possible
to tweak the SAT implementation to replace those rules by a single AtMost
one of rule per *package*, which means the #rules do not grow much by
versions.

The real difficulty of SAT-based solver is the optimization part: many
actually valid solutions are not acceptable, and that's where the
heuristics get more complicated.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[David Cournapeau]

Thanks for pushing this Daniel.

I think we should wait for a bit before making this proposal official. Just
after I made my PR against wheel to implement this, I had some discussions
with Nathaniel Smith from numpy, where he remarked we may want to support
better everything in site-packages model.

At the lowest level, the supported schema supports both usecases. We know
that because distributions like NixOS support the one directory per
package model even though most packages they package use autotools scheme.

But we may want to add higher level support at the same time as this new
scheme to avoid people coming up with their own custom solutions. One idea
that was thrown out was enabling a pkg-config-like mechanism to separate
where files are from how to find information for building things. That
would allow inside site-packages and outside site-packages schemes to work
seamlessly.

I can work on a paperman implementation of this on top of the wheel
installer for the end of this week. I think that would both alleviate some
concerns for people interested in everything in package directory, and
make the discussion more focused.

On Sat, Apr 18, 2015 at 11:24 PM, Daniel Holth dho...@gmail.com wrote:

 I am working on a minor update to the wheel format to add more
 categories under which files can be installed. Constructive comments
 welcome.

 Distutills, setuptools, wheel currently have the best support for
 installing files relative to ('purelib', 'platlib', 'headers',
 'scripts', 'data') with 'data' usually meaning '/' or the root of the
 virtualenv.  In practice only exactly one of the 'purelib' or
 'platlib' locations (which are usually mapped to the same directory on
 disk), and sometimes 'scripts' is used for any given package, and
 Python packages have no good way of loading any of their package files
 at run time if they are installed into any location not relative to
 sys.path.

 This works fairly well for Python libraries, but anyone packaging
 applications for a Linux distribution is required to follow the
 filesystem hierarchy standard or FHS.


 http://docs.fedoraproject.org/en-US/Fedora/14/html/Storage_Administration_Guide/s1-filesystem-fhs.html
 ,
 http://www.debian.org/releases/etch/amd64/apcs02.html,
 http://www.pathname.com/fhs/

 It would help Linux distribution package maintainers and Python
 application (not library) developers if wheel had better support for
 installing files into the FHS, but it would help everyone else who
 wanted to generate cross-platform packages if the FHS was not
 hardcoded as paths in the data/ category. To that end we should come
 up with some additional categories that map, or do not map, to the FHS
 based on the end user's platform.

 Distutils2 had the concept of resource categories:
 http://alexis.notmyidea.org/distutils2/setupcfg.html#resources

 
 Default categories are:

 * config
 * appdata
 * appdata.arch
 * appdata.persistent
 * appdata.disposable
 * help
 * icon
 * scripts
 * doc
 * info
 * man
 

 GNU has directory variables:
 https://www.gnu.org/prep/standards/html_node/Directory-Variables.html
 , for example prefix, exec_prefix, bindir, sbindir.

 Bento has a list based on the GNU paths, and allows new paths to be
 defined:

 prefix: install architecture-independent files
 eprefix: install architecture-dependent files
 bindir: user executables
 sbindir: system admin executables
 libexecdir: program executables
 sysconfdir: read-only single-machine data
 sharedstatedir: modifiable architecture-independent data
 localstatedir: modifiable single-machine data
 libdir: object code libraries
 includedir: C header files
 oldincludedir: C header files for non-gcc
 datarootdir: read-only arch.-independent data root
 datadir: read-only architecture-independent data
 infodir: info documentation
 localedir: locale-dependent data
 mandir: man documentation
 docdir: documentation root
 htmldir: html documentation
 dvidir: dvi documentation
 pdfdir: pdf documentation
 psdir: ps documentation

 I would like to add Bento's list to wheel and to setuptools. We would
 fix the disused setup.py setup(data_files = ) argument so that it
 could be used with $ substitution, or provide a utility function that
 would expand them in setup.py itself:

 data_files = { '$libdir/mylib' : [ 'some_library_file'],
 '$datadir/mydata' : ['some_data_file']}

 We would provide a default configuration file that mapped the
 categories to their installed locations.

 We would store the actual paths used at install time, so a package
 could look for files relative to the $datadir used when it was
 installed.

 Then it would be easier to distribute Python programs that need to
 install some files to paths that are not relative to the site-packages
 directory.

 - Daniel Holth
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[David Cournapeau]

On Sun, Apr 19, 2015 at 6:39 PM, Vinay Sajip vinay_sa...@yahoo.co.uk
wrote:





 From: Paul Moore p.f.mo...@gmail.com
  On 19 April 2015 at 11:55, David Cournapeau courn...@gmail.com wrote:
  I can work on a paperman implementation of this on top of the wheel
  installer for the end of this week. I think that would both alleviate
 some
  concerns for people interested in everything in package directory, and
  make the discussion more focused.

 Is paperman a Disney reference, or something else?


it was used in another project I was involved in, to mean even less than a
straw man. Since the project was handled by native speakers, I assumed it
was a generic term :)

In any case, the implementation I started in wheel was something to *start*
the discussion, not to be used as a reference.

David



  That sounds good. One thing the wheel install command doesn't support

  is per-user installs. I'd appreciate seeing some details of how those
  would be handled - on the assumption that you don't want to deal with
  pip's source just yet, a rough spec would be fine for now.
 I presume the way wheel install works is orthogonal to the scheme for
 handling different categories of data - distil install, for example, does
 per-user installs by default. Are the proposed implementations just a proof
 of concept, to validate the usability of the implemented scheme? What is
 the envisaged timeline for proposing/agreeing specifications for how the
 file categories will work cross-platform? I'm bearing in mind that there
 might be other implementations of installers which would need to
 interoperate with any file category scheme ...

 Regards,

 Vinay Sajip

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pywin32 on wheels [was: Beyond wheels 1.0: helping downstream, FHS and more]

[David Cournapeau]

On Thu, Apr 16, 2015 at 9:58 AM, Paul Moore p.f.mo...@gmail.com wrote:

 On 16 April 2015 at 14:43, Kevin Horn kevin.h...@gmail.com wrote:
  Those that want to use pywin32 in a virtualenv (or just without all the
  system changes) could simply install the wheel (or even an sdist,
 perhaps)
  from the command line using pip, and then perform whatever other steps
 they
  want manually.

 Just as a data point, converting the existing wininst installer for
 pywin32 to a wheel (using wheel convert), and installing that via pip,
 is entirely workable (for the win32api/win32file type basic
 functionality). The pypiwin32 project (started by Glyph as a way of
 providing pywin32 wheels from PyPI) includes wheels for Python 3.x
 which I built that way, so it's certainly seen some level of use.

 The wheels are probably unnecessarily big, as they'll include all of
 pythonwin, and the ActiveX Scripting and service creation support,
 which I guess won't work in that configuration, but they are a good
 starting point for checking precisely what will work unmodified from a
 wheel.


For people interested in a lightweight alternative to pywin32, we have the
pywin32ctypes project, which started as a way to get access to win32
credentials without depending on DLL (to avoid file locking issues with
inplace updates).

The project is on github (https://github.com/enthought/pywin32-ctypes), and
is already used by a few projects. We support both cffi and ctypes backends
(the former to work out of the box on cpython, the latter to work on pypy).

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] name of the dependency problem

[David Cournapeau]

On Wed, Apr 15, 2015 at 11:32 AM, Trishank Karthik Kuppusamy 
trish...@nyu.edu wrote:

 On 15 April 2015 at 11:15, David Cournapeau courn...@gmail.com wrote:


 This is indeed the case. If you want to solve dependencies in a way that
 works well, you want an index that describes all your available package
 versions.

 While solving dependencies is indeed NP complete, they can be fairly fast
 in practice because of various specificities : each rule is generally only
 a few variables, and the rules have a specific form allowing for more
 efficient rule representation (e.g. at most one of variable, etc...). In
 my experience, it is not more difficult than using graph-based algorithms,
 and

 FWIW, at Enthought, we are working on a pure python SAT solver for
 resolving dependencies, to solve some of those issues. I am actually
 hacking on it right at PyCon, we hope to have a first working version end
 of Q2, at which point it will be OSS, and reintegrated in my older project
 depsolver (https://github.com/enthought/depsolver).


 Awesome! Then pip could use that in the near future :)


That's the goal. For various reasons, it ended up easier to develop the
solver within our own package manager enstaller, but once done, extracting
it as a separate library should not be too hard. It is for example designed
to support various versioning schemes (for legacy reasons we can't use
PEP440 just yet).

Regarding speed, initial experiments showed that even for relatively deep
graphs, the running time is taken outside the SAT solver (e.g. to generate
the rules, you need to parse the version of every package you want to
consider, and parsing 1000s of PEP440 versions is slow :) ).

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheel 1.0: more fine-grained installation scheme

[David Cournapeau]

On Tue, Apr 14, 2015 at 9:27 AM, Daniel Holth dho...@gmail.com wrote:

 That's exactly what I would like to do. Then
 distribution-1.0.data/sysconfdir/file in a wheel would install into
 /etc/file in the default scheme, but would probably really wind up in
 $VIRTUAL_ENV/etc/... for most of us web developers.


$prefix would be set to sys.prefix and $eprefix to sys.exec_prefix to
handle automatically.

Should I work on a PEP for wheel 2.0, or a pip implementation first  ?


 IIRC extra package-1.0-data/* directories in wheel are undefined.


pip will actually fail to install any wheel with an undefined directory
package-1.0-data/* (unfortunately this is not detected before installing,
so it ends up with a half installed package).

David


 I
 would have no problem putting fine-grained install schemes in 2.0 and
 putting some of the other wheel 2.0 features into wheel 3.
 Incrementing the major version number would cause older pip to reject
 the newer wheels, incrementing the minor version would produce a
 warning.






 On Tue, Apr 14, 2015 at 8:56 AM, David Cournapeau courn...@gmail.com
 wrote:
  Hi,
 
  I am splitting up the previous thread into one thread / proposal to focus
  the discussion.
 
  Assuming the basis of this proposal does not sound too horrible, I would
  make a proof of concept in a pip branch, so that we can flush out the
  details and then write an actual spec (I guess an updated wheel format
 would
  need a new PEP ?).
 
  The goal of this thread is to flush out a more fine-grained installation
  scheme, so that wheels can install files anywhere they want (at least
 within
  sys.prefix/sys.exec_prefix). I see two issues:
 
  1. defining what the scheme should be
  2. how should it be implemented in wheel: there are different trade-offs
  depending on whether we want this feature to be part of wheel format 1.*
 or
  2.0.
 
  First, my understanding of the current situation:
 
  * per the wheel PEP 427, anything in the wheel top directory and not in
  distribution-1.0.data is installed in site-package
  * every top directory in distribution-1.0.data/ needs to be mapped to the
  scheme as defined in distutils install command.
  * pip rejects any directory in distribution-1.0.data/ which is not in the
  scheme from 2.
 
  My suggestion for a better scheme would be to use an extended version of
 the
  various default directories defined by autotools. The extension would
 handle
  windows-specifics. More concretely:
 
  # Suggested variables
 
  The goal of supporting those variables is to take something that is
 flexible
  enough to support almost any installation scheme, without putting
 additional
  burden on the developer. People who do not want/need the flexibility will
  not need to do anything more than what they do today.
 
  The variables I would suggest are every variable defined in
 
 https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py#L10
 ,
  except for destdir which is not relevant here.
 
  On unix, the defaults follow autotools, and on windows, I mapped almost
  everything relative to sys.exec_prefix, except for the
  bindir/sbindir/libexecdir which map to $prefix\Scripts.
 
  The $sitedir variable would need to be updated to use the value from
  distutils instead of the hardcoded value I put in that file as well.
 
  # How to handle the augmented scheme
 
  Right now, if I wanted to install something in say $prefix/share/doc, I
  would need to put it in distribution-1.0.data/data/share/doc, but this
  prevents use from handling different platforms differently.
 
  OTOH, this is the only way I can see to make the new scheme backward
  compatible with pip versions who would not understand the new scheme. I
  don't have a good sense of what we should do there, the core pip team may
  have a better sense.
 
  For now, I would be happy to just make a proof of concept not caring
 about
  backward compatibility in a pip branch. Does that sound like a workable
  basis to flush out an actual proposal ?
 
  thanks,
  David
 
 
  ___
  Distutils-SIG maillist  -  Distutils-SIG@python.org
  https://mail.python.org/mailman/listinfo/distutils-sig
 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Beyond wheel 1.0: more fine-grained installation scheme

[David Cournapeau]

Hi,

I am splitting up the previous thread into one thread / proposal to focus
the discussion.

Assuming the basis of this proposal does not sound too horrible, I would
make a proof of concept in a pip branch, so that we can flush out the
details and then write an actual spec (I guess an updated wheel format
would need a new PEP ?).

The goal of this thread is to flush out a more fine-grained installation
scheme, so that wheels can install files anywhere they want (at least
within sys.prefix/sys.exec_prefix). I see two issues:

1. defining what the scheme should be
2. how should it be implemented in wheel: there are different trade-offs
depending on whether we want this feature to be part of wheel format 1.* or
2.0.

First, my understanding of the current situation:

* per the wheel PEP 427, anything in the wheel top directory and not
in distribution-1.0.data is installed in site-package
* every top directory in distribution-1.0.data/ needs to be mapped to the
scheme as defined in distutils install command.
* pip rejects any directory in distribution-1.0.data/ which is not in the
scheme from 2.

My suggestion for a better scheme would be to use an extended version of
the various default directories defined by autotools. The extension would
handle windows-specifics. More concretely:

# Suggested variables

The goal of supporting those variables is to take something that is
flexible enough to support almost any installation scheme, without putting
additional burden on the developer. People who do not want/need the
flexibility will not need to do anything more than what they do today.

The variables I would suggest are every variable defined in
https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py#L10,
except for destdir which is not relevant here.

On unix, the defaults follow autotools, and on windows, I mapped almost
everything relative to sys.exec_prefix, except for the
bindir/sbindir/libexecdir which map to $prefix\Scripts.

The $sitedir variable would need to be updated to use the value from
distutils instead of the hardcoded value I put in that file as well.

# How to handle the augmented scheme

Right now, if I wanted to install something in say $prefix/share/doc, I
would need to put it in distribution-1.0.data/data/share/doc, but this
prevents use from handling different platforms differently.

OTOH, this is the only way I can see to make the new scheme backward
compatible with pip versions who would not understand the new scheme. I
don't have a good sense of what we should do there, the core pip team may
have a better sense.

For now, I would be happy to just make a proof of concept not caring about
backward compatibility in a pip branch. Does that sound like a workable
basis to flush out an actual proposal ?

thanks,
David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov
wrote:

 NOTE: I don't work for any of the companies involved -- just a somewhat
 frustrated user... And someone that has been trying for years to make
 things easier for OS-X users.

 I’m not sure what (3) means exactly. What is a “normal” Python, do you
 modify Python in a way that breaks the ABI but which isn’t reflected in the
 standard ABI tag?


 It could be multiple things. The most obvious one is that generally.
 cross-platforms python distributions will try to be relocatable (i.e. the
 whole installation can be moved and still work). This means they require
 python itself to be built a special way. Strictly speaking, it is not an
 ABI issue, but the result is the same though: you can't use libraries from
 anaconda or canopy on top of a normal python


 But why not? -- at least for Anaconda, it's because those libraries likely
 have non-python dependencies, which are expected to be installed in a
 particular way. And really, this is not particular to Anaconda/Canopy at
 all. Python itself has no answer for this issue, and eggs and wheels don't
 help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order
 to build a binary wheel with non-python dependencies (let's say something
 like libjpeg, for instance), you need to either:
  - assume that libjpeg is installed in a standard place -- really no
 solution at all (at least outside of linux)
  - statically link it
  - ship the dynamic lib with the package

 For the most part, the accepted solution for OS-X has been to statically
 link, but:

  - it's a pain to do. The gnu toolchain really likes to use dynamic
 linking, and building a static lib that will run on a
 maybe-older-than-the-build-system machine is pretty tricky.

  - now we end up with multiple copies of the same lib in the python
 install. There are a handful of libs that are used a LOT. Maybe there is no
 real downside -- disk space and memory are cheap these days, but it sure
 feels ugly. And I have yet to feel comfortable with having multiple
 versions of the same lib linked into one python instance -- I can't say
 I've seen a problem, but it makes me nervous.

 On Windows, the choices are the same, except that: It is so much harder to
 build many of the standard open source libs that package authors are more
 likely to do it for folks, and you do get the occasional dll hell issues.

 I had a plan to make some binary wheels for OS-X that were not really
 python packages, but actually just bundled up libs, so that other wheels
 could depend on them. OS-X does allow linking to relative paths, so this
 should have been doable, but I never got anyone else to agree this was a
 good idea, and I never found the roundtoits anyway. And it doesn't really
 fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are
 platform dependent and even worse, build-dependent.

 Meanwhile, conda was chugging along and getting a lot of momentum in the
 Scientific community. And the core thing here is that conda was designed
 from the ground up to support essentially anything, This means is supports
 python packages that depend on non-python packages, but also supports
 packages that have nothing to do with python (Perl, command line tools,
 what have you...)

 So I have been focusing on conda lately.


The whole reason I started this discussion is to make sure wheel has a
standard way to do what is needed for those usecases.

conda, rpm, deb, or eggs as used in enthought are all essentially the same:
an archive with a bunch of metadata. The real issue is standardising on the
exact formats. As you noticed, there is not much missing in the wheel
*spec* to get most of what's needed. We've used eggs for that purpose for
almost 10 years at Enthought, and we did not need that many extensions on
top of the egg format after all.



 Which brings me back to the question: should the python tools (i.e. wheel)
 be extended to support more use-cases, specifically non-python
 dependencies? Or do we just figure that that's a problem better solved by
 projects with a larger scope (i.e. rpm, deb, conda, canopy).


IMO, given that wheels do most of what's needed, it is worth supporting
most simple usecases (compiled libraries required by well known
extensions). Right now, such packages (pyzmq, numpy, cryptography, lxml)
resort to quite horrible custom hacks to support those cases.

Hope that clarifies the intent,

David



 I'm on the fence here. I mostly care about Python, and I think we're
 pretty darn close with allowing wheel to support the non-python
 dependencies, which would allow us all to simply pip install pretty much
 anything -- that would be cool. But maybe it's a bit of a slippery slope,
 and if we go there, we'll end up re-writing conda.

 BTW, while you can't generally install a conda package in/for another
 python, you can generally install a wheel in a conda pythonThere are a
 few issues with 

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

I would advise against using or even reading about our egg extensions, as
the implementation is full of legacy (we've been doing this many years :)
): http://enstaller.readthedocs.org/en/master/reference/egg_format.html

This is what we use on top of setuptools egg:

 - ability to add dependencies which are not python packages (I think most
of it is already handled in metadata 2.0/PEP 426, but I would have to
re-read the PEP carefully).
 - ability to run post/pre install/remove scripts
 - support for all the of the autotools directories, with sensible
mapping on windows
 - a few extensions to the actual binary format (adding support for
symlinks is the only one I can think of ATM).

Everything else is legacy you really don't want to know (see here if you
still want to
http://enstaller.readthedocs.org/en/master/reference/egg_format.html)

David

On Mon, Apr 13, 2015 at 3:55 PM, Daniel Holth dho...@gmail.com wrote:

 On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau courn...@gmail.com
 wrote:
 
 
  On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov
  wrote:
 
  NOTE: I don't work for any of the companies involved -- just a somewhat
  frustrated user... And someone that has been trying for years to make
 things
  easier for OS-X users.
 
  I’m not sure what (3) means exactly. What is a “normal” Python, do you
  modify Python in a way that breaks the ABI but which isn’t reflected
 in the
  standard ABI tag?
 
 
  It could be multiple things. The most obvious one is that generally.
  cross-platforms python distributions will try to be relocatable
 (i.e. the
  whole installation can be moved and still work). This means they
 require
  python itself to be built a special way. Strictly speaking, it is not
 an ABI
  issue, but the result is the same though: you can't use libraries from
  anaconda or canopy on top of a normal python
 
 
  But why not? -- at least for Anaconda, it's because those libraries
 likely
  have non-python dependencies, which are expected to be installed in a
  particular way. And really, this is not particular to Anaconda/Canopy at
  all. Python itself has no answer for this issue, and eggs and wheels
 don't
  help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in
 order to
  build a binary wheel with non-python dependencies (let's say something
 like
  libjpeg, for instance), you need to either:
   - assume that libjpeg is installed in a standard place -- really no
  solution at all (at least outside of linux)
   - statically link it
   - ship the dynamic lib with the package
 
  For the most part, the accepted solution for OS-X has been to statically
  link, but:
 
   - it's a pain to do. The gnu toolchain really likes to use dynamic
  linking, and building a static lib that will run on a
  maybe-older-than-the-build-system machine is pretty tricky.
 
   - now we end up with multiple copies of the same lib in the python
  install. There are a handful of libs that are used a LOT. Maybe there
 is no
  real downside -- disk space and memory are cheap these days, but it sure
  feels ugly. And I have yet to feel comfortable with having multiple
 versions
  of the same lib linked into one python instance -- I can't say I've
 seen a
  problem, but it makes me nervous.
 
  On Windows, the choices are the same, except that: It is so much harder
 to
  build many of the standard open source libs that package authors are
 more
  likely to do it for folks, and you do get the occasional dll hell
 issues.
 
  I had a plan to make some binary wheels for OS-X that were not really
  python packages, but actually just bundled up libs, so that other wheels
  could depend on them. OS-X does allow linking to relative paths, so this
  should have been doable, but I never got anyone else to agree this was a
  good idea, and I never found the roundtoits anyway. And it doesn't
 really
  fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that
 are
  platform dependent and even worse, build-dependent.
 
  Meanwhile, conda was chugging along and getting a lot of momentum in the
  Scientific community. And the core thing here is that conda was designed
  from the ground up to support essentially anything, This means is
 supports
  python packages that depend on non-python packages, but also supports
  packages that have nothing to do with python (Perl, command line tools,
 what
  have you...)
 
  So I have been focusing on conda lately.
 
 
  The whole reason I started this discussion is to make sure wheel has a
  standard way to do what is needed for those usecases.
 
  conda, rpm, deb, or eggs as used in enthought are all essentially the
 same:
  an archive with a bunch of metadata. The real issue is standardising on
 the
  exact formats. As you noticed, there is not much missing in the wheel
 *spec*
  to get most of what's needed. We've used eggs for that purpose for
 almost 10
  years at Enthought, and we did not need that many extensions on top of
 the
  egg format

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

On Mon, Apr 13, 2015 at 5:25 PM, Chris Barker chris.bar...@noaa.gov wrote:

 On Mon, Apr 13, 2015 at 1:19 PM, David Cournapeau courn...@gmail.com
 wrote:

 This is what we use on top of setuptools egg:

  - ability to add dependencies which are not python packages (I think
 most of it is already handled in metadata 2.0/PEP 426, but I would have to
 re-read the PEP carefully).
  - ability to run post/pre install/remove scripts
  - support for all the of the autotools directories, with sensible
 mapping on windows


 Are these inside or outside the python installation? I'm more than a bit
 wary of a wheel that would install stuff outside of the sandbox of the
 python install.


I would always install things relative to sys.prefix, for exactly the
reasons you mention.




 The whole reason I started this discussion is to make sure wheel has a
 standard way to do what is needed for those usecases.

 conda, rpm, deb, or eggs as used in enthought are all essentially the
 same: an archive with a bunch of metadata. The real issue is standardising
 on the exact formats. As you noticed, there is not much missing in the
 wheel *spec* to get most of what's needed.


 hmm -- true. I guess where it seems to get more complicated is beyond the
 wheel (or conda, or...) package itself, to the dependency management,
 installation tools, etc.

 But perhaps you are suggesting that we can extend wheel to support a bt
 more stuff, and leave the rest of the system as separate problem? i.e.
 Canopy can have it's own find, install, manage-dependency tool, but that it
 can use the wheel format for the packages themselves?


Exactly !

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

On Mon, Apr 13, 2015 at 10:17 PM, Robert Collins robe...@robertcollins.net
wrote:

 On 14 April 2015 at 09:35, David Cournapeau courn...@gmail.com wrote:
 ...

 One of the earlier things mentioned here - {pre,post}{install,remove}
 scripts - raises a red flag for me.


That's indeed a good a priori. I myself removed a lot of those scripts
because of the fragility. Anything that needs to run on a end-user machine
can fail, and writing idempotent scripts is hard.

Unfortunately, pure declarative does not really cut it if you want cross
platform support. Sure, you may be able to deal with menu entries,
environment variables, etc... in a cross-platform manner with a significant
effort, but what about COM registration ? pywin32 is one of the most used
package in the python ecosystem, and its post install script is not
trivial. Another difficult case if when a package needs some specific
configuration to run at all, and that configuration requires values known
at install time only (e.g. sys.prefix, as in the iris package).


 I'd really prefer it if we keep wheels 100% declarative, and instead
 focus on defining appropriate metadata for the things you need to
 accomplish {pre,post}{install,remove} of a package.


What about a way for wheels to specify whether their
post,pre/install,remove scripts are declarative or not, with support for
the most common tasks, with an escape, but opt-in mechanism ? This way it
could be a matter of policy to refuse packages that require non-declarative
scripts.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft don...@stufft.io wrote:


 On Apr 13, 2015, at 10:39 AM, David Cournapeau courn...@gmail.com wrote:

 Hi there,

 During pycon, Nick mentioned there was interest in updating the wheel
 format to support downstream distributions. Nick mentioned Linux
 distributions, but I would like to express interest for other kind of
 downstream distributors like Anaconda from Continuum or Canopy from
 Enthought (disclaimer: I work for Enthought).

 Right now, wheels have the following limitations for us:

 1. lack of post/pre install/removing
 2. more fine-grained installation scheme
 3. lack of clarify on which tags vendors should use for custom wheels:
 some packages we provide would not be installable on normal python, and
 it would be nice to have a scheme to avoid confusion there as well.

 At least 1. and 2. are of interest not just for us.

 Regarding 2., it looks like anything in the wheel_name.data/data
 directory will be placed as is in sys.prefix by pip. This is how distutils
 scheme is defined ATM, but I am not sure whether that's by design or
 accident ?

 I would suggest to use something close to autotools, with some tweaks to
 work well on windows. I implemented something like this in my project bento
 (
 https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py),
 but we could of course tweak that.

 For 1., I believe it was a conscious decision not to include them in wheel
 1.0 ? Would it make sense to start a discussion to add it to wheel ?

 I will be at the pycon sprints until wednesday evening, so that we can
 flesh some concrete proposal first, if there is enough interest.

 As a background: at Enthought, we have been using eggs to distribute
 binaries of python packages and other packages (e.g. C libraries, compiled
 binaries, etc...) for a very long time. We had our own extensions to the
 egg format to support this, but I want to get out of eggs so as to make our
 own software more compatible with where the community is going. I would
 also like to avoid making ad-hoc extensions to wheels for our own purposes.


 To my knowledge, (1) was purposely punted until a later revision of Wheel
 just to make it easier to land the “basic” wheel.


Great. Was there any proposal made to support it at all ? Or should I just
work from scratch there ?



 I think (2) is a reasonable thing as long as we can map it sanely on all
 platforms.


Yes. We support all platforms at Enthought, and Windows is important for us
!


 I’m not sure what (3) means exactly. What is a “normal” Python, do you
 modify Python in a way that breaks the ABI but which isn’t reflected in the
 standard ABI tag?


It could be multiple things. The most obvious one is that generally.
cross-platforms python distributions will try to be relocatable (i.e. the
whole installation can be moved and still work). This means they require
python itself to be built a special way. Strictly speaking, it is not an
ABI issue, but the result is the same though: you can't use libraries from
anaconda or canopy on top of a normal python

More generally, we could be modifying python in a way that is not forward
compatible with upstream python: a binary that works on our python may not
work on the python from python.org (though the opposite is true). It would
be nice if one could make sure pip will not try to install those eggs when
installed on top of a python that does not advertise itself as compatible.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

Hi there,

During pycon, Nick mentioned there was interest in updating the wheel
format to support downstream distributions. Nick mentioned Linux
distributions, but I would like to express interest for other kind of
downstream distributors like Anaconda from Continuum or Canopy from
Enthought (disclaimer: I work for Enthought).

Right now, wheels have the following limitations for us:

1. lack of post/pre install/removing
2. more fine-grained installation scheme
3. lack of clarify on which tags vendors should use for custom wheels: some
packages we provide would not be installable on normal python, and it
would be nice to have a scheme to avoid confusion there as well.

At least 1. and 2. are of interest not just for us.

Regarding 2., it looks like anything in the wheel_name.data/data
directory will be placed as is in sys.prefix by pip. This is how distutils
scheme is defined ATM, but I am not sure whether that's by design or
accident ?

I would suggest to use something close to autotools, with some tweaks to
work well on windows. I implemented something like this in my project bento
(
https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py),
but we could of course tweak that.

For 1., I believe it was a conscious decision not to include them in wheel
1.0 ? Would it make sense to start a discussion to add it to wheel ?

I will be at the pycon sprints until wednesday evening, so that we can
flesh some concrete proposal first, if there is enough interest.

As a background: at Enthought, we have been using eggs to distribute
binaries of python packages and other packages (e.g. C libraries, compiled
binaries, etc...) for a very long time. We had our own extensions to the
egg format to support this, but I want to get out of eggs so as to make our
own software more compatible with where the community is going. I would
also like to avoid making ad-hoc extensions to wheels for our own purposes.

thanks,
David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [pycon] Packaging-related discussions

[David Cournapeau]

Do we have a confirmation for Friday ? I have not seen  a BoF board yet.

On Fri, Apr 3, 2015 at 6:16 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On 3 April 2015 at 08:09, Richard Jones rich...@python.org wrote:
  Could the BoF be Friday instead please? Saturday is International
 Tabletop
  Day, and there's a bunch of us will be celebrating that :)

 Sure, Friday works for me, too.

 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [pycon] Packaging-related discussions

[David Cournapeau]

Both Friday and Saturday would work for me.

What does it mean concretely to arrange a room ? Just finding out a room
and communicate the location, or are there other responsibilities ?

David

On Thu, Apr 2, 2015 at 10:46 PM, Nick Coghlan ncogh...@gmail.com wrote:


 On 3 Apr 2015 07:03, Richard Jones rich...@python.org wrote:
 
  I can't speak for any plans others active in the PyPA might have, but
 I'll be using the sprint time to work on Warehouse and hopefully help
 others work on it also.
 
  I'm almost certain that my hallway track time will involve many
 packaging-related discussions, as it always does :)

 A Packaging BoF would be good, it just needs a volunteer to arrange a room
 with the open space organisers. Some time Saturday evening, perhaps?

 Cheers,
 Nick.

 
 
   Richard
 
  On Fri, 3 Apr 2015 at 00:52 David Cournapeau courn...@gmail.com wrote:
 
  Hi there,
 
  I would like to know what are the events/discussions related to
 packaging happening this year at PyCon ?
 
  My employer is sponsoring my trip to the conference this year, and I
 would like to make sure I am not missing an important event. I will also be
 there for most of the sprints,
 
  cheers,
  David
  ___
  Distutils-SIG maillist  -  Distutils-SIG@python.org
  https://mail.python.org/mailman/listinfo/distutils-sig
 
 
  ___
  Distutils-SIG maillist  -  Distutils-SIG@python.org
  https://mail.python.org/mailman/listinfo/distutils-sig
 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] [pycon] Packaging-related discussions

[David Cournapeau]

Hi there,

I would like to know what are the events/discussions related to packaging
happening this year at PyCon ?

My employer is sponsoring my trip to the conference this year, and I would
like to make sure I am not missing an important event. I will also be there
for most of the sprints,

cheers,
David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows?

[David Cournapeau]

Regarding winrm, note that it can be fairly complicated to set it up on the
windows side.

I can confirm it works well, though. We're using an internal hack of fabric
to communicate from linux to windows through winrm at work, and it does the
job.

David

On Sun, Nov 9, 2014 at 12:40 PM, Nick Coghlan ncogh...@gmail.com wrote:


 On 9 Nov 2014 22:28, Nick Coghlan ncogh...@gmail.com wrote:
 
 
  On 9 Nov 2014 22:16, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
  
Thanks, that's very useful feedback. I agree, the need for RDP is
 very
  
Windows-specific - I don't know how common RDP tools are for Unix,
 but
  
  
  
  
   Not uncommon, AFAIK. For example, I use rdesktop on Lubuntu to access
 Windows machines via RDP, and it seems fairly stable. There are alternative
 tools available (such as remmina).
 
  For automated scripting from a *nix host, it's also worth noting that
 Ansible supports remote control of Windows systems, and the underlying
 Python libraries for that connectivity are also open source (they talk to
 the native Windows remote control interfaces, so they don't need anything
 special on the target system).

 Oops, forgot the link: http://docs.ansible.com/intro_windows.html

 Underlying library: https://pypi.python.org/pypi/pywinrm

 Cheers,
 Nick.

 
  Cheers,
  Nick.
 
  
   Regards,
  
   Vinay Sajip
   ___
   Distutils-SIG maillist  -  Distutils-SIG@python.org
   https://mail.python.org/mailman/listinfo/distutils-sig

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows?

[David Cournapeau]

On Sun, Nov 9, 2014 at 6:39 PM, Kevin Horn kevin.h...@gmail.com wrote:

 Regarding remote access to windows machines, there are several options:

 - Remote powershell (not my area, so not sure how viable this is)
 - Use pexec from sysinternals to run cmd.exe remotely (probably on local
 network only, and only from other windows machines, so probably not that
 helpful)
 - Windows comes with a telnet server (obviously not very secure, but you
 could use stunnel/vpn or similar to help here)
 - WinRM (and pywinrm as has been mentioned).  Note that vagrant either
 does or will soon support talking to Windows VMs using this method.


It does (and has for a while).

We at Enthought automate package builds for linux, mac and windows through
fabric (+ winrm on windows) on vagrant-built VMs, and it works well when
you need a consistent interface to those environments. We have been
building packages as involved as Qt, Pyside or scipy with native MS tools
through this way, so I am confident this would work for almost every
situation of interest here.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP425 - Wheel Binary Package Compatibility

[David Cournapeau]

On Tue, Oct 28, 2014 at 10:13 AM, Matthias Hafner hafne...@gmail.com
wrote:

 Hi there,

 Following up on
 https://bitbucket.org/pypa/wheel/issue/124/glibc-incompatibility.

 How should we deal with incompatibility of dynamically linked libraries?
 Doing pip wheel numpy on a Linux 64bit machine results in , which is
 linked dynamically against the GLIBC version installed on the build
 machine.


Wheel by itself simply cannot deal with this, you need something more.

Practically speaking, there is no such a thing as ABI on Linux: even if you
somehow managed to deal with glibc, you would then need to deal with
fortran ABI, then with C++ ABI, etc... Dealing with this at the python
level is simply intractable. That needs to be handled at the distribution
level, or within a sumo distribution like Anaconda from Continuum or
Canopy from Enthought (disclaimer, I work for Enthought), which makes the
problem more tractable by picking up an ABI and stick with it for the full
set of packages.

In theory, you can try to emulate this by building on the oldest version of
each compiled library, and hope that your target platforms are ABI
compatible with this, but that's hard on Linux.

David


 So should the wheel be shipped with GLIBC, or the GLIBC version be
 specified in the wheel name (means to build a new wheel for each GLIBC
 version?). Also, maybe this is not the only library it is dynamically
 linked against?

 Why does that work on MacOS, btw? Are all library versions fixed there for
 one version of OSX?


Yes. Same for windows. The native toolchains there are fairly easy to
configure for targetting a specific minimum version.

David


 Thanks for putting some light into this issue.

 Best regards,
 Matthias

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[David Cournapeau]

On Wed, Oct 1, 2014 at 5:44 PM, David Genest david.gen...@ubisoft.com
wrote:

 Thank you all for the precious info.

 Here are my observations:

 - We are merely writing extension modules with third party dependant code
 packaged in a dll. In my mind, this use case is not the exception, and
 would not necessarily warrant the use of a full blown solution like conda.
 Our deployed environments are self-contained.

 - If you run python setup.py bdist_wheel, the dlls specified in the
 scripts parameter end up in the wheel archive and does what is needed for
 our setup. (the dlls are copied to the scripts directory which is on PATH
 for the activated environment).


One issue with PATH is its global settings: if you copy your dll there and
any other program the user may want to use, you will get into trouble. This
is mostly an issue for well known 3rd party DLLs, though.


 - On the other hand, if you run pip wheel on the same package, the dlls
 are not placed in the archive. Is this a pip bug ?

 In an ideal world, the scripts directory would be called bin, like the
 unix counter-part, and any dependency, being startup scripts or dlls could
 be installed in the bin/ environment global space. This path would be
 added to the python startup sequence (in order to not rely on the env's
 activate).

 I feel that the current state of affairs is not that far, because setup.py
 bdist_wheel works now.

 Knowing that there are alternatives on the way (in metadata 2.0 ?) and
 workarounds, we will go with our current wheel solution using setup.py
 bdist_wheel.

 If the bdist_wheel command ever loses the package binary files in scripts
 dir we have alternatives (listed in order of dependability):

 1) add the dependent dlls to every package that needs it (Steve's answer
 https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html
  concurs that the dependent dll would be loaded only once)


Note Steve's observation regarding manifests. It is not an uncommon issue
when integrating with 3rd party libs in my experience (where renaming the
dll is often impractical).

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[David Cournapeau]

On Tue, Sep 30, 2014 at 3:31 PM, David Genest david.gen...@ubisoft.com
wrote:


  This is not true. Python loads DLLs with LOAD_WITH_ALTERED_SEARCH_PATH,
 to allow them to be located alongside the pyd file. You should therefore be
 able to ship the
  dependent dll in the package directory (which wheels support fine).

  Paul

 Ok, so what if the dll is shared in a given environment (multiple
 extensions use it)?,  the shared dll should be copied to every package?
 Won't that cause multiple loads by the system?


Yes it will, and it is indeed problematic. There are no great solutions:

 - bundle it in your package
 - have a separate wheel and then put it in PATH
 - have a separate wheel but use preload tricks (e.g. using ctypes) to
avoid using PATH

There are better solutions if one can patch python itself, though that's
obviously not practical in most cases.

David


 Thanks for your response,

 D.



 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[David Cournapeau]

On Tue, Sep 30, 2014 at 3:44 PM, Nick Coghlan ncogh...@gmail.com wrote:

 On 1 October 2014 00:37, Paul Moore p.f.mo...@gmail.com wrote:
  On 30 September 2014 15:31, David Genest david.gen...@ubisoft.com
 wrote:
  Ok, so what if the dll is shared in a given environment (multiple
 extensions use it)?,  the shared dll should be copied to every package?
 Won't that cause multiple loads by the system?
 
  I honestly don't know in that case, sorry. You might get a better
  answer on python-list for that, if no-one here can help.
 
  Presumably the usage is all within one distribution, otherwise the
  question would have to be, which distribution ships the DLL? But that
  question ends up leading onto the sort of discussion that starts
  well, I wouldn't design your system the way you have, which isn't
  likely to be of much help to you :-(
 
  Sorry I can't offer any more help.

 Note that this is the external binary dependency problem that the
 scientific folks are currently using conda to address. It's basically
 the point where you cross the line from language specific packaging
 system to multi-language cross-platform platform.


Conda is one such solution, not the solution ;)

I don't know any sumo distribution which solves this problem correctly
ATM, and windows makes this rather difficult to solve.

David


 That said, pip/wheel *may* get some capabilities along these lines in
 the future, it just isn't a high priority at this point.

 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build a wheel with waf instead of setuptools

[David Cournapeau]

That's certainly possible, bento does exactly this. You would write a
setup.py that would like like:

import setuptools
from bento.distutils.monkey_patch import monkey_patch
monkey_patch()
setuptools.setup()

then you can do python setup.py install/build/sdist/etc... without actually
using distutils.

David


On Wed, Jul 30, 2014 at 8:46 AM, Chris Barker chris.bar...@noaa.gov wrote:

 On Fri, Jul 25, 2014 at 7:21 AM, Daniel Holth dho...@gmail.com wrote:

  This kind of thing will require us to implement a flag that tells pip
  setup.py cannot install; go through wheel which is somewhere in the
  plans..


 couldn't you write a file called setup.py, with the core API (i.e
 setup.py build | install), but that called waf instead of distutils to do
 the actual work?

 or does pip doe more than simply call the setup.py script?

 -Chris





   I don’t think there is any plans to tell pip *not* to use a setup.py
 and to
  use a Wheel instead. Rather I think the plans are to enable pluggable
  builders so that a sdist 2.0 package doesn’t rely on setup.py and could
 use
  a waf builder (for instance) plugin.

 Just a flag that tells pip it can't use the install command and has
 to do package - install package on an sdist.
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig




 --

 Christopher Barker, Ph.D.
 Oceanographer

 Emergency Response Division
 NOAA/NOS/ORR(206) 526-6959   voice
 7600 Sand Point Way NE   (206) 526-6329   fax
 Seattle, WA  98115   (206) 526-6317   main reception

 chris.bar...@noaa.gov

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Is build an inherently arbitrary-code process?

[David Cournapeau]

On Thu, Mar 27, 2014 at 7:48 PM, Donald Stufft don...@stufft.io wrote:


 On Mar 27, 2014, at 3:42 PM, Daniel Holth dho...@gmail.com wrote:

  I became convinced that build was an inherently arbitrary-code
  process, and not something to be universally handled by a declarative
  system, when I observed an autotools project under configuration. The
  things spend ten minutes writing and compiling snippets of C code to
  determine which features are and are not supported by the runtime.
  This is *very* arbitrary code and probably one of the tamer things
  people like to do during builds.
 
  As usual I blame distutils, because the separation is not there. So
  when you blame setup.py for having to run procedural code to generate
  the *metadata* you might go to far and think you should also eliminate
  a procedural build - simply because build and metadata are not
  adequately separate in the distutils design. Declarative build systems
  are a nice idea but they are not going to work for everyone.
  ___
  Distutils-SIG maillist  -  Distutils-SIG@python.org
  https://mail.python.org/mailman/listinfo/distutils-sig

 Well for the record auto tools does all of that because there’s near
 zero ability to introspect the compiler to figure out what it supports
 except by compiling little programs that depend on X feature and
 then running them to see if they work or not.

 That being said, yes building code is inherently a process that needs
 to execute things and cannot be made completely static. I believe
 the metadata itself can be static, and instead of an arbitrary build
 script we can declare a build hook, but that’s an argument for a PEP!

 I do believe a declarative build system can work for the 90% case
 though and should probably be the “default” option.


It is indeed possible for most python packages, and I tried to figure out
quite a few things to make that percentage as high as possible in Bento.

The really difficult part is not that, though, but making the system
pluggable to allow pretty much arbitrary extensibility. What makes
distutils very ill suited to that task is the whole subcommand model where
options are not known in advance (that was the #1 issue when integrating
distutils and scons).

For bento, I came up with a system of 'build manifest', where each command
produces a text-based 'protocol' that other commands might use (
https://github.com/pv/bento/blob/master/doc/source/hacking.rst). The
implementation is not great but I think the idea is still sound: this means
that in theory, you could build C extensions with something not involving
python at all (e.g. make), but as long as the build process output that
file, you could build installers from the system. IOW, the json file acts
as 'middleware'.

What any future solution should *not* try to do is inventing yet another
build system

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] waf can be used to compile Python extension modules

[David Cournapeau]

On Fri, Mar 21, 2014 at 1:46 PM, Daniel Holth dho...@gmail.com wrote:

 Bento emulates setup.py:
 https://github.com/cournape/Bento/blob/master/setup.py

 It mostly has to implement egg_info and install to be pip compatible.


Yes. While some things are quite not ideal in Bento, I think that part is a
good one: build a system that goes away completely from distutils, and then
emulate those interfaces (where it makes sense).

This allows compatibility in most cases, while avoiding re-introducing what
makes distutils horrible (commands and compilers).

Before attempting to solve the build problem itself, I would suggest to
have a small library extracted from distutils that does not impose any
'policy' on the build, but expose compilation information (flags, link
options, etc...). This library could be used from distutils, but also
(crucially) by 3rd party tools.

Regarding build, my own experience after having dealt with unholly
scons/waf/distutils/numpy.distutils integration projects:

  - solving the simple cases (simple C extensions) is easy, but moving to
more complex cases makes the problem much harder. Build is really a case
where solving 80 % of the usecases is solving 10 % of the problem. Solving
80 % of the usecases is ok as long as it does not prevent the remaining 20
% to do what they want.
  - neither scons or waf are really usable as libraries (scons much more so
than waf). Even fbuild, which I generally find more approachable, has a
'god' object that makes integration difficult.
  - having well documented, self-contained API to build final artefacts
(sdist, installs, eggs, wheels, bdist_wininst, bdist_msi, etc...) that help
integrating with 3rd party build tools is more critical than the build tool
itself.

David


 In the future we'd like them to implement dist_info (generate a
 .dist-info directory instead of an .egg-info directory) and wheel; an
 sdist would signal that it doesn't implement setup.py install and pip
 would always build and then install a wheel. This would be a
 relatively simple thing to add to pip.

 Once setup.py emulation is working then we could define a Python-level
 plugin interface for build systems that might return metadata dicts,
 and also provide a setup.py that talks to this plugin interface. A
 later version of pip might talk directly to the plugin without
 bothering with setup.py.

 This strategy does not generally try to eliminate arbitrary code
 execution during builds - builds are an inherently arbitrary-code
 process. But once the build has happened most installs should work
 without arbitrary code execution.

 On Fri, Mar 21, 2014 at 6:13 AM, Nick Coghlan ncogh...@gmail.com wrote:
 
  On 20 Mar 2014 23:16, Brett Cannon bcan...@gmail.com wrote:
 
 
 
  On Thu Mar 20 2014 at 12:51:13 AM, Nick Coghlan ncogh...@gmail.com
  wrote:
 
  On 20 March 2014 09:54, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
   Daniel Holth dholth at gmail.com writes:
  
   extensions without using distutils. The problem of invoking the
   compiler has been solved by many build systems and is not just a
   unique and mysterious distutils feature.
  
   Did someone say it was? Building extensions is something distil does
   too, and
   without using distutils or setuptools.
 
  Right, the problem is the lack of a standard interface for how the
  packaging system is supposed to invoke them - that is, *implementation
  independent* docs of what the various setup.py commands are *supposed*
  to do.
 
  The packaging system shouldn't have to care *how* setup.py is
  implemented, but at the moment, the behaviour is underspecified to the
  point where it's a matter of reverse engineering distutils and/or
  implementing what seems necessary and waiting to see if people
  complain.
 
 
  What are the plans for the build step in the grand plan of Python
  packaging? I think previously it has been suggested that once metadata
 is
  done and distribution/installation is taken care of the
 distutils/setuptools
  building part of all of this will be tackled. Is that still accurate?
 
  That's the priority list for my personal focus, but I certainly wouldn't
  complain if there was momentum behind addressing the build side in
 parallel,
  rather than in series. Commenting on PEPs is usually easier than writing
  them, and the starting point for the build side is mostly just defining a
  suitable subset of setup.py commands, options, behaviour and invocation
  context that will make up the fallback legacy interface a new metabuild
  system would need regardless of any other details.
 
  I just don't have the personal bandwidth to champion such an effort at
 this
  point in time.
 
  Cheers,
  Nick.
 
 
  ___
  Distutils-SIG maillist  -  Distutils-SIG@python.org
  https://mail.python.org/mailman/listinfo/distutils-sig
 
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 

Re: [Distutils] How to link to a c library ?

[David Cournapeau]

On Wed, Feb 19, 2014 at 8:30 AM, chaouche yacine
yacinechaou...@yahoo.comwrote:



 What I was trying to accomplish
 ---

 I tried to upgrade numpy
 # pip install --upgrade
 numpy.../chaouche/build/numpy/numpy/numarray/_capi.c:198:undefined
 reference to `log10' collect2: ld returned 1 exit status error: Command
 gcc -pthread -shared -Wl,--as-needed -Wl,--no-undefined -Wl,-z,relro
 -Wl,-O1 -Wl,--build-id -Wl,--enable-new-dtags
 build/temp.linux-i686-2.7/numpy/numarray/_capi.o -L/usr/lib
 -Lbuild/temp.linux-i686-2.7 -lpython2.7 -o
 build/lib.linux-i686-2.7/numpy/numarray/_capi.so failed with exit status 1
  Rolling back uninstall of numpy
 Replacing /usr/lib/python2.7/site-packages/numpy-1.6.1-py2.7.egg-info
 Command /usr/bin/python -c import
 setuptools;__file__='/home/chaouche/build/numpy/setup.py';exec(compile(open(__file__).read().replace('\r\n',
 '\n'), __file__, 'exec')) install --single-version-externally-managed
 --record /tmp/pip-lppaDW-record/install-record.txt failed with error code 1
 Exception information:
 Traceback (most recent call last):
 ...
 InstallationError: Command /usr/bin/python -c import
 setuptools;__file__='/home/chaouche/build/numpy/setup.py';exec(compile(open(__file__).read().replace('\r\n',
 '\n'), __file__, 'exec')) install --single-version-externally-managed
 --record /tmp/pip-lppaDW-record/install-record.txt failed with error code 1



 What the problem was
 

 I eventually figured out that they didn't link to the math library
 where all those symbols where defined (log10, sqrt, sincos etc.). I
 found that out by manually trying the gcc command with the -lm flag and
 it compiled fine.


 My question
 
 What files do I need to edit in order to add this -lm flag and install
 numpy correctly ?


numpy build links against the math library when needed, your problem most
likely comes from somewhere else.

You should contact the numpy mailing list, and post your full build log
there (not just the last lines).

David

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[David Cournapeau]

On Thu, Jan 30, 2014 at 7:50 AM, Paul Moore p.f.mo...@gmail.com wrote:

 On 29 January 2014 22:50, David Cournapeau courn...@gmail.com wrote:
  i.e. it would be nice if anyone setup to build C extensions could just
  build numpy.
 
  This has always been possible, and if not, that's certainly considered
 as a
  bug (I would be eager to fix).

 I don't know if you saw my comment earlier in this thread:

  A quick test later:
  No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the
  rest of the error stack (which I'm frankly not interested in investing
  the time to diagnose and fix) I get RuntimeError: Broken toolchain:
  cannot link a simple C program. Which is utter rubbish - I routinely
  build extensions with this installation.

 This is a straight pip install numpy run in a virtualenv on Windows
 7 64-bit with MSVC 2010 (full edition) installed.


which version of python ?

For 2.x, it is expected to fail at that point, since you can't build C
extensions with something different than 2008 (with python.org builds).

For 3.3, it means there is a bug that needs to be fixed.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[David Cournapeau]

On Thu, Jan 30, 2014 at 10:20 AM, Paul Moore p.f.mo...@gmail.com wrote:

 On 30 January 2014 09:12, David Cournapeau courn...@gmail.com wrote:
 
  On Thu, Jan 30, 2014 at 7:50 AM, Paul Moore p.f.mo...@gmail.com wrote:
 
  On 29 January 2014 22:50, David Cournapeau courn...@gmail.com wrote:
   i.e. it would be nice if anyone setup to build C extensions could
 just
   build numpy.
  
   This has always been possible, and if not, that's certainly considered
   as a
   bug (I would be eager to fix).
 
  I don't know if you saw my comment earlier in this thread:
 
   A quick test later:
   No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the
   rest of the error stack (which I'm frankly not interested in investing
   the time to diagnose and fix) I get RuntimeError: Broken toolchain:
   cannot link a simple C program. Which is utter rubbish - I routinely
   build extensions with this installation.
 
  This is a straight pip install numpy run in a virtualenv on Windows
  7 64-bit with MSVC 2010 (full edition) installed.
 
 
  which version of python ?
 
  For 2.x, it is expected to fail at that point, since you can't build C
  extensions with something different than 2008 (with python.org builds).
 
  For 3.3, it means there is a bug that needs to be fixed.

 Doh. Sorry, I knew I'd forget something. 3.3.


Here we go: https://github.com/numpy/numpy/issues/4245

David

 Paul

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[David Cournapeau]

On Sun, Jan 26, 2014 at 12:29 AM, Nick Coghlan ncogh...@gmail.com wrote:

 Paul's position exactly mirrors my own - I an perfectly fine with the
 recommended advice to scientific users continuing to be NumPy doesn't
 officially support pip and virtualenv because of the way it is built and
 installed, so you will have to get one of the curated scientific stacks,
 bootstrap conda, use the Windows installers or use the version provided by
 your Linux distro vendor.

 The metadata 2.0 standards *will not* be accepted until the pip 1.6 or 1.7
 time frame, and it's more likely the latter, since I don't want to distract
 anyone from the current security work (I know I said otherwise recently,
 but I managed to temporarily forget that the Warehouse transition and
 implementing PEP 458 was next on the to do list when I said that).

 So, if the NumPy community choose to wait for general post-install script
 support in wheel files, they're likely to be waiting at least until the
 release of pip 1.7.

 In the meantime, the failure mode for people attempting to try out the
 Scientific Python stack via pip install numpy in an existing Python
 installation or virtualenv will remain a failure to build with a likely
 cryptic error.

 I do see a few possible workarounds, but none of them would change the
 metadata 2.0 standards:

 1. Add explicit NumPy support *directly* to pip. This would be the quick
 hack, private API support that Oscar is requesting, since it would be a
 special arrangement between the pip devs and the numpy devs, and eventually
 replaced by a general purpose post-install mechanism in metadata 2.0.


I am not speaking for the whole numpy team, but as the former maintainer of
numpy.distutils, I think this will be more hurtful than helpful.

I think the SSE issue is a bit of a side discussion: most people who care
about performance already know how to install numpy. What we care about
here are people who don't care so much about fast eigenvalue decomposition,
but want to use e.g. pandas. Building numpy in a way that supports every
architecture is both doable and acceptable IMO.

2. Add support to pip to request the conversion of available wininst
 installers (and bdist_dumb?) to wheels for installation with pip. Vinay has
 this working from a technical perspective, so it may be something the pip
 devs are interested in exploring for pip 1.6.


Building numpy wheels is not hard, we can do that fairly easily (I have
already done so several times, the hard parts have nothing to do with wheel
or even python, and are related to mingw issues on win 64 bits).


 3. Both of the above options require waiting for pip 1.6 (at the
 earliest), which means neither will improve the behaviour in CPython 3.4
 (which will ship pip 1.5.1). The only folks with the power to improve
 *that* situation are the NumPy devs, who have the ability to choose between
 the doesn't work for anyone except experienced build engineers status quo
 and works for a large percentage of users, but will still fail at runtime
 for users on hardware without SSE2 support.

 To put the but what if the user doesn't have SSE2 support? concern in
 context, it should only affect Intel users with CPUs older than a Pentium 4
 (released 2001), and AMD users with a CPU older than an Opteron or Athlon
 64 (both released 2003). All x86/x86_64 CPUs released in the past decade
 should be able to handle SSE2 binaries, so our caveat can be if your
 computer is more than a decade old, 'pip install numpy' may not work for
 you, but it should do the right thing on newer systems.

 Now, the NumPy devs may feel that persisting with the status quo for
 another 6 to 12 months while waiting for still hypothetical additional
 changes in pip specifically to accommodate NumPy's current installation
 practices is a better alternative than taking option 3. However, from my
 perspective, having NumPy readily available to users using the 
 python.orgWindows installers for Python 3.4 would *significantly* lower the 
 barrier
 to entry to the Scientific Python stack for new users on relatively modern
 systems when compared to the 4 current options (while we accept the Linux
 distro problem is on distutils-sig to deal with, that's far from being a
 NumPy specific problem).


Just to clarify: you actually can install numpy on windows with
python.orginstallers fairly easily by using easy_install already (we
upload a
bdist_wininst compatible binary which should not use any CPU-specific
instructions). It looks like those are missing for 1.8.0, but we can fix
this fairly easily.

David

 Cheers,
 Nick.

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[David Cournapeau]

On Wed, Jan 29, 2014 at 10:52 PM, Donald Stufft don...@stufft.io wrote:

 I don’t see any reason why SSE couldn’t be added as tags in the Wheel
 filename fwiw.


You still need to decide when to install what, but I would be interested in
talking more about that part.



 That doesn’t help for things like MKL though.


Nope, but MKL is actually easy in the sense that it deals with
architectures at runtime. OSS numerical libraries generally don't (lots of
work, and often a non issue when you can build stuff by yourself :) ).

David


 On Jan 29, 2014, at 5:50 PM, David Cournapeau courn...@gmail.com wrote:




 On Wed, Jan 29, 2014 at 10:27 PM, Chris Barker chris.bar...@noaa.govwrote:

 On Wed, Jan 29, 2014 at 2:04 PM, David Cournapeau courn...@gmail.comwrote:

 I think the SSE issue is a bit of a side discussion: most people who
 care about performance already know how to install numpy. What we care
 about here are people who don't care so much about fast eigenvalue
 decomposition, but want to use e.g. pandas. Building numpy in a way that
 supports every architecture is both doable and acceptable IMO.


 Exactly -- I'm pretty sure SSE2 is being suggested because that's the
 lowest common denominator that we expect to see a lot of -- if their really
 are a lot of non-SSE-2 machines out there we could leave that off, too.


 The failure mode is fairly horrible though, and the gain is not that
 substantial anyway compared to really optimized installation (MKL, etc...
 as provided by Continuum or us).


 Building numpy wheels is not hard, we can do that fairly easily (I have
 already done so several times, the hard parts have nothing to do with wheel
 or even python, and are related to mingw issues on win 64 bits).


 David,

 Where is numpy as with building out of the box with the python.orgbinary 
 for Windows, and the standard MS compilers that are used with
 those builds. That used to be an easy python setup.py install away -- has
 that changed? If so, is this a known bug, or a known
 we-aren't-supporting-that?

 i.e. it would be nice if anyone setup to build C extensions could just
 build numpy.


 This has always been possible, and if not, that's certainly considered as
 a bug (I would be eager to fix).

 Numpy is actually fairly easy to build if you have a C Compiler (which is
 the obvious pain point on windows). Scipy, and fortran is where things fall
 apart.

 David
  ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig



 -
 Donald Stufft
 PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Packaging today

[David Cournapeau]

On Thu, Jan 9, 2014 at 10:43 PM, Steve Spicklemire st...@spvi.com wrote:

 So, related question: Should the Mac binaries also work with the
 python.org mac build?


Binary compatibility is best achieved on windows: the platform makes it
easier, and it is so painful to build stuff there that it would be insane
for vendors not to be compatible.

Mac is a whole different story. First, you have frameworks vs non
frameworks builds, and compilers/dev environments keep changing in
incompatible ways: you have gcc, gcc-llvm, clang, SDK which are tied to
xcode versions themselves tied to OS versions, it is a mess :)

Case in point, the python.org binaries are generally not compatible with
the system python provided by Apple (that is you can't use an installer
made for one with the other). At least in the scientific community, we
generally provide binaries for python.org, and I think it is good practice
to avoid touching the system python.

David



 Regarding my quest: I just punted for now (out of time, need something
 working by Monday).

 I'm using Christoph Gohlke's binaries for windows (seem to work well!) and
 cooked up my own custom installer for Mac that loads on top of 
 python.orgpython and our current VPython installer. All working, so crisis 
 averted.
 ;-)

 I'll look into binary wheels when the dust settles.

 thanks for all the suggestions!
 -steve


 On Jan 8, 2014, at 2:09 PM, Steve Spicklemire st...@spvi.com wrote:

  This is good news. I'll experiment!
 
  thanks,
  -steve
 
  On Jan 8, 2014, at 1:48 PM, David Cournapeau courn...@gmail.com wrote:
 
 
 
 
  On Wed, Jan 8, 2014 at 6:44 PM, Chris Barker chris.bar...@noaa.gov
 wrote:
  On Wed, Jan 8, 2014 at 1:48 AM, David Cournapeau courn...@gmail.com
 wrote:
  We don't use mingw to build packages distributed within canopy (at
 least not anymore). We build everything with MSVC 2008, as mixing
 mingw/MSVC often causes trouble.
 
 
  so is Canopy binary-compatible with the python.org builds?
 
  i.e would the same binary wheel work for both?
 
  It should, and we will fix issues if they come up.
 
  David
 
  -Chris
 
 
  --
 
  Christopher Barker, Ph.D.
  Oceanographer
 
  Emergency Response Division
  NOAA/NOS/ORR(206) 526-6959   voice
  7600 Sand Point Way NE   (206) 526-6329   fax
  Seattle, WA  98115   (206) 526-6317   main reception
 
  chris.bar...@noaa.gov
 
 


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Packaging today

[David Cournapeau]

On Mon, Jan 6, 2014 at 9:55 PM, Chris Barker chris.bar...@noaa.gov wrote:

 On Mon, Jan 6, 2014 at 12:26 PM, Steve Spicklemire st...@spvi.com wrote:


 avoid troubling anyone with pesky questions. In that respect I've
 apparently failed, because here comes the question!


 I think this is a use case that  is quite useful for us all to chew on a
 bit...

 1st -- yes Anaconda refers to the distribution from Continuum. Note that
 conda is the packaging system that Anaconda uses, but it can also be used
 independently of the distribution.


 I'm helping out with a python package: vpython http://vpython.org


 [side note: fairly recently  a por tof VPython to wxPython was done -- is
 that what you are using? Notable because I think wxPython is still not
 available for Anaconda...)


 and I'm also teaching an intro scientific computing class this spring.
 I'm mostly a Mac/Linux user, but my students are often windows users. I
 would love to permit my students to use enthought/canopy and/or continuum
 analytics (C.A.) along with vpython.


 Either/or? As an instructor, I'd recommend you pick one and go with it --
 if you need wx, that means Canopy for now. Alternatively, you suggest the
 python.org builds, and point your users to binaries they can get
 elsewhere (Chris Gohlke's site for Windows...)

 At the moment we're creating binary releases of vpython for windows and
 mac and posting them on sourceforge 
 https://sourceforge.net/projects/vpythonwx/.


 Are these for the python.org builds? good for you!

 Bruce has been building the windows binary using VC (no setup.py) in a way
 that's compatible withpython.org python for windows. I've been building
 the mac version using a setup.py script I cobbled together that works on
 MacOSX and Linux.


 Why not distutils for building Windows? I find it really helpful.


 I've noticed that the anaconda system that C.A. installs uses MinGW on
 windows to build extensions.


 I think Canopy does that too -- at least it did a few years ago. but I
 _think_ you can build extensions with either MinGW or MSVC for the same
 binary python -- if it's set up right ;-)


We don't use mingw to build packages distributed within canopy (at least
not anymore). We build everything with MSVC 2008, as mixing mingw/MSVC
often causes trouble.

We do provide mingw so that people can build extensions without depending
on MS toolchain, but that's not a setup I would recommend in general. It
works well for extensions that don't depend much on the C runtime (numpy is
a good example), but as soon as you manipulate IO, etc... you encounter
issues linked to ABI incompatibilities between what you get from the python
runtime and mingw.

David


 I'd love to figure out how to build vpython under this system so that my
 windows users could use them together transparently.


 You want to take a look at conda:

 https://github.com/pydata/conda

 If you can build a conda recipe then you are set to go...

 That being said, it is supposed to be a goal for Anaconda to be binary
 compatible with the python.org binaries -- so you may well be able to
 build the way you are, and give the users a way to install it into
 Anaconda. In theory, binary wheels are the way to do this.

 I'm pretty sure I could work out how to build vpython with continuum
 analytics on the mac (which means building boost + wxPython using the C.A.
 python).


 ahh -- you are using wx -- I'd check with the CA folks and see where they
 are at -- they said they were working on a wxPython packageand I'm sure
 they'd like help and testing...


 Is there any way, *today*, to incorporate dependencies on external
 libraries (e.g., boost) in setup.py?


 no -- except by hand with custom code.


 I'm still a little unclear on your goals here. If you want to simple be
 able to tell your students to use Anaconda  then look into conda and the CA
 help lists -- and conda is more or less designed to solve these sorts of
 problems. also, the odds are good that Anaconda already has boost, and if
 not someone has done a conda recipe for it:

 https://github.com/faircloth-lab/conda-recipes/tree/master/boost

 If you want your users to be able to use any of:

 Anaconda
 Python.org python
 Canopy

 Then I'd look into building binary wheels, and see if you can get them to
 work with Anaconda and Canopy.

 Note: distutils does not address the third-party non-python dependency
 problem -- conda tries does address it (though still not clear to me if it
 helps you build them...) You may also want to look at Gattai, which aims to
 help you build  them:

 http://sourceforge.net/projects/gattai/

 HTH,
   -Chris




 --

 Christopher Barker, Ph.D.
 Oceanographer

 Emergency Response Division
 NOAA/NOS/ORR(206) 526-6959   voice
 7600 Sand Point Way NE   (206) 526-6329   fax
 Seattle, WA  98115   (206) 526-6317   main reception

 chris.bar...@noaa.gov

 ___
 Distutils-SIG maillist  -  

Re: [Distutils] Packaging today

[David Cournapeau]

On Wed, Jan 8, 2014 at 6:44 PM, Chris Barker chris.bar...@noaa.gov wrote:

 On Wed, Jan 8, 2014 at 1:48 AM, David Cournapeau courn...@gmail.comwrote:

 We don't use mingw to build packages distributed within canopy (at least
 not anymore). We build everything with MSVC 2008, as mixing mingw/MSVC
 often causes trouble.


 so is Canopy binary-compatible with the python.org builds?

 i.e would the same binary wheel work for both?


It should, and we will fix issues if they come up.

David


 -Chris


 --

 Christopher Barker, Ph.D.
 Oceanographer

 Emergency Response Division
 NOAA/NOS/ORR(206) 526-6959   voice
 7600 Sand Point Way NE   (206) 526-6329   fax
 Seattle, WA  98115   (206) 526-6317   main reception

 chris.bar...@noaa.gov

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Handling the binary dependency management problem

[David Cournapeau]

On Fri, Dec 6, 2013 at 5:47 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On 6 December 2013 11:52, Donald Stufft don...@stufft.io wrote:
 
  On Dec 5, 2013, at 8:48 PM, Chris Barker - NOAA Federal 
 chris.bar...@noaa.gov wrote:
 
  What would really be best is run-time selection of the appropriate lib
  -- it would solve this problem, and allow users to re-distribute
  working binaries via py2exe, etc. And not require opening a security
  hole in wheels...
 
  Not sure how hard that would be to do, though.
 
  Install time selectors probably isn’t a huge deal as long as there’s a
 way
  to force a particular variant to install and to disable the executing
 code.

 Hmm, I just had an idea for how to do the runtime selection thing. It
 actually shouldn't be that hard, so long as the numpy folks are OK
 with a bit of __path__ manipulation in package __init__ modules.


As Ralf, I think it is overkill. The problem of SSE vs non SSE is because
of one library, ATLAS, which as IMO the design flaw of being arch specific.
I always hoped we could get away from this when I built those special
installers for numpy :)

MKL does not have this issue, and now that openblas (under a BSD license)
can be used as well, we can alleviate this for deployment. Building a
deployment story for this is not justified.

David


 Specifically, what could be done is this:

 - all of the built SSE level dependent modules would move out of their
 current package directories into a suitable named subdirectory (say
 _nosse, _sse2, _sse3)
 - in the __init__.py file for each affected subpackage, you would have
 a snippet like:

 numpy._add_sse_subdir(__path__)

 where _add_sse_subdir would be something like:

 def _add_sse_subdir(search_path):
 if len(search_path)  1:
 return # Assume the SSE dependent dir has already been added
 # Could likely do this SSE availability check once at import time
 if _have_sse3():
 sub_dir = _sse3
 elif _have_sse2():
 sub_dir = _sse2
 else:
 sub_dir = _nosse
 main_dir = search_path[0]
 search_path.append(os.path.join(main_dir, sub_dir)

 With that approach, the existing wheel model would work (no need for a
 variant system), and numpy installations could be freely moved between
 machines (or shared via a network directory).

 To avoid having the implicit namespace packages in 3.3+ cause any
 problems with this approach, the SSE subdirectories should contain
 __init__.py files that explicitly raise ImportError.

 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Handling the binary dependency management problem

[David Cournapeau]

On Fri, Dec 6, 2013 at 12:44 PM, Donald Stufft don...@stufft.io wrote:

 How does conda handle SSE vs SSE2 vs SSE3? I’m digging through it’s
 source code and just installed numpy with it and I can’t seem to find any
 handling of that?


I can't speak for conda, but @enthought, we solve it by using the MKL,
which selects the right implementation at runtime.

Linux distributions have system to cope with it (the hwcap capabtility of
ld), but even there few packages use it. Atlas, libc are the ones I am
aware of. And this breaks anyway when you use static linking obviously.

David


 On Dec 6, 2013, at 7:33 AM, Nick Coghlan ncogh...@gmail.com wrote:

  On 6 December 2013 17:21, Ralf Gommers ralf.gomm...@gmail.com wrote:
  On Fri, Dec 6, 2013 at 6:47 AM, Nick Coghlan ncogh...@gmail.com
 wrote:
  With that approach, the existing wheel model would work (no need for a
  variant system), and numpy installations could be freely moved between
  machines (or shared via a network directory).
 
  Hmm, taking a compile flag and encoding it in the package layout seems
 like
  a fundamentally wrong approach. And in order to not litter the source
 tree
  and all installs with lots of empty dirs, the changes to __init__.py
 will
  have to be made at build time based on whether you're building Windows
  binaries or something else. Path manipulation is usually fragile as
 well. So
  I suspect this is not going to fly.
 
  In the absence of the perfect solution (i.e. picking the right variant
  out of no SSE, SSE2, SSE3 automatically), would it be a reasonable
  compromise to standardise on SSE2 as lowest acceptable common
  denominator?
 
  Users with no sse capability at all or that wanted to take advantage
  of the SSE3 optimisations, would need to grab one of the Windows
  installers or something from conda, but for a lot of users, a pip
  install numpy that dropped the SSE2 version onto their system would
  be just fine, and a much lower barrier to entry than well, first
  install this other packaging system that doesn't interoperate with
  your OS package manager at all
 
  Are we letting perfect be the enemy of better, here? (punting on the
  question for 6 months and seeing if we can deal with the install-time
  variant problem in pip 1.6 is certainly an option, but if we don't
  *need* to wait that long...)
 
  Cheers,
  Nick.
 
  --
  Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia


 -
 Donald Stufft
 PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
 DCFA


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Handling the binary dependency management problem

[David Cournapeau]

On Fri, Dec 6, 2013 at 5:50 PM, Chris Barker chris.bar...@noaa.gov wrote:

 On Fri, Dec 6, 2013 at 5:06 AM, David Cournapeau courn...@gmail.comwrote:

 As Ralf, I think it is overkill. The problem of SSE vs non SSE is because
 of one library, ATLAS, which as IMO the design flaw of being arch specific.


 yup -- really designed for the end user to built it themselves


 MKL does not have this issue, and now that openblas (under a BSD license)
 can be used as well, we can alleviate this for deployment. Building a
 deployment story for this is not justified.


 So Openblas has run-time selection of the right binary? very cool! So are
 we done here?


Not that I know of, but you can easily build one for a given architecture,
which is essentially impossible to do with Atlas reliably.

I did not know about openblas instabilities, though. I guess we will have
to do some more testing.

David


 -Chris


 --

 Christopher Barker, Ph.D.
 Oceanographer

 Emergency Response Division
 NOAA/NOS/ORR(206) 526-6959   voice
 7600 Sand Point Way NE   (206) 526-6329   fax
 Seattle, WA  98115   (206) 526-6317   main reception

 chris.bar...@noaa.gov

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Packaging panel in Montreal next year?

[David Cournapeau]

I could make it this time.

I was considering submitting a talk about a library I have been working on
recently for dependency solving. Would that fall into what you had in mind ?

It is designed to be fairly agnostic toward metadata format, installation
scheme, etc... It handles arbitrary dependency, the concept of
replaces/obsolete, and provides. The current code live at
https://github.com/enthought/depsolver

I haven't announced yet because it still needs some improvements, but a
similar design has been proved to work fairly well for php (composer),
OpenSuse (libzypp) and considered for Fedora (
https://fedoraproject.org/wiki/Features/DNF).

David


On Thu, Sep 12, 2013 at 5:39 PM, Nick Coghlan ncogh...@gmail.com wrote:

 The PyCon submission deadline is just a few days away, and I'm planning to
 submit a packaging QA panel again (same concept as the one this year, but
 with a further 12 months of progress to report).

 Before I do that, though, I just wanted to check there would be at least a
 few other distutils-sig denizens around that I could dragoon into
 participating.

 While it seems like a pretty safe bet, I don't see any reason to rely on
 assumptions when I don't need to :)

 Cheers,
 Nick.

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Python-Dev] distlib updated with resources API

[David Cournapeau]

On Mon, Oct 1, 2012 at 8:22 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
 David Cournapeau cournape at gmail.com writes:

 I noticed that you put the classifiers list as a string (same for
 platform). I think it is expected to be a list, no ?

 That's an oversight; there are doubtless others, too.

Sure. I guess I was just trying to get at getting the code released in
a repo so that we can provide patches :)


 Maybe slightly more controversial, I think the manifest should be
 evaluated. The current system of inclusion + exclusion is too
 baroque to my taste, and makes it near-impossible to make reproducible
 builds.

 Would you care to give a little more detail about what you mean by
 evaluate? I've kept the manifest as it is for backward compatibility (i.e.
 so that my sanity checking of sdist follows the logic as is used by
 distutils/distribute).

If you want to be backward compatible with distutils, then yes, you
have to encode it as is. By evaluating, I meant specifying the list of
files instead of using some higher level logic. Otherwise, the static
format does not specify the actual content (and depends on way too
many parameters).

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Python-Dev] distlib updated with resources API

[David Cournapeau]

On Mon, Oct 1, 2012 at 12:40 AM, Donald Stufft donald.stu...@gmail.com wrote:
 On Sunday, September 30, 2012 at 6:50 PM, David Cournapeau wrote:

 I am not sure I understand the argument: whatever the syntax, if the
 feature is there, you will have the same problem ? The fact that it is
 used by existing solutions tend to convince me it is not a problem
 (cabal works much better than distutils/etc... in pretty much every
 way, including installing things with dependency from their package
 servers, not to speak about RPM).

 I have a site that pulls data from python packages, one of my problems
 of late has been pulling requirements data out of a setuptools setup.py.
 Simply
 fetching the data for my current system is simple, however because of the
 use
 of conditionals that affect the final outcome of the metadata in order to
 make sure I get a true and complete accounting of all the metadata I would
 have to anticipate every combination of conditions used inside that file.

 Now as far as I can tell, the bento.info conditionals have that same problem
 except that they have a much narrow scope, so they have a smaller matrix.

 How would I given a package that has an ``if os(windows)`` conditional, get
 a representation of that files metadata as if it were in windows? I would
 need
 to override what os(windows) is? Now if this same package also has
 conditonals
 for flags, the matrix becomes larger and larger and I need to either create
 my own custom parser that just ignores or annotates the conditionals or I
 need
 to use the official parser and run it through once per item in the matrix to
 get
 the whole story. Am I incorrect in that?

 Additionally, conditionals as they are (to my understanding, having never
 used bento but having read the docs and some of it's source code)
 implemented
 in bento complicate the parser and confuse reading a file with understanding
 a file.

 What additional power or benefit does:

 Library:
 InstallRequires:
 docutils,
 sphinx
 if os(windows):
 pywin32

 provide over:

 {
 Library: {
 InstallRequires: [
 docutils,
 sphinx,
 pywin32; sys.playform == win32
 ]
 }
 }

It is obviously exactly the same, it is just different syntax (which
is why I don't understand the fetching requirements arguments, since
whatever is true for bento is true of your format and vice et versa).


 Maybe i'm missing some powerful construct in the bento.info that warrants
 throwing out a lot of hard work and testing of a lot of individuals for
 these
 common formats, however I just don't see it in merely providing a simple
 conditional statement. The ``bento.parser`` package contains ~1700 SLOC
 now without investigating that terribly deeply I'm willing to bet almost all
 of that could be thrown out in the use of a standard format.

What I am afraid of is that one will need to add a lots of those
markers, not just for requirements, which will lose the point of using
a common format. I would also recommend against json, because it does
not allow for comments, yaml is much better in that regard.

 I know you've mentioned Cabal a lot when talking about bento, and it's
 obvious it's been a very big influence on your design.

Mostly the syntax, actually.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Python-Dev] distlib updated with resources API

[David Cournapeau]

On Mon, Oct 1, 2012 at 8:14 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
 David Cournapeau cournape at gmail.com writes:

 I am not suggesting something very complicated (we don't want to
 re-create a language), but more something like cabal (see conditionals
 in
 http://www.haskell.org/cabal/users-guide/developing-packages.html#package-descriptions),
 or even RPM (http://www.rpm.org/max-rpm/s1-rpm-inside-conditionals.html).
 This has the advantage of using something that has already been
 battle-tested for several years.

 Both the Cabal and RPM conditionals seem fairly narrow in scope (e.g. os/
 arch/impl/flag for Cabal), and therefore it seems that environment markers
 should be able to do the job. Although of course environment markers aren't
 battle tested, it seems worthwhile deploying them on the battlefield to see 
 how
 they perform.

Note that in Cabal at least, those conditionals work not just for
requirements, but for pretty much any section that is not metadata (so
in the python world, you could condition on the package you want to
install, etc...).


 But again, syntax is a really minor point, and what matters the most
 are the features. I was not able to express what I wanted with yaml
 for bento, but would be more than happy to scrap the custom format if
 someone manages to.

 If I can continue working on the YAML format that I've mentioned, I hope I can
 report some progress in this area in due course.

That's great. May I ask you to put this code somewhere ? This would
allow me to try using this inside bento. It would allow using whatever
format you are ending up with directly with existing projects (bento
is is already used by some complex packages such as numpy or scipy).
It would be a good way to ensure the format provides enough semantics.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Python-Dev] distlib updated with resources API

[David Cournapeau]

On Mon, Oct 1, 2012 at 3:15 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
 David Cournapeau cournape at gmail.com writes:

 Note that in Cabal at least, those conditionals work not just for
 requirements, but for pretty much any section that is not metadata (so
 in the python world, you could condition on the package you want to
 install, etc...).

 Right, but the concept of environment markers is simple enough that we should
 be able to extend to other areas. Requirements are just the most obvious
 application.

I think we just need to see how it looks like in real cases to get a
good idea about tradeoffs syntax-wise.


 That's great. May I ask you to put this code somewhere ? This would
 allow me to try using this inside bento. It would allow using whatever
 format you are ending up with directly with existing projects (bento
 is is already used by some complex packages such as numpy or scipy).
 It would be a good way to ensure the format provides enough semantics.

 Which code do you mean? Although I have written some code to produce metadata 
 in
 YAML format, I have not yet got anything that consumes it to a reasonable 
 point
 (apart from sdist generation, which should not be environment-specific).

The code that produces yaml files. The point is precisely that it
would be easy for me to consume this and produce the internal package
representation in bento, which would then allow to configure, build
and install packages using the bento format.

 I will publish what I have as soon as it has reached a reasonable state of
 usefulness. Precisely because it isn't declarative, existing 
 environment-specific
 code in setup.py files in PyPI archives is not easy to convert to environment
 markers automatically :-(

I am curious: do you attempt to parse the setup.py to get those
environment markers ? I personally gave up on this and just run the
setup.py to get whatever values are available in that precise
environment. Given that versions are not normalized, I am afraid
trying to do better is bound to fail, but maybe I am not smart enough
to do it :)

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Python-Dev] distlib updated with resources API

[David Cournapeau]

On Mon, Oct 1, 2012 at 5:06 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
 David Cournapeau cournape at gmail.com writes:

 The code that produces yaml files. The point is precisely that it
 would be easy for me to consume this and produce the internal package
 representation in bento, which would then allow to configure, build
 and install packages using the bento format.

 Well, it's not pretty, but here's a Gist of the method which produces YAML
 metadata from what's passed to setup():

 https://gist.github.com/3812561

Thanks. I don't think it is fair to expect pretty code there in any case.

I noticed that you put the classifiers list as a string (same for
platform). I think it is expected to be a list, no ?

Maybe slightly more controversial, I think the manifest should be
evaluated. The current system of inclusion + exclusion is too
baroque to my taste, and makes it near-impossible to make reproducible
builds.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Python-Dev] distlib updated with resources API

[David Cournapeau]

On Sun, Sep 30, 2012 at 6:03 AM, Donald Stufft donald.stu...@gmail.com wrote:
 I have some unpublished work I should publish.

 Part of the point with what i'm trying to do is to define a standard for
 what is inside
 a package, but not really for how you take a particular set of files and
 turn it into
 that. So if you want to edit yaml you can have a yaml file and have a
 package
 creation tool that turns that yaml into the standard json that gets added to
 the
 package. The same can be said for a python file or a setup.cfg or whatever.
 Ideally
 the roles of package creation, building, and installation should be able to
 be completely
 separate. So my goal is to facilitate that by creating a standard way to
 describe
 all the data about a distribution, including extensible data in a way that
 any tool
 can serialize or deserialize it losslessly.

Note that all this work has already been done in Bento.

I understand the appeal of using an existing format like yaml, but it
is not clear to me how one can handle conditional with it, and I think
you want to handle conditionals in there (for platform-dependent
dependencies).

Bento also has a command to convert setup.py-based projects to bento
internal format, adapting it to use another format should not be too
difficult.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Python-Dev] distlib updated with resources API

[David Cournapeau]

On Sun, Sep 30, 2012 at 4:55 AM, Daniel Holth dho...@gmail.com wrote:
 I like this kind of study. Fixing 1300 packages sounds a lot more
 manageable than fixing 18,000. (I took a similar look at setup.py but
 with the ast module instead of actually running the things. Your
 method is probably more accurate.) It would be very cool to know how
 many packages use if: statements to affect install_requires...

Note that being able to convert a package does not mean the conversion
is working. You need to make sure that installing something from this
new format gives the same thing as installing from the setup.py.
That's harder to test, obviously.


 Is there a good download the latest versions of everything hosted on
 pypi script? Mine was pretty terrible as it could not resume after a
 crash or after the data got stale.

I would be interested in that as well, I wanted to do the same kind of
analysis for Bento's convert command.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Python-Dev] distlib updated with resources API

[David Cournapeau]

On Sun, Sep 30, 2012 at 5:07 PM, Donald Stufft donald.stu...@gmail.com wrote:
 On Sunday, September 30, 2012 at 8:59 AM, David Cournapeau wrote:

 Note that all this work has already been done in Bento.

 I understand the appeal of using an existing format like yaml, but it
 is not clear to me how one can handle conditional with it, and I think
 you want to handle conditionals in there (for platform-dependent
 dependencies).

 Bento also has a command to convert setup.py-based projects to bento
 internal format, adapting it to use another format should not be too
 difficult.

 David

 Instead of conditionals the existing ideas use an environment marker,
 so instead of (pseudo format, I just woke up):

  if python_version  2.6:
 require: simplejson

 You would do:
 require: simplejson; python_version  2.6

Right, in that case, it would work, but in my experience, it is
important to be able to apply conditionals to more than just
requirements (packages definition and so on).

I am not suggesting something very complicated (we don't want to
re-create a language), but more something like cabal (see conditionals
in 
http://www.haskell.org/cabal/users-guide/developing-packages.html#package-descriptions),
or even RPM (http://www.rpm.org/max-rpm/s1-rpm-inside-conditionals.html).
This has the advantage of using something that has already been
battle-tested for several years.

But again, syntax is a really minor point, and what matters the most
are the features. I was not able to express what I wanted with yaml
for bento, but would be more than happy to scrap the custom format if
someone manages to.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Python-Dev] distlib updated with resources API

[David Cournapeau]

On Sun, Sep 30, 2012 at 11:35 PM, Donald Stufft donald.stu...@gmail.com wrote:
 On Sunday, September 30, 2012 at 6:21 PM, David Cournapeau wrote:

 Right, in that case, it would work, but in my experience, it is
 important to be able to apply conditionals to more than just
 requirements (packages definition and so on).

 A significant problem is caused by the allowance of if statements in
 setup.py for trying to generically pull data out. Obviously the limited
 conditionals of Bento make that easier, but you still need to essentially
 maintain a matrix of every possible combination of conditions in order
 to accurately represent that data when you don't know what the target
 system is going to look at. If other fields benefit from the environment
 markers then by all means lets add them to other fields.

 A static piece of metadata that is unchanging is a powerful tool on the
 index side. It allows some of the current stuff i've been experimenting
 with to work very cleanly, such as being able to fetch an entire dependency
 tree in one request.

I am not sure I understand the argument: whatever the syntax, if the
feature is there, you will have the same problem ? The fact that it is
used by existing solutions tend to convince me it is not a problem
(cabal works much better than distutils/etc... in pretty much every
way, including installing things with dependency from their package
servers, not to speak about RPM).

This could be abused of course, but it should not be too difficult to
give suggestions in those cases (one big advantage of
whatever-format-we-want compared to setup.py).

Conditional markers do make the dependency harder to get right, as
conditional in dependencies + provides is what makes dependency graph
building complicated. It seems this can be solved neatly though (see
http://www.slideshare.net/naderman/dependency-resolution-with-sat-symfony-live-2011-paris).

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] GCC versions and binary egg names

[David Cournapeau]

On Sun, Aug 1, 2010 at 1:34 AM, Chris Withers ch...@simplistix.co.uk wrote:
 David Cournapeau wrote:

 On Tue, Jul 27, 2010 at 5:36 PM, Chris Withers ch...@simplistix.co.uk
 wrote:

 Indeed, but the other option requires a more complicated service to
 query.
 Being able to serve packages from a simple folder or from simple folder
 served via svn or Apache is a huge win.

 I don't see how one preclude the other.

 How would you serve your proposed rich environment from Apache?

Debian repositories are mostly purely file-based + a few metadata for
example. I mean, it is not like this issue have not been tackled for
*decades*


 The combination alone makes it complicated very fast. Trying
 to come up with such a scheme in python is foolish IMHO: the problem
 is complicated, and nobody has been able to solve it in a general
 manner anyway.

 I think that's overly pessemistic. What problems do you see with the
 multiple-find-links suggestion I made above?

 It quickly becomes intractable once you have complex requirements.

 Again, I think you're being overly pessimistic. Please can you give an
 actual example?

Well, you do what you want from my suggestion, but I happen to know
extremely well the packages and the issues you are tackling :)

The issue is painstakingly obvious: you have to deal with N1 python
version, N2 C++ ABIs, N3 Fortran ABIs, so you end up having an
exponential explosion of situations.


 I don't see how egg changes anything there compared to rpm/deb to that
 issue. At least, rpm/deb and the infrastructure around have been
 designed to somewhat deal with those issues, whereas egg clearly was
 not.

 rpm/deb focus only on their own operating system. They don't help with
 Windows or MacOS. Eggs work the same on all platforms, from the clients
 perspective, it's just a case of making the right eggs available that
 appears to be problematic.

But egg or rpm/deb is the least of your issue here... The problem is
to make a set of binaries which work everywhere. And that's what is
intractable on Linux in general. The advantage of rpm/deb is that it
gives you a way to correctly manage the thing.


 EPD can use eggs because they pretty much control the whole stack from
 python, hence significantly reducing the issues of dependencies
 issues.

 I don't follow. EPD uses a whole host of dependencies that are non-python...

yes, but most of them are included in EPD. EPD installs its own
python, its own swig, its own blas/lapack libraries, etc... which
means the exponential explosion of situations mentioned before do not
happen.


 Indeed, but as I said before, this is a constrained environment.
 Ubuntu 8.04 or 10.04, each with a controllable list of installed packages.

Then use .deb if you can, it will work so much better. You will
somewhat have a guarantee that everything is built correctly and as
expected.

 With Windows and MacOSX, python packagers seem to go the route of statically
 linking anyway...

The mac situation is pretty awful. There are countless different
python interpreters, each incompatible with each other. For
Numpy/Scipy, we only target the official python.org binary, but even
though, we still have tons of issues.

The static linking is almost always a bad idea, and mostly because of
distutils warts. Certainly, we want to go away from it as soon as
possible (one of the goal of the packaging solution I am working on is
to enable proper linking, but that requires a decent build tool
instead of distutils)

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig