Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker]

On Mon, Jan 11, 2016 at 11:02 AM, David Cournapeau 
wrote:

> If we get all that worked out, we still haven't made any progress toward
>> the non-standard libs that aren't python. This is the big "scipy problem"
>> -- fortran, BLAS, hdf, ad infinitum.
>>
>> I argued for years that we could build binary wheels that hold each of
>> these, and other python packages could depend on them, but pypa never
>> seemed to like that idea.
>>
>
> I don't think that's an accurate statement. There are issues to solve
> around this, but I did not encounter push back, either on the ML or face to
> face w/ various pypa members at Pycon, etc... There may be push backs for a
> particular detail, but making "pip install scipy" or "pip install
> matplotlib" a reality on every platform is something everybody agrees o
>

sure, everyone wants that. But when it gets deeper, they don't want to have
a bunc hof pip-installable binary wheels that are simply clibs re-packaged
as a dependency. And, then you have the problelm of those being "binary
wheel" dependencies, rather than "package" dependencies.

e.g.:

this particular build of pillow depends on the libpng and libjpeg wheels,
but the Pillow package, in general, does not. And you would have different
dependencies on Windows, and OS-X, and Linux.

pip/wheel simply was not designed for that, and I didn't get any warm and
fuzzy feelings from dist-utils sig that the it ever would. And again, then
you are re-designing conda.

So the only way to do all that is to statically link all the dependent libs
in with each binary wheel (or ship a dll). Somehow it always bothered me to
ship the same lib with multiple packages -- isn't that why shared libs
exist?

-  In practice, maybe it doesn't matter, memory is cheap. But I also got
sick of building the damn things! I like that Anaconda comes with libs
someone else has built, and I can use them with my packages, too. And when
it comes to ugly stuff like HDF and GDAL, I'm really happy someone else has
built them!

Anyway -- carry on -- being able to pip install the scipy stack would be
very nice.

-CHB














>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Nathaniel Smith]

On Jan 11, 2016 3:54 PM, "Chris Barker"  wrote:
>
> On Mon, Jan 11, 2016 at 11:02 AM, David Cournapeau 
wrote:
>>>
>>> If we get all that worked out, we still haven't made any progress
toward the non-standard libs that aren't python. This is the big "scipy
problem" -- fortran, BLAS, hdf, ad infinitum.
>>>
>>> I argued for years that we could build binary wheels that hold each of
these, and other python packages could depend on them, but pypa never
seemed to like that idea.
>>
>>
>> I don't think that's an accurate statement. There are issues to solve
around this, but I did not encounter push back, either on the ML or face to
face w/ various pypa members at Pycon, etc... There may be push backs for a
particular detail, but making "pip install scipy" or "pip install
matplotlib" a reality on every platform is something everybody agrees o
>
>
> sure, everyone wants that. But when it gets deeper, they don't want to
have a bunc hof pip-installable binary wheels that are simply clibs
re-packaged as a dependency. And, then you have the problelm of those being
"binary wheel" dependencies, rather than "package" dependencies.
>
> e.g.:
>
> this particular build of pillow depends on the libpng and libjpeg wheels,
but the Pillow package, in general, does not. And you would have different
dependencies on Windows, and OS-X, and Linux.
>
> pip/wheel simply was not designed for that, and I didn't get any warm and
fuzzy feelings from dist-utils sig that the it ever would. And again, then
you are re-designing conda.

I agree that talking about such things on distutils-sig tends to elicit a
certain amount of puzzled incomprehension, but I don't think it matters --
wheels already have everything you need to support this. E.g. wheels for
different platforms can trivially have different dependencies. (They even
go to some lengths to make sure this is possible for pure python packages
where the same wheel can be used on multiple platforms.) When distributing
a library-in-a-wheel then you need a little bit of hackishness to make sure
the runtime loader can find the library, which conda would otherwise handle
for you, but AFAICT it's like 10 lines of code or something.

And in any case we have lots of users who don't use conda and are thus
doomed to support both ecosystems regardless, so we might as well make the
best of it :-).

-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Defining a base linux-64 environment [was: Should I use pip install numpy in linux?]

[Nathaniel Smith]

On Mon, Jan 11, 2016 at 6:06 AM, Robert McGibbon  wrote:
> I started working on a tool for checking linux wheels for "manylinux"
> compatibility, and fixing them up if possible, based on the same ideas as
> Matthew Brett's delocate for OS X. Current WIP code, if anyone wants to help
> / throw penuts, is here: https://github.com/rmcgibbo/deloc8.
>
> It's currently fairly modest and can only list non-whitelisted external
> shared library dependencies, and verify that sufficiently old versioned
> symbols for glibc and its ilk are used.

That is super cool! and also this week David C. @ Enthought
contributed the docker image that they use to actually make compatible
builds, so I guess we have some momentum; let's make this happen :-).
I just made a repo and a mailing list to continue the discussion...

https://github.com/manylinux/manylinux
https://groups.google.com/forum/#!forum/manylinux-discuss

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Robert McGibbon]

> And in any case we have lots of users who don't use conda and are thus
doomed to support both ecosystems regardless, so we might as well make the
best of it :-).

Yes, this is the key. Conda is a great tool for a lot of users / use cases,
but it's not for everyone.

Anyways, I think I've made a pretty good start on the tooling for a wheel
ABI tag for a LSB-style base system that represents a common set of shared
libraries and symbols versions provided by "many" linuxes (previously
discussed by Nathaniel here:
https://code.activestate.com/lists/python-distutils-sig/26272/)

-Robert

On Mon, Jan 11, 2016 at 5:29 PM, Nathaniel Smith  wrote:

> On Jan 11, 2016 3:54 PM, "Chris Barker"  wrote:
> >
> > On Mon, Jan 11, 2016 at 11:02 AM, David Cournapeau 
> wrote:
> >>>
> >>> If we get all that worked out, we still haven't made any progress
> toward the non-standard libs that aren't python. This is the big "scipy
> problem" -- fortran, BLAS, hdf, ad infinitum.
> >>>
> >>> I argued for years that we could build binary wheels that hold each of
> these, and other python packages could depend on them, but pypa never
> seemed to like that idea.
> >>
> >>
> >> I don't think that's an accurate statement. There are issues to solve
> around this, but I did not encounter push back, either on the ML or face to
> face w/ various pypa members at Pycon, etc... There may be push backs for a
> particular detail, but making "pip install scipy" or "pip install
> matplotlib" a reality on every platform is something everybody agrees o
> >
> >
> > sure, everyone wants that. But when it gets deeper, they don't want to
> have a bunc hof pip-installable binary wheels that are simply clibs
> re-packaged as a dependency. And, then you have the problelm of those being
> "binary wheel" dependencies, rather than "package" dependencies.
> >
> > e.g.:
> >
> > this particular build of pillow depends on the libpng and libjpeg
> wheels, but the Pillow package, in general, does not. And you would have
> different dependencies on Windows, and OS-X, and Linux.
> >
> > pip/wheel simply was not designed for that, and I didn't get any warm
> and fuzzy feelings from dist-utils sig that the it ever would. And again,
> then you are re-designing conda.
>
> I agree that talking about such things on distutils-sig tends to elicit a
> certain amount of puzzled incomprehension, but I don't think it matters --
> wheels already have everything you need to support this. E.g. wheels for
> different platforms can trivially have different dependencies. (They even
> go to some lengths to make sure this is possible for pure python packages
> where the same wheel can be used on multiple platforms.) When distributing
> a library-in-a-wheel then you need a little bit of hackishness to make sure
> the runtime loader can find the library, which conda would otherwise handle
> for you, but AFAICT it's like 10 lines of code or something.
>
> And in any case we have lots of users who don't use conda and are thus
> doomed to support both ecosystems regardless, so we might as well make the
> best of it :-).
>
> -n
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Defining a base linux-64 environment [was: Should I use pip install numpy in linux?]

[Robert McGibbon]

I started working on a tool for checking linux wheels for "manylinux"
compatibility, and fixing them up if possible, based on the same ideas as
Matthew Brett's delocate  for OS
X. Current WIP code, if anyone wants to help / throw penuts, is here:
https://github.com/rmcgibbo/deloc8.

It's currently fairly modest and can only list non-whitelisted external
shared library dependencies, and verify that sufficiently old versioned
symbols for glibc and its ilk are used.

-Robert

On Sun, Jan 10, 2016 at 1:19 AM, Robert McGibbon  wrote:

> Hi all,
>
> I followed Nathaniel's advice and restricted the search down to the
> packages included in the Anaconda release (as opposed to all of the
> packages in their repositories), and fixed some technical issues with the
> way I was doing the analysis.
>
> The new list is much smaller. Here are the shared libraries that the
> components of Anaconda require that the system provides on Linux 64:
>
> libpanelw.so.5, libncursesw.so.5, libgcc_s.so.1, libstdc++.so.6,
> libm.so.6, libdl.so.2, librt.so.1, libcrypt.so.1, libc.so.6, libnsl.so.1,
> libutil.so.1, libpthread.so.0, libX11.so.6, libXext.so.6,
> libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0,
> libXrender.so.1, libICE.so.6, libSM.so.6, libGL.so.1.
>
> Many of these libraries are required simply for the interpreter. The
> remaining ones that aren't required by the interpreter are, but are
> required by some other package in Anaconda are:
>
> libgcc_s.so.1, libstdc++.so.6, libXext.so.6, libSM.so.6,
> libgthread-2.0.so.0, libgobject-2.0.so.0, libglib-2.0.so.0, libICE.so.6,
> libXrender.so.1, and libGL.so.1.
>
> Most of these are parts of X11 required by Qt (
> http://doc.qt.io/qt-5/linux-requirements.html).
>
> -Robert
>
>
>
> On Sat, Jan 9, 2016 at 4:42 PM, Robert McGibbon 
> wrote:
>
>> > Maybe a better approach would be to look at what libraries are used on
>> by an up-to-date default Anaconda install (on the assumption that this
>> is the best tested configuration)
>>
>> That's not a bad idea. I also have a couple other ideas about how to
>> filter
>> this based on using debian popularity-contests and the package graph. I
>> will report back when I have more info.
>>
>> -Robert
>>
>> On Sat, Jan 9, 2016 at 3:04 PM, Nathaniel Smith  wrote:
>>
>>> On Sat, Jan 9, 2016 at 3:52 AM, Robert McGibbon 
>>> wrote:
>>> > Hi all,
>>> >
>>> > I went ahead and tried to collect a list of all of the libraries that
>>> could
>>> > be considered to constitute the "base" system for linux-64. The
>>> strategy I
>>> > used was to leverage off the work done by the folks at Continuum by
>>> > searching through their pre-compiled binaries from
>>> > https://repo.continuum.io/pkgs/free/linux-64/ to find shared
>>> libraries that
>>> > were dependened on (according to ldd)  that were not accounted for by
>>> the
>>> > declared dependencies that each package made known to the conda package
>>> > manager.
>>> >
>>> > The full list of these system libraries, sorted in from
>>> > most-commonly-depend-on to rarest, is below. There are 158 of them.
>>> [...]
>>> > So it's not perfect. But it might be a useful starting place.
>>>
>>> Unfortunately, yeah, it looks like there's a lot of false positives in
>>> here :-(. For example your list contains liblzma and libsqlite, but
>>> both of these are shipped as dependencies of python itself. So
>>> probably someone just forgot to declare the dependency explicitly, but
>>> got away with it because the libraries were pulled in anyway.
>>>
>>> Maybe a better approach would be to look at what libraries are used on
>>> by an up-to-date default Anaconda install (on the assumption that this
>>> is the best tested configuration), and then erase from the list all
>>> libraries that are shipped by this configuration (ignoring declared
>>> dependencies since those seem to be unreliable)? It's better to be
>>> conservative here, since the end goal is to come up with a list of
>>> external libraries that we're confident have actually been tested for
>>> compatibility by lots and lots of different users.
>>>
>>> -n
>>>
>>> --
>>> Nathaniel J. Smith -- http://vorpus.org
>>> ___
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion@scipy.org
>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Travis Oliphant]

Anaconda "build environment" was setup by Ilan and me.Aaron helped to
build packages while he was at Continuum but spent most of his time on the
open-source conda project.

It is important to understand the difference between Anaconda and conda in
this respect.   Anaconda is a particular dependency foundation that
Continuum supports and releases -- it will have a particular set of
expected libraries on each platform (we work to keep this fairly limited
and on Linux currently use CentOS 5 as the base).

conda is a general package manager that is open-source and that anyone can
use to produce a set of consistent binaries (there can be many conda-based
distributions).   It solves the problem of multiple binary dependency
chains generally using the concept of "features".  This concept of
"features" allows you to create environments with different base
dependencies.

What packages you install when you  "conda install" depends on which
channels you are pointing to and which features you have "turned on" in the
environment.   It's a general system that extends the notions that were
started by the PyPA.

-Travis


On Sun, Jan 10, 2016 at 10:14 PM, Robert McGibbon 
wrote:

> > > Right. There's a small problem which is that the base linux system
> >> isn't just "CentOS 5", it's "CentOS 5 and here's the list of libraries
> > > that you're allowed to link to: ...", where that list is empirically
> > > chosen to include only stuff that really is installed on ~all linux
> >> machines and for which the ABI really has been stable in practice over
> > > multiple years and distros (so e.g. no OpenSSL).
> > >
> > > Does anyone know who maintains Anaconda's linux build environment?
>
> > I strongly suspect it was originally set up by Aaron Meurer. Who
> maintains it now that he is no longer at Continuum is a good question.
>
> From looking at all of the external libraries referenced by binaries
> included in Anaconda
> and the conda repos, I am not confident that they have a totally strict
> policy here, or at least
> not ones that is enforced by tooling. The sonames I listed here
>  
> cover
> all of the external
> dependencies used by the latest Anaconda release, but earlier releases and
> other
> conda-installable packages from the default channel are not so strict.
>
> -Robert
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

*Travis Oliphant*
*Co-founder and CEO*


@teoliphant
512-222-5440
http://www.continuum.io
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker]

On Fri, Jan 8, 2016 at 7:13 PM, Nathaniel Smith  wrote:

> > that this would potentially be able to let packages like numpy serve
> their
> > linux
> > users better without risking too much junk being uploaded to PyPI.
>
> That will never fly. But like Matthew says, I think we can probably
> get them to accept a PEP saying "here's a new well-specified platform
> tag that means that this wheel works on all linux systems meet the
> following list of criteria: ...", and then allow that new platform tag
> onto PyPI.
>

The second step is a trick though -- how does pip know, when being run on a
client, that the system meets those requirements? Do we put a bunch of code
in that checks for those libs, etc???

If we get all that worked out, we still haven't made any progress toward
the non-standard libs that aren't python. This is the big "scipy problem"
-- fortran, BLAS, hdf, ad infinitum.

I argued for years that we could build binary wheels that hold each of
these, and other python packages could depend on them, but pypa never
seemed to like that idea. In the end, if you did all this right, you'd have
something like conda -- so why not just use conda?

All that being said, if you folks can get the core scipy stack setup to pip
install on OS_X, Windows, and Linux, that would be pretty nice -- so keep
at it !

-CHB









>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Benjamin Root]

The other half of the fun is how to deal with weird binary issues with
libraries like libgeos_c, libhdf5 and such.  You have to get all of the
compile options right for your build of those libraries to get your build
of GDAL and pyhdf working right. You also have packages like gdal and
netcdf4 have diamond dependencies -- not only are they built and linked
against numpy binaries, some of their binary dependencies are built against
numpy binaries as well. Joys!

I don't envy anybody that tries to take on the packaging problem in any
language.

Ben Root

On Mon, Jan 11, 2016 at 1:25 PM, Chris Barker  wrote:

> On Fri, Jan 8, 2016 at 7:13 PM, Nathaniel Smith  wrote:
>
>> > that this would potentially be able to let packages like numpy serve
>> their
>> > linux
>> > users better without risking too much junk being uploaded to PyPI.
>>
>> That will never fly. But like Matthew says, I think we can probably
>> get them to accept a PEP saying "here's a new well-specified platform
>> tag that means that this wheel works on all linux systems meet the
>> following list of criteria: ...", and then allow that new platform tag
>> onto PyPI.
>>
>
> The second step is a trick though -- how does pip know, when being run on
> a client, that the system meets those requirements? Do we put a bunch of
> code in that checks for those libs, etc???
>
> If we get all that worked out, we still haven't made any progress toward
> the non-standard libs that aren't python. This is the big "scipy problem"
> -- fortran, BLAS, hdf, ad infinitum.
>
> I argued for years that we could build binary wheels that hold each of
> these, and other python packages could depend on them, but pypa never
> seemed to like that idea. In the end, if you did all this right, you'd have
> something like conda -- so why not just use conda?
>
> All that being said, if you folks can get the core scipy stack setup to
> pip install on OS_X, Windows, and Linux, that would be pretty nice -- so
> keep at it !
>
> -CHB
>
>
>
>
>
>
>
>
>
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- http://vorpus.org
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion