Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Tony Kelman]

> Right. There's a small problem which is that the base linux system
> isn't just "CentOS 5", it's "CentOS 5 and here's the list of libraries
> that you're allowed to link to: ...", where that list is empirically
> chosen to include only stuff that really is installed on ~all linux
> machines and for which the ABI really has been stable in practice over
> multiple years and distros (so e.g. no OpenSSL).
> 
> So the key next step is for someone to figure out and write down that
> list. Continuum and Enthought both have versions of it that we know
> are good...
> 
> Does anyone know who maintains Anaconda's linux build environment?

I strongly suspect it was originally set up by Aaron Meurer. Who
maintains it now that he is no longer at Continuum is a good question.

We build "generic Linux binaries" for Julia and I co-maintain that
environment. It's using CentOS 5 (for now, until we hit an issue that
is easiest to fix by upgrading the builders to CentOS 6 - I don't think
we have any real users on CentOS 5 any more so no one would notice the
bump), but we avoid using the Red Hat devtoolset. We tried it initially,
but had issues due to the way they attempt to statically link libstdc++
and libgfortran. The -static-libgfortran GCC flag has been broken since
GCC 4.5 because libgfortran now depends on libquadmath, and it requires
rather messy modification of every link line to change -lgfortran to an
explicit absolute path to libgfortran.a (and possibly also libquadmath.a)
to really get libraries like BLAS and LAPACK statically linked. And if
you want to statically link the GCC runtime libraries into a .so, your
distribution's copies of libstdc++.a, libgfortran.a, etc must have been
built with -fPIC, which many are not.

Some of the issues where this was discussed and worked out were:
https://github.com/JuliaLang/julia/issues/8397
https://github.com/JuliaLang/julia/issues/8433
https://github.com/JuliaLang/julia/pull/8442
https://github.com/JuliaLang/julia/pull/10043

So for Julia we wound up at building our own copy of latest GCC from
source on CentOS 5 buildbots, and shipping private shared-library copies
of libgfortran, libstdc++, libgcc_s, and a few others. This works on
pretty much any glibc-using Linux distribution as new or newer than the
buildbot. It sadly doesn't work on musl distributions, ah well. Rust has
been experimenting with truly static libraries/executables that statically
link musl libc in addition to everything else, I'm not sure how practical
that would be for numpy, scipy, etc.

If you go this route of building gcc from source and depending on private
copies of the shared runtime libraries, it ends up important that you
pick a newer version of gcc than any of your users. The reason is for
packages that require compilation on the user's machine. If you build and
distribute a set of shared libraries using GCC 4.8 but then someone on
Ubuntu 15.10 tries to build something else (say as an example, pyipopt
which needs a library that uses both C++ and Fortran) from source using
GCC 5.2, the GCC 4.8 runtime libraries that you built against will get
loaded first, and they won't contain newer-ABI symbols needed by the GCC
5.2-built user library. Sometimes this can be fixed by just deleting the
older bundled copies of the runtime libraries and relying on the users'
system copies, but for example libgfortran typically needs to be manually
installed first.



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Robert McGibbon]

> > Right. There's a small problem which is that the base linux system
>> isn't just "CentOS 5", it's "CentOS 5 and here's the list of libraries
> > that you're allowed to link to: ...", where that list is empirically
> > chosen to include only stuff that really is installed on ~all linux
>> machines and for which the ABI really has been stable in practice over
> > multiple years and distros (so e.g. no OpenSSL).
> >
> > Does anyone know who maintains Anaconda's linux build environment?

> I strongly suspect it was originally set up by Aaron Meurer. Who
maintains it now that he is no longer at Continuum is a good question.

>From looking at all of the external libraries referenced by binaries
included in Anaconda
and the conda repos, I am not confident that they have a totally strict
policy here, or at least
not ones that is enforced by tooling. The sonames I listed here

cover
all of the external
dependencies used by the latest Anaconda release, but earlier releases and
other
conda-installable packages from the default channel are not so strict.

-Robert
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Defining a base linux-64 environment [was: Should I use pip install numpy in linux?]

[Robert McGibbon]

Hi all,

I followed Nathaniel's advice and restricted the search down to the
packages included in the Anaconda release (as opposed to all of the
packages in their repositories), and fixed some technical issues with the
way I was doing the analysis.

The new list is much smaller. Here are the shared libraries that the
components of Anaconda require that the system provides on Linux 64:

libpanelw.so.5, libncursesw.so.5, libgcc_s.so.1, libstdc++.so.6, libm.so.6,
libdl.so.2, librt.so.1, libcrypt.so.1, libc.so.6, libnsl.so.1,
libutil.so.1, libpthread.so.0, libX11.so.6, libXext.so.6,
libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0,
libXrender.so.1, libICE.so.6, libSM.so.6, libGL.so.1.

Many of these libraries are required simply for the interpreter. The
remaining ones that aren't required by the interpreter are, but are
required by some other package in Anaconda are:

libgcc_s.so.1, libstdc++.so.6, libXext.so.6, libSM.so.6,
libgthread-2.0.so.0, libgobject-2.0.so.0, libglib-2.0.so.0, libICE.so.6,
libXrender.so.1, and libGL.so.1.

Most of these are parts of X11 required by Qt (
http://doc.qt.io/qt-5/linux-requirements.html).

-Robert



On Sat, Jan 9, 2016 at 4:42 PM, Robert McGibbon  wrote:

> > Maybe a better approach would be to look at what libraries are used on
> by an up-to-date default Anaconda install (on the assumption that this
> is the best tested configuration)
>
> That's not a bad idea. I also have a couple other ideas about how to filter
> this based on using debian popularity-contests and the package graph. I
> will report back when I have more info.
>
> -Robert
>
> On Sat, Jan 9, 2016 at 3:04 PM, Nathaniel Smith  wrote:
>
>> On Sat, Jan 9, 2016 at 3:52 AM, Robert McGibbon 
>> wrote:
>> > Hi all,
>> >
>> > I went ahead and tried to collect a list of all of the libraries that
>> could
>> > be considered to constitute the "base" system for linux-64. The
>> strategy I
>> > used was to leverage off the work done by the folks at Continuum by
>> > searching through their pre-compiled binaries from
>> > https://repo.continuum.io/pkgs/free/linux-64/ to find shared libraries
>> that
>> > were dependened on (according to ldd)  that were not accounted for by
>> the
>> > declared dependencies that each package made known to the conda package
>> > manager.
>> >
>> > The full list of these system libraries, sorted in from
>> > most-commonly-depend-on to rarest, is below. There are 158 of them.
>> [...]
>> > So it's not perfect. But it might be a useful starting place.
>>
>> Unfortunately, yeah, it looks like there's a lot of false positives in
>> here :-(. For example your list contains liblzma and libsqlite, but
>> both of these are shipped as dependencies of python itself. So
>> probably someone just forgot to declare the dependency explicitly, but
>> got away with it because the libraries were pulled in anyway.
>>
>> Maybe a better approach would be to look at what libraries are used on
>> by an up-to-date default Anaconda install (on the assumption that this
>> is the best tested configuration), and then erase from the list all
>> libraries that are shipped by this configuration (ignoring declared
>> dependencies since those seem to be unreliable)? It's better to be
>> conservative here, since the end goal is to come up with a list of
>> external libraries that we're confident have actually been tested for
>> compatibility by lots and lots of different users.
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- http://vorpus.org
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Sandro Tosi]

On Sun, Jan 10, 2016 at 3:55 AM, Matthew Brett  wrote:
> I updated the page with more on reasons to prefer Debian packages over
> installing with pip:
>
> https://matthew-brett.github.io/pydagogue/installing_on_debian.html
>
> Is that enough to get the message across?

That looks a lot better, thanks! I also kinda agree with all Nathaniel
said on the matter

-- 
Sandro "morph" Tosi
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi
G+: https://plus.google.com/u/0/+SandroTosi
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Matthew Brett]

On Sun, Jan 10, 2016 at 3:40 AM, Sandro Tosi  wrote:
> On Sun, Jan 10, 2016 at 3:55 AM, Matthew Brett  
> wrote:
>> I updated the page with more on reasons to prefer Debian packages over
>> installing with pip:
>>
>> https://matthew-brett.github.io/pydagogue/installing_on_debian.html
>>
>> Is that enough to get the message across?
>
> That looks a lot better, thanks! I also kinda agree with all Nathaniel
> said on the matter

Good so.  You probably saw, I already removed use of sudo on the page as well.

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion