On 20/12/2018 09:44, Alan O'Cais wrote:
Hi all,
We've been essentially doing what Mikael has suggested for a few years
at JSC. I see a lot of merit in how they deal with Python 2/3 at
ComputeCanada, having to currently choose one or the other at install
time creates unnecessary duplication of a lot of software packages.
I've also felt for some time that we could be splitting groups of python
packages off from the core Python installation (for example, at JSC we
already bundle all the SciPy packages), but for this to be effective
without causing lots of headaches it would have to be done unilaterally.
One could I suppose take the approach of GCCcore/GCC (the actual
compiler is in GCCcore, GCC is a bundle that includes this as a dep),
with `Pythoncore` (or whatever) being a dependency of the bundle
`Python` which includes all the relevant packages. Such an approach
would keep the general structure we already have.
At the end of the day though, if we implemented any of this in the
shipped easyconfig repo it would be a major design choice and perhaps
disruptive for a lot of our users. The EB user meeting is probably the
right place to discuss exactly how we might move forward on this and
whether we act in the short or medium term.
Very much plus one on this. It's also a good topic for one of the future
EasyBuild conf calls, but then the people who care most about this (e.g.
Mikael, Bart, Damian) should join.
If we want to make a disruptive change on how we deal with Python
software, we have a great opportunity coming up with EasyBuild v4.0
(which I expect/hope to release by summer 2019).
We could look into both Python at GCCcore + ComputeCanada's approach to
dealing with multiple Python versions, and provide opt-in or opt-out
configuration options for different alternatives, if deemed necessary.
I've added a note on this in
https://github.com/easybuilders/easybuild/issues/447.
regards,
Kenneth
Enjoy the holidays all,
Alan
On Sat, 15 Dec 2018 at 02:43, Mikael Öhman <[email protected]
<mailto:[email protected]>> wrote:
@Bart
While I agree with basically everything you have said, I think there
is diminishing returns here. I'm not to concerned with an extra
numpy, but primarily with theentire dependency graph that gets
pulled up to toolchain level due to basic libpython
@Jack
It has been (and still is )in the works....
https://github.com/easybuilders/easybuild-easyconfigs/pull/4962
https://github.com/easybuilders/easybuild-easyconfigs/pull/5072
Python-2.7.15-GCCcore-7.3.0-bare.eb was included in EB 3.7.1.
(meant as a builddependency only)
Yes, I know of these, but I disagree with this approach; I.M.O the
existance of this package would barely make any difference
whatsoever, as most things affected here, like Mesa, needs a runtime
dep.
Its tricky. ...
I.M.H.O, it's really not tricky at all. Split the current Python
module; put the stuff that can go into GCCcore there (including the
interpreter itself), and keep any package that benefits from the
full toolchain there.
No tricks, no shadowing of libraries, no build-deps, no changes to EB.
For 2018b, I did this for our clusters; I literally just took the
existing Python easyconfig, and split the list of packages. Then as
I encountered Mesa, Qt, etc. that now could then be moved down to
GCCcore, i did so as well.
There was only one actual issue, and that was the link flags for
distutils;
https://github.com/easybuilders/easybuild-easyblocks/pull/1455 (the
fix is very simple)
As a concrete example, I simply think we replace what we basically
have today:
- Python-2.7.15-intel-2018b.eb (incl. numpy and friends)
- Python-2.7.15-foss-2018b.eb (incl. numpy and friends)
- Python-2.7.15-fosscuda-2018b.eb (incl. numpy and friends)
- Mesa-...-intel-2018b-Python-2.7.15.eb + many more
- Mesa-...-foss-2018b-Python-2.7.15.eb + many more
- Mesa-...-fosscuda-2018b-Python-2.7.15.eb + many more
with:
- Python-2.7.15-GCCcore-7.3.0.eb (excl numpy and friends)
- Mesa-...-GCCcore-7.3.0-Python-2.7.15.eb + many more
- numpy-intel-2018b-Python-2.7.15.eb
- numpy-foss-2018b-Python-2.7.15.eb
- numpy-fosscuda-2018b-Python-2.7.15.eb
(or any other equivalent solution which lets libpython exist as a
normal runtime dep at GCCcore level)
As a side-effect of this, there wouldn't be any need to create a
special Python-bare just for build deps.
--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany
Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: [email protected] <mailto:[email protected]>
WWW: http://www.fz-juelich.de/ias/jsc/EN
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
