Hi all, We've been essentially doing what Mikael has suggested for a few years at JSC. I see a lot of merit in how they deal with Python 2/3 at ComputeCanada, having to currently choose one or the other at install time creates unnecessary duplication of a lot of software packages.
I've also felt for some time that we could be splitting groups of python packages off from the core Python installation (for example, at JSC we already bundle all the SciPy packages), but for this to be effective without causing lots of headaches it would have to be done unilaterally. One could I suppose take the approach of GCCcore/GCC (the actual compiler is in GCCcore, GCC is a bundle that includes this as a dep), with `Pythoncore` (or whatever) being a dependency of the bundle `Python` which includes all the relevant packages. Such an approach would keep the general structure we already have. At the end of the day though, if we implemented any of this in the shipped easyconfig repo it would be a major design choice and perhaps disruptive for a lot of our users. The EB user meeting is probably the right place to discuss exactly how we might move forward on this and whether we act in the short or medium term. Enjoy the holidays all, Alan On Sat, 15 Dec 2018 at 02:43, Mikael Öhman <[email protected]<mailto:[email protected]>> wrote: @Bart While I agree with basically everything you have said, I think there is diminishing returns here. I'm not to concerned with an extra numpy, but primarily with theentire dependency graph that gets pulled up to toolchain level due to basic libpython @Jack It has been (and still is )in the works.... https://github.com/easybuilders/easybuild-easyconfigs/pull/4962 https://github.com/easybuilders/easybuild-easyconfigs/pull/5072 Python-2.7.15-GCCcore-7.3.0-bare.eb was included in EB 3.7.1. (meant as a builddependency only) Yes, I know of these, but I disagree with this approach; I.M.O the existance of this package would barely make any difference whatsoever, as most things affected here, like Mesa, needs a runtime dep. Its tricky. ... I.M.H.O, it's really not tricky at all. Split the current Python module; put the stuff that can go into GCCcore there (including the interpreter itself), and keep any package that benefits from the full toolchain there. No tricks, no shadowing of libraries, no build-deps, no changes to EB. For 2018b, I did this for our clusters; I literally just took the existing Python easyconfig, and split the list of packages. Then as I encountered Mesa, Qt, etc. that now could then be moved down to GCCcore, i did so as well. There was only one actual issue, and that was the link flags for distutils; https://github.com/easybuilders/easybuild-easyblocks/pull/1455 (the fix is very simple) As a concrete example, I simply think we replace what we basically have today: - Python-2.7.15-intel-2018b.eb (incl. numpy and friends) - Python-2.7.15-foss-2018b.eb (incl. numpy and friends) - Python-2.7.15-fosscuda-2018b.eb (incl. numpy and friends) - Mesa-...-intel-2018b-Python-2.7.15.eb + many more - Mesa-...-foss-2018b-Python-2.7.15.eb + many more - Mesa-...-fosscuda-2018b-Python-2.7.15.eb + many more with: - Python-2.7.15-GCCcore-7.3.0.eb (excl numpy and friends) - Mesa-...-GCCcore-7.3.0-Python-2.7.15.eb + many more - numpy-intel-2018b-Python-2.7.15.eb - numpy-foss-2018b-Python-2.7.15.eb - numpy-fosscuda-2018b-Python-2.7.15.eb (or any other equivalent solution which lets libpython exist as a normal runtime dep at GCCcore level) As a side-effect of this, there wouldn't be any need to create a special Python-bare just for build deps. -- Dr. Alan O'Cais E-CAM Software Manager Juelich Supercomputing Centre Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49 2461 61 5213 Fax: +49 2461 61 6656 E-mail: [email protected]<mailto:[email protected]> WWW: http://www.fz-juelich.de/ias/jsc/EN ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------
