Hi easybuilders, We have for the 2018b toolchain been running with a shared libpython at GCCcore level, which also let us more a lot of additional packages (e.g. Qt) down to GCCcore level as well, significantly reducing the pointless duplicated packages (primarily from the graphical stack). Mixing python interpreter with packages from gcc + icc works just fine. I'm hoping for something like this to get into upstream EB.
There was a few fixable issues getting EB to build packages with intel, but the main issue is just the name. We used the name "Python-core" @ GCCcore, and had "Python" as a python package bundle on top of this (per toolchain), so it was completely backwards-compatible for users. But, the EB experience is a bit painful; there is quite a few places where the name matters; For example, we don't get %(pyver)s as a result, and there a couple of things of minor annoyances like this. 1. These issues could all be addressed, but the fix isn't always necessarily very beautiful; https://github.com/easybuilders/easybuild-framework/pull/2559 2. Or just exclude numpy, scipy, pandas, ... from the "Python" package allowing it to be moved the GCCcore. Use a separate package for numpy, etc. at the toolchain level. It still doesn't cause an explosion of packages to keep these MPI/BLAS dependent packages separate, as it's just ~5 of them. I would prefer the latter; it requires no hacks in EB, and lets us build these heavy packages separately, which is easier to debug than the bundle. I've also found myself explaining that the Python module includes numpy as often as I have to explain to users that they need to load an additional module, so I don't really think there is much difference on the user perspective. Thoughts? I could live with any solution, but I think libpython simply must get to down to GCCcore one way or another. Best regards, Mikael
