bug#53258: Python unable to find modules within a Singularity container created with guix pack
Hi Konrad, Konrad Hinsen writes: > Konrad Hinsen writes: > >> Patch at https://issues.guix.gnu.org/68241 > > If you want to test it without rebuilding tons of packages, here is a > version that grafts a patched Python onto the existing ones (as substitutes): > > https://codeberg.org/khinsen/guix/src/branch/graft-fix-python-sitecustomize I've applied your patch to core-updates. I trust that it fixed this issue based on your own testing. Closing; we can always reopen if we find it didn't fix every use cases or create a fresh issue. -- Thanks, Maxim
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Konrad Hinsen writes: > Patch at https://issues.guix.gnu.org/68241 If you want to test it without rebuilding tons of packages, here is a version that grafts a patched Python onto the existing ones (as substitutes): https://codeberg.org/khinsen/guix/src/branch/graft-fix-python-sitecustomize Cheers, Konrad
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Konrad Hinsen writes: > I will submit a patch for this. Patch at https://issues.guix.gnu.org/68241 Cheers, Konrad
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Hi everyone, I found this issue while investigating a related one (the error message). It looks like the core problem discussed here has been solved in the meantime: Singularity> python3 -m numpy Error in sitecustomize; set PYTHONVERBOSE for traceback: ValueError: '/gnu/store/kx98dz2vc3gdpjficy4nbd7rscnkvz4s-profile/bin/../../h855kddqbay0pcbwr8a7i8m6ilz67cfn-python-3.10.7/lib/python3.10/site-packages' is not in list /gnu/store/kx98dz2vc3gdpjficy4nbd7rscnkvz4s-profile/bin/python3: No module named numpy.__main__; 'numpy' is a package and cannot be directly executed This error message is to be expected, and NumPy can be imported: Singularity> python3 Error in sitecustomize; set PYTHONVERBOSE for traceback: ValueError: '/gnu/store/kx98dz2vc3gdpjficy4nbd7rscnkvz4s-profile/bin/../../h855kddqbay0pcbwr8a7i8m6ilz67cfn-python-3.10.7/lib/python3.10/site-packages' is not in list Python 3.10.7 (main, Jan 1 1970, 00:00:01) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> What remains is the error message, which comes from a bug in Guix' sitecustomize.py. It compares a raw path (sys.prefix) with paths that have been normalized (os.path.normpath). That comparison fails if sys.prefix contains "../", as it does when run from a Singularity container. I will submit a patch for this. Cheers, Konrad
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Hi, On jeu., 02 févr. 2023 at 10:40, Maxim Cournoyer wrote: >> So, upstream hasn't replied at all to the above proposal. How do we move >> forward? I have a patch that still applies on Python 3.9.9 that we can >> put into core-updates. Would that be fine? > > I haven't revisited that patch, but if you are confident it wouldn't > break anything in the Python world, core-updates would be a fine place, > yes. Maybe for testing this change, you could create one Python interpreter variant containing the patch. Then using ’package-with-explicit-python’ package transformation from (guix build-system python) we could build Python packages and check if it correctly fixes and also incrementally check if nothing break. It would avoid a full world rebuild. Cheers, simon
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Hi Josselin, Josselin Poiret writes: > Hi Ludo, > > Ludovic Courtès writes: > >> Thanks for the investigation and for reporting it upstream! >> >> Is there a workaround we could apply locally, ideally one that does not >> lead to a world rebuild? > > So, upstream hasn't replied at all to the above proposal. How do we move > forward? I have a patch that still applies on Python 3.9.9 that we can > put into core-updates. Would that be fine? I haven't revisited that patch, but if you are confident it wouldn't break anything in the Python world, core-updates would be a fine place, yes. -- Thanks, Maxim
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Hi Ludo, Ludovic Courtès writes: > Thanks for the investigation and for reporting it upstream! > > Is there a workaround we could apply locally, ideally one that does not > lead to a world rebuild? So, upstream hasn't replied at all to the above proposal. How do we move forward? I have a patch that still applies on Python 3.9.9 that we can put into core-updates. Would that be fine? Best, -- Josselin Poiret
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Hi Josselin, Josselin Poiret skribis: > After the IRC investigation, it turned out the use of abspath in > Lib/site.py was in fact the culprit. Replacing all its uses with > realpath and doing `./pre-inst-env guix pack -RR python python-numpy -S > /bin=bin -S /etc=etc` did in fact resolve the issue! > > I've raised the issue upstream at [1]. The fix itself is pretty simple, > but it's theoretically a breaking change, so we'll see how Python > maintainers want to move forward with it. We could backport this fix > for the next core-updates cycle, whether or not it gets accepted > upstream, and we will also need to modify sitecustomize.py as well to > call os.path.realpath on python_site. > [1] https://bugs.python.org/issue46653 Thanks for the investigation and for reporting it upstream! Is there a workaround we could apply locally, ideally one that does not lead to a world rebuild? Ludo’.
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Silly me forgetting the link, here it is: [1] https://bugs.python.org/issue46653 Apologies for the noise, -- Josselin Poiret
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Hello everyone, After the IRC investigation, it turned out the use of abspath in Lib/site.py was in fact the culprit. Replacing all its uses with realpath and doing `./pre-inst-env guix pack -RR python python-numpy -S /bin=bin -S /etc=etc` did in fact resolve the issue! I've raised the issue upstream at [1]. The fix itself is pretty simple, but it's theoretically a breaking change, so we'll see how Python maintainers want to move forward with it. We could backport this fix for the next core-updates cycle, whether or not it gets accepted upstream, and we will also need to modify sitecustomize.py as well to call os.path.realpath on python_site. Best, -- Josselin Poiret
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Hello! Ludovic Courtès skribis: >>From Marek’s message, it would seem that somewhere file names are > constructed by appending “..”. Given Unix’s weird dot-dot semantics, > relative symlinks could be interfering: Some news! Here’s a recap of the IRC discussion earlier today: ‘sys.prefix’ looks like /tmp/pack2/bin/../../zmv1alp7dzjd6hkc7l6fjz817n2mfzzw-python-3.9.9R. ‘sys.normpath’ does lexical dot-dot resolution (which is incorrect on POSIX): https://docs.python.org/3.11/library/os.path.html#os.path.normpath Thus, passing the prefix above to ‘sys.normpath’ leads “pack2” to be erased. Josselin mentioned “path_hooks” as a possible solution, though I’m not sure what that means; could you explain? :-) Thanks! Ludo’.
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Hello folks, Recently, I have come around this issue. I created a Singularity container using the following `guix pack` command: `guix pack -f squashfs bash coreutils python python-numpy` Then, I run a bash shell within the container: `singularity exec test.gz.squashfs bash` Now, if I want to use numpy, `python3 -m numpy` it gives me the following error and the numpy module IS NOT found: Error in sitecustomize; set PYTHONVERBOSE for traceback: ValueError: '/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages' is not in list /gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No module named numpy However, if I do, `export PYTHONPATH=$GUIX_PYTHONPATH` the numpy module IS found: Error in sitecustomize; set PYTHONVERBOSE for traceback: ValueError: '/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages' is not in list /gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No module named numpy.__main__; 'numpy' is a package and cannot be directly executed Still, the sitecustomize error persists. See the output of `guix describe` below (only the guix channel): Pokolenie 3 14. január 2022 14:33:06 (súčasné) guix 5c5d9e5 zdroj repozitára: https://git.savannah.gnu.org/git/guix.git vetva: master úprava: 5c5d9e5a3208fa51351c6ad8267eba3d96fa2232 Thanks, Marek
bug#53258: Python unable to find modules within a Singularity container created with guix pack
Hi Marek,
Marek Felšöci skribis:
> Recently, I have come around this issue. I created a Singularity
> container using the following `guix pack` command:
>
> `guix pack -f squashfs bash coreutils python python-numpy`
>
> Then, I run a bash shell within the container:
>
> `singularity exec test.gz.squashfs bash`
>
> Now, if I want to use numpy,
>
> `python3 -m numpy`
>
> it gives me the following error and the numpy module IS NOT found:
>
> Error in sitecustomize; set PYTHONVERBOSE for traceback:
> ValueError:
> '/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages'
>
> is not in list
> /gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No
> module named numpy
I can kinda reproduce via a relocatable pack (relocatable packs and
Singularity packs have in common that they use relative symlinks in
their profile, to allow for relocation):
guix pack -RR python python-numpy -S /bin=bin -S /etc=etc
Then I unpack the thing:
mkdir /tmp/pack; cd /tmp/pack; tar xf /gnu/store/…-pack.tar.gz
And from there I use the same trick as in
‘tests/guix-pack-relocatable.sh’ to “hide” /gnu/store:
--8<---cut here---start->8---
$ unshare -mrf sh -c 'mount -t tmpfs -o ro none /gnu/store; .
/tmp/pack/etc/profile; echo $GUIX_PYTHONPATH ; /tmp/pack/bin/python3 '
/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages
Python 3.9.6 (default, Jan 1 1970, 00:00:01)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'numpy'
>>> import os
>>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages")
['setuptools-56.0.0.dist-info', 'distutils-precedence.pth', '_distutils_hack',
'setuptools', 'pip-21.1.3.dist-info', 'sitecustomize.py', 'README.txt', 'pip',
'numpy-1.20.3-py3.9.egg-info', 'numpy', 'pkg_resources', '__pycache__']
>>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages/numpy")
['__init__.pyi', 'doc', 'matlib.py', 'ctypeslib.pyi', 'linalg', 'ctypeslib.py',
'compat', 'typing', '__init__.pxd', 'lib', 'char.pyi', 'version.py',
'__config__.py', 'core', '_distributor_init.py', 'distutils', 'tests',
'emath.pyi', '_pytesttester.py', 'fft', '__init__.cython-30.pxd',
'conftest.py', '__init__.py', 'py.typed', '_globals.py', 'rec.pyi', 'testing',
'LICENSE.txt', 'polynomial', 'f2py', 'matrixlib', 'random', 'ma', 'dual.py',
'setup.py', '__pycache__']
--8<---cut here---end--->8---
So in this case ‘GUIX_PYTHONPATH’ is correctly set, but it looks as
though ‘sitecustomize.py’ wasn’t loaded. (I don’t get the “Error in
sitecustomize” message that Marek sees though, weird!)
>From Marek’s message, it would seem that somewhere file names are
constructed by appending “..”. Given Unix’s weird dot-dot semantics,
relative symlinks could be interfering:
--8<---cut here---start->8---
$ ls -l
/tmp/pack/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages/
totalo 48
lrwxrwxrwx 1 ludo users 102 Jan 1 1970 _distutils_hack ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/_distutils_hack
lrwxrwxrwx 1 ludo users 111 Jan 1 1970 distutils-precedence.pth ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/distutils-precedence.pth
lrwxrwxrwx 1 ludo users 99 Jan 1 1970 numpy ->
../../../../mlccgh05bf8cdinq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.9/site-packages/numpy
lrwxrwxrwx 1 ludo users 121 Jan 1 1970 numpy-1.20.3-py3.9.egg-info ->
../../../../mlccgh05bf8cdinq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.9/site-packages/numpy-1.20.3-py3.9.egg-info
lrwxrwxrwx 1 ludo users 90 Jan 1 1970 pip ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pip
lrwxrwxrwx 1 ludo users 107 Jan 1 1970 pip-21.1.3.dist-info ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pip-21.1.3.dist-info
lrwxrwxrwx 1 ludo users 100 Jan 1 1970 pkg_resources ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pkg_resources
lrwxrwxrwx 1 ludo users 98 Jan 1 1970 __pycache__ ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/__pycache__
lrwxrwxrwx 1 ludo users 97 Jan 1 1970 README.txt ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/README.txt
lrwxrwxrwx 1 ludo users 97 Jan 1 1970 setuptools ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/setuptools
lrwxrwxrwx 1 ludo users 114 Jan 1 1970 setuptools-56.0.0.dist-info ->
