Re: PYTHONPATH woes
Am 24.02.2018 um 11:44 schrieb Hartmut Goebel: > you may read all of the mail as the techniques are a bit complex. No offense meant, please ignore the sentence. When I started writing the mail I though my explanation would be much more complex. -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
Re: PYTHONPATH woes
Am 23.02.2018 um 17:59 schrieb Pjotr Prins: > I may misunderstand how you are doing this, but I think it is going to > cause problems. Python makes assumptions about the environment and > virtualenv kinda redirects those. On top of that we have guix profiles > and people who need to redirect PYTHONPATH (temporarily). Adding > virtualenv into the mix will complicate things. Then there are people > using virtualenv on top of Guix ... I have to admit that my description was a bit terse. I understand your concerns, but don't worry. This answer is a bit lengthy and you may read all of the mail as the techniques are a bit complex. 0) I'm proposing to replace the wrapper scripts by a private virtual environment. 1) This only effects the ways scripts are run, the package is still available as normal. 2) The venv-hack I posted a few days ago is proposing something different and is aiming to solve a different problem. 3) AFAIK virtualenvs can NOT be layered, they can only share the system site-packages. This needs to be investigated further, but only effects the venv-hack I posted a few days ago. Re 1): This only effects the way *scripts* are run. Instead of using a wrapper script, this uses a private virtual env - for the script only! Any package is still available as normal. Instead of using the python-executable of the profile, the script uses the one in its private virtual env. This will make the script being run in its own environment, which we also could name "profile". The generated layout is as follows: /gnu/store/…-my-app-0.1 + lib/python3.5/site-packages/my_package/__init__.py # unchanged + bin/my-app # no wrapper! uses private venv's python + share/guix-venv/my-app-0.1 +- pyvenv.cfg # tells bin/python this is a virtual environment +- bin/python -> /gnu/store/…-python-3.5.3/bin/python +- lib/python3.5/site-packages/my-app.pth # simulates PYTHONPATH This comes down to something like pyvenv-3.5 /gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1 /gnu/store/…-my-app-0.1/bin/pip install my-app-0.1.tar.gz ln -s /gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1/bin/my-app \ /gnu/store/…-my-app-0.1/bin/my-app ln -s /gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1/lib/python3.5/site-packages/my-app \ /gnu/store/…-my-app-0.1/lib/python3.5/site-packages/my-app This solves several issues: * - python-applications in one profile can use conflicting python-packages, since each application's dependencies are enclosed in this application's private environment * no more problems since the scripts file-name is .my-app-real (see e.g. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26752) * packages include references to their dependencies the gc can now pick up (only valid if the package includes a script, though) Re 2) and 3): > I think the problem of mixing module versions has to be resolved > through profiles. That should just work(tm). Mixing different package-versions for the same python version can never ever being solved by profile-means Python supports only a single package version in sys.path. Python has this pkg_resources.get_distribution()-stuff, but this never really took of and was superseded by virtual environments. > The problem of mixing interpreter versions can be resolved through > profiles. Though there is the danger that people mix them into one > profile. Mixing versions of different Python version can by solved by the venv-hack I posted a few days ago. This basically makes the profile into a virtual environment. Thus there is no need for setting PYTHONPATH in the profile anymore (we can even remove the search-path) and packages of different Python versions will no longer be mixed. > For this I suggest we tell Python2 to only use PYTHONPATH2. That way > there is no interference. Python2 is being phased out (it is obsolete) > and upstream should consider such a solution too. No other distribution is using something like PYTHONPATH2, neither does upstream. Upstream will never introduce such a heavy incompatible change. Our problems are caused by misusing PYTHONPATH for collection the system/profile site-packages. We could of course implement search-paths GUIX-PYTHON-SITE-PACKAGE-2 and GUIX-PYTHON-SITE-PACKAGE-3 (instead of PYTHONPATH) and change "site.py" to honor them. I'll look into whether this or my venv-hack are a better solution, esp. regarding "stacked" virtual environments. -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
Re: PYTHONPATH woes
On Fri, Feb 23, 2018 at 08:36:46PM +0100, Ricardo Wurmus wrote: > Patching Python 2 is still an option, but I’d like to explore (and > understand) upstream mechanisms first. I don't think it will be radical and something upstream can adopt. We can at least suggest it ;) > > With Ruby we have a similar interpreter issue - even more fine grained > > between versions. It is a pain. But there is no real solution other > > than using profiles properly. > > Do Ruby *executables* also suffer from accidental dependency injection > as Ribodiff does in this case? Main problem is that gems are stored in major versions, e.g. 2.4. So Ruby 2.4.1 stores modules in the same path as 2.4.2. I think that is a mistake. Theoretically gems are compatible... But you can see the potential mess.
Re: PYTHONPATH woes
Pjotr Prinswrites: > Then there are people > using virtualenv on top of Guix ... Pjotr brings up a good point: can virtualenvs be composed? Do users still have a way to override modules via PYTHONPATH or some other mechanism if we were to create virtualenvs by default? > I think the problem of mixing module versions has to be resolved > through profiles. That should just work(tm). @Pjotr: But we have seen that this doesn’t work as is and currently requires user intervention. We cannot expect users to separate all Python 2 things from all Python 3 things because it is not even obvious in all cases that a tool results in Python modules to be installed to the profile. > For this I suggest we tell Python2 to only use PYTHONPATH2. That way > there is no interference. Python2 is being phased out (it is obsolete) > and upstream should consider such a solution too. I’d like to avoid radical patching when Python seems to have some support for ignoring directories on the PYTHONPATH that don’t include the correct version number. Patching Python 2 is still an option, but I’d like to explore (and understand) upstream mechanisms first. > With Ruby we have a similar interpreter issue - even more fine grained > between versions. It is a pain. But there is no real solution other > than using profiles properly. Do Ruby *executables* also suffer from accidental dependency injection as Ribodiff does in this case? -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: PYTHONPATH woes
Am 22.02.2018 um 16:30 schrieb Ricardo Wurmus: > (This wouldn’t help us much for wrapper scripts, though.) Attached please fins an approach for solving this issue. The basic idea is make the application/script use a python within a virtual environment. I nee to rethink some details and check whether this will work out. See the comments in the code for how it is intended to work. -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | ;;; Copyright © 2018 Hartmut Goebel(use-modules (guix) (gnu packages python) (guix licenses)) ;; TODO decide which path to use (define VENV-DIR "/share/guix-venv/") ; trailing slash! ;; python-uniseg is a reasonable small package without any dependencies and ;; including a script (package (inherit python-uniseg) (name "my-python-uniseg") (version (package-version python-uniseg)) (arguments `(#:tests? #f ; copied from base package #:phases (modify-phases %standard-phases (replace 'wrap (lambda* (#:key inputs outputs #:allow-other-keys) ;; This sets up a small virtual environment for the package ;; and wraps the scripts to ;; ../share/guix-venv/my-site-display-0.1 ;; +- pyvenv.cfg ;; +- bin/python ;; +- lib/pythonX.Y/site-packages/NAME.pth (let* ((out (assoc-ref outputs "out")) (venv (string-append out ,VENV-DIR ,name "-" ,version)) (python (assoc-ref inputs "python")) (site-dir (string-append "/lib/python" "3.5" ;; FIXME (get-python-version python) "/site-packages")) (PYTHONPATH ;; FIXME: Why is ther only one entry? (cons (string-append out site-dir) (search-path-as-string->list (or (getenv "PYTHONPATH") "") (mkdir-p (string-append venv "/bin")) (mkdir-p (string-append venv site-dir)) ;; The existance of a pyvenv.cfg file marks this as being a virtual ;; environment (call-with-output-file (string-append venv "/pyvenv.cfg") (lambda (p) (format p "#include-system-site-packages = false"))) ;; Link to all required packages using a .pth file (call-with-output-file (string-append venv site-dir "/" ,name ".pth") (lambda (p) (for-each (lambda (pkg) (format p "~a~%" pkg)) PYTHONPATH))) ;; Create the python "executable" within the virtual environment (for-each (lambda (name) (symlink (string-append python "/bin/" name) (string-append venv "/bin/" name))) `("python" ,"python3")) ))) ;; Re-wrap the scripts to use the python within the virtual env (add-after 'patch-shebangs 'patch-python-venv-shebang (lambda* (#:key outputs inputs #:allow-other-keys) (let* ((out (assoc-ref outputs "out")) (venv (string-append out ,VENV-DIR ,name "-" ,version)) (python (assoc-ref inputs "python"))) (define (venv-program file) (substitute* file (((string-append "#!" python "/bin/python([0-9](\\.[0-9])?)?")) (string-append "#!" venv "/bin/python" ;; FIXME: Use logic from python-build-system to find scripts (venv-program (string-append out "/bin/uniseg-dbpath")) )) ;; Some requirement to check if all required pathes are included into the ;; .pth-file. (inputs `(("req" ,python-simplejson))) )
Re: PYTHONPATH woes
Hello, On Thu, Feb 22, 2018 at 9:42 PM, Hartmut Goebelwrote: > (synopsis "Python venc-hack for Guix") Looks like a typo... s/venc/venv/ -- Vincent Legoll
Re: PYTHONPATH woes
Hi, Am 22.02.2018 um 16:30 schrieb Ricardo Wurmus: >> re 2): As soon as Python2 and Python3 are installed in the same profile, >> we put both site-packages for both versions into PYTHONPATH, which >> obviously is wrong. >> >> $ PYTHONPATH= guix environment --ad-hoc python@2 python >> […] >> [guix] $ echo $PYTHONPATH >> /gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python2.7/site-packages:/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python3.5/site-packages: >> >> A simple work-around would be to make the profile a (pseudo) virtual >> environment, which is a easy as creating a file >> "/gnu/store/…-profile/pyvenv.cfg". This will trigger a mechanism in >> site.py to insert /gnu/store/…-profile/lib/pythonX.Y/site-packages" into >> sys.path - for the current python version only! > This is very good to know. I haven’t tested this (as I’m not editing > the store by hand), but I suppose we could create an empty pyvenv.cfg in > a profile hook when Python packages are installed. Below please find a simple package which adds this hack. After installing it, unset PYTHONPATH: export PYTHONPATH= In the long run both python@2 and python@3 could propagate this hack-package, thus is would be installed whenever python is installed. (This need further investigation or a guix guru to help.) > (This wouldn’t help us much for wrapper scripts, though.) I have another hack in petto :) But one step at a time :-) Now here is the cat > guix-python-venv-hack.scm <<"EOF" ;;; Copyright © 2018 Hartmut Goebel(use-modules (guix) (guix build-system trivial) (guix licenses)) (package (name "guix-python-venv-hack") (version "0.1") (source #f) (build-system trivial-build-system) (arguments `(#:modules ((guix build utils)) #:builder (begin (use-modules (guix build utils)) (let* ((out (assoc-ref %outputs "out")) (pyvenv.cfg (string-append out "/pyvenv.cfg"))) (mkdir-p out) (call-with-output-file pyvenv.cfg (lambda (p) (format p "#"))) ;;(chmod index.php #o555) (synopsis "Python venc-hack for Guix") (description "See ...") (home-page #f) (license gpl3+)) EOF guix package --install-from-file guix-python-venv-hack.scm unset PYTHONPATH -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
Re: PYTHONPATH woes
Hi Ricardo, you are welcome, I've taken the challenge :-) I always have fun digging in the internals of Python - if I find time. Thanks a lot for the data. I started investigating it and this raised some more questions. Some behavior is curious. E.g. why does "python -c 'import statmodules'" start Thus it would be easiest if I would work interactively with that environment. Can you please send me the information I need to rebuild this environment (see below) - assistance would be great! Alternatively I could download a export/copy of that profile - if guix has some means for this. Could I get interactive access to that profile? (Details to be discussed of-list) This would allow me to investigate some of the curious behavior. Nevertheless I would also like to rebuild the profile, so I can hack the store-objects to work towards a solution. For rebuilding the I assume I need the following installation - relevant parts of "guix package --list-installed" - relevant installation dates from "guix package --list-generations" - guix version used would be helpful -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
Re: PYTHONPATH woes
Hi Hartmut, I was hoping for you to show up and give your input. Thank you! I appreciate you taking the time. > re 1): This is extremely curious: Why is the older package imported, > where the newer one is expected to be first in path? > > To analyze this in detail, please make a copy of the resp. wrapper > script and change the last command into: > > exec -a "$0" python3 -m site > > and run the changed wrapper. This should show all paths defined in the > wrapper first. Since this is Python 2 I used this line: exec -a "$0" /gnu/store/0n8ni2l…-python-2.7.13/bin/python -m site The full wrapper looks like this: --8<---cut here---start->8--- #!/gnu/store/f8k940vy9gck66m9r4id5m098w3hxgka-bash-minimal-4.4.12/bin/bash export PYTHONPATH="/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages:/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/site-packages:/gnu/store/95xlp80dp36m6nllaifndvc1vspnxwb2-python2-mock-1.0.1/lib/python2.7/site-packages:/gnu/store/8dgqb88wabnqscri4brwhasar7i1an1h-python2-nose-1.3.7/lib/python2.7/site-packages:/gnu/store/xd796qc9kjaslw85jk7f81fxkql021x0-python2-numpy-1.12.0/lib/python2.7/site-packages:/gnu/store/l1gdwsx4gr23gzy8zw6n588icdhhw8z4-python2-matplotlib-2.0.2/lib/python2.7/site-packages:/gnu/store/prvq91x42x7wpxbjgx83y78n23jx3jd5-python2-scipy-0.19.1/lib/python2.7/site-packages:/gnu/store/ng6srvhfmkz9i6g2ilbg6zksbh0v9yik-python2-statsmodels-0.8.0/lib/python2.7/site-packages:/gnu/store/vls5bci2wk15sz080g6wgycj6fmm44w9-python2-cairocffi-0.8.0/lib/python2.7/site-packages:/gnu/store/lfnzcj977hldqhkcpivag1hwqqqzk4gr-python2-six-1.10.0/lib/python2.7/site-packages:/gnu/store/vcs6yjy3851zn350gc90ipw2if3rg2vj-python2-pytz-2017.3/lib/python2.7/site-packages:/gnu/store/5w9r2c3dcz353n9rp56pcwvbks8s2hva-python2-pillow-3.3.3/lib/python2.7/site-packages:/gnu/store/hl1ly62q6gsiwd287gqy354dbgrq5sxk-python2-dateutil-2.6.0/lib/python2.7/site-packages:/gnu/store/8jwxgbcym5i1accf88bq7i8fgzg2z68q-python2-pyparsing-2.2.0/lib/python2.7/site-packages:/gnu/store/ikx5his9njw7r7df06gkwlayhgl2780a-python2-cycler-0.10.0/lib/python2.7/site-packages:/gnu/store/jknf4w4s9w5w71ampcymgn4d14hfwx3h-python-2.7.13-tk/lib/python2.7/site-packages:/gnu/store/gdgy38ylfm4jaz4cmq6c6650i8iga21l-python2-subprocess32-3.2.7/lib/python2.7/site-packages:/gnu/store/pdlc7mwmm1vfrkgcpscps5kj0p1gwa14-python2-pygobject-2.28.6/lib/python2.7/site-packages:/gnu/store/s8sn8r41jyn35aginp63z2232sw9g23a-python2-functools32-3.2.3-2/lib/python2.7/site-packages:/gnu/store/bj4jlrnai5qlic1arjijyp47isvipca7-python2-pycairo-1.10.0/lib/python2.7/site-packages:/gnu/store/rhc5rpihy35ss7i9fvc0knqa798br9yj-python2-patsy-0.4.1/lib/python2.7/site-packages:/gnu/store/mrm8hn8rv04hm80sl63c1w33xs5gg147-python2-pandas-0.19.2/lib/python2.7/site-packages:/gnu/store/q977dgwlvmak15qn0w2kjk3q322mbrwi-python2-xcffib-0.5.1/lib/python2.7/site-packages:/gnu/store/gmdlgpm3jyfi608fyjmj6g7svpp5y8lc-python2-cffi-1.11.2/lib/python2.7/site-packages:/gnu/store/7fziyn0m8wjc98j0g056bms4yv94mi5q-python2-pycparser-2.17/lib/python2.7/site-packages${PYTHONPATH:+:}$PYTHONPATH" #exec -a "$0" "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/bin/.TE.py-real" "$@" exec -a "$0" /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/bin/python -m site --8<---cut here---end--->8--- While PYTHONPATH is set this prints *nothing* at all. Only after “unset PYTHONPATH” I get this: --8<---cut here---start->8--- sys.path = [ '/home/uzinnal', '/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages', '/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/site-packages', '/gnu/store/95xlp80dp36m6nllaifndvc1vspnxwb2-python2-mock-1.0.1/lib/python2.7/site-packages', '/gnu/store/8dgqb88wabnqscri4brwhasar7i1an1h-python2-nose-1.3.7/lib/python2.7/site-packages', '/gnu/store/xd796qc9kjaslw85jk7f81fxkql021x0-python2-numpy-1.12.0/lib/python2.7/site-packages', '/gnu/store/l1gdwsx4gr23gzy8zw6n588icdhhw8z4-python2-matplotlib-2.0.2/lib/python2.7/site-packages', '/gnu/store/prvq91x42x7wpxbjgx83y78n23jx3jd5-python2-scipy-0.19.1/lib/python2.7/site-packages', '/gnu/store/ng6srvhfmkz9i6g2ilbg6zksbh0v9yik-python2-statsmodels-0.8.0/lib/python2.7/site-packages', '/gnu/store/vls5bci2wk15sz080g6wgycj6fmm44w9-python2-cairocffi-0.8.0/lib/python2.7/site-packages', '/gnu/store/lfnzcj977hldqhkcpivag1hwqqqzk4gr-python2-six-1.10.0/lib/python2.7/site-packages', '/gnu/store/vcs6yjy3851zn350gc90ipw2if3rg2vj-python2-pytz-2017.3/lib/python2.7/site-packages', '/gnu/store/5w9r2c3dcz353n9rp56pcwvbks8s2hva-python2-pillow-3.3.3/lib/python2.7/site-packages', '/gnu/store/hl1ly62q6gsiwd287gqy354dbgrq5sxk-python2-dateutil-2.6.0/lib/python2.7/site-packages',
Re: PYTHONPATH woes
Hi, thanks for raising the PYTHONPATH issue. Looks like we have a major problem here. Am 20.02.2018 um 11:53 schrieb Ricardo Wurmus: > I don’t know why this happens. I find it puzzling that in this > particular case the user’s profile contains an *older* version of > statsmodels (0.6.1). The wrapper includes the correct version of > statsmodels (0.8.0) in the PYTHONPATH. Here’s the backtrace: You are addressing three issues here: 1) Why is the older package imported, where the newer one is expected to be first in path? 2) We are mixing Python2 and Python3 paths 3) Is the way we use PYTHONPATH in the wrapper the correct way? re 1): This is extremely curious: Why is the older package imported, where the newer one is expected to be first in path? To analyze this in detail, please make a copy of the resp. wrapper script and change the last command into: exec -a "$0" python3 -m site and run the changed wrapper. This should show all paths defined in the wrapper first. If this does not give any insight, change it into exec -a "$0" python3 -v -c "import statsmodels" and then exec -a "$0" python3 -v -c "import ribodiff.estimatedisp" and try to find some insight there. re 2): As soon as Python2 and Python3 are installed in the same profile, we put both site-packages for both versions into PYTHONPATH, which obviously is wrong. $ PYTHONPATH= guix environment --ad-hoc python@2 python […] [guix] $ echo $PYTHONPATH /gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python2.7/site-packages:/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python3.5/site-packages: A simple work-around would be to make the profile a (pseudo) virtual environment, which is a easy as creating a file "/gnu/store/…-profile/pyvenv.cfg". This will trigger a mechanism in site.py to insert /gnu/store/…-profile/lib/pythonX.Y/site-packages" into sys.path - for the current python version only! Try it: sudo touch $GUIX_ENVIRONMENT/pyvenv.cfg $ PYTHONPATH= guix environment --ad-hoc python@2 python […] [guix] $ PYTHONPATH=/tmp/foo:/tmp/bar python3 -m site […] '/gnu/store/…-profile/lib/python3.5/site-packages', re 3) When running PYTHONPATH=/tmp/foo:/tmp/bar python -m site on e.g. Debian, one can see that the order in sys.path is as follows: - $PWD - $PYTHONPATH elements - built-in paths (e.g. /usr/lib64/python3.5) - site-packages The idea seems to be that PYTHONPATH can overwrite all packages, but site-package can not. This can be seen as if in the wrapper-scripts we are not using PYTHONPATH as indented: The user can not overwrite site-packages. We ought to think if this is what we want, as this is how Python works. Depending on the result of the analysis for (1) and if we implement (2), we can investigate how to solve (3). One idea I already looked at this evening is to replace the wrappers by a minimal virtualenv. -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
Re: PYTHONPATH woes
On Tue, Feb 20, 2018 at 04:18:40PM +0100, Andy Wingo wrote: > In Guix we don't set LD_LIBRARY_PATH but we do set PYTHONPATH, so it's > not quite the same I don't think. Not exactly the same, but close enough ;). But it is clearer now that mixing is the problem. Ricardos .pth may be an option if that works, or we introduce a separate GUIX_PYTHONPATH2 and GUIX_PYTHONPATH for 3. Python2 is at end of life, so we may set an example there for others. Even so, I don't think it will solve the particular conflict that Ricardo was describing. There will always be hairy mix-ins. Pj. --
Re: PYTHONPATH woes
Hi Pjotr, > On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote: >> Would it be good to make the wrappers for Python scripts stricter and >> not accept any user-set PYTHONPATH? > > I think that is a bad idea. You need to be able to opt out. Also > people need to experiment with modules without understanding Guix per > se. In my upcomping blog I would emphasize packaging at the point you > become a serious user. > > That should come with a health warning ;). Similarly we should allow > for LD_LIBRARY_PATH etc. It is what they exist for, even if it is > dangerous. While I agree that it must be *possible* to opt out, I think the defaults are wrong here. We don’t ever set LD_LIBRARY_PATH when building a profile, but we do set PYTHONPATH. Having PYTHONPATH set (even without the knowledge of the user) leads to problems here when it shouldn’t. I’d argue that the number of users who need to be able to override individual Python packages for a tool like Ribodiff is way lower than the number of those who accidentally get into a situation where PYTHONPATH is set purely because of the contents of their profile, leading to breakage of unrelated packages that just happen to use Python modules. >> How do we approach the problem of having both Python 2 modules and >> Python 3 modules in the same profile? PYTHONPATH will be set to refer >> to the site-packages directories of both versions, which is never good. >> Does Python offer us a way to do better? Can we make use of pth files >> to get around this problem somehow? > > Python should have created PYTHONPATH2 to split them out. We could > patch python2 to do just that. I think there’s an alternative, but I don’t know it well. The official way is to use “.pth” files instead of setting PYTHONPATH. Maybe there’s a way that doesn’t involve setting PYTHONPATH. Instead we could nudge Python towards reading the profile’s “.pth” file and read the package locations from there. This would require a new profile hook and possibly a patch to Python to add a way to tell it to read the “.pth” file from a location provided by a Guix environment variable. > Even so, the real solution is separate profiles. I get that with > versions of Ruby too. Right, but it is not obvious what packages must be treated with extra care. When I install the Ribodiff package I don’t know or care that it’s written in Python. That shouldn’t matter at all. But now I actually have to pay attention to this and install Ribodiff in a profile that doesn’t contain Python 3 things. That’s rather complicated and unfriendly for users. I don’t want the users here to be anxious about installing software, just because a new tool may be using Python and thus might break when installed to a profile containing Python things. We made these wrappers precisely to isolate the tools from the current environment. Allowing them to be disturbed so easily is making wrappers much less useful. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: PYTHONPATH woes
On Tue 20 Feb 2018 16:01, Pjotr Prinswrites: > On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote: >> Would it be good to make the wrappers for Python scripts stricter and >> not accept any user-set PYTHONPATH? > > I think that is a bad idea. You need to be able to opt out. Why? I am not sure this is the case for programs that just happen to be written in Python. > That should come with a health warning ;). Similarly we should allow > for LD_LIBRARY_PATH etc. It is what they exist for, even if it is > dangerous. In Guix we don't set LD_LIBRARY_PATH but we do set PYTHONPATH, so it's not quite the same I don't think. Andy
Re: PYTHONPATH woes
On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote: > Would it be good to make the wrappers for Python scripts stricter and > not accept any user-set PYTHONPATH? I think that is a bad idea. You need to be able to opt out. Also people need to experiment with modules without understanding Guix per se. In my upcomping blog I would emphasize packaging at the point you become a serious user. That should come with a health warning ;). Similarly we should allow for LD_LIBRARY_PATH etc. It is what they exist for, even if it is dangerous. > How do we approach the problem of having both Python 2 modules and > Python 3 modules in the same profile? PYTHONPATH will be set to refer > to the site-packages directories of both versions, which is never good. > Does Python offer us a way to do better? Can we make use of pth files > to get around this problem somehow? Python should have created PYTHONPATH2 to split them out. We could patch python2 to do just that. Even so, the real solution is separate profiles. I get that with versions of Ruby too. Pj.