Re: PYTHONPATH woes

[Hartmut Goebel]

Am 24.02.2018 um 11:44 schrieb Hartmut Goebel:
>  you may read all of the mail as the techniques are a bit complex.

No offense meant, please ignore the sentence. When I started writing the
mail I though my explanation would be much more complex.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH woes

[Hartmut Goebel]

Am 23.02.2018 um 17:59 schrieb Pjotr Prins:
> I may misunderstand how you are doing this, but I think it is going to
> cause problems. Python makes assumptions about the environment and
> virtualenv kinda redirects those. On top of that we have guix profiles
> and people who need to redirect PYTHONPATH (temporarily). Adding
> virtualenv into the mix will complicate things. Then there are people
> using virtualenv on top of Guix ...

I have to admit that my description was a bit terse. I understand your
concerns, but don't worry. This answer is a bit lengthy and you may read
all of the mail as the techniques are a bit complex.

0) I'm proposing to replace the wrapper scripts by a private virtual
environment.

1) This only effects the ways scripts are run, the package is still
available as normal.

2) The venv-hack I posted a few days ago is proposing something
different and is aiming to solve a different problem.

3) AFAIK virtualenvs can NOT be layered, they can only share the system
site-packages. This needs to be investigated  further, but only effects
the venv-hack I posted a few days ago.

Re 1):

This only effects the way *scripts* are run. Instead of using a wrapper
script, this uses a private virtual env - for the script only! Any
package is still available as normal.

Instead of using the python-executable of the profile, the script uses
the one in its private virtual env. This will make the script being run
in its own environment, which we also could name "profile".

The generated layout is as follows:

/gnu/store/…-my-app-0.1
+ lib/python3.5/site-packages/my_package/__init__.py  # unchanged
+ bin/my-app   # no wrapper! uses private venv's python
+ share/guix-venv/my-app-0.1
  +- pyvenv.cfg  # tells bin/python this is a virtual environment
  +- bin/python -> /gnu/store/…-python-3.5.3/bin/python
  +- lib/python3.5/site-packages/my-app.pth  # simulates PYTHONPATH

This comes down to something like

pyvenv-3.5 /gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1
/gnu/store/…-my-app-0.1/bin/pip install my-app-0.1.tar.gz
ln -s /gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1/bin/my-app \
  /gnu/store/…-my-app-0.1/bin/my-app
ln -s 
/gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1/lib/python3.5/site-packages/my-app
 \
  /gnu/store/…-my-app-0.1/lib/python3.5/site-packages/my-app

This solves several issues:

  * - python-applications in one profile can use conflicting
python-packages, since each application's dependencies are enclosed
in this application's private environment
  * no more problems since the scripts file-name is .my-app-real (see
e.g. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26752)
  * packages include references to their dependencies the gc can now
pick up (only valid if the package includes a script, though)


Re 2) and 3):

> I think the problem of mixing module versions has to be resolved
> through profiles. That should just work(tm).

Mixing different package-versions for the same python version can never
ever being solved by profile-means Python supports only a single package
version in sys.path. Python has this
pkg_resources.get_distribution()-stuff, but this never really took of
and was superseded by virtual environments.

> The problem of mixing interpreter versions can be resolved through
> profiles. Though there is the danger that people mix them into one
> profile.
Mixing versions of different Python version can by solved by the
venv-hack I posted a few days ago. This basically makes the profile into
a virtual environment. Thus there is no need for setting PYTHONPATH in
the profile anymore (we can even remove the search-path) and packages of
different Python versions will no longer be mixed.

> For this I suggest we tell Python2 to only use PYTHONPATH2. That way
> there is no interference. Python2 is being phased out (it is obsolete)
> and upstream should consider such a solution too.

No other distribution is using something like PYTHONPATH2, neither does
upstream. Upstream will never introduce such a heavy incompatible change.

Our problems are caused by misusing PYTHONPATH for collection the
system/profile site-packages. We could of course implement search-paths
GUIX-PYTHON-SITE-PACKAGE-2 and GUIX-PYTHON-SITE-PACKAGE-3 (instead of
PYTHONPATH) and change "site.py" to honor them. I'll look into whether
this or my venv-hack are a better solution, esp. regarding "stacked"
virtual environments.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH woes

[Pjotr Prins]

On Fri, Feb 23, 2018 at 08:36:46PM +0100, Ricardo Wurmus wrote:
> Patching Python 2 is still an option, but I’d like to explore (and
> understand) upstream mechanisms first.

I don't think it will be radical and something upstream can adopt. We
can at least suggest it ;)

> > With Ruby we have a similar interpreter issue - even more fine grained
> > between versions. It is a pain. But there is no real solution other
> > than using profiles properly.
> 
> Do Ruby *executables* also suffer from accidental dependency injection
> as Ribodiff does in this case?

Main problem is that gems are stored in major versions, e.g. 2.4. So
Ruby 2.4.1 stores modules in the same path as 2.4.2. I think that is a
mistake. Theoretically gems are compatible... But you can see the
potential mess.

Re: PYTHONPATH woes

[Ricardo Wurmus]

Pjotr Prins  writes:

> Then there are people
> using virtualenv on top of Guix ...

Pjotr brings up a good point: can virtualenvs be composed?  Do users
still have a way to override modules via PYTHONPATH or some other
mechanism if we were to create virtualenvs by default?

> I think the problem of mixing module versions has to be resolved
> through profiles. That should just work(tm).

@Pjotr: But we have seen that this doesn’t work as is and currently
requires user intervention.  We cannot expect users to separate all
Python 2 things from all Python 3 things because it is not even obvious
in all cases that a tool results in Python modules to be installed to
the profile.

> For this I suggest we tell Python2 to only use PYTHONPATH2. That way
> there is no interference. Python2 is being phased out (it is obsolete)
> and upstream should consider such a solution too.

I’d like to avoid radical patching when Python seems to have some
support for ignoring directories on the PYTHONPATH that don’t include
the correct version number.

Patching Python 2 is still an option, but I’d like to explore (and
understand) upstream mechanisms first.

> With Ruby we have a similar interpreter issue - even more fine grained
> between versions. It is a pain. But there is no real solution other
> than using profiles properly.

Do Ruby *executables* also suffer from accidental dependency injection
as Ribodiff does in this case?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: PYTHONPATH woes

[Hartmut Goebel]

Am 22.02.2018 um 16:30 schrieb Ricardo Wurmus:
> (This wouldn’t help us much for wrapper scripts, though.)

Attached please fins an approach for solving this issue.

The basic idea is make the application/script use a python within a
virtual environment. I nee to rethink some details and check whether
this will work out.

See the comments in the code for how it is intended to work.


-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

;;; Copyright © 2018 Hartmut Goebel 

(use-modules (guix)
 (gnu packages python)
 (guix licenses))

;; TODO decide which path to use
(define VENV-DIR "/share/guix-venv/") ; trailing slash!

;; python-uniseg is a reasonable small package without any dependencies and
;; including a script
(package
  (inherit python-uniseg)
  (name "my-python-uniseg")
  (version (package-version python-uniseg))
  (arguments
   `(#:tests? #f  ; copied from base package
 #:phases
 (modify-phases %standard-phases
   (replace 'wrap
 (lambda* (#:key inputs outputs #:allow-other-keys)
   ;; This sets up a small virtual environment for the package
   ;; and wraps the scripts to 
   ;; ../share/guix-venv/my-site-display-0.1
   ;; +- pyvenv.cfg
   ;; +- bin/python
   ;; +- lib/pythonX.Y/site-packages/NAME.pth
  (let* ((out  (assoc-ref outputs "out"))
 (venv (string-append out ,VENV-DIR ,name "-" ,version))
 (python (assoc-ref inputs "python"))
 (site-dir (string-append "/lib/python"
  "3.5" ;; FIXME (get-python-version python)
  "/site-packages"))
 (PYTHONPATH ;; FIXME: Why is ther only one entry?
  (cons (string-append out site-dir)
 (search-path-as-string->list
  (or (getenv "PYTHONPATH") "")
(mkdir-p (string-append venv "/bin"))
(mkdir-p (string-append venv site-dir))
;; The existance of a pyvenv.cfg file marks this as being a virtual
;; environment
(call-with-output-file (string-append venv "/pyvenv.cfg")
  (lambda (p)
(format p "#include-system-site-packages = false")))
;; Link to all required packages using a .pth file
(call-with-output-file (string-append venv site-dir "/" ,name ".pth")
  (lambda (p)
(for-each (lambda (pkg)
(format p "~a~%" pkg))
  PYTHONPATH)))
;; Create the python "executable" within the virtual environment
(for-each
 (lambda (name)
   (symlink (string-append python "/bin/" name)
(string-append venv  "/bin/" name)))
 `("python" ,"python3"))
)))
   ;; Re-wrap the scripts to use the python within the virtual env
   (add-after 'patch-shebangs 'patch-python-venv-shebang
 (lambda* (#:key outputs inputs #:allow-other-keys)
  
   (let* ((out (assoc-ref outputs "out"))
  (venv (string-append out ,VENV-DIR ,name "-" ,version))
  (python (assoc-ref inputs "python")))
 (define (venv-program file)
 (substitute* file
   (((string-append "#!" python "/bin/python([0-9](\\.[0-9])?)?"))
(string-append "#!" venv "/bin/python"
 ;; FIXME: Use logic from python-build-system to find scripts
 (venv-program (string-append out "/bin/uniseg-dbpath"))
 ))
  ;; Some requirement to check if all required pathes are included into the
  ;; .pth-file.
  (inputs
   `(("req" ,python-simplejson)))
  )

Re: PYTHONPATH woes

[Vincent Legoll]

Hello,

On Thu, Feb 22, 2018 at 9:42 PM, Hartmut Goebel
 wrote:
>   (synopsis "Python venc-hack for Guix")

Looks like a typo...

s/venc/venv/

-- 
Vincent Legoll

Re: PYTHONPATH woes

[Hartmut Goebel]

Hi,

Am 22.02.2018 um 16:30 schrieb Ricardo Wurmus:
>> re 2): As soon as Python2 and Python3 are installed in the same profile,
>> we put both site-packages for both versions into PYTHONPATH, which
>> obviously is wrong.
>>
>> $ PYTHONPATH= guix environment --ad-hoc python@2 python
>> […]
>> [guix] $ echo $PYTHONPATH
>> /gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python2.7/site-packages:/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python3.5/site-packages:
>>
>> A simple work-around would be to make the profile a (pseudo) virtual
>> environment, which is a easy as creating a file
>> "/gnu/store/…-profile/pyvenv.cfg". This will trigger a mechanism in
>> site.py to insert /gnu/store/…-profile/lib/pythonX.Y/site-packages" into
>> sys.path - for the current python version only!
> This is very good to know.  I haven’t tested this (as I’m not editing
> the store by hand), but I suppose we could create an empty pyvenv.cfg in
> a profile hook when Python packages are installed.

Below please find a simple package which adds this hack. After
installing it, unset PYTHONPATH:
export PYTHONPATH=

In the long run both python@2 and python@3 could propagate this
hack-package, thus is would be installed whenever python is installed.
(This need further investigation or a guix guru to help.)

> (This wouldn’t help us much for wrapper scripts, though.)

I have another hack in petto :) But one step at a time :-)


Now here is the

cat > guix-python-venv-hack.scm <<"EOF"
;;; Copyright © 2018 Hartmut Goebel 

(use-modules (guix)
 (guix build-system trivial)
 (guix licenses))

(package
  (name "guix-python-venv-hack")
  (version "0.1")
  (source #f)
  (build-system trivial-build-system)
  (arguments
   `(#:modules ((guix build utils))
 #:builder
 (begin
   (use-modules (guix build utils))
   (let* ((out   (assoc-ref %outputs "out"))
  (pyvenv.cfg (string-append out "/pyvenv.cfg")))
 (mkdir-p out)
 (call-with-output-file pyvenv.cfg
   (lambda (p)
 (format p "#")))
 ;;(chmod index.php #o555)
 
  (synopsis "Python venc-hack for Guix")
  (description "See ...")
  (home-page #f)
  (license gpl3+))
EOF
guix package --install-from-file guix-python-venv-hack.scm
unset PYTHONPATH

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH woes

[Hartmut Goebel]

Hi Ricardo,

you are welcome, I've taken the challenge :-) I always have fun digging
in the internals of Python - if I find time.

Thanks a lot for the data. I started investigating it and this raised
some more questions. Some behavior is curious. E.g. why does "python -c
'import statmodules'" start

 Thus it would be easiest if I would work interactively with that
environment. Can you please send me the information I need to rebuild
this environment (see below) - assistance would be great!

Alternatively I could download a export/copy of that profile - if guix
has some means for this.

Could I get interactive access to that profile? (Details to be discussed
of-list) This would allow me to investigate some of the curious
behavior. Nevertheless I would also like to rebuild the profile, so I
can hack the store-objects to work towards a solution.

For rebuilding  the I assume I need the following installation
- relevant parts of "guix package --list-installed"
- relevant installation dates from "guix package --list-generations"
- guix version used would be helpful

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH woes

[Ricardo Wurmus]

Hi Hartmut,

I was hoping for you to show up and give your input.  Thank you!  I
appreciate you taking the time.

> re 1): This is extremely curious: Why is the older package imported,
> where the newer one is expected to be first in path?
>
> To analyze this in detail, please make a copy of the resp. wrapper
> script and change the last command into:
>
> exec -a "$0" python3 -m site
>
> and run the changed wrapper. This should show all paths defined in the
> wrapper first.

Since this is Python 2 I used this line:

exec -a "$0" /gnu/store/0n8ni2l…-python-2.7.13/bin/python -m site

The full wrapper looks like this:

--8<---cut here---start->8---
#!/gnu/store/f8k940vy9gck66m9r4id5m098w3hxgka-bash-minimal-4.4.12/bin/bash
export 
PYTHONPATH="/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages:/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/site-packages:/gnu/store/95xlp80dp36m6nllaifndvc1vspnxwb2-python2-mock-1.0.1/lib/python2.7/site-packages:/gnu/store/8dgqb88wabnqscri4brwhasar7i1an1h-python2-nose-1.3.7/lib/python2.7/site-packages:/gnu/store/xd796qc9kjaslw85jk7f81fxkql021x0-python2-numpy-1.12.0/lib/python2.7/site-packages:/gnu/store/l1gdwsx4gr23gzy8zw6n588icdhhw8z4-python2-matplotlib-2.0.2/lib/python2.7/site-packages:/gnu/store/prvq91x42x7wpxbjgx83y78n23jx3jd5-python2-scipy-0.19.1/lib/python2.7/site-packages:/gnu/store/ng6srvhfmkz9i6g2ilbg6zksbh0v9yik-python2-statsmodels-0.8.0/lib/python2.7/site-packages:/gnu/store/vls5bci2wk15sz080g6wgycj6fmm44w9-python2-cairocffi-0.8.0/lib/python2.7/site-packages:/gnu/store/lfnzcj977hldqhkcpivag1hwqqqzk4gr-python2-six-1.10.0/lib/python2.7/site-packages:/gnu/store/vcs6yjy3851zn350gc90ipw2if3rg2vj-python2-pytz-2017.3/lib/python2.7/site-packages:/gnu/store/5w9r2c3dcz353n9rp56pcwvbks8s2hva-python2-pillow-3.3.3/lib/python2.7/site-packages:/gnu/store/hl1ly62q6gsiwd287gqy354dbgrq5sxk-python2-dateutil-2.6.0/lib/python2.7/site-packages:/gnu/store/8jwxgbcym5i1accf88bq7i8fgzg2z68q-python2-pyparsing-2.2.0/lib/python2.7/site-packages:/gnu/store/ikx5his9njw7r7df06gkwlayhgl2780a-python2-cycler-0.10.0/lib/python2.7/site-packages:/gnu/store/jknf4w4s9w5w71ampcymgn4d14hfwx3h-python-2.7.13-tk/lib/python2.7/site-packages:/gnu/store/gdgy38ylfm4jaz4cmq6c6650i8iga21l-python2-subprocess32-3.2.7/lib/python2.7/site-packages:/gnu/store/pdlc7mwmm1vfrkgcpscps5kj0p1gwa14-python2-pygobject-2.28.6/lib/python2.7/site-packages:/gnu/store/s8sn8r41jyn35aginp63z2232sw9g23a-python2-functools32-3.2.3-2/lib/python2.7/site-packages:/gnu/store/bj4jlrnai5qlic1arjijyp47isvipca7-python2-pycairo-1.10.0/lib/python2.7/site-packages:/gnu/store/rhc5rpihy35ss7i9fvc0knqa798br9yj-python2-patsy-0.4.1/lib/python2.7/site-packages:/gnu/store/mrm8hn8rv04hm80sl63c1w33xs5gg147-python2-pandas-0.19.2/lib/python2.7/site-packages:/gnu/store/q977dgwlvmak15qn0w2kjk3q322mbrwi-python2-xcffib-0.5.1/lib/python2.7/site-packages:/gnu/store/gmdlgpm3jyfi608fyjmj6g7svpp5y8lc-python2-cffi-1.11.2/lib/python2.7/site-packages:/gnu/store/7fziyn0m8wjc98j0g056bms4yv94mi5q-python2-pycparser-2.17/lib/python2.7/site-packages${PYTHONPATH:+:}$PYTHONPATH"
#exec -a "$0" 
"/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/bin/.TE.py-real" 
"$@"
exec -a "$0" 
/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/bin/python -m site
--8<---cut here---end--->8---

While PYTHONPATH is set this prints *nothing* at all.  Only after “unset
PYTHONPATH” I get this:

--8<---cut here---start->8---
sys.path = [
'/home/uzinnal',

'/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages',

'/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/site-packages',

'/gnu/store/95xlp80dp36m6nllaifndvc1vspnxwb2-python2-mock-1.0.1/lib/python2.7/site-packages',

'/gnu/store/8dgqb88wabnqscri4brwhasar7i1an1h-python2-nose-1.3.7/lib/python2.7/site-packages',

'/gnu/store/xd796qc9kjaslw85jk7f81fxkql021x0-python2-numpy-1.12.0/lib/python2.7/site-packages',

'/gnu/store/l1gdwsx4gr23gzy8zw6n588icdhhw8z4-python2-matplotlib-2.0.2/lib/python2.7/site-packages',

'/gnu/store/prvq91x42x7wpxbjgx83y78n23jx3jd5-python2-scipy-0.19.1/lib/python2.7/site-packages',

'/gnu/store/ng6srvhfmkz9i6g2ilbg6zksbh0v9yik-python2-statsmodels-0.8.0/lib/python2.7/site-packages',

'/gnu/store/vls5bci2wk15sz080g6wgycj6fmm44w9-python2-cairocffi-0.8.0/lib/python2.7/site-packages',

'/gnu/store/lfnzcj977hldqhkcpivag1hwqqqzk4gr-python2-six-1.10.0/lib/python2.7/site-packages',

'/gnu/store/vcs6yjy3851zn350gc90ipw2if3rg2vj-python2-pytz-2017.3/lib/python2.7/site-packages',

'/gnu/store/5w9r2c3dcz353n9rp56pcwvbks8s2hva-python2-pillow-3.3.3/lib/python2.7/site-packages',

'/gnu/store/hl1ly62q6gsiwd287gqy354dbgrq5sxk-python2-dateutil-2.6.0/lib/python2.7/site-packages',

Re: PYTHONPATH woes

[Hartmut Goebel]

Hi,

thanks for raising the PYTHONPATH issue. Looks like we have a major
problem here.

Am 20.02.2018 um 11:53 schrieb Ricardo Wurmus:
> I don’t know why this happens.  I find it puzzling that in this
> particular case the user’s profile contains an *older* version of
> statsmodels (0.6.1).  The wrapper includes the correct version of
> statsmodels (0.8.0) in the PYTHONPATH.  Here’s the backtrace:

You are addressing three issues here:
1) Why is the older package imported, where the newer one is expected to
be first in path?
2) We are mixing Python2 and Python3 paths
3) Is the way we use PYTHONPATH in the wrapper the correct way?

re 1): This is extremely curious: Why is the older package imported,
where the newer one is expected to be first in path?

To analyze this in detail, please make a copy of the resp. wrapper
script and change the last command into:

exec -a "$0" python3 -m site

and run the changed wrapper. This should show all paths defined in the
wrapper first.

If this does not give any insight, change it into

exec -a "$0" python3 -v -c "import statsmodels"

and then

exec -a "$0" python3 -v -c "import ribodiff.estimatedisp"

and try to find some insight there.


re 2): As soon as Python2 and Python3 are installed in the same profile,
we put both site-packages for both versions into PYTHONPATH, which
obviously is wrong.

$ PYTHONPATH= guix environment --ad-hoc python@2 python
[…]
[guix] $ echo $PYTHONPATH
/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python2.7/site-packages:/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python3.5/site-packages:

A simple work-around would be to make the profile a (pseudo) virtual
environment, which is a easy as creating a file
"/gnu/store/…-profile/pyvenv.cfg". This will trigger a mechanism in
site.py to insert /gnu/store/…-profile/lib/pythonX.Y/site-packages" into
sys.path - for the current python version only!

Try it:

sudo touch $GUIX_ENVIRONMENT/pyvenv.cfg
$ PYTHONPATH= guix environment --ad-hoc python@2 python
[…]
[guix] $ PYTHONPATH=/tmp/foo:/tmp/bar python3 -m site
[…]
    '/gnu/store/…-profile/lib/python3.5/site-packages',


re 3) When running

PYTHONPATH=/tmp/foo:/tmp/bar python -m site

on e.g. Debian, one can see that the order in sys.path is as follows:

- $PWD
- $PYTHONPATH elements
- built-in paths (e.g. /usr/lib64/python3.5)
- site-packages

The idea seems to be that PYTHONPATH can overwrite all packages, but
site-package can not. This can be seen as if in the wrapper-scripts we
are not using PYTHONPATH as indented: The user can not overwrite
site-packages. We ought to think if this is what we want, as this is how
Python works.

Depending on the result of the analysis for (1) and if we implement (2),
we can investigate how to solve (3). One idea I already looked at this
evening is to replace the wrappers by a minimal virtualenv.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH woes

[Pjotr Prins]

On Tue, Feb 20, 2018 at 04:18:40PM +0100, Andy Wingo wrote:
> In Guix we don't set LD_LIBRARY_PATH but we do set PYTHONPATH, so it's
> not quite the same I don't think.

Not exactly the same, but close enough ;). But it is clearer now that
mixing is the problem. Ricardos .pth may be an option if that works,
or we introduce a separate GUIX_PYTHONPATH2 and GUIX_PYTHONPATH for 3.
Python2 is at end of life, so we may set an example there for others.

Even so, I don't think it will solve the particular conflict that
Ricardo was describing. There will always be hairy mix-ins.

Pj.
-- 

Re: PYTHONPATH woes

[Ricardo Wurmus]

Hi Pjotr,

> On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote:
>> Would it be good to make the wrappers for Python scripts stricter and
>> not accept any user-set PYTHONPATH?
>
> I think that is a bad idea. You need to be able to opt out. Also
> people need to experiment with modules without understanding Guix per
> se. In my upcomping blog I would emphasize packaging at the point you
> become a serious user.
>
> That should come with a health warning ;). Similarly we should allow
> for LD_LIBRARY_PATH etc. It is what they exist for, even if it is
> dangerous.

While I agree that it must be *possible* to opt out, I think the
defaults are wrong here.  We don’t ever set LD_LIBRARY_PATH when
building a profile, but we do set PYTHONPATH.  Having PYTHONPATH set
(even without the knowledge of the user) leads to problems here when it
shouldn’t.

I’d argue that the number of users who need to be able to override
individual Python packages for a tool like Ribodiff is way lower than
the number of those who accidentally get into a situation where
PYTHONPATH is set purely because of the contents of their profile,
leading to breakage of unrelated packages that just happen to use Python
modules.

>> How do we approach the problem of having both Python 2 modules and
>> Python 3 modules in the same profile?  PYTHONPATH will be set to refer
>> to the site-packages directories of both versions, which is never good.
>> Does Python offer us a way to do better?  Can we make use of pth files
>> to get around this problem somehow?
>
> Python should have created PYTHONPATH2 to split them out. We could
> patch python2 to do just that.

I think there’s an alternative, but I don’t know it well.  The official
way is to use “.pth” files instead of setting PYTHONPATH.  Maybe there’s
a way that doesn’t involve setting PYTHONPATH.  Instead we could nudge
Python towards reading the profile’s “.pth” file and read the package
locations from there.

This would require a new profile hook and possibly a patch to Python to
add a way to tell it to read the “.pth” file from a location provided by
a Guix environment variable.

> Even so, the real solution is separate profiles. I get that with
> versions of Ruby too.

Right, but it is not obvious what packages must be treated with extra
care.  When I install the Ribodiff package I don’t know or care that
it’s written in Python.  That shouldn’t matter at all.  But now I
actually have to pay attention to this and install Ribodiff in a profile
that doesn’t contain Python 3 things.

That’s rather complicated and unfriendly for users.  I don’t want the
users here to be anxious about installing software, just because a new
tool may be using Python and thus might break when installed to a
profile containing Python things.  We made these wrappers precisely to
isolate the tools from the current environment.  Allowing them to be
disturbed so easily is making wrappers much less useful.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: PYTHONPATH woes

[Andy Wingo]

On Tue 20 Feb 2018 16:01, Pjotr Prins  writes:

> On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote:
>> Would it be good to make the wrappers for Python scripts stricter and
>> not accept any user-set PYTHONPATH?
>
> I think that is a bad idea. You need to be able to opt out.

Why?  I am not sure this is the case for programs that just happen to be
written in Python.

> That should come with a health warning ;). Similarly we should allow
> for LD_LIBRARY_PATH etc. It is what they exist for, even if it is
> dangerous.

In Guix we don't set LD_LIBRARY_PATH but we do set PYTHONPATH, so it's
not quite the same I don't think.

Andy

Re: PYTHONPATH woes

[Pjotr Prins]

On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote:
> Would it be good to make the wrappers for Python scripts stricter and
> not accept any user-set PYTHONPATH?

I think that is a bad idea. You need to be able to opt out. Also
people need to experiment with modules without understanding Guix per
se. In my upcomping blog I would emphasize packaging at the point you
become a serious user.

That should come with a health warning ;). Similarly we should allow
for LD_LIBRARY_PATH etc. It is what they exist for, even if it is
dangerous.

> How do we approach the problem of having both Python 2 modules and
> Python 3 modules in the same profile?  PYTHONPATH will be set to refer
> to the site-packages directories of both versions, which is never good.
> Does Python offer us a way to do better?  Can we make use of pth files
> to get around this problem somehow?

Python should have created PYTHONPATH2 to split them out. We could
patch python2 to do just that.

Even so, the real solution is separate profiles. I get that with
versions of Ruby too.

Pj.