Re: [osgeo4w-dev] [osgeo4w] #692: Python 3.9 on v2 installer breaks part of stdlib

[OSGeo4W]

#692: Python 3.9 on v2 installer breaks part of stdlib
---+
Reporter:  akominlsfi  |   Owner:  osgeo4w-dev@…
Type:  defect  |  Status:  new
Priority:  major   |   Component:  Installer
 Version:  |  Resolution:
Keywords:  |
---+
Comment (by Andreas Müller):

 I created the file sitecustomize.py with this code:

 {{{
 import os
 for p in os.getenv("PATH").split(";"):
 if os.path.exists(p):
 os.add_dll_directory(p)
 }}}

 and it seems to fix issues I had with an IDE I use (PyScripter using
 PyQt5) plus using pip in a virtual environment (were I got a ssl issue).
-- 
Ticket URL: 
OSGeo4W 
OSGeo4W is the Windows installer and package environment for the OSGeo stack.___
osgeo4w-dev mailing list
osgeo4w-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev

Re: [osgeo4w-dev] [osgeo4w] #692: Python 3.9 on v2 installer breaks part of stdlib

[OSGeo4W]

#692: Python 3.9 on v2 installer breaks part of stdlib
---+
Reporter:  akominlsfi  |   Owner:  osgeo4w-dev@…
Type:  defect  |  Status:  new
Priority:  major   |   Component:  Installer
 Version:  |  Resolution:
Keywords:  |
---+

Comment (by akominlsfi):

 Replying to [comment:5 jef]:
 > How would that work for multiple installed versions of QGIS?

 Oh I see, its possible to select multiple different suffix version of QGIS
 to the same install tree (qgis, qgis-ltr, qgis-ltr-dev etc)? It seems
 adding the pth file and dll path for qgis cant work properly in that case,
 so probably no reason for different behaviour if only one qgis exists.
 That can be easily manually patched in venvs or directly to the install if
 necessary, since correct dll paths will be added by qgis module itself and
 only a pth file is necessary for the module to be found.

 How about the bin path added always as a dll path, and qt if installed?

 Since currently neither the bin or apps python exe works without some
 environment setup beforehand, dll path setup should allow direct execution
 of apps exe and still keep any osgeo shell or qgis bat wrapped calls to
 the bin exe since those set the PYTHONHOME.

 PyQt5 setups its own dll paths only if the ddls are found in PATH, so that
 could benefit  from a sitecustomize so the PATH does not necessarily need
 to contain the qt dlls.

-- 
Ticket URL: 
OSGeo4W 
OSGeo4W is the Windows installer and package environment for the OSGeo stack.
___
osgeo4w-dev mailing list
osgeo4w-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev

Re: [osgeo4w-dev] [osgeo4w] #692: Python 3.9 on v2 installer breaks part of stdlib

[OSGeo4W]

#692: Python 3.9 on v2 installer breaks part of stdlib
---+
Reporter:  akominlsfi  |   Owner:  osgeo4w-dev@…
Type:  defect  |  Status:  new
Priority:  major   |   Component:  Installer
 Version:  |  Resolution:
Keywords:  |
---+

Comment (by jef):

 Replying to [comment:4 akominlsfi]:
 > Replying to [comment:3 jef]:
 > > we have multiple versions of qgis.

 > Well yes, that was specifically for my install since the package name
 and the paths would anyway need templating at install time. I assume
 something like a pre-post install hook could be useful to add/remove
 necessary lines in sitecustomize.py or to create/remove necessary pth
 files?

 How would that work for multiple installed versions of QGIS?

-- 
Ticket URL: 
OSGeo4W 
OSGeo4W is the Windows installer and package environment for the OSGeo stack.
___
osgeo4w-dev mailing list
osgeo4w-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev

Re: [osgeo4w-dev] [osgeo4w] #692: Python 3.9 on v2 installer breaks part of stdlib

[OSGeo4W]

#692: Python 3.9 on v2 installer breaks part of stdlib
---+
Reporter:  akominlsfi  |   Owner:  osgeo4w-dev@…
Type:  defect  |  Status:  new
Priority:  major   |   Component:  Installer
 Version:  |  Resolution:
Keywords:  |
---+

Comment (by akominlsfi):

 Replying to [comment:3 jef]:
 > we have multiple versions of qgis.

 Well yes, that was specifically for my install since the package name and
 the paths would anyway need templating at install time. I assume something
 like a pre-post install hook could be useful to add/remove necessary lines
 in sitecustomize.py or to create/remove necessary pth files?

 Its not necessarily specific to QGIS (it just appeared using the python-
 qgis wrapper bat), but installing only python+gdal has the same issue,
 python module in site-packages and dlls in bin. venvs are broken if using
 the bin exe, or dll paths need to be setup if using the apps exe.

 Does this sound like a viable idea? It would help development a lot if the
 venvs can be supported. If you can give some pointers how this could be
 implemented in the scripts we could possibly make a draft pull request
 later of the proposed change, we'd need this when updating to the next
 QGIS LTR later this year. As it is possible to patch the things on own
 machine after install its not really a blocker, but I think this could
 benefit osgeo4w also?

-- 
Ticket URL: 
OSGeo4W 
OSGeo4W is the Windows installer and package environment for the OSGeo stack.
___
osgeo4w-dev mailing list
osgeo4w-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev

Re: [osgeo4w-dev] [osgeo4w] #692: Python 3.9 on v2 installer breaks part of stdlib

[OSGeo4W]

#692: Python 3.9 on v2 installer breaks part of stdlib
---+
Reporter:  akominlsfi  |   Owner:  osgeo4w-dev@…
Type:  defect  |  Status:  new
Priority:  major   |   Component:  Installer
 Version:  |  Resolution:
Keywords:  |
---+

Comment (by jef):

 Replying to [comment:2 akominlsfi]:
 > I tested this theory on a fresh v2 network install with only qgis-ltr-
 full (3.16.11) selected.
 >
 > Creating a file `/apps/python39/sitecustomize.py` with
 these contents:

 > os.add_dll_directory('/apps/qgis-ltr/bin')

 we have multiple versions of qgis.

-- 
Ticket URL: 
OSGeo4W 
OSGeo4W is the Windows installer and package environment for the OSGeo stack.
___
osgeo4w-dev mailing list
osgeo4w-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev

Re: [osgeo4w-dev] [osgeo4w] #692: Python 3.9 on v2 installer breaks part of stdlib

[OSGeo4W]

#692: Python 3.9 on v2 installer breaks part of stdlib
---+
Reporter:  akominlsfi  |   Owner:  osgeo4w-dev@…
Type:  defect  |  Status:  new
Priority:  major   |   Component:  Installer
 Version:  |  Resolution:
Keywords:  |
---+

Comment (by akominlsfi):

 I tested this theory on a fresh v2 network install with only qgis-ltr-full
 (3.16.11) selected.

 Creating a file `/apps/python39/sitecustomize.py` with
 these contents:

 {{{#!python
 import os

 os.add_dll_directory('/bin')
 os.add_dll_directory('/apps/qgis-ltr/bin')
 os.add_dll_directory('/apps/Qt5/bin')
 }}}

 and a file `/apps/python39/lib/site-packages/qgis.pth` with
 these contents:

 {{{#!python
 /apps/qgis-ltr/python
 }}}

 Then executing python directly from `/apps/python39/python.exe` works with all the imports I figured
 out to test
 (qgis,PyQt5,osgeo.gdal,osgeo.ogr,scipy,numpy,pandas,matplotlib,psycopg2,pyproj)
 that have .pyds and may have required dlls somewhere else in the install
 tree. Not sure about what can be best way to bake in the env variables
 that would be setup by `/etc/ini/*.bat` files, eg.
 GDAL_DATA, PROJ_LIB etc. that might be necessary for the environment to
 function in some cases? Could that be something that is also made in
 customization script, set variable if not already present?

 Also with this setup creating a venv works both without system site
 packages (qgis etc. not importable as would be expected), and with system
 site packages (qgis etc. importable), since the dll path patch is found
 also when executing the venv python exe.

-- 
Ticket URL: 
OSGeo4W 
OSGeo4W is the Windows installer and package environment for the OSGeo stack.
___
osgeo4w-dev mailing list
osgeo4w-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev

Re: [osgeo4w-dev] [osgeo4w] #692: Python 3.9 on v2 installer breaks part of stdlib

[OSGeo4W]

#692: Python 3.9 on v2 installer breaks part of stdlib
---+
Reporter:  akominlsfi  |   Owner:  osgeo4w-dev@…
Type:  defect  |  Status:  new
Priority:  major   |   Component:  Installer
 Version:  |  Resolution:
Keywords:  |
---+

Comment (by akominlsfi):

 Regarding the patching of dll directories I found this
 [https://www.python.org/dev/peps/pep-0648/ PEP #648] for site
 customizations, which seems to be what osgeo4w packaging could benefit a
 lot from (each package that gets installed and needs for example shared
 dlls could add their respective script to the directory).

 Since that PEP is still a draft, sitecustomize or a custom .pth file/files
 (using [https://docs.python.org/3/library/site.html site module]) could be
 way to go for now regarding the patching of dll directories? Location of
 these could be in site-packages or next to the `apps/python37/python.exe`.
 Creating a venv without the `--system-site-packages` flag needs the
 patching anyway if stdlib dlls are not duplicated (for ssl and sqlite3 to
 work), so the customization could be done always by using the exe path.

 This change should preserve the current usage (osgeo4w shell, python-
 qgis.bats, anything else?) and also the python environment would work now
 as a standalone installation directly from the exe that just happens to
 include a lot of stuff in system site-packages without any special preset
 environment needed before execution.

 \\
 Side note: It could also help if the QGIS install added a .pth file to the
 site-packages. The module is currently resolved by the wrapper bat
 extending PYTHONPATH to include `apps/qgis/python`, but since `import
 qgis` now works if its found at all even if the wrapper bat is not used
 (fixed [https://github.com/qgis/QGIS/pull/43345 here]), it could be used
 as if it was a normal site-package in the python environment with the help
 of a .pth file pointing to the module path.

-- 
Ticket URL: 
OSGeo4W 
OSGeo4W is the Windows installer and package environment for the OSGeo stack.
___
osgeo4w-dev mailing list
osgeo4w-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev