Re: Python3 modules not built for all supported Python versions
Hi Emilio, * Emilio Pozuelo Monfort [2020-03-30 17:47]: I've heard pybuild now has a cmake backend, so theoretically you could do something like %: dh $@ --buildsystem=pybuild --system=cmake [..] I don't know if I'm missing an argument to dh_python3 so that it knows the python version, or even if there's a better workaround. But perhaps pybuild should be doing this automatically between the dh_auto_install calls so that this kind of workarounds aren't necessary. Thanks for looking into this and providing a patch! I polished it a little: https://salsa.debian.org/science-team/ros-geometry2/-/commit/11739091abe58007485772492fddbdc1a12a59c6 and uploaded a working version to the archive. Will fix all ros-* packages now. Cheers Jochen signature.asc Description: PGP signature
Re: Python3 modules not built for all supported Python versions
> I don't know if I'm missing an argument to dh_python3 so that it knows the > python version, or even if there's a better workaround. But perhaps pybuild better workaround is to force cmake to install into versioned dist-packages (that's what I do in distutils) as I didn't find a reliable way to read Python version/architecture/tag/etc. from .so file > should be doing this automatically between the dh_auto_install calls so that > this kind of workarounds aren't necessary. it's actually a nice idea - pybuild can check if /usr/lib/python3.X/dist-packages/ is empty and there are .so files in /usr/lib/python3/dist-packages/ and move them temporarily to /usr/lib/python3.X/dist-packages/ (it knows which interpreter is used at this point) and let dh_python3 rename them later > Piotr, is this a bug in pybuild, or am I doing something wrong? well, cmake is not really easy to work with, they provide -DPYTHON_LIBRARY:FILEPATH… but they don't use it or use completely different name… it basically is different for each library I checked (and in different official documentation versions I read, sic!)
Re: Python3 modules not built for all supported Python versions
On 30/03/2020 16:08, Simon McVittie wrote: > On Mon, 30 Mar 2020 at 15:30:01 +0200, Johannes Schauer wrote: >> does this mean that build-depending on python3-dev is wrong in general and >> should instead be replaced by build-depending on python3-all-dev? > > It is only wrong for packages that build Python 3 extensions (binary > modules) that are intended to be loadable by all supported Python > 3 versions (roughly: `find /usr/lib/python3/dist-packages -name '*.so'`). > > For packages that embed Python 3, like the versions of vim that > have Python scripting support, or packages that use a Python 3 > extension as an internal implementation detail of some tool, like > gobject-introspection, my understanding is that build-depending > on python3-dev continues to be appropriate. These extensions would > ideally be installed in a private directory, like gobject-introspection's > /usr/lib/x86_64-linux-gnu/gobject-introspection/giscanner/_giscanner.cpython-38-x86_64-linux-gnu.so > - but I know some upstreams and some downstream maintainers (arguably > incorrectly) package private extensions as though they were public > extensions, because the mechanics of doing so are much simpler. > >> For example the package src:ros-geometry2 has a super simple >> dh-style rules file, basically just doing: >> >> %: >> dh $@ --buildsystem=cmake --with python3 >> >> What would I have to change to successfully fix this problem? > > The general answer is that you would have to build it repeatedly in a > loop, with each supported version of Python 3 in turn. I am not aware > of a way to do this in a similarly simple rules file. I've heard pybuild now has a cmake backend, so theoretically you could do something like %: dh $@ --buildsystem=pybuild --system=cmake Since I couldn't find any package in the archive that used pybuild with the cmake backend, I have looked at this specific package and came up with the attached debdiff. I added the dh_auto_install hack because the cmake build system creates the python extension as _tf2.so, and is only renamed by dh_python3 (as can be seen in current build logs). However when building for both python3.7 and python3.8, the last installation will override the former, and only one _tf2.so will be available. However that workaround proved to not be enough, because dh_install3 renames both the 3.7 and 3.8 versions as 3.8: make[2]: Leaving directory '/build/ros-geometry2-0.6.6/.pybuild/cpython3_3.7/build' I: pybuild pybuild:310: dh_python3 I: dh_python3 fs:343: renaming _tf2.so to _tf2.cpython-38-x86_64-linux-gnu.so [...] make[2]: Leaving directory '/build/ros-geometry2-0.6.6/.pybuild/cpython3_3.8/build' I: pybuild pybuild:310: dh_python3 W: dh_python3 fs:340: destination file exist, cannot rename _tf2.so to _tf2.cpython-38-x86_64-linux-gnu.so Resulting in these installed files: -rw-r--r-- root/root 69112 2020-03-30 14:56 ./usr/lib/python3/dist-packages/tf2_py/_tf2.cpython-38-x86_64-linux-gnu.so -rw-r--r-- root/root 69128 2020-03-30 14:56 ./usr/lib/python3/dist-packages/tf2_py/_tf2.so I don't know if I'm missing an argument to dh_python3 so that it knows the python version, or even if there's a better workaround. But perhaps pybuild should be doing this automatically between the dh_auto_install calls so that this kind of workarounds aren't necessary. Piotr, is this a bug in pybuild, or am I doing something wrong? Cheers, Emilio diff -Nru ros-geometry2-0.6.6/debian/changelog ros-geometry2-0.6.6/debian/changelog --- ros-geometry2-0.6.6/debian/changelog2020-01-18 21:51:17.0 +0100 +++ ros-geometry2-0.6.6/debian/changelog2020-03-30 16:56:18.0 +0200 @@ -1,3 +1,10 @@ +ros-geometry2 (0.6.6-1.1) UNRELEASED; urgency=medium + + * Non-maintainer upload. + * Build for all supported versions of python3. + + -- Emilio Pozuelo Monfort Mon, 30 Mar 2020 16:56:18 +0200 + ros-geometry2 (0.6.6-1) unstable; urgency=medium * New upstream version 0.6.6 diff -Nru ros-geometry2-0.6.6/debian/control ros-geometry2-0.6.6/debian/control --- ros-geometry2-0.6.6/debian/control 2020-01-18 21:51:17.0 +0100 +++ ros-geometry2-0.6.6/debian/control 2020-03-30 16:56:18.0 +0200 @@ -6,7 +6,7 @@ Leopold Palomo-Avellaneda Build-Depends: debhelper-compat (= 12), dh-exec, catkin (>= 0.7.14-4), libroscpp-core-dev, ros-message-generation, libstd-msgs-dev, - python3-dev, python3-setuptools, dh-python, + python3-all-dev, python3-setuptools, dh-python, libgeometry-msgs-dev, ros-actionlib-msgs, libconsole-bridge-dev, python3-rospy (>= 1.14.3+ds1-7), python3-rosgraph (>= 1.14.3+ds1-7), libactionlib-dev, librosconsole-dev, diff -Nru ros-geometry2-0.6.6/debian/rules ros-geometry2-0.6.6/debian/rules --- ros-geometry2-0.6.6/debian/rules2019-10-25 17:04:01.0 +0200 +++ ros-geometry2-0.6.6/debian/rules2020-03-30 16:56:18.0 +0200
Re: Python3 modules not built for all supported Python versions
On Mon, 30 Mar 2020 at 15:30:01 +0200, Johannes Schauer wrote: > does this mean that build-depending on python3-dev is wrong in general and > should instead be replaced by build-depending on python3-all-dev? It is only wrong for packages that build Python 3 extensions (binary modules) that are intended to be loadable by all supported Python 3 versions (roughly: `find /usr/lib/python3/dist-packages -name '*.so'`). For packages that embed Python 3, like the versions of vim that have Python scripting support, or packages that use a Python 3 extension as an internal implementation detail of some tool, like gobject-introspection, my understanding is that build-depending on python3-dev continues to be appropriate. These extensions would ideally be installed in a private directory, like gobject-introspection's /usr/lib/x86_64-linux-gnu/gobject-introspection/giscanner/_giscanner.cpython-38-x86_64-linux-gnu.so - but I know some upstreams and some downstream maintainers (arguably incorrectly) package private extensions as though they were public extensions, because the mechanics of doing so are much simpler. > For example the package src:ros-geometry2 has a super simple > dh-style rules file, basically just doing: > > %: > dh $@ --buildsystem=cmake --with python3 > > What would I have to change to successfully fix this problem? The general answer is that you would have to build it repeatedly in a loop, with each supported version of Python 3 in turn. I am not aware of a way to do this in a similarly simple rules file. smcv
Re: Python3 modules not built for all supported Python versions
Hi, Quoting Emilio Pozuelo Monfort (2020-03-30 13:24:01) > We've just finished the transition to python3.8 as the default python3 > interpreter, which was a bit difficult due to some autopkgtest regressions in > a few rdeps, and to the fact that many modules only build their extensions > for the default python version, which means they have a strict dependency on > the python3 version[1] and they need to be rebuilt and migrated in lockstep > with python3-defaults. > > I have analyzed this based on current sid amd64 contents and have come up with > the following packages that don't ship extensions for both py3.7 and 3.8 > (which > are the currently supported versions). Note that pure python packages that > don't > build C extensions are not affected. > > It would be great if this situation can be improved in order to help with > future > python transitions. Building for all the supported python versions can be done > by build-depending on python3-all-dev and compiling your package (or just the > python bits) with PYTHON pointing to each version. Depending on your package's > build system, this could be largely automated using some helper, such as > pybuild. If you don't know how to add support for your package, feel free to > ask. does this mean that build-depending on python3-dev is wrong in general and should instead be replaced by build-depending on python3-all-dev? Could wrong build dependencies or binary packages which have a strict dependency on a python3 version not easily be detected by lintian? I'm not an expert in Python module packaging -- it just so happens that some packages I maintain offer Python bindings. Do you have a diff of a source package that successfully made the transition so that I know what I would have to change? For example the package src:ros-geometry2 has a super simple dh-style rules file, basically just doing: %: dh $@ --buildsystem=cmake --with python3 What would I have to change to successfully fix this problem? Thanks! cheers, josch signature.asc Description: signature
Re: Python3 modules not built for all supported Python versions
On 3/30/20 1:24 PM, Emilio Pozuelo Monfort wrote: > Hi, > > We've just finished the transition to python3.8 as the default python3 > interpreter, which was a bit difficult due to some autopkgtest regressions in > a > few rdeps, and to the fact that many modules only build their extensions for > the > default python version, which means they have a strict dependency on the > python3 > version[1] and they need to be rebuilt and migrated in lockstep with > python3-defaults. > > I have analyzed this based on current sid amd64 contents and have come up with > the following packages that don't ship extensions for both py3.7 and 3.8 > (which > are the currently supported versions). Note that pure python packages that > don't > build C extensions are not affected. > > It would be great if this situation can be improved in order to help with > future > python transitions. Building for all the supported python versions can be done > by build-depending on python3-all-dev and compiling your package (or just the > python bits) with PYTHON pointing to each version. Depending on your package's > build system, this could be largely automated using some helper, such as > pybuild. If you don't know how to add support for your package, feel free to > ask. Yes, that would be useful. However there are only limited time windows when you can do *and* check such work, i.e. when the python3-defaults package has two supported python3 versions. Filing and fixing these issues would be nice. Plus for most of those packages you have to modify the build system to build the package for each python3 version, or to just build the extension for the non-default version. I assume we still could keep 3.7 as a supported python3 version for some time, if people want to work on those issues. But people probably won't have the time to work on that when we introduce 3.9. Matthias
Python3 modules not built for all supported Python versions
Hi, We've just finished the transition to python3.8 as the default python3 interpreter, which was a bit difficult due to some autopkgtest regressions in a few rdeps, and to the fact that many modules only build their extensions for the default python version, which means they have a strict dependency on the python3 version[1] and they need to be rebuilt and migrated in lockstep with python3-defaults. I have analyzed this based on current sid amd64 contents and have come up with the following packages that don't ship extensions for both py3.7 and 3.8 (which are the currently supported versions). Note that pure python packages that don't build C extensions are not affected. It would be great if this situation can be improved in order to help with future python transitions. Building for all the supported python versions can be done by build-depending on python3-all-dev and compiling your package (or just the python bits) with PYTHON pointing to each version. Depending on your package's build system, this could be largely automated using some helper, such as pybuild. If you don't know how to add support for your package, feel free to ask. Cheers, Emilio [1] e.g. python3 (>= 3.7), python3 (<< 3.8) "Adam C. Powell, IV" netgen (U) A. Maitland Bottoms gr-air-modes gr-fcdproplus (U) gr-fosphor gr-gsm (U) gr-iio gr-iqbal gr-limesdr (U) gr-osmosdr gr-rds quisk (U) uhd Adam Borowski btrfs-progs Agustin Henze logbook Alan Boudreault mapserver (U) Alastair McKinstry ecflow pyferret xdmf Anders Waananen nordugrid-arc (U) Andreas Bombe gr-limesdr (U) soapysdr (U) Andreas Metzler hugin (U) libvigraimpex (U) Andreas Tille atropos (U) conda-package-handling (U) epigrass (U) libsbml (U) obitools (U) python-thinc (U) umis (U) Andrew Bartlett samba (U) Andrius Merkys openbabel (U) Anthony Wong pycangjie (U) Anton Gladky python-demgengeo (U) Aron Xu ukui-menus (U) Axel Beckert gnudatalanguage (U) Balint Reczey libcec (U) Barak A. Pearlmutter mlpack (U) Bas Couwenberg mapserver (U) qgis (U) Bastien Roucariès pythonmagick (U) Benjamin Drung rdma-core Bernd Zeimetz ceph (U) Boyuan Yang libplist (U) Carl Fürstenberg pythonmagick (U) Carsten Schoenert kicad (U) kopanocore (U) Ceph Packaging Team ceph Christoph Berg gr-limesdr (U) gr-soapy (U) postgresql-multicorn (U) quisk (U) Christoph Egger fife (U) python-enet Christopher Schramm blueman Daniel Kahn Gillmor fontforge (U) Daniel Leidert openbabel (U) Danny Edel borgbackup (U) Davide Viti fontforge (U) Debian 3D-Printer Packaging Team <3dprinter-gene...@lists.alioth.debian.org> printrun Debian Astronomy Team astrometry.net gnudatalanguage Debian Borg Collective borgbackup Debian DNS Team ldns Debian Edu Packaging Team sdaps Debian Electronics Team kicad Debian Fonts Task Force fontforge Debian Fonts Task Force psautohint Debian Games Team cegui-mk2 fife Debian GIS Project mapserver qgis saga Debian GNOME Maintainers glom libpwquality Debian Hamradio Maintainers gr-fcdproplus gr-gsm gr-limesdr gr-soapy quisk soapysdr Debian Input Method Team pycangjie Debian LibreOffice Maintainers libixion liborcus Debian Med Packaging Team atropos biosig4c++ conda-package-handling epigrass gdcm insighttoolkit4 libsbml obitools pymia simpleitk umis vtk-dicom Debian Multimedia Maintainers csound libopenshot openvdb Debian PhotoTools Maintainers hugin opencolorio openimageio Debian PostgreSQL Maintainers postgresql-multicorn Debian Printing Team hplip Debian Python Modules Team portio pyodbc (U) pythonmagick Debian QA Group link-grammar Debian Samba Maintainers ldb samba talloc tdb Debian Science Maintainers caffe libvigraimpex mlpack netgen openturns orocos-kdl python-thinc ros-geometry2 ros-image-common ros-ros-comm ros-rviz ros-vision-opencv siconos veusz Debian Science Team apertium apertium-lex-tools apriltag cg3 cryptominisat getfem++ lttoolbox morse-simulator neuron opencv plplot python-demgengeo sagemath vtk7 Debian Security Tools libbde libesedb libevt libevtx libewf libfsapfs libfsntfs libfvde libfwnt libfwsi liblnk libmsiecf libolecf libpff libqcow libregf libscca libsigscan libsmdev libsmraw libvhdi libvmdk libvshadow libvslvm Debian SSSD Team sssd Debian SSSD Team pam-wrapper Debichem Team avogadrolibs chemps2 openbabel rdkit Deepak Tripathi pyodbc Denis Barbier openturns (U) Didier Raboud hplip (U) Dima Kogan apriltag (U) Dmitry Smirnov
