Re: PYTHONPATH with cmake build
Hi PICCA (2022.08.30_16:02:01_+)
> cd build
> cmake ../modules/dxtbx
> cmake --build . --target install
> pip install ../modules/dxtbx
This package essentially has 2 build systems, camke for the C++ stuff
and setuptools for the pure-python. I don't see any integration between
the two (which is quite annoying, the upstream really should fix that).
pybuild knows how to drive distutils, and knows how to drive cmake, but
not both together. Ideally, you do want to use pybuild when building
Python extensions, so you can do them for all supported Python versions.
So, can I suggest something like this?
#! /usr/bin/make -f
export PYBUILD_NAME=dxtbx
export PYBUILD_SYSTEM=distutils
export PYBUILD_AFTER_CONFIGURE=cmake -DPython_EXECUTABLE=/usr/bin/{interpreter}
-S . -B {build_dir}/lib
export PYBUILD_AFTER_BUILD=make -C {build_dir}/lib
export PYBUILD_BEFORE_TEST=cp {build_dir}/lib/lib/*.so {build_dir}
%:
dh $@ --with python3 --buildsystem=pybuild
You'll need something for the install too, but I didn't get that far, as
the tests fail (due to missing dependencies in Debian).
SR
--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272
Re: PYTHONPATH with cmake build
On Tue, 30 Aug 2022 19:02:24 +0200 (CEST) PICCA Frederic-Emmanuel wrote: > > /usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just > > obj-$DEB_HOST_GNU_TYPE. > > > Thanks for the info, if I an not wrong during the build process we > can setup a new builddir. So is is possible to obtain the real > builddir during the build ? PYTHONPATH can be a relative path, there is no need to try to identify the full build path (that can introduce reproducibility issues). No need to force a change to the build dir either. So just obj-$DEB_HOST_GNU_TYPE -- Neil Williams li...@codehelp.co.uk https://linux.codehelp.co.uk pgpby3iw0Kq0m.pgp Description: OpenPGP digital signature
Re: Lintian info message "hardening-no-bindnow" with vanilla debian/rules
I: python3-pyxdameraulevenshtein: hardening-no-bindnow [usr/lib/python3/dist-packages/pyxdameraulevenshtein.cpython-310-x86_64-linux-gnu.so] and there is nothing about CFLAGS or the like in the setup.py file. So if having this hardening flag enabled is a good thing, it should probably be enabled somewhere within the pybuild system, rather than every individual package with an extension file doing it. Hardening is generally a good thing, but can break code in subtle ways. I suppose that's why it was decided that enabling it by default in Debian was deemed too risky. Enabling it is quite easy, though: Just add export DEB_BUILD_MAINT_OPTIONS = hardening=+all near the top of your d/rules file. Some build systems may require additional flags, as documented here: https://wiki.debian.org/Hardening Also, note that hardening-no-bindnow is an Informational message, so not strictly something that needs to be acted upon: https://lintian.debian.org/tags/hardening-no-bindnow YMMV.
Re: PYTHONPATH with cmake build
> Oh. In this case setting PYTHONPATH (if it works, and Im not 100% sure > it > will) sounds like a better option. > So, the cmake build path is, as far as I know, defined in > /usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just > obj-$DEB_HOST_GNU_TYPE. thanks for the info. It seems to me that the builddir can be set directly in the ruels file with --builddir= So do you know a way to extract the real builddir during the build process ? Cheers
Re: PYTHONPATH with cmake build
> /usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just > obj-$DEB_HOST_GNU_TYPE. Thanks for the info, if I an not wrong during the build process we can setup a new builddir. So is is possible to obtain the real builddir during the build ?
Re: PYTHONPATH with cmake build
On Tue, Aug 30, 2022 at 06:02:01PM +0200, PICCA Frederic-Emmanuel wrote: > > When trying to run tests you should look how does the upstream intend to > run them. > > > Yes they are building inplace the module like this > > from https://github.com/dials/dxtbx/blob/main/.azure-pipelines/unix-build.yml > > > # Build dxtbx > - bash: | > set -e > . conda_base/bin/activate > set -ux > mkdir build > cd build > cmake ../modules/dxtbx > cmake --build . --target install > pip install ../modules/dxtbx Oh. In this case setting PYTHONPATH (if it works, and I'm not 100% sure it will) sounds like a better option. So, the cmake build path is, as far as I know, defined in /usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just "obj-$DEB_HOST_GNU_TYPE". -- WBR, wRAR signature.asc Description: PGP signature
Re: PYTHONPATH with cmake build
> When trying to run tests you should look how does the upstream intend to
run them.
Yes they are building inplace the module like this
from https://github.com/dials/dxtbx/blob/main/.azure-pipelines/unix-build.yml
# Build dxtbx
- bash: |
set -e
. conda_base/bin/activate
set -ux
mkdir build
cd build
cmake ../modules/dxtbx
cmake --build . --target install
pip install ../modules/dxtbx
displayName: Build dxtbx
workingDirectory: $(Pipeline.Workspace)
then
# Finally, run the full regression test suite
- bash: |
set -e
. conda_base/bin/activate
set -ux
export DIALS_DATA=${PWD}/data
cd modules/dxtbx
export PYTHONDEVMODE=1
pytest -v -ra -n auto --basetemp="$(Pipeline.Workspace)/tests"
--durations=10 \
--cov=dxtbx --cov-report=html --cov-report=xml --cov-branch \
--timeout=5400 --regression || echo "##vso[task.complete
result=Failed;]Some tests failed"
displayName: Run tests
workingDirectory: $(Pipeline.Workspace)
--
WBR, wRAR
Re: PYTHONPATH with cmake build
On Tue, Aug 30, 2022 at 05:26:01PM +0200, PICCA Frederic-Emmanuel wrote: > Hello Andrey > > > Does the same happen when you run the test in the source tree manually? > > I do not know When trying to run tests you should look how does the upstream intend to run them. -- WBR, wRAR signature.asc Description: PGP signature
Re: PYTHONPATH with cmake build
Hello Andrey > Does the same happen when you run the test in the source tree manually? I do not know, I am in the process to build the package in sbuild so I just try to fix the build process. If I have hard code the lib path the test are running failling for other missing modules, but this is ok. so my question is more how can I obtain the cmake default build path from the rules file ? cheers
Re: PYTHONPATH with cmake build
On Tue, Aug 30, 2022 at 03:58:26PM +0200, PICCA Frederic-Emmanuel wrote: > Hello, I am packaging a python extension which use cmake as build system > > here the repo > > https://salsa.debian.org/science-team/dxtbx > > I try to activate the test but this cause this kind of trouble > > During handling of the above exception, another exception occurred: > /usr/lib/python3.10/importlib/__init__.py:126: in import_module > return _bootstrap._gcd_import(name[level:], package, level) > tests/serialize/test_xds.py:8: in > from dxtbx.imageset import ImageSetFactory > dxtbx/imageset.py:5: in > import dxtbx.format.image # noqa: F401, import dependency for unpickling > dxtbx/format/image.py:8: in > from dxtbx_format_image_ext import * # type: ignore # isort:skip # noqa: > F403 > E ModuleNotFoundError: No module named 'dxtbx_format_image_ext' > > Indeed the extension are build here. > > cd /<>/obj-x86_64-linux-gnu/src/dxtbx && /usr/bin/cmake -E > cmake_link_script CMakeFiles/dxtbx_flumpy.dir/link.txt --verbose=1 > /usr/bin/c++ -fPIC -g -O2 -ffile-prefix-map=/<>=. > -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time > -D_FORTIFY_SOURCE=2 -flto -Wl,-z,relro -shared -o > ../../lib/dxtbx_flumpy.cpython-310-x86_64-linux-gnu.so > CMakeFiles/dxtbx_flumpy.dir/boost_python/flumpy.cc.o > /usr/lib/x86_64-linux-gnu/libboost_python310.so.1.74.0 > lto-wrapper: warning: using serial compilation of 4 LTRANS jobs > lto-wrapper: note: see the ‘-flto’ option documentation for more information > [ 90%] Linking CXX shared module ../../lib/dxtbx_format_image_ext.so > > so I need to add the /<>/obj-x86_64-linux-gnu/lib directory to > the PYTHONPATH. > > do you know how to do this ? Does the same happen when you run the test in the source tree manually? -- WBR, wRAR signature.asc Description: PGP signature
PYTHONPATH with cmake build
Hello, I am packaging a python extension which use cmake as build system here the repo https://salsa.debian.org/science-team/dxtbx I try to activate the test but this cause this kind of trouble During handling of the above exception, another exception occurred: /usr/lib/python3.10/importlib/__init__.py:126: in import_module return _bootstrap._gcd_import(name[level:], package, level) tests/serialize/test_xds.py:8: in from dxtbx.imageset import ImageSetFactory dxtbx/imageset.py:5: in import dxtbx.format.image # noqa: F401, import dependency for unpickling dxtbx/format/image.py:8: in from dxtbx_format_image_ext import * # type: ignore # isort:skip # noqa: F403 E ModuleNotFoundError: No module named 'dxtbx_format_image_ext' Indeed the extension are build here. cd /<>/obj-x86_64-linux-gnu/src/dxtbx && /usr/bin/cmake -E cmake_link_script CMakeFiles/dxtbx_flumpy.dir/link.txt --verbose=1 /usr/bin/c++ -fPIC -g -O2 -ffile-prefix-map=/<>=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -flto -Wl,-z,relro -shared -o ../../lib/dxtbx_flumpy.cpython-310-x86_64-linux-gnu.so CMakeFiles/dxtbx_flumpy.dir/boost_python/flumpy.cc.o /usr/lib/x86_64-linux-gnu/libboost_python310.so.1.74.0 lto-wrapper: warning: using serial compilation of 4 LTRANS jobs lto-wrapper: note: see the ‘-flto’ option documentation for more information [ 90%] Linking CXX shared module ../../lib/dxtbx_format_image_ext.so so I need to add the /<>/obj-x86_64-linux-gnu/lib directory to the PYTHONPATH. do you know how to do this ? thanks for your help Fred
Re: Join the team
Hi Claudius (2022.08.30_09:56:12_+) > I would like to join the python team to help maintain packages, currently > related to tpm2-pytss but will potentially extend to other python packages > like python-cryptography and others. Added, welcome! SR -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
Join the team
Hi, I would like to join the python team to help maintain packages, currently related to tpm2-pytss but will potentially extend to other python packages like python-cryptography and others. My salsa account is: cmhe And I read and agree to the Debian Python Team Policy [1] cheers, Claudius [1] https://salsa.debian.org/python-team/tools/python-modules/blob/master/policy.rst
Lintian info message "hardening-no-bindnow" with vanilla debian/rules
Hi! A package I maintain within the team (python3-pyxdameraulevenshtein) gives the following lintian message: I: python3-pyxdameraulevenshtein: hardening-no-bindnow [usr/lib/python3/dist-packages/pyxdameraulevenshtein.cpython-310-x86_64-linux-gnu.so] The debian/rules file is very bland, essentially: %: dh $@ --buildsystem=pybuild and there is nothing about CFLAGS or the like in the setup.py file. So if having this hardening flag enabled is a good thing, it should probably be enabled somewhere within the pybuild system, rather than every individual package with an extension file doing it. Or have I missed something? Best wishes, Julian
Re: Joining the team
Hi Emanuele (2022.08.29_19:39:17_+) > I'd like to join the debian-python team to help with general QA work on > all packages. > > My salsa login is ema. Added, welcome! SR -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
