Re: Compiled C modules are not found by test suite (Was: Help needed for python-biopython which splits up modules into two packages per Python version)
Hi Piotr,
On Sat, Mar 15, 2014 at 12:52:41PM +0100, Piotr Ożarowski wrote:
pybuild takes care of distutils cannot find files it build in previous
step and has to build them again problem. I just checked
python-biopython and all .so files are in the right location
(yey pybuild! :).
The problem is package's root directory has higher priority on sys.path
and your package has both Bio and BioSQL it there (instead of in src or lib
directory, like other upstreams do... you know... Python standards)
anyway, that's one of the reasons I do this in pybuild:
cd BUILD_DIRECTORY; run_tests
this way files from you package's root directory are not taken into account.
Ahhh, thanks for the explanation.
To fix your problem apply this change (not tested, because clean target
is not doing the right thing and I'm too lazy to fix it):
Sorry for the broken clean target (which is only broken if the build
interrupts in the test itself.) I fixed this in SVN.
- dh_auto_test -- --test --system=custom --test-args='env
DIALIGN2_DIR=/usr/share/dialign EMBOSS_ROOT=/usr/lib/emboss
PYTHONPATH={build_dir} {interpreter} setup.py test'
+ dh_auto_test -- --test --system=custom --test-args='env
DIALIGN2_DIR=/usr/share/dialign EMBOSS_ROOT=/usr/lib/emboss; cd {build_dir};
{interpreter} setup.py test'
Unfortunately this does not work since
env setvariables ;
doese not survive the
cd {build_dir}
I tried
dh_auto_test -- --test --system=custom --test-args='cd {build_dir}; env
DIALIGN2_DIR=/usr/share/dialign EMBOSS_ROOT=/usr/lib/emboss {interpreter}
../setup.py test'
instead but this fails as well. :-(
Any better idea?
Thanks for any help
Andreas.
--
http://fam-tille.de
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140317134447.gn10...@an3as.eu
Re: Compiled C modules are not found by test suite (Was: Help needed for python-biopython which splits up modules into two packages per Python version)
Hi Éric, On Fri, Mar 14, 2014 at 07:15:26PM -0400, Éric Araujo wrote: There are two ways to fix that: either make distutils build the extensions alongside the Python modules, or make the test import all code from the build directory. I’m surprised that this isn’t a common issue for PMPT. +1 I admit that I was somehow afraid to ask a FAQ and a quite simple answer to it since running test suitws should be normal and there are several packages containing C modules. I tried the first solution and did not see a difference in tests: both times, 223 tests were run successfully. I’d like to reproduce the errors you mention to be sure that any change I make is actually a fix. I guess this is the solution since if the C modules are not found the test is just skipped (and does not fail). If you could provide this very patch I'd be really glad. Unfortunately, there is a long-standing bug in distutils where extensions modules built alongside the Python modules are not found by the install command. This means that you’d have to compile twice (once in-place to run the tests, one to a build directory to install to debian/tmp/etc.), or that you’d prefer the other fix. I think I'll try what you tested to get 223 tests running and see how far this helps. Kind regards Andreas. -- http://fam-tille.de -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140315094319.gg15...@an3as.eu
Re: Compiled C modules are not found by test suite (Was: Help needed for python-biopython which splits up modules into two packages per Python version)
Hi, I tried the first solution and did not see a difference in tests: both times, 223 tests were run successfully. I’d like to reproduce the errors you mention to be sure that any change I make is actually a fix. I guess this is the solution since if the C modules are not found the test is just skipped (and does not fail). I expected to see fewer tests run when the C modules are found. I was not able to find one example test that runs if a C module is found and skipped if not. If you could provide this very patch I'd be really glad. Run “python build_ext --inplace build test” for tests, “python install” to copy all files to the debian/tmp/etc directory. This will avoid the other bug I mention (http://bugs.python.org/issue5977#msg213606). I did not use the debian/rules makefile, but modifying line 45 to add “build_ext --inplace build” before “test” should do the trick. (Putting the option in a setup.cfg file as I initially advised would cause the error I alluded to when running “setup.py install”.) Cheers -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/53242318.4090...@netwok.org
Re: Compiled C modules are not found by test suite (Was: Help needed for python-biopython which splits up modules into two packages per Python version)
On 15.03.2014 00:15, Éric Araujo wrote: Hello, I got the code and the debian directory. I confirmed that extension modules are in the build directory, not alongside the Python modules, so they can’t be imported from tests. There are two ways to fix that: either make distutils build the extensions alongside the Python modules, or make the test import all code from the build directory. I’m surprised that this isn’t a common issue for PMPT. a typical approach is to not test in dh_auto_test but to install and then test from debian/tmp by setting PYTHONPATH -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/532430c0.9030...@googlemail.com
Re: Compiled C modules are not found by test suite (Was: Help needed for python-biopython which splits up modules into two packages per Python version)
[Éric Araujo, 2014-03-15]
Hi,
I tried the first solution and did not see a difference in tests:
both times, 223 tests were run successfully. I’d like to reproduce
the errors you mention to be sure that any change I make is actually
a fix.
I guess this is the solution since if the C modules are not found the
test is just skipped (and does not fail).
I expected to see fewer tests run when the C modules are found. I was not
able to find one example test that runs if a C module is found and skipped
if not.
If you could provide this very patch I'd be really glad.
Run “python build_ext --inplace build test” for tests, “python install” to
copy all files to the debian/tmp/etc directory. This will avoid the other
bug I mention (http://bugs.python.org/issue5977#msg213606).
I did not use the debian/rules makefile, but modifying line 45 to add
“build_ext --inplace build” before “test” should do the trick.
(Putting the option in a setup.cfg file as I initially advised would cause
the error I alluded to when running “setup.py install”.)
pybuild takes care of distutils cannot find files it build in previous
step and has to build them again problem. I just checked
python-biopython and all .so files are in the right location
(yey pybuild! :).
The problem is package's root directory has higher priority on sys.path
and your package has both Bio and BioSQL it there (instead of in src or lib
directory, like other upstreams do... you know... Python standards)
anyway, that's one of the reasons I do this in pybuild:
cd BUILD_DIRECTORY; run_tests
this way files from you package's root directory are not taken into account.
To fix your problem apply this change (not tested, because clean target
is not doing the right thing and I'm too lazy to fix it):
- dh_auto_test -- --test --system=custom --test-args='env
DIALIGN2_DIR=/usr/share/dialign EMBOSS_ROOT=/usr/lib/emboss
PYTHONPATH={build_dir} {interpreter} setup.py test'
+ dh_auto_test -- --test --system=custom --test-args='env
DIALIGN2_DIR=/usr/share/dialign EMBOSS_ROOT=/usr/lib/emboss; cd {build_dir};
{interpreter} setup.py test'
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
signature.asc
Description: Digital signature
Re: Compiled C modules are not found by test suite (Was: Help needed for python-biopython which splits up modules into two packages per Python version)
On Wed, Mar 12, 2014 at 01:20:38AM -0400, Éric Araujo wrote:
Le 05/03/2014 10:31, Andreas Tille a écrit :
I have noted another problem: I worked a bit
on the tests and noticed that those tests that are including compiled C
code are failing. Is there anything in addition I need to do to make
the C object code visible to the Python modules?
Can non-DD see the source code somewhere?
Theoretically you should get the code via
debcheckout python-biopython
Unfortunately currently this is a bit broken (I have filed a bug report
about this) since the URL redirector on svn.debian.org needs an explicit
/svn dir after the host name and thus you need to replace
svn co
svn://svn.debian.org/debian-med/trunk/packages/python-biopython/trunk/
python-biopython
by
svn co
svn://svn.debian.org/svn/debian-med/trunk/packages/python-biopython/trunk/
python-biopython
Since debcheckout failed in this step you manually need to get the
source via
apt-get source python-biopython
and replace the debian/ dir in it by the current status in SVN.
IIUC you have a Python package and C extensions; when running the
tests during the package build (or is it against the installed
package?),
It is my exact problem that I'm running the test at package build
time.
the C extensions are not found. One explanation for that
could be that the C extensions are built in a “build” directory
instead of inside the Python package, and thus aren’t readily
importable. If the build uses distutils/setuptools, there is an
option to control that:
# file setup.cfg next to setup.py
[build_ext]
inplace = 1
This hint sounds promising but setup.py only says
setup_args = {
name: 'biopython',
version: __version__,
author: 'The Biopython Consortium',
author_email: 'biopyt...@biopython.org',
url: 'http://www.biopython.org/',
description: 'Freely available tools for computational molecular
biology.',
download_url: 'http://biopython.org/DIST/',
cmdclass: {
install: install_biopython,
build_py: build_py_biopython,
build_ext: build_ext_biopython,
test: test_biopython,
},
...
Perhaps I could set build_ext_biopython somehow properly? Since I have
no experience with distutils/setuptools a proper patch to the file would
save me some time with experimenting.
Thanks for your comments in any case
Andreas.
--
http://fam-tille.de
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140314100715.gm25...@an3as.eu
Re: Compiled C modules are not found by test suite (Was: Help needed for python-biopython which splits up modules into two packages per Python version)
Hello, I got the code and the debian directory. I confirmed that extension modules are in the build directory, not alongside the Python modules, so they can’t be imported from tests. There are two ways to fix that: either make distutils build the extensions alongside the Python modules, or make the test import all code from the build directory. I’m surprised that this isn’t a common issue for PMPT. I tried the first solution and did not see a difference in tests: both times, 223 tests were run successfully. I’d like to reproduce the errors you mention to be sure that any change I make is actually a fix. Unfortunately, there is a long-standing bug in distutils where extensions modules built alongside the Python modules are not found by the install command. This means that you’d have to compile twice (once in-place to run the tests, one to a build directory to install to debian/tmp/etc.), or that you’d prefer the other fix. Cheers -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/53238d8e.4070...@netwok.org
Re: Compiled C modules are not found by test suite (Was: Help needed for python-biopython which splits up modules into two packages per Python version)
Hello, Le 05/03/2014 10:31, Andreas Tille a écrit : I have noted another problem: I worked a bit on the tests and noticed that those tests that are including compiled C code are failing. Is there anything in addition I need to do to make the C object code visible to the Python modules? Can non-DD see the source code somewhere? IIUC you have a Python package and C extensions; when running the tests during the package build (or is it against the installed package?), the C extensions are not found. One explanation for that could be that the C extensions are built in a “build” directory instead of inside the Python package, and thus aren’t readily importable. If the build uses distutils/setuptools, there is an option to control that: # file setup.cfg next to setup.py [build_ext] inplace = 1 Regards -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/531feea6.6040...@netwok.org
Compiled C modules are not found by test suite (Was: Help needed for python-biopython which splits up modules into two packages per Python version)
Hi again On Tue, Mar 04, 2014 at 07:18:08PM +0100, Andreas Tille wrote: you can force pybuild to move BioSQL files to a different location (hint: --ext-dest-dir --ext-pattern or equivalent PYBUILD_* vars) but IMO it's cleaner to use *.install files for that Hmmm, you say *.install files are cleaner - but this exactly is my problem since the *.install files do a really bad job in this case and mix up the Python versions. I do not have the slightest idea why. Besides that the mysterious riddle why the python2 modules all end up in the python3 packages and vice versa is not solved yet and I have no idea why this happens since this $ grep Bio debian/*.install debian/python3-biopython.install:usr/lib/python3*/*-packages/Bio debian/python3-biopython-sql.install:usr/lib/python3*/*-packages/BioSQL debian/python-biopython.install:usr/lib/python[^3]*/*-packages/Bio debian/python-biopython-sql.install:usr/lib/python[^3]*/*-packages/BioSQL seems to me the correct way to get files in the proper location (please correct me if I'm wrong) I have noted another problem: I worked a bit on the tests and noticed that those tests that are including compiled C code are failing. Is there anything in addition I need to do to make the C object code visible to the Python modules? Kind regards Andreas. -- http://fam-tille.de -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140305153122.gk...@an3as.eu
