Hi again,
I am just wondering how to do unit testing on extensions built using
Cython with py.test.
My problem: The extension module I am working on is installed in our
global python environment (inside the build slaves as well as on local
machines) already. Before installing a new version of the extension
module, I'd like to run the unit tests. However, that way the original
module is tested.
I am trying to illustrate.
Doing TDD I create a new module which will fail the test:
torsten@sharokan:~/foo$ cat foo.pyx
def it_works():
return False
torsten@sharokan:~/foo$ cat tests/test_foo.py
import pprint, foo
def test_foo():
pprint.pprint(foo.__file__)
assert foo.it_works()
torsten@sharokan:~/foo$ py.test -v tests
============================================================ test
session starts
============================================================
platform linux2 -- Python 2.7.3 -- pytest-2.3.4 --
/opt/dynasdk/loco2-precise/bin/python
plugins: cov, capturelog
collected 0 items / 1 errors
==================================================================
ERRORS
===================================================================
____________________________________________________ ERROR
collecting tests/test_foo.py
_____________________________________________________
tests/test_foo.py:1: in <module>
> import sys, pprint, foo
E ImportError: No module named foo
========================================================== 1 error
in 0.01 seconds
==========================================================
Sure, module foo does not exists. I could use pyximport, but I want to
check that the extension itself is correctly built.
Easily done:
torsten@sharokan:~/foo$ python setup.py build_ext -i
running build_ext
cythoning foo.pyx to foo.c
building 'foo' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3
-Wall -Wstrict-prototypes -fPIC
-I/opt/dynasdk/loco2-precise/include/python2.7 -c foo.c -o
build/temp.linux-x86_64-2.7/foo.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/foo.o
-L/opt/dynasdk/loco2-precise/lib -lpython2.7 -o /home/torsten/foo/foo.so
But alas, py.test still will not find foo.so, which is now installed
into the current directory (due to using the "-i" flag to build_ext).
Workaround: Overwrite PYTHONPATH:
torsten@sharokan:~/foo$ PYTHONPATH=`pwd` py.test
============================================================ test
session starts
============================================================
platform linux2 -- Python 2.7.3 -- pytest-2.3.4
plugins: cov, capturelog
collected 1 items
tests/test_foo.py F
=================================================================
FAILURES
==================================================================
_________________________________________________________________
test_foo
__________________________________________________________________
def test_foo():
pprint.pprint(foo.__file__)
> assert foo.it_works()
E assert <built-in function it_works>()
E + where <built-in function it_works> = foo.it_works
tests/test_foo.py:5: AssertionError
--------------------------------------------------------------
Captured stdout
--------------------------------------------------------------
'/home/torsten/foo/foo.so'
========================================================= 1 failed
in 0.02 seconds
==========================================================
Now it really loads our extension module and the failure is actually
genuine. Let's assume we have the old version installed in our current
Python environment:
torsten@sharokan:~/foo$ python setup.py install
[...]
creating
/usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg
Extracting foo-0.0-py2.7-linux-x86_64.egg to
/usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages
Adding foo 0.0 to easy-install.pth file
Installed
/usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg
[...]
Great. Now let's implement the feature that our test checks:
torsten@sharokan:~/foo$ vim foo.pyx
torsten@sharokan:~/foo$ cat foo.pyx
def it_works():
return True
The unit tests should pass now, after we rebuilt the extension:
torsten@sharokan:~/foo$ python setup.py build_ext -i
running build_ext
cythoning foo.pyx to foo.c
building 'foo' extension
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3
-Wall -Wstrict-prototypes -fPIC
-I/opt/dynasdk/loco2-precise/include/python2.7 -c foo.c -o
build/temp.linux-x86_64-2.7/foo.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/foo.o
-L/opt/dynasdk/loco2-precise/lib -lpython2.7 -o /home/torsten/foo/foo.so
torsten@sharokan:~/foo$ PYTHONPATH=`pwd` py.test
============================================================ test
session starts
============================================================
platform linux2 -- Python 2.7.3 -- pytest-2.3.4
plugins: cov, capturelog
collected 1 items
tests/test_foo.py F
=================================================================
FAILURES
==================================================================
_________________________________________________________________
test_foo
__________________________________________________________________
def test_foo():
pprint.pprint(foo.__file__)
> assert foo.it_works()
E assert <built-in function it_works>()
E + where <built-in function it_works> = foo.it_works
tests/test_foo.py:5: AssertionError
--------------------------------------------------------------
Captured stdout
--------------------------------------------------------------
'/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg/foo.so'
========================================================= 1 failed
in 0.02 seconds
==========================================================
Unfortunately, it doesn't. The unit tests actually uses the installed
version of the library, which we only want to replace after our tests pass.
Let's try it another way: By creating a virtualenv just for our tests,
we should be fine:
torsten@sharokan:~/foo$ virtualenv --system-site-packages fooenv
New python executable in fooenv/bin/python
Please make sure you remove any previous custom paths from your
/home/torsten/.pydistutils.cfg file.
Installing setuptools............done.
Installing pip...............done.
torsten@sharokan:~/foo$ . fooenv/bin/activate
(fooenv)torsten@sharokan:~/foo$ pip install --upgrade .
Unpacking /home/torsten/foo
Running setup.py egg_info for package from file:///home/torsten/foo
Downloading/unpacking Cython from
http://pypi.python.org/packages/source/C/Cython/Cython-0.19.tar.gz#md5=76989337dee4cf7afdcb5cde514423f8
(from foo==0.0)
Downloading Cython-0.19.tar.gz (1.4MB): 270kB downloaded
...
Good start, but I don't want to recreate the whole thing inside the
virtualenv fooenv. This would pull Cython, numpy, scipy, paramiko and more.
Any hint how to ensure that we are testing the local version of that
extension instead of the installed one?
Thanks, Torsten
--
DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH
Torsten Landschoff
Office Dresden
Tel: +49-(0)351-4519587
Fax: +49-(0)351-4519561
mailto:torsten.landsch...@dynamore.de
http://www.dynamore.de
DYNAmore Gesellschaft für FEM Ingenieurdienstleistungen mbH
Registration court: Stuttgart, HRB 733694
Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz
_______________________________________________
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
