Hi all, particularly Floris and Carl,
i have finally arrived at the V2 resource-API draft based on the very valuable
feedback you gave to the first version. The document implements a largely
changed approach, see the "Changes from V1 to V2" at the beginning, and
focuses on usage-level documentation instead of internal details.
I have also uploaded this doc as HTML which makes it a bit more colorful
to read, and also contains some cross-references:
http://pytest.org/dev/resources.html
Please find the the source txt-file also attached for your
inline-commenting usage. Before i target the actual (substantial)
refactoring, i'd actually be very grateful for some more of your time
and comments on this new version.
I believe that the new resource parametrization facilities are a major
step forward - they should allow test writers to much more seldomly having
to resort to pytest_* hooks, and make access and working with parametrized
resources straight forward, irrespective of previous xUnit/pytest background.
Plugin writers, of course, may still use the hooks for good value.
best & thanks,
holger
V2: Creating and working with parametrized test resources
===============================================================
Abstract: pytest-2.X provides generalized scoping and parametrization
of resource setup. It does so by introducing new scoping and parametrization
capabilities directly to to funcarg factories and by enhancing
xUnit-style setup_X methods to directly accept funcarg resources.
Moreover, new xUnit setup_directory() and setup_session() methods allow
fixture code (and resource usage) at previously unavailable scopes.
Pre-existing test suites and plugins written to work for previous pytest
versions shall run unmodified.
This V2 draft is based on incorporating feedback provided by Floris Bruynooghe,
Carl Meyer and Ronny Pfannschmidt. It remains as draft documentation, pending
further refinements and changes according to implementation or backward
compatibility issues. The main changes to V1 are:
* changed approach: now based on improving ``pytest_funcarg__``
factories and extending setup_X methods to directly accept
funcarg resources, also including a new per-directory
"setup_directory()" and setup_session() function for respectively
scoped setup.
* the "funcarg" versus "resource" naming issue is disregarded in favour
of keeping with "funcargs" and talking about "funcarg resources"
to ease a later possible renaming (whose value is questionable)
* The register_factory/getresource methods are moved to an
implementation section for now, drawing a clear boundary between
usage-level docs and impl-level ones.
* use "2.X" as the version for introduction (might be 2.3, else 2.4)
.. currentmodule:: _pytest
Shortcomings of the previous pytest_funcarg__ mechanism
---------------------------------------------------------
The previous funcarg mechanism calls a factory each time a
funcarg for a test function is requested. If a factory wants
t re-use a resource across different scopes, it often used
the ``request.cached_setup()`` helper to manage caching of
resources. Here is a basic example how we could implement
a per-session Database object::
# content of conftest.py
class Database:
def __init__(self):
print ("database instance created")
def destroy(self):
print ("database instance destroyed")
def pytest_funcarg__db(request):
return request.cached_setup(setup=DataBase,
teardown=lambda db: db.destroy,
scope="session")
There are some problems with this approach:
1. Scoping resource creation is not straight forward, instead one must
understand the intricate cached_setup() method mechanics.
2. parametrizing the "db" resource is not straight forward:
you need to apply a "parametrize" decorator or implement a
:py:func:`~hookspec.pytest_generate_tests` hook
calling :py:func:`~python.Metafunc.parametrize` which
performs parametrization at the places where the resource
is used. Moreover, you need to modify the factory to use an
``extrakey`` parameter containing ``request.param`` to the
:py:func:`~python.Request.cached_setup` call.
3. the current implementation is inefficient: it performs factory discovery
each time a "db" argument is required. This discovery wrongly happens at
setup-time.
4. there is no way how you can use funcarg factories, let alone
parametrization, when your tests use the xUnit setup_X approach.
5. there is no way to specify a per-directory scope for caching.
In the following sections, API extensions are presented to solve
each of these problems.
Direct scoping of funcarg factories
--------------------------------------------------------
Instead of calling cached_setup(), you can decorate your factory
to state its scope::
@pytest.mark.factory_scope("session")
def pytest_funcarg__db(request):
# factory will only be invoked once per session -
db = DataBase()
request.addfinalizer(db.destroy) # destroy when session is finished
return db
This factory implementation does not need to call ``cached_setup()`` anymore
because it will only be invoked once per session. Moreover, the
``request.addfinalizer()`` registers a finalizer according to the specified
factory scope on which this factory is operating. With this new
scoping, the still existing ``cached_setup()`` should be much less used
but will remain for compatibility reasons and for the case where you
still want to have your factory get called on a per-item basis.
Direct parametrization of funcarg factories
----------------------------------------------------------
Previously, funcarg factories could not directly cause parametrization.
You needed to specify a ``@parametrize`` or implement a
``pytest_generate_tests`` hook to perform parametrization, i.e. calling a test
multiple times
with different value sets. pytest-2.X introduces a decorator for use
on the factory itself::
@pytest.mark.factory_parametrize(["mysql", "pg"])
def pytest_funcarg__db(request):
...
Here the factory will be invoked twice (with the respective "mysql"
and "pg" values set as ``request.param`` attributes) and and all of
the tests requiring "db" will run twice as well. The "mysql" and
"pg" values will also be used for reporting the test-invocation variants.
This new way of directly parametrizing funcarg factories should
in many cases allow to use already written factories because effectively
``@factory_parametrize`` sets the same ``request.param`` that a
:py:func:`~_pytest.python.Metafunc.parametrize()` call causes to be set.
Of course it's perfectly to combine parametrization and scoping::
@pytest.mark.factory_parametrize(["mysql", "pg"])
@pytest.mark.factory_scope("session")
def pytest_funcarg__db(request):
if request.param == "mysql":
db = MySQL()
elif request.param == "pg":
db = PG()
request.addfinalizer(db.destroy) # destroy when session is finished
return db
This would execute all tests requiring the per-session "db" resource twice,
receiving the values respectively created by the two invocations to the
``db`` factory.
Using funcarg resources in xUnit setup methods
------------------------------------------------------------
For a long time, pytest has recommended the usage of funcarg resource
factories as a primary means for managing resources in your test run.
It is a better approach than the jUnit-based approach in many cases, even
more with the new pytest-2.X features, because the funcarg resource factory
provides a single place to determine scoping and parametrization. Your tests
do not need to encode setup/teardown details in every test file's
setup_module/class/method.
However, the jUnit methods originally introduced by pytest to Python,
remain popoular with nose and unittest-based test suites. And in any
case, there are large existing test suites using this paradigm. pytest-2.X
recognizes this fact and now offers direct integration with funcarg resources.
Here is a basic example for getting a per-module tmpdir::
def setup_module(mod, tmpdir):
mod.tmpdir = tmpdir
This will make pytest's funcarg resource mechanism to create a value of
"tmpdir" which can then be used through the module as a global.
Note that the new extension to setup_X methods also works in case a
resource is parametrized. Let's consider an setup_class example using
our "db" resource::
class TestClass:
def setup_class(cls, db):
cls.db = db
# perform some extra things on db
# so that test methods can work with it
With pytest-2.X the setup* methods will be discovered at collection-time,
allowing to seemlessly integrate this approach with parametrization.
Note again, that the setup_class itself does not itself need to
be aware of the fact that "db" might be a mysql/PG database.
Note that if the specified resource is provided only as a per-testfunction
resource, collection will already report a ScopingMismatch error.
support for setup_session and setup_directory
------------------------------------------------------
pytest for a long time offered a pytest_configure and a pytest_sessionstart
hook which are often used to setup global resources. This suffers from
several problems:
1. in distributed testing the master process would setup test resources
that are never needed because it only co-ordinates the test run
activities of the slave processes.
2. if you only perform a collection (with "--collectonly")
resource-setup will still be executed.
3. If a pytest_sessionstart is contained in some subdirectories
conftest.py file, it will not be called. This stems from the
fact that this hook is actually used for reporting, in particular
the test-header with platform/custom information.
4. there is no direct way how you can restrict setup to a directory scope.
It follows that these hooks are not a good place to implement session
or directory-based setup. pytest-2.X offers new "setup_X" methods, accepting
funcargs, which you can put into conftest.py or plugins files::
# content of conftest.py
def setup_session(db):
... use db resource or do some initial global init stuff
... before any test is run.
def setup_directory(db):
# called when the first test in the directory tree is about to execute
For obvious reasons, the used funcarg-resources be provided on a per-directory
or per-session basis.
the "directory" caching scope
--------------------------------------------
All API accepting a scope (:py:func:`cached_setup()` nd
@mark.factory_scope currently) now also accepts a "directory"
specification. This allows to restrict/cache resource values on a
per-directory level.
funcarg and setup discovery now happens at collection time
---------------------------------------------------------------------
pytest-2.X takes care to discover resource factories and setup_X methods
at collection time. This is more efficient especially for large test suites.
Moreover, a call to "py.test --collectonly" should be able to show
a lot of setup-information and thus presents a nice method to get an
overview of resource management in your project.
Implementation level
===================================================================
To implement the above new features, pytest-2.X grows some new hooks and
methods. At the time of writing V2 and without actually implementing
it, it is not clear how much of this new internal API will also be
exposed and advertised e. g. for plugin writers.
the "request" object incorporates scope-specific behaviour
------------------------------------------------------------------
funcarg factories receive a request object to help with implementing
finalization and inspection of the requesting-context. If there is
no scoping is in effect, nothing much will change of the API behaviour.
However, with scoping the request object represents the according context.
Let's consider this example::
@pytest.mark.factory_scope("class")
def pytest_funcarg__db(request):
# ...
request.getfuncargvalue(...)
#
request.addfinalizer(db)
Due to the class-scope, the request object will:
- provide a ``None`` value for the ``request.function`` attribute.
- default to per-class finalization with the addfinalizer() call.
- raise a ScopeMismatchError if a more broadly scoped factory
wants to use a more tighly scoped factory (e.g. per-function)
In fact, the request object is likely going to provide a "node"
attribute, denoting the current collection node on which it internally
operates. (Prior to pytest-2.3 there already was an internal
_pyfuncitem).
As these are rather intuitive extensions, not much friction is expected
for test/plugin writers using the new scoping and parametrization mechanism.
It's, however, a serious internal effort to reorganize the pytest
implementation.
node.register_factory/getresource() methods
--------------------------------------------------------
In order to implement factory- and setup-method discovery at
collection time, a new node API will be introduced to allow
for factory registration and a getresource() call to obtain
created values. The exact details of this API remain subject
to experimentation. The basic idea is to introduce two new
methods to the Session class which is already available on all nodes
through the ``node.session`` attribute::
class Session:
def register_resource_factory(self, name, factory_or_list, scope):
""" register a resource factory for the given name.
:param name: Name of the resource.
:factory_or_list: a function or a list of functions creating
one or multiple resource values.
:param scope: a node instance. The factory will be only visisble
available for all descendant nodes.
specify the "session" instance for global availability
"""
def getresource(self, name, node):
""" get a named resource for the give node.
This method looks up a matching funcarg resource factory
and calls it.
"""
.. todo::
XXX While this new API (or some variant of it) may suffices to implement
all of the described new usage-level features, it remains unclear how the
existing "@parametrize" or "metafunc.parametrize()" calls will map to it.
These parametrize-approaches tie resource parametrization to the
function/funcargs-usage rather than to the factories.
.. todo::
Will this hook be called
_______________________________________________
py-dev mailing list
py-dev@codespeak.net
http://codespeak.net/mailman/listinfo/py-dev
