Re: [py-dev] RFC: V2 of the new resource setup/parametrization facilities
Hello Holger, On 06/29/2012 04:55 AM, holger krekel wrote: I believe that the new resource parametrization facilities are a major step forward - they should allow test writers to much more seldomly having to resort to pytest_* hooks, and make access and working with parametrized resources straight forward, irrespective of previous xUnit/pytest background. I think so too! Very excited about the more straightforward parametrization and cache-scope for resources. I'm of course fairly new to pytest and so not as familiar with history or internal implementation, but perhaps that new perspective is also useful. On the whole I think the proposal is excellent. I would also prefer to see pytest.mark.factory_scope and pytest.mark.factory_parametrize combined into a single decorator, but I don't like the name factoryattr. For one thing, I find the usage of factory a bit confusing (there's no clarity about what sort of factory). And second, I would go even further and allow this decorator itself (even with no arguments) to mark a function as a funcarg, even if the factory function is not named according to the pytest_funcarg__foo convention. So for instance: pytest.mark.resource() def database(request): # ... (The decorator could also be pytest.mark.funcarg() if you're not ready to move towards the resource naming. I think resource is more intuitive than funcarg for newcomers, but I recognize the costs of shifting.) And then this decorator would of course also accept the scope and parametrize keyword arguments. To my mind, this would be a really nice unification and could become the canonical way to mark a test resource, with the naming convention becoming either a deprecated backwards-compatibility feature, or an approved and supported shortcut for the no-extra-parameters case, depending how you feel about it. (Personally I prefer explicit marking to naming conventions, but I realize this preference may not be in the spirit of pytest). Regarding the naming of scope and the dual scopes (lifecycle and visibility), personally I don't think it's a big issue to just use the name scope; does the name scope actually occur in the API related to the visibility variety? I find cachescope irritating and a bit misleading. If scope is overloaded, what about lifetime? Speaking of the lifetime of an instance of a test resource feels more natural to me than speaking of caching it. Carl ___ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev
Re: [py-dev] RFC: V2 of the new resource setup/parametrization facilities
On Sat, Jun 30, 2012 at 12:26 +0100, Floris Bruynooghe wrote:
On Sat, Jun 30, 2012 at 01:23 +0100, Floris Bruynooghe wrote:
On Fri, Jun 29, 2012 at 10:55:23AM +, holger krekel wrote:
def setup_directory(db):
# called when the first test in the directory tree is about
to execute
I think the naming of these functions break the py.test convention,
normally the only functions picked up from conftest.py start with
pytest_. I can certainly imagine a conftest.py or plugin already
having a setup_session function. These are new functions and do not
provide a compatibility API with other testing frameworks, so I think
they would be better named pytest_setup_session and
pytest_setup_directory.
I think using pytest_* hooks also has consistency problems:
* hooks cannot usually receive arbitrary funcargs
This is why a signature with a request/node for these might be better::
def pytest_setup_session(session):
session.getresource('db') # or .getfuncargvalue()?
...
* xUnit-style consistency: consider explaining the new functions
to someone only knowing setup_module/ class etc.
As I tried to say before, they do not come for xUnit so I don't think
this is too important. I think the consistency inside conftest.py is
more important.
Well, pytest introduced setup_module/class and nose/unittest ported it.
I consider setup_directory (or setup_session) to be xUnit consistent
from a user perspective and maybe nose/unittest will also add it - i guess
there are extensions there implementing something like this already.
In generaly, if we go the route of making setup_X more powerful there
probably is less need for pytest_runtest_setup calls. The only difference
would be that runtest_setup is called in plugins/conftest.py whereas
setup_function/method need to be defined around the actual test code.
(TBH i wonder if setup_module/class/function/method could be allowed in
conftest.py files as well - in many cases there are easier to handle
than pytest_runtest_item(item) which does not even guarantee that
the item is a python function - could be a PEP8 checker Item).
I am wondering, however, do we even need a setup_session? setup_directory
should usually be enough, i guess, and it's more unlikely people used
that name already (and we could warn about setup_session in 2.X to
reserve introducing it in 2.X+1).
Maybe not, but if you don't provide setup_session (or
pytest_setup_session) then pytest_sessionstart will be used again
when someone thinks of a reason to use it. And that's what you wanted
to avoid.
We definitely need to provide prominent examples for whole-session setup
to avoid further usage of sessionstart or configure.
[...]
If a setup-function has no body, then tests could just require it themselves
and that'd be enough. If there is a need, we could introduce a marker for
requiring funcarg-resources such that tests do not need to require it
in their signature.
I'm not sure what that would save, either the test function must
request the resource or must be marked to need the resource. If
anything the second takes more work.
As an aside however, one of my usecases for merged request/item
objects was so I could put setup in a session-wide scoped funcarg but
also automatically request this funcarg based on a mark::
def pytest_runtest_setup(item):
if 'needsdb' in item.keywords: # or a more explicit API
item.getresource('db')
I understand that this will still be possible via::
def pytest_runtest_setup(item):
if 'needsdb' in item.keywords:
item.session.getresource('db')
Or something similar to that.
It'd probably be best if this requirement is know at collection time so
--collectonly can present the complete picture. This could look like this::
def pytest_itemcollected(item):
if 'needsdb' in item.keywords:
item.applymarker(pytest.mark.needsresource(db))
Also, the above issue of requiring a global resource could be expressed
like this::
def pytest_collection_finish(session):
session.applymarker(pytest.mark.needsresource(db))
This should also in principle work well in case of a parametrized db
so that all tests requiring db can be run multiple times.
(I am not sure if the above already existing hooks fit well for that, however.)
best,
holger
___
py-dev mailing list
py-dev@codespeak.net
http://codespeak.net/mailman/listinfo/py-dev
Re: [py-dev] RFC: V2 of the new resource setup/parametrization facilities
Hi Floris, some preliminary notes, i'll probably think some more about your feedback ... On Sat, Jun 30, 2012 at 01:23 +0100, Floris Bruynooghe wrote: Hello Holger, On Fri, Jun 29, 2012 at 10:55:23AM +, holger krekel wrote: [...] Direct scoping of funcarg factories [...] Direct parametrization of funcarg factories These two seem fine, but personally I would prefer them to use the same marker with keyword-only arguments:: @pytest.mark.factory(scope='session', parametrize=['mysql', 'pg']) def pytest_funcarg__db(request): ... This seems like a more natural API which collects the different functions, certainly when using both for one funcarg. I'll consider it, probably under the name of factoryattr or so. However it bothers me that funcargs now have two types of scope: an implied scope derived from where it is defined and which defines their visibility (e.g. only inside a class, module, directory). And then this new scope which is essentially a caching/teardown scope. The fact that the ScopeMismatch exception is needed is a result of this I think. previously, the scope-mismatch could happen as well and go unnoticed:: def pytest_funcarg__Y(request): return request.function.__name__ def pytest_funcarg__X(request): def setup(): return request.getfuncargvalue(Y) return request.cached_setup(setup, scope=session) The result will depend on which test function is first requested. In the future, we might want to try raise a ScopeMismatchError here as well. The previous resource/funcarg split avoided this confusion. a) What about just naming it cachescope? b) i moved register_factory/getresource to implementation details not the least because Carl Meyer as a relatively recent pytest user expressed his expectation of a consistent pytest_funcarg__ factory story - and if we are going to anyway have to support the existing ones, i'd now like to focus on extending it and only go for a usage-level visible paradigm change if it's really needed. Does this make general sense to you? Lastly, when do scoped funcarg resources get invoked? Only at the time a test function requests it or always at the time when the scope is entered? factories are invoked when a test function or one of its involved setup methods needs it. A scope is only entered if there is a test to be executed within it. Does this clarify? support for setup_session and setup_directory -- [...] # content of conftest.py def setup_session(db): ... use db resource or do some initial global init stuff ... before any test is run. def setup_directory(db): # called when the first test in the directory tree is about to execute I think the naming of these functions break the py.test convention, normally the only functions picked up from conftest.py start with pytest_. I can certainly imagine a conftest.py or plugin already having a setup_session function. These are new functions and do not provide a compatibility API with other testing frameworks, so I think they would be better named pytest_setup_session and pytest_setup_directory. I think using pytest_* hooks also has consistency problems: * hooks cannot usually receive arbitrary funcargs * xUnit-style consistency: consider explaining the new functions to someone only knowing setup_module/ class etc. I am wondering, however, do we even need a setup_session? setup_directory should usually be enough, i guess, and it's more unlikely people used that name already (and we could warn about setup_session in 2.X to reserve introducing it in 2.X+1). And what what about putting setup_directory into an __init__.py file? I don't really like requiring __init__ files, but am fine to go with it if you and others prefer that. I would guess, that using the already directory-scoped conftest.py file feels fine to someone coming new to pytest. It also feels slightly weird that they do not get their respective Node passed in. This is a little inconsistent with the current setup_X method which all take a module, class or method argument. I can't think of an immediate use for it as you can push out pretty much everything you need to do to a properly scoped funcarg resource. We can certainly add modulenode, classnode etc. to the respective setup-methods because they participate in the funcarg-protocol (which allows accepting less parameters than are available). And following that reasoning the setup function would end up having no body at all, which also seems weird. If a setup-function has no body, then tests could just require it themselves and that'd be enough. If there is a need, we could introduce a marker for requiring funcarg-resources such that tests do not need to require it in their signature. Implementation level
Re: [py-dev] RFC: V2 of the new resource setup/parametrization facilities
On Sat, Jun 30, 2012 at 08:08:41AM +, holger krekel wrote:
On Sat, Jun 30, 2012 at 01:23 +0100, Floris Bruynooghe wrote:
On Fri, Jun 29, 2012 at 10:55:23AM +, holger krekel wrote:
previously, the scope-mismatch could happen as well and go unnoticed::
def pytest_funcarg__Y(request):
return request.function.__name__
def pytest_funcarg__X(request):
def setup():
return request.getfuncargvalue(Y)
return request.cached_setup(setup, scope=session)
The result will depend on which test function is first requested.
In the future, we might want to try raise a ScopeMismatchError here
as well.
Oh, never noticed this. That's an improvement then.
The previous resource/funcarg split avoided this confusion.
a) What about just naming it cachescope?
Maybe. I wouldn't take my word for it that just scope is not
sufficient, see what other people say. I'd probably get annoyed at
the extra typing for cachescope after a while, maybe even
@pytest.mark.factory(cache=session) is an option? It would avoid
having two things called scope.
b) i moved register_factory/getresource to implementation details
not the least because Carl Meyer as a relatively recent pytest user
expressed his expectation of a consistent pytest_funcarg__ factory
story - and if we are going to anyway have to support the existing ones,
i'd now like to focus on extending it and only go for a usage-level visible
paradigm change if it's really needed. Does this make general
sense to you?
Yes this does make sense, in general I do like this approach.
Lastly, when do scoped funcarg resources get invoked? Only at the
time a test function requests it or always at the time when the scope
is entered?
factories are invoked when a test function or one of its involved setup
methods needs it. A scope is only entered if there is a test to be executed
within it. Does this clarify?
It does.
support for setup_session and setup_directory
--
[...]
# content of conftest.py
def setup_session(db):
... use db resource or do some initial global init stuff
... before any test is run.
def setup_directory(db):
# called when the first test in the directory tree is about
to execute
I think the naming of these functions break the py.test convention,
normally the only functions picked up from conftest.py start with
pytest_. I can certainly imagine a conftest.py or plugin already
having a setup_session function. These are new functions and do not
provide a compatibility API with other testing frameworks, so I think
they would be better named pytest_setup_session and
pytest_setup_directory.
I think using pytest_* hooks also has consistency problems:
* hooks cannot usually receive arbitrary funcargs
This is why a signature with a request/node for these might be better::
def pytest_setup_session(session):
session.getresource('db') # or .getfuncargvalue()?
...
* xUnit-style consistency: consider explaining the new functions
to someone only knowing setup_module/ class etc.
As I tried to say before, they do not come for xUnit so I don't think
this is too important. I think the consistency inside conftest.py is
more important.
I am wondering, however, do we even need a setup_session? setup_directory
should usually be enough, i guess, and it's more unlikely people used
that name already (and we could warn about setup_session in 2.X to
reserve introducing it in 2.X+1).
Maybe not, but if you don't provide setup_session (or
pytest_setup_session) then pytest_sessionstart will be used again
when someone thinks of a reason to use it. And that's what you wanted
to avoid.
And what what about putting setup_directory into an __init__.py file?
I don't really like requiring __init__ files, but am fine to go with it if
you and others prefer that. I would guess, that using the already
directory-scoped conftest.py file feels fine to someone coming new to pytest.
I agree, requiring __init__.py is worse then just putting it in
conftest.py. I think it would be best if it fits inside conftest.py.
If a setup-function has no body, then tests could just require it themselves
and that'd be enough. If there is a need, we could introduce a marker for
requiring funcarg-resources such that tests do not need to require it
in their signature.
I'm not sure what that would save, either the test function must
request the resource or must be marked to need the resource. If
anything the second takes more work.
As an aside however, one of my usecases for merged request/item
objects was so I could put setup in a session-wide scoped funcarg but
also automatically request this funcarg based on a mark::
def pytest_runtest_setup(item):
if 'needsdb' in item.keywords: # or a more explicit API
[py-dev] RFC: V2 of the new resource setup/parametrization facilities
Hi all, particularly Floris and Carl, i have finally arrived at the V2 resource-API draft based on the very valuable feedback you gave to the first version. The document implements a largely changed approach, see the Changes from V1 to V2 at the beginning, and focuses on usage-level documentation instead of internal details. I have also uploaded this doc as HTML which makes it a bit more colorful to read, and also contains some cross-references: http://pytest.org/dev/resources.html Please find the the source txt-file also attached for your inline-commenting usage. Before i target the actual (substantial) refactoring, i'd actually be very grateful for some more of your time and comments on this new version. I believe that the new resource parametrization facilities are a major step forward - they should allow test writers to much more seldomly having to resort to pytest_* hooks, and make access and working with parametrized resources straight forward, irrespective of previous xUnit/pytest background. Plugin writers, of course, may still use the hooks for good value. best thanks, holger V2: Creating and working with parametrized test resources === Abstract: pytest-2.X provides generalized scoping and parametrization of resource setup. It does so by introducing new scoping and parametrization capabilities directly to to funcarg factories and by enhancing xUnit-style setup_X methods to directly accept funcarg resources. Moreover, new xUnit setup_directory() and setup_session() methods allow fixture code (and resource usage) at previously unavailable scopes. Pre-existing test suites and plugins written to work for previous pytest versions shall run unmodified. This V2 draft is based on incorporating feedback provided by Floris Bruynooghe, Carl Meyer and Ronny Pfannschmidt. It remains as draft documentation, pending further refinements and changes according to implementation or backward compatibility issues. The main changes to V1 are: * changed approach: now based on improving ``pytest_funcarg__`` factories and extending setup_X methods to directly accept funcarg resources, also including a new per-directory setup_directory() and setup_session() function for respectively scoped setup. * the funcarg versus resource naming issue is disregarded in favour of keeping with funcargs and talking about funcarg resources to ease a later possible renaming (whose value is questionable) * The register_factory/getresource methods are moved to an implementation section for now, drawing a clear boundary between usage-level docs and impl-level ones. * use 2.X as the version for introduction (might be 2.3, else 2.4) .. currentmodule:: _pytest Shortcomings of the previous pytest_funcarg__ mechanism - The previous funcarg mechanism calls a factory each time a funcarg for a test function is requested. If a factory wants t re-use a resource across different scopes, it often used the ``request.cached_setup()`` helper to manage caching of resources. Here is a basic example how we could implement a per-session Database object:: # content of conftest.py class Database: def __init__(self): print (database instance created) def destroy(self): print (database instance destroyed) def pytest_funcarg__db(request): return request.cached_setup(setup=DataBase, teardown=lambda db: db.destroy, scope=session) There are some problems with this approach: 1. Scoping resource creation is not straight forward, instead one must understand the intricate cached_setup() method mechanics. 2. parametrizing the db resource is not straight forward: you need to apply a parametrize decorator or implement a :py:func:`~hookspec.pytest_generate_tests` hook calling :py:func:`~python.Metafunc.parametrize` which performs parametrization at the places where the resource is used. Moreover, you need to modify the factory to use an ``extrakey`` parameter containing ``request.param`` to the :py:func:`~python.Request.cached_setup` call. 3. the current implementation is inefficient: it performs factory discovery each time a db argument is required. This discovery wrongly happens at setup-time. 4. there is no way how you can use funcarg factories, let alone parametrization, when your tests use the xUnit setup_X approach. 5. there is no way to specify a per-directory scope for caching. In the following sections, API extensions are presented to solve each of these problems. Direct scoping of funcarg factories Instead of calling cached_setup(), you can decorate your factory to state its scope:: @pytest.mark.factory_scope(session) def pytest_funcarg__db(request):
Re: [py-dev] RFC: V2 of the new resource setup/parametrization facilities
Hello Holger, On Fri, Jun 29, 2012 at 10:55:23AM +, holger krekel wrote: [...] Direct scoping of funcarg factories [...] Direct parametrization of funcarg factories These two seem fine, but personally I would prefer them to use the same marker with keyword-only arguments:: @pytest.mark.factory(scope='session', parametrize=['mysql', 'pg']) def pytest_funcarg__db(request): ... This seems like a more natural API which collects the different functions, certainly when using both for one funcarg. However it bothers me that funcargs now have two types of scope: an implied scope derived from where it is defined and which defines their visibility (e.g. only inside a class, module, directory). And then this new scope which is essentially a caching/teardown scope. The fact that the ScopeMismatch exception is needed is a result of this I think. The previous resource/funcarg split avoided this confusion. Lastly, when do scoped funcarg resources get invoked? Only at the time a test function requests it or always at the time when the scope is entered? support for setup_session and setup_directory -- [...] # content of conftest.py def setup_session(db): ... use db resource or do some initial global init stuff ... before any test is run. def setup_directory(db): # called when the first test in the directory tree is about to execute I think the naming of these functions break the py.test convention, normally the only functions picked up from conftest.py start with pytest_. I can certainly imagine a conftest.py or plugin already having a setup_session function. These are new functions and do not provide a compatibility API with other testing frameworks, so I think they would be better named pytest_setup_session and pytest_setup_directory. It also feels slightly weird that they do not get their respective Node passed in. This is a little inconsistent with the current setup_X method which all take a module, class or method argument. I can't think of an immediate use for it as you can push out pretty much everything you need to do to a properly scoped funcarg resource. And following that reasoning the setup function would end up having no body at all, which also seems weird. Implementation level === [...] the request object incorporates scope-specific behaviour -- [...] In fact, the request object is likely going to provide a node attribute, denoting the current collection node on which it internally operates. (Prior to pytest-2.3 there already was an internal _pyfuncitem). Does this mean you will revert the currently checked-in behaviour where a Node is actually a Request subclass and is the object passed to the funcarg resource factories? Hope this was helpful feedback, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev
