Re: [pytest-dev] Proposal: python namespacign for markings/marker objects

[Bruno Oliveira]

On Thu, Oct 20, 2016 at 12:33 PM Ronny Pfannschmidt <
opensou...@ronnypfannschmidt.de> wrote:

I'm still not getting how that fits with the collection of *tests*...
consider this module:

# contents of test_foo.py
from pytest_blocker import Blocker

@Blocker(123)
def test_foo():
pass

fist that usage is wrong, a *mark* object in my proposal is neither usable
as decorator,
 nor aware of mark mechanism, it can literally be any object

a *mark* is orthogonal to the process of marking, mixing those 2 concepts
just creates a huge mess

after collection you have the test items, those have all the markers,
from the markers of test items one can infer the markers used in the
collected tests




Oh I see, my bad. You mean this then:

from pytest_blocker import Blocker

@pytest.mark(Blocker(123))
def test_foo():
pass
___
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev

Re: [pytest-dev] Proposal: python namespacign for markings/marker objects

[Ronny Pfannschmidt]

On 20.10.2016 15:14, Bruno Oliveira wrote:
> Hey Ronny,
>
>
> On Thu, Oct 20, 2016 at 10:55 AM Ronny Pfannschmidt
>  > wrote:
>
>> b) Another suggestion from Ronny: running collection and getting
>> used marker names. I'm not entirely sure what this means exactly,
>> since the original proposal doesn't use "names" at all, only the
>> objects directly.
> in my proposal the "name" of a marker would be either the current
> string or a type
>
>
> I'm still not getting how that fits with the collection of *tests*...
> consider this module:
>
> # contents of test_foo.py
> from pytest_blocker import Blocker
>
> @Blocker(123)
> def test_foo():
> pass
>
fist that usage is wrong, a *mark* object in my proposal is neither
usable as decorator,
 nor aware of mark mechanism, it can literally be any object

a *mark* is orthogonal to the process of marking, mixing those 2
concepts just creates a huge mess

after collection you have the test items, those have all the markers,
from the markers of test items one can infer the markers used in the
collected tests
-- Ronny
> How does pytest know that the file "test_foo" uses "Blocker", and that
> "Blocker" is a marker? One possible solution would be to inspect the
> namespace and see if any of the objects are a pytest.Mark subclass.
> I'm not suggesting that, just trying to illustrate what I mean by my
> question.
>
> another important part of my proposal is, that i want to decouple
> from the namespace we put into pytest.mark,
> after all there is now a multitude of plugins that register
> certain markers, sometimes for *different* usages and incompatible
> signatures
>
>
> That's the "flat namespaces" part of the discussion, if I'm
> understanding your proposal correctly. :)
>  
>
>> IMHO the discussion by email at this point is a little hard to
>> digest and to track all points/replies/proposals. Perhaps we
>> should move this discussion to a different venue? I propose an
>> issue on GitHub because the main issue containing the proposal
>> can be updated as the discussion progresses, although I'm not
>> sure if it would be any easier to track the discussion itself.
>> Perhaps using the Wiki would also be possible?
>>
> i could make a wiki page
>
>
> I would like to see what others think first. People might be OK with
> the current format, or have other suggestions.
>
> Cheers
> Bruno.

___
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev

Re: [pytest-dev] Proposal: python namespacign for markings/marker objects

[Bruno Oliveira]

Hey Ronny,


On Thu, Oct 20, 2016 at 10:55 AM Ronny Pfannschmidt <
opensou...@ronnypfannschmidt.de> wrote:

> b) Another suggestion from Ronny: running collection and getting used
> marker names. I'm not entirely sure what this means exactly, since the
> original proposal doesn't use "names" at all, only the objects directly.
>
> in my proposal the "name" of a marker would be either the current string
> or a type
>

I'm still not getting how that fits with the collection of *tests*...
consider this module:

# contents of test_foo.py
from pytest_blocker import Blocker

@Blocker(123)
def test_foo():
pass

How does pytest know that the file "test_foo" uses "Blocker", and that
"Blocker" is a marker? One possible solution would be to inspect the
namespace and see if any of the objects are a pytest.Mark subclass. I'm not
suggesting that, just trying to illustrate what I mean by my question.

another important part of my proposal is, that i want to decouple from the
> namespace we put into pytest.mark,
> after all there is now a multitude of plugins that register certain
> markers, sometimes for *different* usages and incompatible signatures
>

That's the "flat namespaces" part of the discussion, if I'm understanding
your proposal correctly. :)


> IMHO the discussion by email at this point is a little hard to digest and
> to track all points/replies/proposals. Perhaps we should move this
> discussion to a different venue? I propose an issue on GitHub because the
> main issue containing the proposal can be updated as the discussion
> progresses, although I'm not sure if it would be any easier to track the
> discussion itself. Perhaps using the Wiki would also be possible?
>
> i could make a wiki page
>

I would like to see what others think first. People might be OK with the
current format, or have other suggestions.

Cheers
Bruno.
___
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev

Re: [pytest-dev] Proposal: python namespacign for markings/marker objects

[Ronny Pfannschmidt]

HI Bruno


On 20.10.2016 14:20, Bruno Oliveira wrote:
> On Thu, Oct 20, 2016 at 7:12 AM Ronny Pfannschmidt
>  > wrote:
>
>
> On 20.10.2016 10:56, holger krekel wrote:
> > Hey Ronny,
> >
> > i
> >> while trying to turn various internal markers of a work project
> >> into public plugins, i noticed a very plain problems -
> different other
> >> plugins used the same generic marker name for different
> purposes/intents
> >>
> >> such a flat namespace really doesn't scale
> > The more plugins there are the more it causes potential clashes
> of marker and fixture names.  We've had some discussions at the
> sprint about it IIRC.  I suggest that whatever namespacing we come
> up with it should be a) backward-compatible b) work for both
> markers and fixtures.
> python packages are pretty perfect for name-spacing markers, and its
> backward compatible
>
>
> Perhaps we should clearly state the points of Ronny's proposal so we
> can have a more structured discussion. Also, I suggest we try to
> discuss the proposal using an example which we are all familiar with,
> like the "skipif" marker. This also lets us see how we could
> eventually replace the current marker's implementation in the core
> (hopefully improving them by making them easier to understand) while
> also maintaining backward compatibility.
>
> * Problem 1: Flat namespace for markers
>
> This seems to be the main point of Ronny's proposal, the fact that a
> flat namespace for markers doesn't scale, with the potential of
> different plugins using markers with the same name causing problems
> and confusion.
>
> Ronny's proposal: import and use "marker objects"
>
> from myplugin import SkipIf
>
> @pytest.mark(SkipIf(sys.platform != 'win32', reason='win32 only'))
> def test_foo():
> pass
>
> * Problem 2: How plugins use markers and handle their arguments
>
> Currently plugins using markers have to deal with the *args and
> **kwargs parameters themselves, which is often messy and done incorrectly.
>
> Currently:
>
> def pytest_collection_modifyitems(items):
> for item in items:
> m = item.getmarker('skipif')
> condition = m.args[0] if skip.args else m.kwargs['condition']
> ...
>
> If getmarker also supports a `type` as parameter, this becomes possible:
>
> def pytest_collection_modifyitems(items):
> for item in items:
> skipif = item.getmarker(SkipIf)
> m.condition  # naturally available as an attribute
> ...
>
> IMO the API `item.getmarker(x)` can be kept backward compatible by
> accepting either a mark "global name" or a marker `type`. More on how
> to declare/register markers below.
>
> Some questions:
>
> 1. How to declare those markers?
>
> a) One of Ronny's suggestion is a new entry point: not entirely clear
> to me how it would be done. I personally don't like this idea, entry
> points are used to declare *plugins* only, and plugins use other
> mechanisms to declare fixtures, markers, hooks, etc. I don't think we
> should overload this.
>


> b) Another suggestion from Ronny: running collection and getting used
> marker names. I'm not entirely sure what this means exactly, since the
> original proposal doesn't use "names" at all, only the objects directly.
in my proposal the "name" of a marker would be either the current string
or a type
>
> One mechanism for declaring markers that I believe we discussed in the
> sprint was to provide a new hook which could be used to declare new
> markers. The hook would return a dict of marker names and opaque
> types, which could then be accessed by plugins using the current
> `item.getmarker(name)` API. For example, using "skipif":
>
> class SkipIf:
> def __init__(self, condition, *, reason=None):
> pass
>
> The hook would look like this:
>
> def pytest_register_markers(config):
> return {'skipif': SkipIf}
>
> This is similar to what Floris mentioned:
>
> @pytest.marker
> def skipif(condition, *, reason=None):
> return SkipIf(condition, reason=reason)
>
> In both cases, `@pytest.mark.skipif` would instantiate a new `SkipIf`
> instance with the parameters given and make it available to items
> using the `item.getmarker(name)` API.
another important part of my proposal is, that i want to decouple from
the namespace we put into pytest.mark,
after all there is now a multitude of plugins that register certain
markers, sometimes for *different* usages and incompatible signatures

>
> I really like both the hook idea as well as the `@pytest.marker`
> decorator as a way to improve how the current markers work because
> they fix the "handle *args and **kwargs" problem nicely. 
>
> This of course doesn`t address the "flat namespace" problem, but I
> think they both can co-exist.
>
> 2. How to declare markers at the module and class level.
>
> Ronny suggests using the `pytestmark` mechanism for that, which I
> think works and is backwar

Re: [pytest-dev] Proposal: python namespacign for markings/marker objects

[Bruno Oliveira]

On Thu, Oct 20, 2016 at 7:12 AM Ronny Pfannschmidt <
opensou...@ronnypfannschmidt.de> wrote:


On 20.10.2016 10:56, holger krekel wrote:
> Hey Ronny,
>
> i
>> while trying to turn various internal markers of a work project
>> into public plugins, i noticed a very plain problems - different other
>> plugins used the same generic marker name for different purposes/intents
>>
>> such a flat namespace really doesn't scale
> The more plugins there are the more it causes potential clashes of marker
and fixture names.  We've had some discussions at the sprint about it
IIRC.  I suggest that whatever namespacing we come up with it should be a)
backward-compatible b) work for both markers and fixtures.
python packages are pretty perfect for name-spacing markers, and its
backward compatible


Perhaps we should clearly state the points of Ronny's proposal so we can
have a more structured discussion. Also, I suggest we try to discuss the
proposal using an example which we are all familiar with, like the "skipif"
marker. This also lets us see how we could eventually replace the current
marker's implementation in the core (hopefully improving them by making
them easier to understand) while also maintaining backward compatibility.

* Problem 1: Flat namespace for markers

This seems to be the main point of Ronny's proposal, the fact that a flat
namespace for markers doesn't scale, with the potential of different
plugins using markers with the same name causing problems and confusion.

Ronny's proposal: import and use "marker objects"

from myplugin import SkipIf

@pytest.mark(SkipIf(sys.platform != 'win32', reason='win32 only'))
def test_foo():
pass

* Problem 2: How plugins use markers and handle their arguments

Currently plugins using markers have to deal with the *args and **kwargs
parameters themselves, which is often messy and done incorrectly.

Currently:

def pytest_collection_modifyitems(items):
for item in items:
m = item.getmarker('skipif')
condition = m.args[0] if skip.args else m.kwargs['condition']
...

If getmarker also supports a `type` as parameter, this becomes possible:

def pytest_collection_modifyitems(items):
for item in items:
skipif = item.getmarker(SkipIf)
m.condition  # naturally available as an attribute
...

IMO the API `item.getmarker(x)` can be kept backward compatible by
accepting either a mark "global name" or a marker `type`. More on how to
declare/register markers below.

Some questions:

1. How to declare those markers?

a) One of Ronny's suggestion is a new entry point: not entirely clear to me
how it would be done. I personally don't like this idea, entry points are
used to declare *plugins* only, and plugins use other mechanisms to declare
fixtures, markers, hooks, etc. I don't think we should overload this.

b) Another suggestion from Ronny: running collection and getting used
marker names. I'm not entirely sure what this means exactly, since the
original proposal doesn't use "names" at all, only the objects directly.

One mechanism for declaring markers that I believe we discussed in the
sprint was to provide a new hook which could be used to declare new
markers. The hook would return a dict of marker names and opaque types,
which could then be accessed by plugins using the current
`item.getmarker(name)` API. For example, using "skipif":

class SkipIf:
def __init__(self, condition, *, reason=None):
pass

The hook would look like this:

def pytest_register_markers(config):
return {'skipif': SkipIf}

This is similar to what Floris mentioned:

@pytest.marker
def skipif(condition, *, reason=None):
return SkipIf(condition, reason=reason)

In both cases, `@pytest.mark.skipif` would instantiate a new `SkipIf`
instance with the parameters given and make it available to items using the
`item.getmarker(name)` API.

I really like both the hook idea as well as the `@pytest.marker` decorator
as a way to improve how the current markers work because they fix the
"handle *args and **kwargs" problem nicely.

This of course doesn`t address the "flat namespace" problem, but I think
they both can co-exist.

2. How to declare markers at the module and class level.

Ronny suggests using the `pytestmark` mechanism for that, which I think
works and is backward compatible.

I would like to comment that marks are often used in test suites (as
opposed to in plugins) to mark some tests for test selection using `-m`,
for example `@pytest.mark.integration` or `@pytest.mark.units`. In this use
case, the user usually doesn't pass any arguments and doesn't formally
declare markers. But I think this is still compatible with the proposal so
far, because `item.getmarker` in those cases could return a general
`Marker(*args, **kwargs)` object when `pytest.mark.integration` is called
and no "integration" marker has been registered.

---

So far I think we can address both the "flat namespace" and "argument
handling" aspects of markers in a co

Re: [pytest-dev] Proposal: python namespacign for markings/marker objects

[Ronny Pfannschmidt]

Hi Holger,


On 20.10.2016 10:56, holger krekel wrote:
> Hey Ronny,
>
> i'd like to get back to your original suggestion ...
>
> On Wed, Sep 07, 2016 at 11:38 +0200, Ronny Pfannschmidt wrote:
>> Hi all,
>>
>> while trying to turn various internal markers of a work project
>> into public plugins, i noticed a very plain problems - different other
>> plugins used the same generic marker name for different purposes/intents
>>
>> such a flat namespace really doesn't scale
> The more plugins there are the more it causes potential clashes of marker and 
> fixture names.  We've had some discussions at the sprint about it IIRC.  I 
> suggest that whatever namespacing we come up with it should be a) 
> backward-compatible b) work for both markers and fixtures.
python packages are pretty perfect for name-spacing markers, and its
backward compatible

as for fixture namespaces, if we base them in plain string names and
continue doing so,
i feel absolutely certain it will break in a messy way and it has
different referencing needs than markers

so making 2 different things use the same mechanism is a guarantee that
we end up in a broken mess :(

i'd much rather start a discussion about using different means even for
fixtures,
but thats for a different topic that i hope to start this weekend (and
it would solve  lot of massive headaches wrt getting rid of py.path.local)


>> as such i would like to propose having marker types and objects as
>> importable objects
>>
>>
>> for example
>>
>>
>>
>> import pytest
>> from pytest_bugzilla import Blocker
>>
>>
>> @pytest.mark(Blocker(123))
>> def test_fun():
>>   pass
>>
>> that way we can do both, reuse python name-spacing *and* use more
>> meaningful objects for markings
> a few questions:
>
> - How would you integrate this with interactive help such as "py.test 
> --markers"? 
i see 2 sensible paths
a) a new entry-point to make them discover-able
b) running collection and getting a set of the used marker names

>
> - how would you access the Blocker marker from a hook?  New API?
   basically a mark name is either a identifier or a type

getmarker(Blocker) ->mark collection of Blocker objects
   
   this api break is needed anyway because what we have now is
demonstrably broken and unusable
   in various situations, usage and parameters we get different kinds of
objects with different behaviors
   
>
> - how would you apply it at module or class level?
pytestmark = [ Blocker(123) ]


>
> holger
> ___
> pytest-dev mailing list
> pytest-dev@python.org
> https://mail.python.org/mailman/listinfo/pytest-dev
-- Ronny

___
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev

Re: [pytest-dev] Proposal: python namespacign for markings/marker objects

[holger krekel]

Hey Ronny,

i'd like to get back to your original suggestion ...

On Wed, Sep 07, 2016 at 11:38 +0200, Ronny Pfannschmidt wrote:
> Hi all,
> 
> while trying to turn various internal markers of a work project
> into public plugins, i noticed a very plain problems - different other
> plugins used the same generic marker name for different purposes/intents
> 
> such a flat namespace really doesn't scale

The more plugins there are the more it causes potential clashes of marker and 
fixture names.  We've had some discussions at the sprint about it IIRC.  I 
suggest that whatever namespacing we come up with it should be a) 
backward-compatible b) work for both markers and fixtures.

> as such i would like to propose having marker types and objects as
> importable objects
> 
> 
> for example
> 
> 
> 
> import pytest
> from pytest_bugzilla import Blocker
> 
> 
> @pytest.mark(Blocker(123))
> def test_fun():
>   pass
> 
> that way we can do both, reuse python name-spacing *and* use more
> meaningful objects for markings

a few questions:

- How would you integrate this with interactive help such as "py.test 
--markers"? 

- how would you access the Blocker marker from a hook?  New API?

- how would you apply it at module or class level?

holger
___
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev

Re: [pytest-dev] Proposal: python namespacign for markings/marker objects

[Floris Bruynooghe]

On 7 September 2016 at 10:38, Ronny Pfannschmidt  wrote:
> Hi all,
>
> while trying to turn various internal markers of a work project
> into public plugins, i noticed a very plain problems - different other
> plugins used the same generic marker name for different purposes/intents
>
> such a flat namespace really doesn't scale
>
> as such i would like to propose having marker types and objects as
> importable objects
>
> for example
>
> import pytest
> from pytest_bugzilla import Blocker
>
> @pytest.mark(Blocker(123))
> def test_fun():
>   pass
>
> that way we can do both, reuse python name-spacing *and* use more meaningful
> objects for markings

I'm rather lukewarm for this proposal to be honest.  We already have
an API which allows creating of mark objects for later re-use:

blocker = pytest.mark.bugzilla_blocker

@blocker
def test_fun():
pass

So instead of changing the way way pytest.mark behaves I think it
might be more interesting to look at alternative ways of creating
marker objects.  E.g. if you could do:

# pytest_bugzilla.py
class Blocker(pytest.MarkerObject):
def __init__(self, n):
pass

# test_fun.py
from pytest_bugzilla import Blocker

@Blocker(123)
def test_fun():
pass

Then you can still import your marker and you also get more strict
namespacing since you created a unique object in your module.  But the
benefit is that from the point of view of using markers not much has
changed.  There's just a new way of creating markers.

What do you reckon about such an approach?


Finally, and I only just remembered, Holger has some lingering code
somewhere which does something like:

@pytest.marker
def blocker(n):
return {'issue no': int(n)}  # or whatever object you want your marker to be

@pytest.mark.blocker(123)
def test_fun():
pass

This uses symmetry with how we create fixtures, which is nice.  But it
does not solve the namespacing issue, which to some extend also exists
in fixtures.  Just brainstorming to add something namespace like to
that would be:

@pytest.marker(ns='bugzilla')
def blocker(n):
return int(n)

@pytest.mark.bugzilla.blocker(123)
def test_fun():
pass

I admit this does not re-use the python module namespaces, which is
probably a downside.


Floris
___
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev

Re: [pytest-dev] Proposal: python namespacign for markings/marker objects

[Bruno Oliveira]

Hey Ronny,

On Wed, Sep 7, 2016 at 10:45 AM Ronny Pfannschmidt 
wrote:

> such a flat namespace really doesn't scale
>
> as such i would like to propose having marker types and objects as
> importable objects
>

I'm not against adding support for it, but I wouldn't change it to be the
"recommended way" to do it: breaks backward compatibility and having more
than one way to do it might be confusing to users. Perhaps mentioning it in
some advanced session in the docs.


> import pytest
> from pytest_bugzilla import Blocker
>
>
> @pytest.mark(Blocker(123))
> def test_fun():
>   pass
>

What do you think would be the requirements for the Blocker object?

Cheers,
Bruno.
___
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev