On 10/06/2014 11:21 PM, John Dennis wrote:
On 10/03/2014 09:24 AM, Petr Viktorin wrote:

Our test suite is currently not very maintainable. I want to dedicate
some time to improve this.
The biggest part of this effort will be switching to a different test
framework, [pytest]. Compared to Nose, it makes complicated tests easier
to write -- at the expense of using more "magic".

Just looking for information, why pytest?

Nose is not enough, and knowing nothing clearly better (easier, more supportable, more popular*) than pytest, I went with what I know.

* unittest/nose is more popular than pytest, of course

Here is why I'm asking. After leaving IPA I went to another large Python
based project, OpenStack. OpenStack is really a number of loosely
affiliated autonomous projects, each of which gets to decide how they
are going to write and run their unit tests. As might be expected it's a
bit of a mess. But there has been some convergence lately using tox,
testr and testtools in combination.Python like Java has no shortage of
test frameworks and that diversity is actually a pain point. What
happens is very few folks actually understand the intricacies of test
construction and execution (because there are so many options, it can be
convoluted, there are many layers and each project does things
differently). The net result is a lot of wasted time, I groan every time
tests get ported to the test framework de jour, which seems to happen
with alarming frequency.

I'd really like to see some convergence in the Python community on test
frameworks so as a developer you only have to be an expert in one
framework because debugging test failures (including simply reproducing
the failure) is currently a time sink and just when you think you
understand what the heck is going on someone announces we're moving to
the next big thing. The proliferation of test frameworks is further
compounded by weak documentation. If something goes "boom" it can
literally take days to piece together enough information to diagnose the
issue (often because there are so many layers).

I sympathize.
Do you have a solution, though? Staying with Nose is not one at this point. I'm not too convinced by convergence in OpenStack; it's still just one project. If you have some other arguments for testr/testtools, let me know; I don't see the advantages over pytest.

So how does pytest fit in with everything else? Is stable, long term and
widely adopted? What about mocking? Is there a consensus on fixtures vs.
mocks (that isn't religious). Can pytest handle both?

I'm not sure I understand correctly what you mean, but you can wrap mocks/monkeypatching in fixtures. So yes, pytest can handle both.

I guess what I'm asking is there some convergence with Python test
tools? Or with testing are we in the equivalent of the SCM wars of 5-7
years ago until most everyone figured out git was superior and had the
most traction?

Yeah, I guess you could say the testing war is raging on.
But even the SCM situation is not clear-cut; e.g. Mozilla or Python use Mercurial, and given the availability of bridge tools, they aren't likely to switch any time soon. And the Linux package manager war is also raging on, with no clear winner.

Picking the test framework other than the one that bubbles to the top
and becomes the de facto standard has consequences. I clearly do not
have enough information to make a judgment here and any additional
information would be very much appreciated.

Perhaps it's because I'm in Europe, but every major Python conference I go to has one or two pytest talks. Also, pytest was written by the same people as tox, which (I dare say) is the de-facto standard for test *orchestration* in Python.
So in the test framework war I'm betting on this one.


Freeipa-devel mailing list

Reply via email to