On 2/25/2014 7:46 PM, Matt Riedemann wrote:
On 2/12/2014 1:57 PM, Matthew Treinish wrote:
On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:
On 1/17/2014 8:34 AM, Matthew Treinish wrote:
On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:
On 01/16/2014 10:56 PM, Matthew Treinish wrote:
Hi everyone,
With some recent changes made to Tempest compatibility with
nosetests is going
away. We've started using newer features that nose just doesn't
support. One
example of this is that we've started using testscenarios and
we're planning to
do this in more places moving forward.
So at Icehouse-3 I'm planning to push the patch out to remove
nosetests from the
requirements list and all the workarounds and references to nose
will be pulled
out of the tree. Tempest will also start raising an unsupported
exception when
you try to run it with nose so that there isn't any confusion on
this moving
forward. We talked about doing this at summit briefly and I've
brought it up a
couple of times before, but I believe it is time to do this now. I
feel for
tempest to move forward we need to do this now so that there isn't
any ambiguity
as we add even more features and new types of testing.
I'm with you up to here.
Now, this will have implications for people running tempest with
python 2.6
since up until now we've set nosetests. There is a workaround for
getting
tempest to run with python 2.6 and testr see:
https://review.openstack.org/#/c/59007/1/README.rst
but essentially this means that when nose is marked as unsupported
on tempest
python 2.6 will also be unsupported by Tempest. (which honestly it
basically has
been for while now just we've gone without making it official)
The way we handle different runners/os can be categorized as "tested
in gate", "unsupported" (should work, possibly some hacks needed),
and "hostile". At present, both nose and py2.6 I would say are in
the unsupported category. The title of this message and the content
up to here says we are moving nose to the hostile category. With
only 2 months to feature freeze I see no justification in moving
py2.6 to the hostile category. I don't see what new testing features
scheduled for the next two months will be enabled by saying that
tempest cannot and will not run on 2.6. It has been agreed I think
by all projects that py2.6 will be dropped in J. It is OK that py2.6
will require some hacks to work and if in the next few months it
needs a few more then that is ok. If I am missing another connection
between the py2.6 and nose issues, please explain.
So honestly we're already at this point in tempest. Nose really just
doesn't
work with tempest, and we're adding more features to tempest, your
negative test
generator being one of them, that interfere further with nose. I've
seen several
I disagree here, my team is running Tempest API, CLI and scenario
tests every day with nose on RHEL 6 with minimal issues. I had to
workaround the negative test discovery by simply sed'ing that out of
the tests before running it, but that's acceptable to me until we
can start testing on RHEL 7. Otherwise I'm completely OK with
saying py26 isn't really supported and isn't used in the gate, and
it's a buyer beware situation to make it work, which includes
pushing up trivial patches to make it work (which I did a few of
last week, and they were small syntax changes or usages of
testtools).
I don't understand how the core projects can be running unit tests
in the gate on py26 but our functional integration project is going
to actively go out and make it harder to run Tempest with py26, that
sucks.
If we really want to move the test project away from py26, let's
make the concerted effort to get the core projects to move with it.
So as I said before the python 2.6 story for tempest remains the same
after this
change. The only thing that we'll be doing is actively preventing nose
from
working with tempest.
And FWIW, I tried the discover.py patch with unittest2 and
testscenarios last week and either I botched it, it's not documented
properly on how to apply it, or I screwed something up, but it
didn't work for me, so I'm not convinced that's the workaround.
What's the other option for running Tempest on py26 (keeping RHEL 6
in mind)? Using tox with testr and pip? I'm doing this all
single-node.
Yes, that is what the discover patch is used to enable. By disabling
nose the
only path to run tempest with py2.6 is to use testr. (which is what it
always
should have been)
Attila confirmed it was working here:
http://fpaste.org/76651/32143139/
in that example he applies 2 patches the second one is currently in
the gate for
tempest. (https://review.openstack.org/#/c/72388/ ) So all that needs
to be done
is to apply that discover patch:
https://code.google.com/p/unittest-ext/issues/detail?id=79
(which I linked to before)
Then tempest should run more or less the same between 2.7 and 2.6.
(The only
difference I've seen is in how skips are handled)
patches this cycle that attempted to introduce incorrect behavior
while trying
to fix compatibility with nose. That's why I think we need a clear
message on
this sooner than later. Which is why I'm proposing actively raising
an error
when things are run with nose upfront so there isn't any illusion
that things
are expected to work.
This doesn't necessarily mean we're moving python 2.6 to the hostile
category.
Nose support is independent of python 2.6 support. Py26 I would
still consider
to be unsupported, the issue is that the hack to make py26 work is
outside of
tempest. This is why we've recommended that people using python 2.6
run with
nose, which really is no longer an option. Attila's abandoned patch
that I
linked above documents points to this bug with a patch to discover
which is
need to get python 2.6 working with tempest and testr:
https://code.google.com/p/unittest-ext/issues/detail?id=79
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
One question I had was is there an easy way to setup a config file to
specify the test bucket and what can be excluded, like you can with
nose.cfg and nose? We used that for filtering out API tests that didn't
work with the PowerVM driver in Nova but I'm not aware of something
similar with testr.
Another question I guess, and I know this is too late now, but the
nosexcover package allowed us to get xunit xml results for the test runs
and then jenkins has a nice way to process that xml and keep a history
of failures and dig into the stack traces, how long the tests take,
trends, etc. I'm assuming there is nothing like that available with testr?
--
Thanks,
Matt Riedemann
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
