On Thu, Apr 27, 2017 at 11:56 PM, Matt Riedemann <[email protected]>
wrote:
On 4/21/2017 8:36 AM, Ferenc Horváth wrote:
> Dear OpenStackerz,
>
> I'd like to improve the coverage of the current test suite over some
> special code parts in Nova.
> My main target is to add a few scenarios [1] that would exercise the
> AggregateMultiTenancyIsolation scheduler filter.
> I'm also planning on adding new test cases for other filters and for
> some libvirt related features [2] as well.
>
> Unfortunately, [1] and [2] could not be merged into Tempest for
various
> reasons, hence I started working on functional tests in Nova.
> However, since functional tests cannot be used to verify that a
deployed
> system behaves correctly, we still need end to end tests.
> Therefore, I'm proposing a new Tempest test plug-in [3,4] that
would be
> the home of currently out of tree tests.
> The idea is that these tests would run separately on a weekly basis
to
> not use too much resources, but the rest of the questions are still
open.
>
> Therefore, I'd appreciate any advice or review on this topic.
>
> Thank You all in advance.
>
> Best regards,
> Ferenc Horváth
>
> [1] https://review.openstack.org/#/c/374887/
> [2] https://review.openstack.org/#/c/315786/
> [3] https://review.openstack.org/#/c/448482/
> [4] https://review.openstack.org/#/c/451227/
>
>
__________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
[email protected]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
As discussed in the nova meeting today [1] I think some of this could
make sense as a Tempest plugin, or an in-tree functional test using
fixtures, or an in-tree dsvm integration style job that's not Tempest
but runs real tests against a live devstack configuration.
For the scheduler filter testing, I think that's something that is
totally doable with the in-tree functional tests using our fixtures.
You
don't need devstack for that since it's just testing the logic of the
scheduler filters. We have running services, database, and a live API
in
those tests, and external services like cinder/glance/neutron are
stubbed out. An example of a test like this is here [2]. I understand
you have some internal requirements for how these tests are performed
though, so I can't really help you there. Keep in mind if you do it
with
an in-tree functional test, it gets tested on every change and is
voting, whereas a periodic job is not and you only find out it's
broken
after we break it.
I think the best of both words would be to set up a 3pp CI to run the
extra tempest tests. This way the upstream CI doesn't have to spend
extra resources to run the tests but we could have e2e test coverage.
At the moment I don't think this will be realized in the near future.
So as a compromise we will provide filter tests in nova functional test
environment. I think hferenc already started adding such coverage.
For testing the libvirt watchdog action we obviously need a real
deployment with running libvirt. I think you could do that as a
tempest
plugin or as an in-tree dsvm integration job, much like how the
novaclient functional tests work (those aren't tempest but they run
against a live devstack and execute real APIs via the nova CLI). The
downside is we don't have any dsvm-integration infrastructure setup in
Nova today, so you'd have to blaze that trail. But we've talked about
this as a need for a long time, so it'd be great if someone worked on
it. Alternatively it could be an in-tree Tempest plugin...but then we
can't test things like the libvirt image cache which is something we
could do with a dsvm-integration job I think, or the evacuate API
(we'd
have to run that in serial so it doesn't break other concurrently
running tests). Actually testing evacuate would be awesome though.
I think dsvm-integration-libvirt is a nice idea so we will look the
novaclient env to learn how that works and we will try to implement
something similar in nova tree.
Thanks for the feedback!
Cheers,
gibi
In general I feel like writing a CI job for a very specific scheduler
filter configuration is overdoing it, unless you also made that job
re-configurable on the fly, like how the live migration job works [3].
[1]
http://eavesdrop.openstack.org/meetings/nova/2017/nova.2017-04-27-21.00.log.html#l-111
[2]
https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py
[3]
https://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/run_tests.sh
--
Thanks,
Matt
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
