Hi Adam,
I think the steps you've completed so far are the foundation for testers like
myself to be able to really dive in and start kicking the tires of Ironic.
Having an easily repeatable way to generate test environments makes a huge
impact in where I spend my time (environments vs actual testing). As for
functional coverage, to be completely honest I'm still working on fully context
switching from Nova, but I've looked at your etherpad and I definitely have
some thoughts and additions to share.
I also wouldn't rule the idea completely of communicating with other management
resources. While this used to be an idea I strongly and loudly opposed, one of
the realizations I had during my time testing Nova was that ignoring the
backend was somewhat of a mistake, at least from a root cause analysis of
failures perspective. While working with Nova and XenServer, I ran into enough
situations where XenServer had a much clearer perspective of faults and
failures that I began to add optional duplicate validations to the XenServer
API to help better track where the real source of a failure was. While I'm
still not sure if this would make sense to implement in any Ironic tests, I
don't think completely closing the door on the idea would be wise yet.
As for dealing with narrowing the scope of Tempest tests by Nova driver, I
think there's a few easy options here. What I've done in my own testing outside
of Tempest is to use and expand on the Hypervisor/Driver Feature Matrix [1] in
combination with a driver configuration value to determine at run time which
tests should be executed. When I first started developing Tempest, I
implemented a much weaker solution, which was to provide a per feature flag to
enable certain tests that use individual features, which still exists today.
The concept of feature sets isn't a problem limited to Ironic (other
hypervisors and drivers and similar problems), so I'd like to reengage the
Tempest community with this concept and see how it's received.
Daryl
[1]
https://wiki.openstack.org/wiki/HypervisorSupportMatrix#Hypervisor_feature_support_matrix
On Mar 20, 2014, at 9:18 PM, Adam Gandelman
ad...@ubuntu.commailto:ad...@ubuntu.com wrote:
Hi--
We've made some progress over the last week or so toward more thorough Ironic
CI:
* Initial devstack support has been merged enabling an all-in-one Ironic
environment that uses libvirt+VMs+OVS to support baremetal deployments [1]
* Devananda and myself have been getting patches pushed into infra. that add an
experimental devstack gate check using the new support
While we continue to work out the kinks in the devstack gate, I've started
thinking about what functional Ironic tests in Tempest would look like. I've
been capturing my thoughts in an etherpad [2] and invite anyone interested to
add theirs.
I've put together a Tempest scenario test that validates instance spawn and
corresponding Ironic orchestration [3]. This initial test assumes its being
tested against the pxe_ssh driver (which devstack enrolls nodes with) and
verifies assumptions accordingly. A longer term goal would be for this same
test to be run against other non-devstack environments (ie, TripleO undercloud)
and verify other things specific to the drivers in use there.
I'm curious to know what others think should be, or even can be, stressed and
tested from the outside by Tempest. Since Tempest cannot assume it has access
to poke at management resources like libvirt, IPMI, etc. it can really only
inspect what is provided by the APIs. Validating that nova/neutron/etc injects
the correct data into Ironic node/port/etc objects seems like the most that can
happen beyond simply spawning an instance and waiting for it to show up on the
network.
On a related note, running Tempest in the gate against Ironic presents some
interesting challenges that we'll need to work with the Tempest team to solve.
The API and smoke tests that are run assume many things about the supported
features of the running compute driver. Many of these are not supported in
Ironic (eg,, boot from volume). This not specific to Ironic (eg, lxc) but will
require some discussion and work before Tempest is just point-and-shoot against
Ironic. I'm wondering if it would make sense to lean on the
soon-to-be-deprecated devstack exercises for verification in the short-term,
while we work through the larger issues in Tempest.
-Adam
[1] https://review.openstack.org/#/c/70348/
[2] https://etherpad.openstack.org/p/IronicCI
[2] https://review.openstack.org/#/c/81958/
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
