On Aug 27, 2014, at 1:47 AM, Salvatore Orlando <sorla...@nicira.com> wrote:

> TL; DR
> A few folks are proposing to stop running tests for neutron advanced services 
> [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on the neutron 
> gate.
> Reason: projects like nova are 100% orthogonal to neutron advanced services. 
> Also, there have been episodes in the past of unreliability of tests for 
> these services, and it would be good to limit affected projects considering 
> that more api tests and scenarios are being added.

Given how many rechecks I’ve had to do to merge what are effectively no-op 
patches to infra/config, most often due to the full neutron job exhibiting 
sporadic failures, I fully support this change.  I think we need time to 
stabilize the tests for advanced services against just neutron before we 
consider slowing down merges for other projects.

> -----
> So far the neutron full job runs tests (api and scenarios) for neutron "core" 
> functionality as well as neutron "advanced services", which run as neutron 
> service plugin.
> It's highly unlikely, if not impossible, that changes in projects such as 
> nova, glance or ceilometer can have an impact on the stability of these 
> services.
> On the other hand, instability in these services can trigger gate failures in 
> unrelated projects as long as tests for these services are run in the neutron 
> full job in the integrated gate. There have already been several 
> gate-breaking bugs in lbaas scenario tests are firewall api tests.
> Admittedly, advanced services do not have the same level of coverage as core 
> neutron functionality. Therefore as more tests are being added, there is an 
> increased possibility of unearthing dormant bugs.
> For this reason we are proposing to not run anymore tests for neutron 
> advanced services in the integrated gate, but keep them running on the 
> neutron gate.
> This means we will have two neutron jobs:
> 1) check-tempest-dsvm-neutron-full which will run only "core" neutron 
> functionality
> 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full 
> job is today.
> The former will be part of the integrated gate, the latter will be part of 
> the neutron gate.
> Considering that other integrating services should not have an impact on 
> neutron advanced services, this should not make gate testing asymmetric.
> However, there might be exceptions for:
> - "orchestration" project like heat which in the future might leverage 
> capabilities like load balancing
> - oslo-* libraries, as changes in them might have an impact on neutron 
> advanced services, since they consume those libraries
> Another good question is whether "extended" tests should be performed as part 
> of functional or tempest checks. My take on this is that scenario tests 
> should always be part of tempest. On the other hand I reckon API tests should 
> exclusively be part of functional tests, but as so far tempest is running a 
> gazillion of API tests, this is probably a discussion for the medium/long 
> term. 

As you say, tempest should retain responsibility for ‘golden-path’ integration 
tests involving other OpenStack services (’scenario tests’).  Everything else 
should eventually be in-tree, though the transition period to achieve this is 
likely to be multi-cycle.


> In order to add this new job there are a few patches under review:
> [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it.
> [3] Are the patches implementing a blueprint which will enable us to specify 
> for which extensions test should be executed.
> Finally, one more note about smoketests. Although we're planning to get rid 
> of them soon, we still have failures in the pg job because of [4]. For this 
> reasons smoketests are still running for postgres in the integrated gate. As 
> load balancing and firewall API tests are part of it, they should be removed 
> from the smoke test executed on the integrated gate ([5], [6]). This is a 
> temporary measure until the postgres issue is fixed.
> Regards,
> Salvatore
> [1] https://review.openstack.org/#/c/114933/
> [2] https://review.openstack.org/#/c/114932/
> [3] 
> https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z
> [4] https://bugs.launchpad.net/nova/+bug/1305892
> [5] https://review.openstack.org/#/c/115022/
> [6] https://review.openstack.org/#/c/115023/
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack-dev mailing list

Reply via email to