Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate
On Wed, 2014-09-03 at 23:10 +0200, Salvatore Orlando wrote: > > > On 3 September 2014 22:10, Joe Gordon wrote: > > > > On Tue, Aug 26, 2014 at 4:47 PM, Salvatore Orlando > wrote: > TL; DR > A few folks are proposing to stop running tests for > neutron advanced services [ie: (lb|vpn|fw)aas] in the > integrated gate, and run them only on the neutron > gate. > > > Reason: projects like nova are 100% orthogonal to > neutron advanced services. Also, there have been > episodes in the past of unreliability of tests for > these services, and it would be good to limit affected > projects considering that more api tests and scenarios > are being added. > > > - > > > So far the neutron full job runs tests (api and > scenarios) for neutron "core" functionality as well as > neutron "advanced services", which run as neutron > service plugin. > > > It's highly unlikely, if not impossible, that changes > in projects such as nova, glance or ceilometer can > have an impact on the stability of these services. > On the other hand, instability in these services can > trigger gate failures in unrelated projects as long as > tests for these services are run in the neutron full > job in the integrated gate. There have already been > several gate-breaking bugs in lbaas scenario tests are > firewall api tests. > > Admittedly, advanced services do not have the same > level of coverage as core neutron functionality. > Therefore as more tests are being added, there is an > increased possibility of unearthing dormant bugs. > > > > > I support this split but for slightly different reasons. I am > under the impression that neutron advanced services are not > ready for prime time. If that is correct I don't think we > should be gating on things that aren't ready. > > > I deliberately avoided going into this field in my first post as I did > not want my personal opinions to appear as those of the Neutron > project core team. > Neutron has so far 5 "service" plugins. Of those I believe l3 and > metering are part of what is neutron core functionality, and, as > stated by Sean, should be tested as part of the integrated gate. > Metering is a bit of an accessory service so I'm +/- 0 on whether it > should be part or not of integrated gate tests. > > > For load balancing, v1 has been considered fairly stable for a while. > However, as it's being overhauled with lbaas v2 activities, I might > question its production readiness. > For VPN, we just do not have yet enough data points to assess its > stability in the gate (no scenario test), or production readiness. > For firewall, my impression is that it still considered an > experimental feature, but I might be mistaken. Coming from a guy who wrote a good portion of the LBaaS V2 code, I agree with Salvatore that I don't think it is production ready either, mainly because there is no driver support for it yet and the reference implementation is not scalable yet. It would be naive and reckless to think it was ready for production. So it is going into the incubator. > > > Considering the above I would also subscribe to Joe's point - under > the assumption that only things that are production ready should be > tested in the integrated gate. > > > > > For this reason we are proposing to not run anymore > tests for neutron advanced services in the integrated > gate, but keep them running on the neutron gate. > This means we will have two neutron jobs: > 1) check-tempest-dsvm-neutron-full which will run only > "core" neutron functionality > 2) check-tempest-dsvm-neutron-full-ext which will be > what the neutron full job is today. > > > Using my breakdown, the extended job would include > experimental neutron features. > > > > The former will be part of the integrated gate, the > latter will be part of the neutron gate. > Considering that other integrating services should not > have an impact on neutron advanced services, this > should not make gate
Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate
On Wed, 2014-09-03 at 15:41 -0500, Kyle Mestery wrote: > On Wed, Sep 3, 2014 at 3:22 PM, Sean Dague wrote: > > On 08/26/2014 07:47 PM, Salvatore Orlando wrote: > >> TL; DR > >> A few folks are proposing to stop running tests for neutron advanced > >> services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only > >> on the neutron gate. > >> > >> Reason: projects like nova are 100% orthogonal to neutron advanced > >> services. Also, there have been episodes in the past of unreliability of > >> tests for these services, and it would be good to limit affected > >> projects considering that more api tests and scenarios are being added. > >> > >> - > >> > >> So far the neutron full job runs tests (api and scenarios) for neutron > >> "core" functionality as well as neutron "advanced services", which run > >> as neutron service plugin. > >> > >> It's highly unlikely, if not impossible, that changes in projects such > >> as nova, glance or ceilometer can have an impact on the stability of > >> these services. > >> On the other hand, instability in these services can trigger gate > >> failures in unrelated projects as long as tests for these services are > >> run in the neutron full job in the integrated gate. There have already > >> been several gate-breaking bugs in lbaas scenario tests are firewall api > >> tests. > >> Admittedly, advanced services do not have the same level of coverage as > >> core neutron functionality. Therefore as more tests are being added, > >> there is an increased possibility of unearthing dormant bugs. > >> > >> For this reason we are proposing to not run anymore tests for neutron > >> advanced services in the integrated gate, but keep them running on the > >> neutron gate. > >> This means we will have two neutron jobs: > >> 1) check-tempest-dsvm-neutron-full which will run only "core" neutron > >> functionality > >> 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron > >> full job is today. > >> > >> The former will be part of the integrated gate, the latter will be part > >> of the neutron gate. > >> Considering that other integrating services should not have an impact on > >> neutron advanced services, this should not make gate testing asymmetric. > >> > >> However, there might be exceptions for: > >> - "orchestration" project like heat which in the future might leverage > >> capabilities like load balancing > >> - oslo-* libraries, as changes in them might have an impact on neutron > >> advanced services, since they consume those libraries > >> > >> Another good question is whether "extended" tests should be performed as > >> part of functional or tempest checks. My take on this is that scenario > >> tests should always be part of tempest. On the other hand I reckon API > >> tests should exclusively be part of functional tests, but as so far > >> tempest is running a gazillion of API tests, this is probably a > >> discussion for the medium/long term. > >> > >> In order to add this new job there are a few patches under review: > >> [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it. > >> [3] Are the patches implementing a blueprint which will enable us to > >> specify for which extensions test should be executed. > >> > >> Finally, one more note about smoketests. Although we're planning to get > >> rid of them soon, we still have failures in the pg job because of [4]. > >> For this reasons smoketests are still running for postgres in the > >> integrated gate. As load balancing and firewall API tests are part of > >> it, they should be removed from the smoke test executed on the > >> integrated gate ([5], [6]). This is a temporary measure until the > >> postgres issue is fixed. > >> > >> Regards, > >> Salvatore > >> > >> [1] https://review.openstack.org/#/c/114933/ > >> [2] https://review.openstack.org/#/c/114932/ > >> [3] > >> https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z > >> [4] https://bugs.launchpad.net/nova/+bug/1305892 > >> [5] https://review.openstack.org/#/c/115022/ > >> [6] https://review.openstack.org/#/c/115023/ > > > > +1 > > > > I realistically think that we should think about neutron as 2 things. > > The L2/L3 services, and the advanced services. L2/L3 seem appropriate to > > ensure are tightly integrated to the rest of OpenStack. The advanced > > services really are a different beast (and honestly might be better as a > > separate OpenStack service that's not neutron). > > > There is talk about spinning these out into a separate > repository/project under the networking program. LBaaS V2, for > example, is almost certainly going to end up this way. It's not > unreasonable to think we could do the same with the other services as > well. Spinning out is definitely a goal for the LBaaS team. This is not only beneficial to LBaaS but also beneficial to Neutron because Neutron's scope will shrink and it's focus more precise. Getting to that point will require
Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate
On 3 September 2014 22:10, Joe Gordon wrote: > > > > On Tue, Aug 26, 2014 at 4:47 PM, Salvatore Orlando > wrote: > >> TL; DR >> A few folks are proposing to stop running tests for neutron advanced >> services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on >> the neutron gate. >> >> Reason: projects like nova are 100% orthogonal to neutron advanced >> services. Also, there have been episodes in the past of unreliability of >> tests for these services, and it would be good to limit affected projects >> considering that more api tests and scenarios are being added. >> >> - >> >> So far the neutron full job runs tests (api and scenarios) for neutron >> "core" functionality as well as neutron "advanced services", which run as >> neutron service plugin. >> >> It's highly unlikely, if not impossible, that changes in projects such as >> nova, glance or ceilometer can have an impact on the stability of these >> services. >> On the other hand, instability in these services can trigger gate >> failures in unrelated projects as long as tests for these services are run >> in the neutron full job in the integrated gate. There have already been >> several gate-breaking bugs in lbaas scenario tests are firewall api tests. >> Admittedly, advanced services do not have the same level of coverage as >> core neutron functionality. Therefore as more tests are being added, there >> is an increased possibility of unearthing dormant bugs. >> >> > I support this split but for slightly different reasons. I am under the > impression that neutron advanced services are not ready for prime time. If > that is correct I don't think we should be gating on things that aren't > ready. > I deliberately avoided going into this field in my first post as I did not want my personal opinions to appear as those of the Neutron project core team. Neutron has so far 5 "service" plugins. Of those I believe l3 and metering are part of what is neutron core functionality, and, as stated by Sean, should be tested as part of the integrated gate. Metering is a bit of an accessory service so I'm +/- 0 on whether it should be part or not of integrated gate tests. For load balancing, v1 has been considered fairly stable for a while. However, as it's being overhauled with lbaas v2 activities, I might question its production readiness. For VPN, we just do not have yet enough data points to assess its stability in the gate (no scenario test), or production readiness. For firewall, my impression is that it still considered an experimental feature, but I might be mistaken. Considering the above I would also subscribe to Joe's point - under the assumption that only things that are production ready should be tested in the integrated gate. > > > >> For this reason we are proposing to not run anymore tests for neutron >> advanced services in the integrated gate, but keep them running on the >> neutron gate. >> This means we will have two neutron jobs: >> 1) check-tempest-dsvm-neutron-full which will run only "core" neutron >> functionality >> 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron >> full job is today. >> > > Using my breakdown, the extended job would include experimental neutron > features. > > >> >> The former will be part of the integrated gate, the latter will be part >> of the neutron gate. >> Considering that other integrating services should not have an impact on >> neutron advanced services, this should not make gate testing asymmetric. >> >> However, there might be exceptions for: >> - "orchestration" project like heat which in the future might leverage >> capabilities like load balancing >> - oslo-* libraries, as changes in them might have an impact on neutron >> advanced services, since they consume those libraries >> > > Once another service starts consuming an advanced feature I think it makes > sense to move it to the main neutron-full job. Especially if we assume that > things will only depend on neutron features that are not too experimental. > Correct. Shifting services from neutron's full-ext to the integrated gate full job should be easy especially if these projects are spun out. > >> >> Another good question is whether "extended" tests should be performed as >> part of functional or tempest checks. My take on this is that scenario >> tests should always be part of tempest. On the other hand I reckon API >> tests should exclusively be part of functional tests, but as so far tempest >> is running a gazillion of API tests, this is probably a discussion for the >> medium/long term. >> >> In order to add this new job there are a few patches under review: >> [1] and [2] Introduces the 'full-ext' job and devstack-gate support for >> it. >> [3] Are the patches implementing a blueprint which will enable us to >> specify for which extensions test should be executed. >> >> Finally, one more note about smoketests. Although we're planning to get >> rid of them soon, we still have failures in the p
Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate
On Wed, Sep 3, 2014 at 3:22 PM, Sean Dague wrote: > On 08/26/2014 07:47 PM, Salvatore Orlando wrote: >> TL; DR >> A few folks are proposing to stop running tests for neutron advanced >> services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only >> on the neutron gate. >> >> Reason: projects like nova are 100% orthogonal to neutron advanced >> services. Also, there have been episodes in the past of unreliability of >> tests for these services, and it would be good to limit affected >> projects considering that more api tests and scenarios are being added. >> >> - >> >> So far the neutron full job runs tests (api and scenarios) for neutron >> "core" functionality as well as neutron "advanced services", which run >> as neutron service plugin. >> >> It's highly unlikely, if not impossible, that changes in projects such >> as nova, glance or ceilometer can have an impact on the stability of >> these services. >> On the other hand, instability in these services can trigger gate >> failures in unrelated projects as long as tests for these services are >> run in the neutron full job in the integrated gate. There have already >> been several gate-breaking bugs in lbaas scenario tests are firewall api >> tests. >> Admittedly, advanced services do not have the same level of coverage as >> core neutron functionality. Therefore as more tests are being added, >> there is an increased possibility of unearthing dormant bugs. >> >> For this reason we are proposing to not run anymore tests for neutron >> advanced services in the integrated gate, but keep them running on the >> neutron gate. >> This means we will have two neutron jobs: >> 1) check-tempest-dsvm-neutron-full which will run only "core" neutron >> functionality >> 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron >> full job is today. >> >> The former will be part of the integrated gate, the latter will be part >> of the neutron gate. >> Considering that other integrating services should not have an impact on >> neutron advanced services, this should not make gate testing asymmetric. >> >> However, there might be exceptions for: >> - "orchestration" project like heat which in the future might leverage >> capabilities like load balancing >> - oslo-* libraries, as changes in them might have an impact on neutron >> advanced services, since they consume those libraries >> >> Another good question is whether "extended" tests should be performed as >> part of functional or tempest checks. My take on this is that scenario >> tests should always be part of tempest. On the other hand I reckon API >> tests should exclusively be part of functional tests, but as so far >> tempest is running a gazillion of API tests, this is probably a >> discussion for the medium/long term. >> >> In order to add this new job there are a few patches under review: >> [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it. >> [3] Are the patches implementing a blueprint which will enable us to >> specify for which extensions test should be executed. >> >> Finally, one more note about smoketests. Although we're planning to get >> rid of them soon, we still have failures in the pg job because of [4]. >> For this reasons smoketests are still running for postgres in the >> integrated gate. As load balancing and firewall API tests are part of >> it, they should be removed from the smoke test executed on the >> integrated gate ([5], [6]). This is a temporary measure until the >> postgres issue is fixed. >> >> Regards, >> Salvatore >> >> [1] https://review.openstack.org/#/c/114933/ >> [2] https://review.openstack.org/#/c/114932/ >> [3] >> https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z >> [4] https://bugs.launchpad.net/nova/+bug/1305892 >> [5] https://review.openstack.org/#/c/115022/ >> [6] https://review.openstack.org/#/c/115023/ > > +1 > > I realistically think that we should think about neutron as 2 things. > The L2/L3 services, and the advanced services. L2/L3 seem appropriate to > ensure are tightly integrated to the rest of OpenStack. The advanced > services really are a different beast (and honestly might be better as a > separate OpenStack service that's not neutron). > There is talk about spinning these out into a separate repository/project under the networking program. LBaaS V2, for example, is almost certainly going to end up this way. It's not unreasonable to think we could do the same with the other services as well. Thanks, Kyle > -Sean > > -- > Sean Dague > http://dague.net > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate
On 08/26/2014 07:47 PM, Salvatore Orlando wrote: > TL; DR > A few folks are proposing to stop running tests for neutron advanced > services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only > on the neutron gate. > > Reason: projects like nova are 100% orthogonal to neutron advanced > services. Also, there have been episodes in the past of unreliability of > tests for these services, and it would be good to limit affected > projects considering that more api tests and scenarios are being added. > > - > > So far the neutron full job runs tests (api and scenarios) for neutron > "core" functionality as well as neutron "advanced services", which run > as neutron service plugin. > > It's highly unlikely, if not impossible, that changes in projects such > as nova, glance or ceilometer can have an impact on the stability of > these services. > On the other hand, instability in these services can trigger gate > failures in unrelated projects as long as tests for these services are > run in the neutron full job in the integrated gate. There have already > been several gate-breaking bugs in lbaas scenario tests are firewall api > tests. > Admittedly, advanced services do not have the same level of coverage as > core neutron functionality. Therefore as more tests are being added, > there is an increased possibility of unearthing dormant bugs. > > For this reason we are proposing to not run anymore tests for neutron > advanced services in the integrated gate, but keep them running on the > neutron gate. > This means we will have two neutron jobs: > 1) check-tempest-dsvm-neutron-full which will run only "core" neutron > functionality > 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron > full job is today. > > The former will be part of the integrated gate, the latter will be part > of the neutron gate. > Considering that other integrating services should not have an impact on > neutron advanced services, this should not make gate testing asymmetric. > > However, there might be exceptions for: > - "orchestration" project like heat which in the future might leverage > capabilities like load balancing > - oslo-* libraries, as changes in them might have an impact on neutron > advanced services, since they consume those libraries > > Another good question is whether "extended" tests should be performed as > part of functional or tempest checks. My take on this is that scenario > tests should always be part of tempest. On the other hand I reckon API > tests should exclusively be part of functional tests, but as so far > tempest is running a gazillion of API tests, this is probably a > discussion for the medium/long term. > > In order to add this new job there are a few patches under review: > [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it. > [3] Are the patches implementing a blueprint which will enable us to > specify for which extensions test should be executed. > > Finally, one more note about smoketests. Although we're planning to get > rid of them soon, we still have failures in the pg job because of [4]. > For this reasons smoketests are still running for postgres in the > integrated gate. As load balancing and firewall API tests are part of > it, they should be removed from the smoke test executed on the > integrated gate ([5], [6]). This is a temporary measure until the > postgres issue is fixed. > > Regards, > Salvatore > > [1] https://review.openstack.org/#/c/114933/ > [2] https://review.openstack.org/#/c/114932/ > [3] > https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z > [4] https://bugs.launchpad.net/nova/+bug/1305892 > [5] https://review.openstack.org/#/c/115022/ > [6] https://review.openstack.org/#/c/115023/ +1 I realistically think that we should think about neutron as 2 things. The L2/L3 services, and the advanced services. L2/L3 seem appropriate to ensure are tightly integrated to the rest of OpenStack. The advanced services really are a different beast (and honestly might be better as a separate OpenStack service that's not neutron). -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate
On Tue, Aug 26, 2014 at 4:47 PM, Salvatore Orlando wrote: > TL; DR > A few folks are proposing to stop running tests for neutron advanced > services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on > the neutron gate. > > Reason: projects like nova are 100% orthogonal to neutron advanced > services. Also, there have been episodes in the past of unreliability of > tests for these services, and it would be good to limit affected projects > considering that more api tests and scenarios are being added. > > - > > So far the neutron full job runs tests (api and scenarios) for neutron > "core" functionality as well as neutron "advanced services", which run as > neutron service plugin. > > It's highly unlikely, if not impossible, that changes in projects such as > nova, glance or ceilometer can have an impact on the stability of these > services. > On the other hand, instability in these services can trigger gate failures > in unrelated projects as long as tests for these services are run in the > neutron full job in the integrated gate. There have already been several > gate-breaking bugs in lbaas scenario tests are firewall api tests. > Admittedly, advanced services do not have the same level of coverage as > core neutron functionality. Therefore as more tests are being added, there > is an increased possibility of unearthing dormant bugs. > > I support this split but for slightly different reasons. I am under the impression that neutron advanced services are not ready for prime time. If that is correct I don't think we should be gating on things that aren't ready. > For this reason we are proposing to not run anymore tests for neutron > advanced services in the integrated gate, but keep them running on the > neutron gate. > This means we will have two neutron jobs: > 1) check-tempest-dsvm-neutron-full which will run only "core" neutron > functionality > 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full > job is today. > Using my breakdown, the extended job would include experimental neutron features. > > The former will be part of the integrated gate, the latter will be part of > the neutron gate. > Considering that other integrating services should not have an impact on > neutron advanced services, this should not make gate testing asymmetric. > > However, there might be exceptions for: > - "orchestration" project like heat which in the future might leverage > capabilities like load balancing > - oslo-* libraries, as changes in them might have an impact on neutron > advanced services, since they consume those libraries > Once another service starts consuming an advanced feature I think it makes sense to move it to the main neutron-full job. Especially if we assume that things will only depend on neutron features that are not too experimental. > > Another good question is whether "extended" tests should be performed as > part of functional or tempest checks. My take on this is that scenario > tests should always be part of tempest. On the other hand I reckon API > tests should exclusively be part of functional tests, but as so far tempest > is running a gazillion of API tests, this is probably a discussion for the > medium/long term. > > In order to add this new job there are a few patches under review: > [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it. > [3] Are the patches implementing a blueprint which will enable us to > specify for which extensions test should be executed. > > Finally, one more note about smoketests. Although we're planning to get > rid of them soon, we still have failures in the pg job because of [4]. For > this reasons smoketests are still running for postgres in the integrated > gate. As load balancing and firewall API tests are part of it, they should > be removed from the smoke test executed on the integrated gate ([5], [6]). > This is a temporary measure until the postgres issue is fixed. > ++ > > Regards, > Salvatore > > [1] https://review.openstack.org/#/c/114933/ > [2] https://review.openstack.org/#/c/114932/ > [3] > https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z > [4] https://bugs.launchpad.net/nova/+bug/1305892 > [5] https://review.openstack.org/#/c/115022/ > [6] https://review.openstack.org/#/c/115023/ > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate
On Aug 27, 2014, at 1:47 AM, Salvatore Orlando wrote: > TL; DR > A few folks are proposing to stop running tests for neutron advanced services > [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on the neutron > gate. > > Reason: projects like nova are 100% orthogonal to neutron advanced services. > Also, there have been episodes in the past of unreliability of tests for > these services, and it would be good to limit affected projects considering > that more api tests and scenarios are being added. Given how many rechecks I’ve had to do to merge what are effectively no-op patches to infra/config, most often due to the full neutron job exhibiting sporadic failures, I fully support this change. I think we need time to stabilize the tests for advanced services against just neutron before we consider slowing down merges for other projects. > > - > > So far the neutron full job runs tests (api and scenarios) for neutron "core" > functionality as well as neutron "advanced services", which run as neutron > service plugin. > > It's highly unlikely, if not impossible, that changes in projects such as > nova, glance or ceilometer can have an impact on the stability of these > services. > On the other hand, instability in these services can trigger gate failures in > unrelated projects as long as tests for these services are run in the neutron > full job in the integrated gate. There have already been several > gate-breaking bugs in lbaas scenario tests are firewall api tests. > Admittedly, advanced services do not have the same level of coverage as core > neutron functionality. Therefore as more tests are being added, there is an > increased possibility of unearthing dormant bugs. > > For this reason we are proposing to not run anymore tests for neutron > advanced services in the integrated gate, but keep them running on the > neutron gate. > This means we will have two neutron jobs: > 1) check-tempest-dsvm-neutron-full which will run only "core" neutron > functionality > 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full > job is today. > > The former will be part of the integrated gate, the latter will be part of > the neutron gate. > Considering that other integrating services should not have an impact on > neutron advanced services, this should not make gate testing asymmetric. > > However, there might be exceptions for: > - "orchestration" project like heat which in the future might leverage > capabilities like load balancing > - oslo-* libraries, as changes in them might have an impact on neutron > advanced services, since they consume those libraries > > Another good question is whether "extended" tests should be performed as part > of functional or tempest checks. My take on this is that scenario tests > should always be part of tempest. On the other hand I reckon API tests should > exclusively be part of functional tests, but as so far tempest is running a > gazillion of API tests, this is probably a discussion for the medium/long > term. As you say, tempest should retain responsibility for ‘golden-path’ integration tests involving other OpenStack services (’scenario tests’). Everything else should eventually be in-tree, though the transition period to achieve this is likely to be multi-cycle. m. > > In order to add this new job there are a few patches under review: > [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it. > [3] Are the patches implementing a blueprint which will enable us to specify > for which extensions test should be executed. > > Finally, one more note about smoketests. Although we're planning to get rid > of them soon, we still have failures in the pg job because of [4]. For this > reasons smoketests are still running for postgres in the integrated gate. As > load balancing and firewall API tests are part of it, they should be removed > from the smoke test executed on the integrated gate ([5], [6]). This is a > temporary measure until the postgres issue is fixed. > > Regards, > Salvatore > > [1] https://review.openstack.org/#/c/114933/ > [2] https://review.openstack.org/#/c/114932/ > [3] > https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z > [4] https://bugs.launchpad.net/nova/+bug/1305892 > [5] https://review.openstack.org/#/c/115022/ > [6] https://review.openstack.org/#/c/115023/ > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate
TL; DR A few folks are proposing to stop running tests for neutron advanced services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on the neutron gate. Reason: projects like nova are 100% orthogonal to neutron advanced services. Also, there have been episodes in the past of unreliability of tests for these services, and it would be good to limit affected projects considering that more api tests and scenarios are being added. - So far the neutron full job runs tests (api and scenarios) for neutron "core" functionality as well as neutron "advanced services", which run as neutron service plugin. It's highly unlikely, if not impossible, that changes in projects such as nova, glance or ceilometer can have an impact on the stability of these services. On the other hand, instability in these services can trigger gate failures in unrelated projects as long as tests for these services are run in the neutron full job in the integrated gate. There have already been several gate-breaking bugs in lbaas scenario tests are firewall api tests. Admittedly, advanced services do not have the same level of coverage as core neutron functionality. Therefore as more tests are being added, there is an increased possibility of unearthing dormant bugs. For this reason we are proposing to not run anymore tests for neutron advanced services in the integrated gate, but keep them running on the neutron gate. This means we will have two neutron jobs: 1) check-tempest-dsvm-neutron-full which will run only "core" neutron functionality 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full job is today. The former will be part of the integrated gate, the latter will be part of the neutron gate. Considering that other integrating services should not have an impact on neutron advanced services, this should not make gate testing asymmetric. However, there might be exceptions for: - "orchestration" project like heat which in the future might leverage capabilities like load balancing - oslo-* libraries, as changes in them might have an impact on neutron advanced services, since they consume those libraries Another good question is whether "extended" tests should be performed as part of functional or tempest checks. My take on this is that scenario tests should always be part of tempest. On the other hand I reckon API tests should exclusively be part of functional tests, but as so far tempest is running a gazillion of API tests, this is probably a discussion for the medium/long term. In order to add this new job there are a few patches under review: [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it. [3] Are the patches implementing a blueprint which will enable us to specify for which extensions test should be executed. Finally, one more note about smoketests. Although we're planning to get rid of them soon, we still have failures in the pg job because of [4]. For this reasons smoketests are still running for postgres in the integrated gate. As load balancing and firewall API tests are part of it, they should be removed from the smoke test executed on the integrated gate ([5], [6]). This is a temporary measure until the postgres issue is fixed. Regards, Salvatore [1] https://review.openstack.org/#/c/114933/ [2] https://review.openstack.org/#/c/114932/ [3] https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z [4] https://bugs.launchpad.net/nova/+bug/1305892 [5] https://review.openstack.org/#/c/115022/ [6] https://review.openstack.org/#/c/115023/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
