Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate

2014-09-03 Thread Joe Gordon
On Tue, Aug 26, 2014 at 4:47 PM, Salvatore Orlando sorla...@nicira.com
wrote:

 TL; DR
 A few folks are proposing to stop running tests for neutron advanced
 services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on
 the neutron gate.

 Reason: projects like nova are 100% orthogonal to neutron advanced
 services. Also, there have been episodes in the past of unreliability of
 tests for these services, and it would be good to limit affected projects
 considering that more api tests and scenarios are being added.

 -

 So far the neutron full job runs tests (api and scenarios) for neutron
 core functionality as well as neutron advanced services, which run as
 neutron service plugin.

 It's highly unlikely, if not impossible, that changes in projects such as
 nova, glance or ceilometer can have an impact on the stability of these
 services.
 On the other hand, instability in these services can trigger gate failures
 in unrelated projects as long as tests for these services are run in the
 neutron full job in the integrated gate. There have already been several
 gate-breaking bugs in lbaas scenario tests are firewall api tests.
 Admittedly, advanced services do not have the same level of coverage as
 core neutron functionality. Therefore as more tests are being added, there
 is an increased possibility of unearthing dormant bugs.


I support this split but for slightly different reasons.  I am under the
impression that neutron advanced services are not ready for prime time. If
that is correct I don't think we should be gating on things that aren't
ready.



 For this reason we are proposing to not run anymore tests for neutron
 advanced services in the integrated gate, but keep them running on the
 neutron gate.
 This means we will have two neutron jobs:
 1) check-tempest-dsvm-neutron-full which will run only core neutron
 functionality
 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full
 job is today.


Using my breakdown, the extended job would include experimental neutron
features.



 The former will be part of the integrated gate, the latter will be part of
 the neutron gate.
 Considering that other integrating services should not have an impact on
 neutron advanced services, this should not make gate testing asymmetric.

 However, there might be exceptions for:
 - orchestration project like heat which in the future might leverage
 capabilities like load balancing
 - oslo-* libraries, as changes in them might have an impact on neutron
 advanced services, since they consume those libraries


Once another service starts consuming an advanced feature I think it makes
sense to move it to the main neutron-full job. Especially if we assume that
things will only depend on neutron features that are not too experimental.



 Another good question is whether extended tests should be performed as
 part of functional or tempest checks. My take on this is that scenario
 tests should always be part of tempest. On the other hand I reckon API
 tests should exclusively be part of functional tests, but as so far tempest
 is running a gazillion of API tests, this is probably a discussion for the
 medium/long term.

 In order to add this new job there are a few patches under review:
 [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it.
 [3] Are the patches implementing a blueprint which will enable us to
 specify for which extensions test should be executed.

 Finally, one more note about smoketests. Although we're planning to get
 rid of them soon, we still have failures in the pg job because of [4]. For
 this reasons smoketests are still running for postgres in the integrated
 gate. As load balancing and firewall API tests are part of it, they should
 be removed from the smoke test executed on the integrated gate ([5], [6]).
 This is a temporary measure until the postgres issue is fixed.


++



 Regards,
 Salvatore

 [1] https://review.openstack.org/#/c/114933/
 [2] https://review.openstack.org/#/c/114932/
 [3]
 https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z
 [4] https://bugs.launchpad.net/nova/+bug/1305892
 [5] https://review.openstack.org/#/c/115022/
 [6] https://review.openstack.org/#/c/115023/


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate

2014-09-03 Thread Sean Dague
On 08/26/2014 07:47 PM, Salvatore Orlando wrote:
 TL; DR
 A few folks are proposing to stop running tests for neutron advanced
 services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only
 on the neutron gate.
 
 Reason: projects like nova are 100% orthogonal to neutron advanced
 services. Also, there have been episodes in the past of unreliability of
 tests for these services, and it would be good to limit affected
 projects considering that more api tests and scenarios are being added.
 
 -
 
 So far the neutron full job runs tests (api and scenarios) for neutron
 core functionality as well as neutron advanced services, which run
 as neutron service plugin.
 
 It's highly unlikely, if not impossible, that changes in projects such
 as nova, glance or ceilometer can have an impact on the stability of
 these services.
 On the other hand, instability in these services can trigger gate
 failures in unrelated projects as long as tests for these services are
 run in the neutron full job in the integrated gate. There have already
 been several gate-breaking bugs in lbaas scenario tests are firewall api
 tests.
 Admittedly, advanced services do not have the same level of coverage as
 core neutron functionality. Therefore as more tests are being added,
 there is an increased possibility of unearthing dormant bugs.
 
 For this reason we are proposing to not run anymore tests for neutron
 advanced services in the integrated gate, but keep them running on the
 neutron gate.
 This means we will have two neutron jobs:
 1) check-tempest-dsvm-neutron-full which will run only core neutron
 functionality
 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron
 full job is today.
 
 The former will be part of the integrated gate, the latter will be part
 of the neutron gate.
 Considering that other integrating services should not have an impact on
 neutron advanced services, this should not make gate testing asymmetric.
 
 However, there might be exceptions for:
 - orchestration project like heat which in the future might leverage
 capabilities like load balancing
 - oslo-* libraries, as changes in them might have an impact on neutron
 advanced services, since they consume those libraries
 
 Another good question is whether extended tests should be performed as
 part of functional or tempest checks. My take on this is that scenario
 tests should always be part of tempest. On the other hand I reckon API
 tests should exclusively be part of functional tests, but as so far
 tempest is running a gazillion of API tests, this is probably a
 discussion for the medium/long term. 
 
 In order to add this new job there are a few patches under review:
 [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it.
 [3] Are the patches implementing a blueprint which will enable us to
 specify for which extensions test should be executed.
 
 Finally, one more note about smoketests. Although we're planning to get
 rid of them soon, we still have failures in the pg job because of [4].
 For this reasons smoketests are still running for postgres in the
 integrated gate. As load balancing and firewall API tests are part of
 it, they should be removed from the smoke test executed on the
 integrated gate ([5], [6]). This is a temporary measure until the
 postgres issue is fixed.
 
 Regards,
 Salvatore
 
 [1] https://review.openstack.org/#/c/114933/
 [2] https://review.openstack.org/#/c/114932/
 [3] 
 https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z
 [4] https://bugs.launchpad.net/nova/+bug/1305892
 [5] https://review.openstack.org/#/c/115022/
 [6] https://review.openstack.org/#/c/115023/

+1

I realistically think that we should think about neutron as 2 things.
The L2/L3 services, and the advanced services. L2/L3 seem appropriate to
ensure are tightly integrated to the rest of OpenStack. The advanced
services really are a different beast (and honestly might be better as a
separate OpenStack service that's not neutron).

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate

2014-09-03 Thread Salvatore Orlando
On 3 September 2014 22:10, Joe Gordon joe.gord...@gmail.com wrote:




 On Tue, Aug 26, 2014 at 4:47 PM, Salvatore Orlando sorla...@nicira.com
 wrote:

 TL; DR
 A few folks are proposing to stop running tests for neutron advanced
 services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on
 the neutron gate.

 Reason: projects like nova are 100% orthogonal to neutron advanced
 services. Also, there have been episodes in the past of unreliability of
 tests for these services, and it would be good to limit affected projects
 considering that more api tests and scenarios are being added.

 -

 So far the neutron full job runs tests (api and scenarios) for neutron
 core functionality as well as neutron advanced services, which run as
 neutron service plugin.

 It's highly unlikely, if not impossible, that changes in projects such as
 nova, glance or ceilometer can have an impact on the stability of these
 services.
 On the other hand, instability in these services can trigger gate
 failures in unrelated projects as long as tests for these services are run
 in the neutron full job in the integrated gate. There have already been
 several gate-breaking bugs in lbaas scenario tests are firewall api tests.
 Admittedly, advanced services do not have the same level of coverage as
 core neutron functionality. Therefore as more tests are being added, there
 is an increased possibility of unearthing dormant bugs.


 I support this split but for slightly different reasons.  I am under the
 impression that neutron advanced services are not ready for prime time. If
 that is correct I don't think we should be gating on things that aren't
 ready.


I deliberately avoided going into this field in my first post as I did not
want my personal opinions to appear as those of the Neutron project core
team.
Neutron has so far 5 service plugins. Of those I believe l3 and metering
are part of what is neutron core functionality, and, as stated by Sean,
should be tested as part of the integrated gate. Metering is a bit of an
accessory service so I'm +/- 0 on whether it should be part or not of
integrated gate tests.

For load balancing, v1 has been considered fairly stable for a while.
However, as it's being overhauled with lbaas v2 activities, I might
question its production readiness.
For VPN, we just do not have yet enough data points to assess its stability
in the gate (no scenario test), or production readiness.
For firewall, my impression is that it still considered an experimental
feature, but I might be mistaken.

Considering the above I would also subscribe to Joe's point - under the
assumption that only things that are production ready should be tested in
the integrated gate.





 For this reason we are proposing to not run anymore tests for neutron
 advanced services in the integrated gate, but keep them running on the
 neutron gate.
 This means we will have two neutron jobs:
 1) check-tempest-dsvm-neutron-full which will run only core neutron
 functionality
 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron
 full job is today.


 Using my breakdown, the extended job would include experimental neutron
 features.




 The former will be part of the integrated gate, the latter will be part
 of the neutron gate.
 Considering that other integrating services should not have an impact on
 neutron advanced services, this should not make gate testing asymmetric.

 However, there might be exceptions for:
 - orchestration project like heat which in the future might leverage
 capabilities like load balancing
 - oslo-* libraries, as changes in them might have an impact on neutron
 advanced services, since they consume those libraries


 Once another service starts consuming an advanced feature I think it makes
 sense to move it to the main neutron-full job. Especially if we assume that
 things will only depend on neutron features that are not too experimental.


Correct. Shifting services from neutron's full-ext to the integrated gate
full job should be easy especially if these projects are spun out.




 Another good question is whether extended tests should be performed as
 part of functional or tempest checks. My take on this is that scenario
 tests should always be part of tempest. On the other hand I reckon API
 tests should exclusively be part of functional tests, but as so far tempest
 is running a gazillion of API tests, this is probably a discussion for the
 medium/long term.

 In order to add this new job there are a few patches under review:
 [1] and [2] Introduces the 'full-ext' job and devstack-gate support for
 it.
 [3] Are the patches implementing a blueprint which will enable us to
 specify for which extensions test should be executed.

 Finally, one more note about smoketests. Although we're planning to get
 rid of them soon, we still have failures in the pg job because of [4]. For
 this reasons smoketests are still running for postgres in the integrated
 gate. As load balancing 

Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate

2014-09-03 Thread Brandon Logan
On Wed, 2014-09-03 at 15:41 -0500, Kyle Mestery wrote:
 On Wed, Sep 3, 2014 at 3:22 PM, Sean Dague s...@dague.net wrote:
  On 08/26/2014 07:47 PM, Salvatore Orlando wrote:
  TL; DR
  A few folks are proposing to stop running tests for neutron advanced
  services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only
  on the neutron gate.
 
  Reason: projects like nova are 100% orthogonal to neutron advanced
  services. Also, there have been episodes in the past of unreliability of
  tests for these services, and it would be good to limit affected
  projects considering that more api tests and scenarios are being added.
 
  -
 
  So far the neutron full job runs tests (api and scenarios) for neutron
  core functionality as well as neutron advanced services, which run
  as neutron service plugin.
 
  It's highly unlikely, if not impossible, that changes in projects such
  as nova, glance or ceilometer can have an impact on the stability of
  these services.
  On the other hand, instability in these services can trigger gate
  failures in unrelated projects as long as tests for these services are
  run in the neutron full job in the integrated gate. There have already
  been several gate-breaking bugs in lbaas scenario tests are firewall api
  tests.
  Admittedly, advanced services do not have the same level of coverage as
  core neutron functionality. Therefore as more tests are being added,
  there is an increased possibility of unearthing dormant bugs.
 
  For this reason we are proposing to not run anymore tests for neutron
  advanced services in the integrated gate, but keep them running on the
  neutron gate.
  This means we will have two neutron jobs:
  1) check-tempest-dsvm-neutron-full which will run only core neutron
  functionality
  2) check-tempest-dsvm-neutron-full-ext which will be what the neutron
  full job is today.
 
  The former will be part of the integrated gate, the latter will be part
  of the neutron gate.
  Considering that other integrating services should not have an impact on
  neutron advanced services, this should not make gate testing asymmetric.
 
  However, there might be exceptions for:
  - orchestration project like heat which in the future might leverage
  capabilities like load balancing
  - oslo-* libraries, as changes in them might have an impact on neutron
  advanced services, since they consume those libraries
 
  Another good question is whether extended tests should be performed as
  part of functional or tempest checks. My take on this is that scenario
  tests should always be part of tempest. On the other hand I reckon API
  tests should exclusively be part of functional tests, but as so far
  tempest is running a gazillion of API tests, this is probably a
  discussion for the medium/long term.
 
  In order to add this new job there are a few patches under review:
  [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it.
  [3] Are the patches implementing a blueprint which will enable us to
  specify for which extensions test should be executed.
 
  Finally, one more note about smoketests. Although we're planning to get
  rid of them soon, we still have failures in the pg job because of [4].
  For this reasons smoketests are still running for postgres in the
  integrated gate. As load balancing and firewall API tests are part of
  it, they should be removed from the smoke test executed on the
  integrated gate ([5], [6]). This is a temporary measure until the
  postgres issue is fixed.
 
  Regards,
  Salvatore
 
  [1] https://review.openstack.org/#/c/114933/
  [2] https://review.openstack.org/#/c/114932/
  [3] 
  https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z
  [4] https://bugs.launchpad.net/nova/+bug/1305892
  [5] https://review.openstack.org/#/c/115022/
  [6] https://review.openstack.org/#/c/115023/
 
  +1
 
  I realistically think that we should think about neutron as 2 things.
  The L2/L3 services, and the advanced services. L2/L3 seem appropriate to
  ensure are tightly integrated to the rest of OpenStack. The advanced
  services really are a different beast (and honestly might be better as a
  separate OpenStack service that's not neutron).
 
 There is talk about spinning these out into a separate
 repository/project under the networking program. LBaaS V2, for
 example, is almost certainly going to end up this way. It's not
 unreasonable to think we could do the same with the other services as
 well.

Spinning out is definitely a goal for the LBaaS team.  This is not only
beneficial to LBaaS but also beneficial to Neutron because Neutron's
scope will shrink and it's focus more precise.  Getting to that point
will require effort from both sides.

 
 Thanks,
 Kyle
 
  -Sean
 
  --
  Sean Dague
  http://dague.net
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  

Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate

2014-09-03 Thread Brandon Logan
On Wed, 2014-09-03 at 23:10 +0200, Salvatore Orlando wrote:
 
 
 On 3 September 2014 22:10, Joe Gordon joe.gord...@gmail.com wrote:
 
 
 
 On Tue, Aug 26, 2014 at 4:47 PM, Salvatore Orlando
 sorla...@nicira.com wrote:
 TL; DR
 A few folks are proposing to stop running tests for
 neutron advanced services [ie: (lb|vpn|fw)aas] in the
 integrated gate, and run them only on the neutron
 gate.
 
 
 Reason: projects like nova are 100% orthogonal to
 neutron advanced services. Also, there have been
 episodes in the past of unreliability of tests for
 these services, and it would be good to limit affected
 projects considering that more api tests and scenarios
 are being added.
 
 
 -
 
 
 So far the neutron full job runs tests (api and
 scenarios) for neutron core functionality as well as
 neutron advanced services, which run as neutron
 service plugin.
 
 
 It's highly unlikely, if not impossible, that changes
 in projects such as nova, glance or ceilometer can
 have an impact on the stability of these services.
 On the other hand, instability in these services can
 trigger gate failures in unrelated projects as long as
 tests for these services are run in the neutron full
 job in the integrated gate. There have already been
 several gate-breaking bugs in lbaas scenario tests are
 firewall api tests.
 
 Admittedly, advanced services do not have the same
 level of coverage as core neutron functionality.
 Therefore as more tests are being added, there is an
 increased possibility of unearthing dormant bugs.
 
 
 
 
 I support this split but for slightly different reasons.  I am
 under the impression that neutron advanced services are not
 ready for prime time. If that is correct I don't think we
 should be gating on things that aren't ready.
 
 
 I deliberately avoided going into this field in my first post as I did
 not want my personal opinions to appear as those of the Neutron
 project core team.
 Neutron has so far 5 service plugins. Of those I believe l3 and
 metering are part of what is neutron core functionality, and, as
 stated by Sean, should be tested as part of the integrated gate.
 Metering is a bit of an accessory service so I'm +/- 0 on whether it
 should be part or not of integrated gate tests.
 
 
 For load balancing, v1 has been considered fairly stable for a while.
 However, as it's being overhauled with lbaas v2 activities, I might
 question its production readiness.
 For VPN, we just do not have yet enough data points to assess its
 stability in the gate (no scenario test), or production readiness.
 For firewall, my impression is that it still considered an
 experimental feature, but I might be mistaken.

Coming from a guy who wrote a good portion of the LBaaS V2 code, I agree
with Salvatore that I don't think it is production ready either, mainly
because there is no driver support for it yet and the reference
implementation is not scalable yet. It would be naive and reckless to
think it was ready for production. So it is going into the incubator.
 
 
 Considering the above I would also subscribe to Joe's point - under
 the assumption that only things that are production ready should be
 tested in the integrated gate.
  
 
 
  
 For this reason we are proposing to not run anymore
 tests for neutron advanced services in the integrated
 gate, but keep them running on the neutron gate.
 This means we will have two neutron jobs:
 1) check-tempest-dsvm-neutron-full which will run only
 core neutron functionality
 2) check-tempest-dsvm-neutron-full-ext which will be
 what the neutron full job is today.
 
 
 Using my breakdown, the extended job would include
 experimental neutron features. 
  
 
 
 The former will be part of the integrated gate, the
 latter will be part of the neutron gate.
 Considering that other integrating services should not
 have an impact on neutron advanced services, this
 should not make gate testing asymmetric.
 
 
 

Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate

2014-09-01 Thread Maru Newby

On Aug 27, 2014, at 1:47 AM, Salvatore Orlando sorla...@nicira.com wrote:

 TL; DR
 A few folks are proposing to stop running tests for neutron advanced services 
 [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on the neutron 
 gate.
 
 Reason: projects like nova are 100% orthogonal to neutron advanced services. 
 Also, there have been episodes in the past of unreliability of tests for 
 these services, and it would be good to limit affected projects considering 
 that more api tests and scenarios are being added.

Given how many rechecks I’ve had to do to merge what are effectively no-op 
patches to infra/config, most often due to the full neutron job exhibiting 
sporadic failures, I fully support this change.  I think we need time to 
stabilize the tests for advanced services against just neutron before we 
consider slowing down merges for other projects.


 
 -
 
 So far the neutron full job runs tests (api and scenarios) for neutron core 
 functionality as well as neutron advanced services, which run as neutron 
 service plugin.
 
 It's highly unlikely, if not impossible, that changes in projects such as 
 nova, glance or ceilometer can have an impact on the stability of these 
 services.
 On the other hand, instability in these services can trigger gate failures in 
 unrelated projects as long as tests for these services are run in the neutron 
 full job in the integrated gate. There have already been several 
 gate-breaking bugs in lbaas scenario tests are firewall api tests.
 Admittedly, advanced services do not have the same level of coverage as core 
 neutron functionality. Therefore as more tests are being added, there is an 
 increased possibility of unearthing dormant bugs.
 
 For this reason we are proposing to not run anymore tests for neutron 
 advanced services in the integrated gate, but keep them running on the 
 neutron gate.
 This means we will have two neutron jobs:
 1) check-tempest-dsvm-neutron-full which will run only core neutron 
 functionality
 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full 
 job is today.
 
 The former will be part of the integrated gate, the latter will be part of 
 the neutron gate.
 Considering that other integrating services should not have an impact on 
 neutron advanced services, this should not make gate testing asymmetric.
 
 However, there might be exceptions for:
 - orchestration project like heat which in the future might leverage 
 capabilities like load balancing
 - oslo-* libraries, as changes in them might have an impact on neutron 
 advanced services, since they consume those libraries
 
 Another good question is whether extended tests should be performed as part 
 of functional or tempest checks. My take on this is that scenario tests 
 should always be part of tempest. On the other hand I reckon API tests should 
 exclusively be part of functional tests, but as so far tempest is running a 
 gazillion of API tests, this is probably a discussion for the medium/long 
 term. 

As you say, tempest should retain responsibility for ‘golden-path’ integration 
tests involving other OpenStack services (’scenario tests’).  Everything else 
should eventually be in-tree, though the transition period to achieve this is 
likely to be multi-cycle.


m.

 
 In order to add this new job there are a few patches under review:
 [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it.
 [3] Are the patches implementing a blueprint which will enable us to specify 
 for which extensions test should be executed.
 
 Finally, one more note about smoketests. Although we're planning to get rid 
 of them soon, we still have failures in the pg job because of [4]. For this 
 reasons smoketests are still running for postgres in the integrated gate. As 
 load balancing and firewall API tests are part of it, they should be removed 
 from the smoke test executed on the integrated gate ([5], [6]). This is a 
 temporary measure until the postgres issue is fixed.
 
 Regards,
 Salvatore
 
 [1] https://review.openstack.org/#/c/114933/
 [2] https://review.openstack.org/#/c/114932/
 [3] 
 https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z
 [4] https://bugs.launchpad.net/nova/+bug/1305892
 [5] https://review.openstack.org/#/c/115022/
 [6] https://review.openstack.org/#/c/115023/
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev