Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Brian Haley]

On 10/14/2016 05:53 PM, Clark Boylan wrote:

On Thu, Oct 13, 2016, at 05:47 PM, Emilien Macchi wrote:

Greetings OpenStack,

== Background

Since the beginning of OpenStack (or almost), devstack has been used
as a common tool to deploy OpenStack in CI environment. Most of
OpenStack projects (if not all) that are written in Python use it to
deploy the different components.
While devstack became popular and the reference in term of deployment
tool for continuous integration, devstack doesn't deploy OpenStack in
production (versus some tools like Kolla, Fuel, TripleO, Juju, etc).
It means things might (and did) break when deploying OpenStack outside
devstack, for different reasons. Some examples:

* until recently, SSL was not tested, and I believe some projects
still don't test with SSL enabled.
* IPv6 is not tested everywhere.




IPv6 testing likely means two different things to two different groups
of people. First is whether or not the cloud endpoints are hosted over
IPv6 and the other is whether or not instances launched by openstack are
assigned IPv6 addresses. The second thing has been tested for quite a
while now (tempest has had tests for it for almost 2 years though it
hasn't been enabled in the gate for that long). We should definitely
ensure that we are testing with openstack servers listening on IPv6
addresses as well.


The first item - IPv6 service endpoints, is actually covered by an experimental 
job (tempest-dsvm-neutron-serviceipv6), and devstack supports it in local.conf:


SERVICE_IP_VERSION=6

Last year it was working great, bug I've noticed there are failures now, I'll 
take a crack at getting it all working again.  Maybe it's something we could 
then promote to voting?


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Clark Boylan]

On Thu, Oct 13, 2016, at 05:47 PM, Emilien Macchi wrote:
> Greetings OpenStack,
> 
> == Background
> 
> Since the beginning of OpenStack (or almost), devstack has been used
> as a common tool to deploy OpenStack in CI environment. Most of
> OpenStack projects (if not all) that are written in Python use it to
> deploy the different components.
> While devstack became popular and the reference in term of deployment
> tool for continuous integration, devstack doesn't deploy OpenStack in
> production (versus some tools like Kolla, Fuel, TripleO, Juju, etc).
> It means things might (and did) break when deploying OpenStack outside
> devstack, for different reasons. Some examples:
> 
> * until recently, SSL was not tested, and I believe some projects
> still don't test with SSL enabled.
> * IPv6 is not tested everywhere.
> * Production scenarios, with HA (HAproxy or/and Pacemaker) are not
> tested.
> 

I am a little late, but wanted to clarify some things here. You are
correct that SSL was not tested until very recently. However, any
project running the "base" neutron and nova net tempest jobs is now
being tested with the tls-proxy service enabled in devstack (for
>=ocata). There is also a growing list of other jobs that are adding
support for this. My hope is that during the ocata cycle we would enable
it by default in devstack and projects that explicitly do not work
disable it (rather than the current explicit enable where it functions).

IPv6 testing likely means two different things to two different groups
of people. First is whether or not the cloud endpoints are hosted over
IPv6 and the other is whether or not instances launched by openstack are
assigned IPv6 addresses. The second thing has been tested for quite a
while now (tempest has had tests for it for almost 2 years though it
hasn't been enabled in the gate for that long). We should definitely
ensure that we are testing with openstack servers listening on IPv6
addresses as well.

I mention this because I think these are both things that devstack
should support and test, and while experimental jobs to run other
deployment tests can only help they are not a substitute for fixing
these things in devstack.

With that out of the way I think having a diverse set of testing can
only help make OpenStack better especially for your more production like
scenarios. Devstack has very explicitly been a development tool so not
sure that those scenarios need to be supported by it.

Clark




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Miles Gould]

On 14/10/16 12:33, Sean Dague wrote:

I kind of wonder if that hints to a better model here instead of the
deployments running services from master. Instead running periodics and
moving forward reference hashes every day if tests pass, and not if they
fail. That would let deployment tools automatically move forward, quite
close to master, but not get tightly coupled into every change.


+1. This would also make debugging gate failures easier: when the only 
thing that has changed is the project under test (and not every other 
service it depends on), the source of the breakage is much more likely 
to be the patch being tested.


Miles

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Jesse Pretorius]

On 10/14/16, 12:33 PM, "Sean Dague"  wrote:

> I kind of wonder if that hints to a better model here instead of the
> deployments running services from master. Instead running periodics and
> moving forward reference hashes every day if tests pass, and not if they
> fail. That would let deployment tools automatically move forward, quite
> close to master, but not get tightly coupled into every change.

FWIW That’s pretty much what OpenStack-Ansible does for our ‘integrated build’.

We have the ‘production style’ deployment pinned to upstream SHA’s which Are 
updated once every two weeks. This allows us to get on with our own development 
instead of being disrupted every time something causes a break upstream. We 
then only have to deal with upstream-related issues when we update SHA pins.

However we do also do more focused, slightly less complex builds on a per role 
(per upstream service) basis which build directly from the upstream branch. 
These have a broader testing spectrum (more platforms, more backends) and due 
to the lower level of complexity in the number of components used in the builds 
they are generally successful unless our own patch is bad.

From my own standpoint I think that if production-type build tests (with 
functional testing) are executed using deployment projects there would need to 
be:

1. A good track record of the builds succeeding without failures due to 
circumstances that are specific to the deployment tool project. Ie The builds 
must be at least as successful as DevStack in its results with as few false 
negatives as possible.
2. Each project using these tests would need a designated liaison to the 
deployment project in order to expedite the triage of issues that arise from 
the tests. The service project person understands the service and the 
deployment project liaison understands the deployment project. Effectively they 
should pair up to quickly triage and resolve any check failures that arise.
3. The tests should not just be a deployment test – they should have some level 
of functional testing too. If this is not the case then all that’s happening is 
that the deployment tooling is being tested – not the effect of the patch to 
the service project.

J



Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Emilien Macchi]

On Thu, Oct 13, 2016 at 10:30 PM, Davanum Srinivas  wrote:
> Please see below:
>
> On Thu, Oct 13, 2016 at 9:33 PM, Matt Riedemann
>  wrote:
>> On 10/13/2016 7:47 PM, Emilien Macchi wrote:
>>>
>>> Greetings OpenStack,
>>>
>>> == Background
>>>
>>> Since the beginning of OpenStack (or almost), devstack has been used
>>> as a common tool to deploy OpenStack in CI environment. Most of
>>> OpenStack projects (if not all) that are written in Python use it to
>>> deploy the different components.
>>> While devstack became popular and the reference in term of deployment
>>> tool for continuous integration, devstack doesn't deploy OpenStack in
>>> production (versus some tools like Kolla, Fuel, TripleO, Juju, etc).
>>> It means things might (and did) break when deploying OpenStack outside
>>> devstack, for different reasons. Some examples:
>>>
>>> * until recently, SSL was not tested, and I believe some projects
>>> still don't test with SSL enabled.
>>> * IPv6 is not tested everywhere.
>>> * Production scenarios, with HA (HAproxy or/and Pacemaker) are not tested.
>>>
>>> My point here, is that devstack has been doing very good job for its
>>> simplicity (written in bash) and its large adoption by projects to
>>> make CI, though we might consider adding more coverage to make sure it
>>> works outside devstack.
>>> As an example, Puppet OpenStack modules CI is using a devstack-like
>>> job (with 3 scenarios), called puppet-openstack-integration [1] but we
>>> also run TripleO and Fuel CI jobs, to increase coverage and give a
>>> better feedback on testing.
>>>
>>>
>>> == Proposal
>>>
>>> This is not about removing devstack! The idea is to add more coverage
>>> in an iterative way, with some other tools.
>>> We started some months ago by running TripleO CI jobs in Ironic and
>>> Heat gates (experimental pipeline) because TripleO is high consumer of
>>> Ironic and Heat.
>>> Also, we recently added our TripleO multinode job in Nova experimental
>>> pipeline (doc here [2]).
>>> Now, we are moving forward with python-openstackclient and osc-lib.
>>>
>>> I started to draft a document about how we could increase coverage in
>>> different projects:
>>>
>>> https://docs.google.com/spreadsheets/d/1bLg-uEGrQXyRZ-FuR6pf1WT4XN0-6MrlfqEShI7xMxg/edit#gid=0
>>>
>>> (feel free to add your project and give your opinion directly in the
>>> spreadsheet).
>>>
>>> The intention here is to discuss with teams interested by such CI
>>> coverage. We don't want to slow down or break your gate (example with
>>> TripleO, our jobs are non-voting outside TripleO and take ~45 min);
>>> but reduce the feedback loop between developers and deployment tools
>>> used in production.
>>> We don't expect developers to investigate why new CI jobs would fail
>>> (ex: a Nova developer to look at TripleO CI job), it would be unfair.
>>> Just some horizontal communication would be enough, IRC or email, to
>>> inform that a patch might break a CI job outside devstack.
>>> I also want to mention that the proposal is not only about TripleO. I
>>> used this tool in my examples because I'm working on it but obviously
>>> this proposal should be open to Big Tent projects that also deploy
>>> OpenStack.
>>>
>>> Please give any feedback, and let's make OpenStack testing stronger!
>>> Thanks for reading so far,
>>>
>>> [1] https://github.com/openstack/puppet-openstack-integration
>>> [2] https://review.openstack.org/#/c/381838/
>>>
>>
>> I don't really have a problem with wanting to run non-devstack deployment
>> tool jobs against project changes on-demand (experimental queue job). That's
>> why I approved the change to add that TripleO job to nova's experimental
>> queue.
>>
>> The experimental queue is only on-demand though, so reviewers have to be
>> conscious of running it and even then people don't think to check the
>> results, or a failure might not be obvious as to what caused it (my patch,
>> or is this job always broken and is thus in the experimental queue, like the
>> nova-lxc job?).
>>
>> For better or worse devstack is at least universally used and it's THE
>> default thing we point newcomers to when getting started if they want to
>> quickly and easily get a development environment with a running openstack on
>> a single-node up and running to kick the tires. I can't say the same for the
>> plethora of other deployment projects out there like kolla, ansible, salt,
>> puppet, chef, tripleo/packstack/rdo, fuel, etc, etc. I think that's what's
>> really caused the lack of universal adoption of anything besides devstack in
>> our CI environment. And love it or hate it, I think anyone that's been
>> around for awhile and tries to debug gate failures is at least used to
>> hacking on devstack and knows how it works to a certain extent.
>>
>> Anyway, as I said, I've got no problem with getting some additional optional
>> non-voting coverage in other projects besides devstack to at least try and
>> prevent breaking 

Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Emilien Macchi]

On Thu, Oct 13, 2016 at 11:59 PM, Steve Martinelli
 wrote:
> On Thu, Oct 13, 2016 at 10:30 PM, Davanum Srinivas 
> wrote:
>>
>>
>> Matt, Emilien,
>>
>> there's one more option, run jobs daily and add the output to the
>> openstack health dashboard.
>> example -
>> http://status.openstack.org/openstack-health/#/g/build_name/periodic-ironic-py35-with-oslo-master
>
>
> But that only runs against what is merged on master right? I think Emilien
> wants to catch changes that break before they are merged.

Exactly.

> Emilien, I think it's also worth noting which projects you intend to make
> these changes to? Just "core" projects + projects that tripleO uses (heat +
> ironic) + projects that have burned you in the past (OSC)? Or do you plan on
> covering all projects?

Not all projects, but those in which we had some breakages over the
last 6 months. We think installers can bring useful feedback to these
projects.

> Finding a balance between not enough testing, and overusing infra resources
> is tricky, we should only aim for high value targets like the ones listed
> above. I can't imagine this being run on all check jobs everywhere.

Right, we don't want to waste Infra resources, that's why I started
this discussion to see if it's really worth it and get the feedback
from our community.

Thanks,
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Emilien Macchi]

On Fri, Oct 14, 2016 at 7:33 AM, Sean Dague  wrote:
> On 10/13/2016 11:59 PM, Steve Martinelli wrote:
>>
>> On Thu, Oct 13, 2016 at 10:30 PM, Davanum Srinivas > > wrote:
>>
>>
>> Matt, Emilien,
>>
>> there's one more option, run jobs daily and add the output to the
>> openstack health dashboard.
>> example -
>>
>> http://status.openstack.org/openstack-health/#/g/build_name/periodic-ironic-py35-with-oslo-master
>>
>> 
>>
>>
>> But that only runs against what is merged on master right? I think
>> Emilien wants to catch changes that break before they are merged.
>>
>> Emilien, I think it's also worth noting which projects you intend to
>> make these changes to? Just "core" projects + projects that tripleO uses
>> (heat + ironic) + projects that have burned you in the past (OSC)? Or do
>> you plan on covering all projects?
>>
>> Finding a balance between not enough testing, and overusing infra
>> resources is tricky, we should only aim for high value targets like the
>> ones listed above. I can't imagine this being run on all check jobs
>> everywhere.
>
>
> It's not just overuse of infra resources. Anything that's more complicated
> than unit tests has > 0% chance of failure. So every new complex full stack
> test is going to -1 some set of good patches.
>
> The best thing to do when there is a break is to figure out what the root
> cause was, and push testing back as low in the stack as possible. Was there
> a good unit test that could have prevented it? That's super cheap, and helps
> a ton going forward. Or realize that projects are using undefined behavior
> of other tools, and we need to define that behavior.
>
> The deployment teams are releasing their final version of things weeks after
> the newton release hits. By nature there is some following time.
>
> I kind of wonder if that hints to a better model here instead of the
> deployments running services from master. Instead running periodics and
> moving forward reference hashes every day if tests pass, and not if they
> fail. That would let deployment tools automatically move forward, quite
> close to master, but not get tightly coupled into every change.
>
> Because I do get concerned for deciding that every developer needs to
> understand tripleo, and ansible, and fuel and the configuration decisions
> they made, the bugs they have, the way their services are deployed, to be
> able to land any patches. Because I think that will further reduce the pool
> of developers that can contribute.

I explicitly mentioned the opposite in the proposal, let me copy/paste it again:

"We don't expect developers to investigate why new CI jobs would fail
(ex: a Nova developer to look at TripleO CI job), it would be unfair.
Just some horizontal communication would be enough, IRC or email, to
inform that a patch might break a CI job outside devstack."

If a CI job is breaking because of a patch, deployment tools folks
would look at the CI job and tell to the project why it fails, who
would decide if whether or not it's a blocker (ie: indeed, the patch
is breaking backward compatibility, or no, the patch is good,
installer has to be updated, etc).

I hope it's clear that we don't expect project to spend time on this
thing and of course we don't want to reduce the pool of devs here.

-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Sean Dague]

On 10/13/2016 11:59 PM, Steve Martinelli wrote:

On Thu, Oct 13, 2016 at 10:30 PM, Davanum Srinivas > wrote:


Matt, Emilien,

there's one more option, run jobs daily and add the output to the
openstack health dashboard.
example -

http://status.openstack.org/openstack-health/#/g/build_name/periodic-ironic-py35-with-oslo-master




But that only runs against what is merged on master right? I think
Emilien wants to catch changes that break before they are merged.

Emilien, I think it's also worth noting which projects you intend to
make these changes to? Just "core" projects + projects that tripleO uses
(heat + ironic) + projects that have burned you in the past (OSC)? Or do
you plan on covering all projects?

Finding a balance between not enough testing, and overusing infra
resources is tricky, we should only aim for high value targets like the
ones listed above. I can't imagine this being run on all check jobs
everywhere.


It's not just overuse of infra resources. Anything that's more 
complicated than unit tests has > 0% chance of failure. So every new 
complex full stack test is going to -1 some set of good patches.


The best thing to do when there is a break is to figure out what the 
root cause was, and push testing back as low in the stack as possible. 
Was there a good unit test that could have prevented it? That's super 
cheap, and helps a ton going forward. Or realize that projects are using 
undefined behavior of other tools, and we need to define that behavior.


The deployment teams are releasing their final version of things weeks 
after the newton release hits. By nature there is some following time.


I kind of wonder if that hints to a better model here instead of the 
deployments running services from master. Instead running periodics and 
moving forward reference hashes every day if tests pass, and not if they 
fail. That would let deployment tools automatically move forward, quite 
close to master, but not get tightly coupled into every change.


Because I do get concerned for deciding that every developer needs to 
understand tripleo, and ansible, and fuel and the configuration 
decisions they made, the bugs they have, the way their services are 
deployed, to be able to land any patches. Because I think that will 
further reduce the pool of developers that can contribute.


-Sean

--
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Steve Martinelli]

On Thu, Oct 13, 2016 at 10:30 PM, Davanum Srinivas 
wrote:

>
> Matt, Emilien,
>
> there's one more option, run jobs daily and add the output to the
> openstack health dashboard.
> example - http://status.openstack.org/openstack-health/#/g/build_
> name/periodic-ironic-py35-with-oslo-master


But that only runs against what is merged on master right? I think Emilien
wants to catch changes that break before they are merged.

Emilien, I think it's also worth noting which projects you intend to make
these changes to? Just "core" projects + projects that tripleO uses (heat +
ironic) + projects that have burned you in the past (OSC)? Or do you plan
on covering all projects?

Finding a balance between not enough testing, and overusing infra resources
is tricky, we should only aim for high value targets like the ones listed
above. I can't imagine this being run on all check jobs everywhere.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Davanum Srinivas]

Please see below:

On Thu, Oct 13, 2016 at 9:33 PM, Matt Riedemann
 wrote:
> On 10/13/2016 7:47 PM, Emilien Macchi wrote:
>>
>> Greetings OpenStack,
>>
>> == Background
>>
>> Since the beginning of OpenStack (or almost), devstack has been used
>> as a common tool to deploy OpenStack in CI environment. Most of
>> OpenStack projects (if not all) that are written in Python use it to
>> deploy the different components.
>> While devstack became popular and the reference in term of deployment
>> tool for continuous integration, devstack doesn't deploy OpenStack in
>> production (versus some tools like Kolla, Fuel, TripleO, Juju, etc).
>> It means things might (and did) break when deploying OpenStack outside
>> devstack, for different reasons. Some examples:
>>
>> * until recently, SSL was not tested, and I believe some projects
>> still don't test with SSL enabled.
>> * IPv6 is not tested everywhere.
>> * Production scenarios, with HA (HAproxy or/and Pacemaker) are not tested.
>>
>> My point here, is that devstack has been doing very good job for its
>> simplicity (written in bash) and its large adoption by projects to
>> make CI, though we might consider adding more coverage to make sure it
>> works outside devstack.
>> As an example, Puppet OpenStack modules CI is using a devstack-like
>> job (with 3 scenarios), called puppet-openstack-integration [1] but we
>> also run TripleO and Fuel CI jobs, to increase coverage and give a
>> better feedback on testing.
>>
>>
>> == Proposal
>>
>> This is not about removing devstack! The idea is to add more coverage
>> in an iterative way, with some other tools.
>> We started some months ago by running TripleO CI jobs in Ironic and
>> Heat gates (experimental pipeline) because TripleO is high consumer of
>> Ironic and Heat.
>> Also, we recently added our TripleO multinode job in Nova experimental
>> pipeline (doc here [2]).
>> Now, we are moving forward with python-openstackclient and osc-lib.
>>
>> I started to draft a document about how we could increase coverage in
>> different projects:
>>
>> https://docs.google.com/spreadsheets/d/1bLg-uEGrQXyRZ-FuR6pf1WT4XN0-6MrlfqEShI7xMxg/edit#gid=0
>>
>> (feel free to add your project and give your opinion directly in the
>> spreadsheet).
>>
>> The intention here is to discuss with teams interested by such CI
>> coverage. We don't want to slow down or break your gate (example with
>> TripleO, our jobs are non-voting outside TripleO and take ~45 min);
>> but reduce the feedback loop between developers and deployment tools
>> used in production.
>> We don't expect developers to investigate why new CI jobs would fail
>> (ex: a Nova developer to look at TripleO CI job), it would be unfair.
>> Just some horizontal communication would be enough, IRC or email, to
>> inform that a patch might break a CI job outside devstack.
>> I also want to mention that the proposal is not only about TripleO. I
>> used this tool in my examples because I'm working on it but obviously
>> this proposal should be open to Big Tent projects that also deploy
>> OpenStack.
>>
>> Please give any feedback, and let's make OpenStack testing stronger!
>> Thanks for reading so far,
>>
>> [1] https://github.com/openstack/puppet-openstack-integration
>> [2] https://review.openstack.org/#/c/381838/
>>
>
> I don't really have a problem with wanting to run non-devstack deployment
> tool jobs against project changes on-demand (experimental queue job). That's
> why I approved the change to add that TripleO job to nova's experimental
> queue.
>
> The experimental queue is only on-demand though, so reviewers have to be
> conscious of running it and even then people don't think to check the
> results, or a failure might not be obvious as to what caused it (my patch,
> or is this job always broken and is thus in the experimental queue, like the
> nova-lxc job?).
>
> For better or worse devstack is at least universally used and it's THE
> default thing we point newcomers to when getting started if they want to
> quickly and easily get a development environment with a running openstack on
> a single-node up and running to kick the tires. I can't say the same for the
> plethora of other deployment projects out there like kolla, ansible, salt,
> puppet, chef, tripleo/packstack/rdo, fuel, etc, etc. I think that's what's
> really caused the lack of universal adoption of anything besides devstack in
> our CI environment. And love it or hate it, I think anyone that's been
> around for awhile and tries to debug gate failures is at least used to
> hacking on devstack and knows how it works to a certain extent.
>
> Anyway, as I said, I've got no problem with getting some additional optional
> non-voting coverage in other projects besides devstack to at least try and
> prevent breaking changes. I worry about trying to move various deployment
> jobs into the check queue for multiple projects though, as I think that
> would put a pretty serious strain on resources 

Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Matt Riedemann]

On 10/13/2016 7:47 PM, Emilien Macchi wrote:

Greetings OpenStack,

== Background

Since the beginning of OpenStack (or almost), devstack has been used
as a common tool to deploy OpenStack in CI environment. Most of
OpenStack projects (if not all) that are written in Python use it to
deploy the different components.
While devstack became popular and the reference in term of deployment
tool for continuous integration, devstack doesn't deploy OpenStack in
production (versus some tools like Kolla, Fuel, TripleO, Juju, etc).
It means things might (and did) break when deploying OpenStack outside
devstack, for different reasons. Some examples:

* until recently, SSL was not tested, and I believe some projects
still don't test with SSL enabled.
* IPv6 is not tested everywhere.
* Production scenarios, with HA (HAproxy or/and Pacemaker) are not tested.

My point here, is that devstack has been doing very good job for its
simplicity (written in bash) and its large adoption by projects to
make CI, though we might consider adding more coverage to make sure it
works outside devstack.
As an example, Puppet OpenStack modules CI is using a devstack-like
job (with 3 scenarios), called puppet-openstack-integration [1] but we
also run TripleO and Fuel CI jobs, to increase coverage and give a
better feedback on testing.


== Proposal

This is not about removing devstack! The idea is to add more coverage
in an iterative way, with some other tools.
We started some months ago by running TripleO CI jobs in Ironic and
Heat gates (experimental pipeline) because TripleO is high consumer of
Ironic and Heat.
Also, we recently added our TripleO multinode job in Nova experimental
pipeline (doc here [2]).
Now, we are moving forward with python-openstackclient and osc-lib.

I started to draft a document about how we could increase coverage in
different projects:
https://docs.google.com/spreadsheets/d/1bLg-uEGrQXyRZ-FuR6pf1WT4XN0-6MrlfqEShI7xMxg/edit#gid=0

(feel free to add your project and give your opinion directly in the
spreadsheet).

The intention here is to discuss with teams interested by such CI
coverage. We don't want to slow down or break your gate (example with
TripleO, our jobs are non-voting outside TripleO and take ~45 min);
but reduce the feedback loop between developers and deployment tools
used in production.
We don't expect developers to investigate why new CI jobs would fail
(ex: a Nova developer to look at TripleO CI job), it would be unfair.
Just some horizontal communication would be enough, IRC or email, to
inform that a patch might break a CI job outside devstack.
I also want to mention that the proposal is not only about TripleO. I
used this tool in my examples because I'm working on it but obviously
this proposal should be open to Big Tent projects that also deploy
OpenStack.

Please give any feedback, and let's make OpenStack testing stronger!
Thanks for reading so far,

[1] https://github.com/openstack/puppet-openstack-integration
[2] https://review.openstack.org/#/c/381838/



I don't really have a problem with wanting to run non-devstack 
deployment tool jobs against project changes on-demand (experimental 
queue job). That's why I approved the change to add that TripleO job to 
nova's experimental queue.


The experimental queue is only on-demand though, so reviewers have to be 
conscious of running it and even then people don't think to check the 
results, or a failure might not be obvious as to what caused it (my 
patch, or is this job always broken and is thus in the experimental 
queue, like the nova-lxc job?).


For better or worse devstack is at least universally used and it's THE 
default thing we point newcomers to when getting started if they want to 
quickly and easily get a development environment with a running 
openstack on a single-node up and running to kick the tires. I can't say 
the same for the plethora of other deployment projects out there like 
kolla, ansible, salt, puppet, chef, tripleo/packstack/rdo, fuel, etc, 
etc. I think that's what's really caused the lack of universal adoption 
of anything besides devstack in our CI environment. And love it or hate 
it, I think anyone that's been around for awhile and tries to debug gate 
failures is at least used to hacking on devstack and knows how it works 
to a certain extent.


Anyway, as I said, I've got no problem with getting some additional 
optional non-voting coverage in other projects besides devstack to at 
least try and prevent breaking changes. I worry about trying to move 
various deployment jobs into the check queue for multiple projects 
though, as I think that would put a pretty serious strain on resources 
for non-voting jobs, which we'd like to avoid I think.


My two cents.

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

[openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Emilien Macchi]

Greetings OpenStack,

== Background

Since the beginning of OpenStack (or almost), devstack has been used
as a common tool to deploy OpenStack in CI environment. Most of
OpenStack projects (if not all) that are written in Python use it to
deploy the different components.
While devstack became popular and the reference in term of deployment
tool for continuous integration, devstack doesn't deploy OpenStack in
production (versus some tools like Kolla, Fuel, TripleO, Juju, etc).
It means things might (and did) break when deploying OpenStack outside
devstack, for different reasons. Some examples:

* until recently, SSL was not tested, and I believe some projects
still don't test with SSL enabled.
* IPv6 is not tested everywhere.
* Production scenarios, with HA (HAproxy or/and Pacemaker) are not tested.

My point here, is that devstack has been doing very good job for its
simplicity (written in bash) and its large adoption by projects to
make CI, though we might consider adding more coverage to make sure it
works outside devstack.
As an example, Puppet OpenStack modules CI is using a devstack-like
job (with 3 scenarios), called puppet-openstack-integration [1] but we
also run TripleO and Fuel CI jobs, to increase coverage and give a
better feedback on testing.


== Proposal

This is not about removing devstack! The idea is to add more coverage
in an iterative way, with some other tools.
We started some months ago by running TripleO CI jobs in Ironic and
Heat gates (experimental pipeline) because TripleO is high consumer of
Ironic and Heat.
Also, we recently added our TripleO multinode job in Nova experimental
pipeline (doc here [2]).
Now, we are moving forward with python-openstackclient and osc-lib.

I started to draft a document about how we could increase coverage in
different projects:
https://docs.google.com/spreadsheets/d/1bLg-uEGrQXyRZ-FuR6pf1WT4XN0-6MrlfqEShI7xMxg/edit#gid=0

(feel free to add your project and give your opinion directly in the
spreadsheet).

The intention here is to discuss with teams interested by such CI
coverage. We don't want to slow down or break your gate (example with
TripleO, our jobs are non-voting outside TripleO and take ~45 min);
but reduce the feedback loop between developers and deployment tools
used in production.
We don't expect developers to investigate why new CI jobs would fail
(ex: a Nova developer to look at TripleO CI job), it would be unfair.
Just some horizontal communication would be enough, IRC or email, to
inform that a patch might break a CI job outside devstack.
I also want to mention that the proposal is not only about TripleO. I
used this tool in my examples because I'm working on it but obviously
this proposal should be open to Big Tent projects that also deploy
OpenStack.

Please give any feedback, and let's make OpenStack testing stronger!
Thanks for reading so far,

[1] https://github.com/openstack/puppet-openstack-integration
[2] https://review.openstack.org/#/c/381838/
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev