Re: [openstack-dev] [qa] New upgrade test tool proposal.

2017-06-27 Thread Sean Dague 
On 06/27/2017 03:19 PM, Dean Troyer wrote:
> On Tue, Jun 27, 2017 at 1:37 PM, Jay Pipes  wrote:
>> Hi Castulo, sorry for the delayed response on this. Has your team moved
>> forward on any of this?
> 
> IIRC this work was impacted by the OSIC shutdown, I believe it is not
> currently on anyone's radar.
> 
>> What about the Grenade testing framework that uses devstack as its
>> deployment system was not useful or usable for you?
> 
> I can take at least part of the blame in encouraging them to not
> attempt to leverage Grenade directly.  Grenade needs to be replaced as
> it has far out-lived its expected life[0].
> 
> Grenade was built to do static in-place upgrades and the fact that it
> has been pushed as far as it has is a happy surprise to me.  However,
> it is fundamentally limited in its abilities as a test orchestrator,
> implementing robust multi-node capabilities and the granularity that
> is required to properly do upgrade testing really needs a reboot.  In
> a well-funded world that would include replacing DevStack too, which
> while nice is not strictly necessary to achieve the testing goals they
> had.
> 
> The thing that Grenade and DevStack have going for them besides
> inertia is that they are not otherwise tied to any deployment
> strategy.  Starting over from scratch really is not an option at this
> point, something existing really does need to be leveraged even though
> it may hurt some feelings along the way for the project(s) not chosen.
> 
> dt
> 
> 
> [0] Seriously, I never expected Grenade (or DevStack for that matter)
> to have survived this long, but they have mostly because they were/are
> just barely good enough that nobody wants to fund replacing them.

Well, we also lengthened their life and usefulness by adding an external
plugin interface, that let people leverage what we had without having to
wait on review queues.

But, I also agree that grenade is largely pushed to its limits. That
also includes the fact that I'm more or less the only "active" reviewer
at this point. The only way this is even vaguely tenable is by the
complexity being capped so it's straight forward enough to think through
implications of patches.

The complicated upgrade orchestration including multiple things
executing at the same time, breaks that. It's cool if someone wants to
do it, however it definitely needs more dedicated folks on the review
side to be successful.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] New upgrade test tool proposal.

2017-06-27 Thread Dean Troyer 
On Tue, Jun 27, 2017 at 1:37 PM, Jay Pipes  wrote:
> Hi Castulo, sorry for the delayed response on this. Has your team moved
> forward on any of this?

IIRC this work was impacted by the OSIC shutdown, I believe it is not
currently on anyone's radar.

> What about the Grenade testing framework that uses devstack as its
> deployment system was not useful or usable for you?

I can take at least part of the blame in encouraging them to not
attempt to leverage Grenade directly.  Grenade needs to be replaced as
it has far out-lived its expected life[0].

Grenade was built to do static in-place upgrades and the fact that it
has been pushed as far as it has is a happy surprise to me.  However,
it is fundamentally limited in its abilities as a test orchestrator,
implementing robust multi-node capabilities and the granularity that
is required to properly do upgrade testing really needs a reboot.  In
a well-funded world that would include replacing DevStack too, which
while nice is not strictly necessary to achieve the testing goals they
had.

The thing that Grenade and DevStack have going for them besides
inertia is that they are not otherwise tied to any deployment
strategy.  Starting over from scratch really is not an option at this
point, something existing really does need to be leveraged even though
it may hurt some feelings along the way for the project(s) not chosen.

dt


[0] Seriously, I never expected Grenade (or DevStack for that matter)
to have survived this long, but they have mostly because they were/are
just barely good enough that nobody wants to fund replacing them.

-- 

Dean Troyer
dtro...@gmail.com

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] New upgrade test tool proposal.

2017-06-27 Thread Jay Pipes 

On 04/05/2017 04:22 PM, Martinez, Castulo wrote:

Hi,

As you might know, the TC introduced new tags [1] for upgrade processes
quite some time ago, which reflect the level of maturity of an upgrade.
Therefore there is the growing need of test tools which help to exercise
and validate the upgrade approaches defined by the community.

During the PTG at Atlanta there were discussions around options for
covering this gap in testing tools, one proposal was to add this
functionality to Grenade, but this idea was dismissed since it was
determined that Grenade was originally designed for a different purpose
and is already pushing its limits, so the consensus was towards
creating a new tool to fill this gap.

We are proposing a new toolset for testing an OpenStack environment
before, during, and after an upgrade process. The toolset answers the
question ³how does OpenStack behave across upgrades from one release N
to a release N+1, or from the latest official release to master?². It
also provides information that can be used to assess if an upgrade
complies with the requirements for being recognized as a rolling
upgrade, a zero downtime upgrade or a zero impact upgrade.

Before starting this effort, we would like to hear feedback from
everybody, are there any concerns with this approach? Are we missing
something?

You can find details of this proposal here:
https://review.openstack.org/#/c/449295/3


Hi Castulo, sorry for the delayed response on this. Has your team moved 
forward on any of this?


From first glance at the spec, it does seem that you are biting off a 
lot -- deploying, healthchecking, upgrading, monitoring, data/report 
storage, and configuration management are all parts of this tool.


What about the Grenade testing framework that uses devstack as its 
deployment system was not useful or usable for you?


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa] New upgrade test tool proposal.

2017-04-05 Thread Martinez, Castulo 
Hi,

As you might know, the TC introduced new tags [1] for upgrade processes
quite some time ago, which reflect the level of maturity of an upgrade.
Therefore there is the growing need of test tools which help to exercise
and validate the upgrade approaches defined by the community.

During the PTG at Atlanta there were discussions around options for
covering this gap in testing tools, one proposal was to add this
functionality to Grenade, but this idea was dismissed since it was
determined that Grenade was originally designed for a different purpose
and is already pushing its limits, so the consensus was towards
creating a new tool to fill this gap.

We are proposing a new toolset for testing an OpenStack environment
before, during, and after an upgrade process. The toolset answers the
question ³how does OpenStack behave across upgrades from one release N
to a release N+1, or from the latest official release to master?². It
also provides information that can be used to assess if an upgrade
complies with the requirements for being recognized as a rolling
upgrade, a zero downtime upgrade or a zero impact upgrade.

Before starting this effort, we would like to hear feedback from
everybody, are there any concerns with this approach? Are we missing
something?

You can find details of this proposal here:
https://review.openstack.org/#/c/449295/3

Thanks

--
Castulo J. Martinez


[1] 
https://governance.openstack.org/tc/reference/tags/assert_supports-rolling-
upgrade.html

https://governance.openstack.org/tc/reference/tags/assert_supports-zero-dow
ntime-upgrade.html

https://governance.openstack.org/tc/reference/tags/assert_supports-zero-imp
act-upgrade.html


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev