Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Amrith Kumar]

> -Original Message-
> From: Sean McGinnis [mailto:sean.mcgin...@gmx.com]
> Sent: Thursday, June 1, 2017 4:48 PM
> To: OpenStack Development Mailing List (not for usage questions)  d...@lists.openstack.org>
> Subject: Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests
> 
> >
> > And yes, I agree with the argument that we should be fair and treat
> > all projects the same way. If we're going to move tests out of the
> > tempest repository, we should move all of them. The QA team can still
> > help maintain the test suites for whatever projects they want, even if
> > those tests are in plugins.
> >
> > Doug
> >
> 
> +1

+1


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Doug Hellmann]

Excerpts from Matthew Treinish's message of 2017-06-01 20:51:24 -0400:
> On Thu, Jun 01, 2017 at 11:57:00AM -0400, Doug Hellmann wrote:
> > Excerpts from Thierry Carrez's message of 2017-06-01 11:51:50 +0200:
> > > Graham Hayes wrote:
> > > > On 01/06/17 01:30, Matthew Treinish wrote:
> > > >> TBH, it's a bit premature to have the discussion. These additional 
> > > >> programs do
> > > >> not exist yet, and there is a governance road block around this. Right 
> > > >> now the
> > > >> set of projects that can be used defcore/interopWG is limited to the 
> > > >> set of 
> > > >> projects in:
> > > >>
> > > >> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html
> > > > 
> > > > Sure - but that is a solved problem, when the interop committee is
> > > > ready to propose them, they can add projects into that tag. Or am I
> > > > misunderstanding [1] (again)?
> > > 
> > > I think you understand it well. The Board/InteropWG should propose
> > > additions/removals of this tag, which will then be approved by the TC:
> > > 
> > > https://governance.openstack.org/tc/reference/tags/tc_approved-release.html#tag-application-process
> > > 
> > > > [...]
> > > >> We had a forum session on it (I can't find the etherpad for the 
> > > >> session) which
> > > >> was pretty speculative because it was about planning the new programs. 
> > > >> Part of
> > > >> that discussion was around the feasibility of using tests in plugins 
> > > >> and whether
> > > >> that would be desirable. Personally, I was in favor of doing that for 
> > > >> some of
> > > >> the proposed programs because of the way they were organized it was a 
> > > >> good fit.
> > > >> This is because the proposed new programs were extra additions on top 
> > > >> of the
> > > >> base existing interop program. But it was hardly a definitive 
> > > >> discussion.
> > > > 
> > > > Which will create 2 classes of testing for interop programs.
> > > 
> > > FWIW I would rather have a single way of doing "tests used in trademark
> > > programs" without differentiating between old and new trademark programs.
> > > 
> > > I fear that we are discussing solutions before defining the problem. We
> > > want:
> > > 
> > > 1- Decentralize test maintenance, through more tempest plugins, to
> > > account for limited QA resources
> > > 2- Additional codereview constraints and approval rules for tests that
> > > happen to be used in trademark programs
> > > 3- Discoverability/ease-of-install of the set of tests that happen to be
> > > used in trademark programs
> > > 4- A git repo layout that can be simply explained, for new teams to
> > > understand
> > > 
> > > It feels like the current git repo layout (result of that 2016-05-04
> > > resolution) optimizes for 2 and 3, which kind of works until you add
> > > more trademark programs, at which point it breaks 1 and 4.
> > > 
> > > I feel like you could get 2 and 3 without necessarily using git repo
> > > boundaries (using Gerrit approval rules and some tooling to install/run
> > > subset of tests across multiple git repos), which would allow you to
> > > optimize git repo layout to get 1 and 4...
> > > 
> > > Or am I missing something ?
> > > 
> > 
> > Right. The point of having the trademark tests "in tempest" was not
> > to have them "in the tempest repo", that was just an implementation
> > detail of the policy of "put them in a repository managed by people
> > who understand the expanded review rules".
> 
> There was more to it than this, a big part was duplication of effort as well.
> Tempest itself is almost a perfect fit for the scope of the testing defcore is
> doing. While tempest does additional testing that defcore doesn't use, a large
> subset is exactly what they want.

That does explain why Tempest was appealing to the DefCore folks.
I was trying to explain my motivation for writing the resolution
saying that we did not want DefCore using tests scattered throughout
a bunch of plugin repositories managed by different reviewer teams.

> > There were a lot of unexpected issues when we started treating the
> > test suite as a production tool for validating a cloud.  We have
> > to be careful about how we change the behavior of tests, for example,
> > even if the API responses are expected to be the same.  It's not
> > fair to vendors or operators who get trademark approval with one
> > release to have significant changes in behavior in the exact same
> > tests for the next release.
> 
> I actually find this to be kinda misleading. Tempest has always had
> running on any cloud as part of it's mission. I think you're referring
> to the monster defcore thread from last summer about proprietary nova 
> extensions
> adding on to API responses. This is honestly a completely separate problem
> which is not something I want to dive into again, because that was a much more
> nuanced problem that involved much more than just code review.

That may have been the situation I'm thinking of, and I agree,

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Masayuki Igawa]

On Fri, Jun 2, 2017, at 09:51 AM, Matthew Treinish wrote:
> On Thu, Jun 01, 2017 at 11:57:00AM -0400, Doug Hellmann wrote:
> > Excerpts from Thierry Carrez's message of 2017-06-01 11:51:50 +0200:
> > > Graham Hayes wrote:
> > > > On 01/06/17 01:30, Matthew Treinish wrote:
> > > >> TBH, it's a bit premature to have the discussion. These additional 
> > > >> programs do
> > > >> not exist yet, and there is a governance road block around this. Right 
> > > >> now the
> > > >> set of projects that can be used defcore/interopWG is limited to the 
> > > >> set of 
> > > >> projects in:
> > > >>
> > > >> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html
> > > > 
> > > > Sure - but that is a solved problem, when the interop committee is
> > > > ready to propose them, they can add projects into that tag. Or am I
> > > > misunderstanding [1] (again)?
> > > 
> > > I think you understand it well. The Board/InteropWG should propose
> > > additions/removals of this tag, which will then be approved by the TC:
> > > 
> > > https://governance.openstack.org/tc/reference/tags/tc_approved-release.html#tag-application-process
> > > 
> > > > [...]
> > > >> We had a forum session on it (I can't find the etherpad for the 
> > > >> session) which
> > > >> was pretty speculative because it was about planning the new programs. 
> > > >> Part of
> > > >> that discussion was around the feasibility of using tests in plugins 
> > > >> and whether
> > > >> that would be desirable. Personally, I was in favor of doing that for 
> > > >> some of
> > > >> the proposed programs because of the way they were organized it was a 
> > > >> good fit.
> > > >> This is because the proposed new programs were extra additions on top 
> > > >> of the
> > > >> base existing interop program. But it was hardly a definitive 
> > > >> discussion.
> > > > 
> > > > Which will create 2 classes of testing for interop programs.
> > > 
> > > FWIW I would rather have a single way of doing "tests used in trademark
> > > programs" without differentiating between old and new trademark programs.
> > > 
> > > I fear that we are discussing solutions before defining the problem. We
> > > want:
> > > 
> > > 1- Decentralize test maintenance, through more tempest plugins, to
> > > account for limited QA resources
> > > 2- Additional codereview constraints and approval rules for tests that
> > > happen to be used in trademark programs
> > > 3- Discoverability/ease-of-install of the set of tests that happen to be
> > > used in trademark programs
> > > 4- A git repo layout that can be simply explained, for new teams to
> > > understand
> > > 
> > > It feels like the current git repo layout (result of that 2016-05-04
> > > resolution) optimizes for 2 and 3, which kind of works until you add
> > > more trademark programs, at which point it breaks 1 and 4.
> > > 
> > > I feel like you could get 2 and 3 without necessarily using git repo
> > > boundaries (using Gerrit approval rules and some tooling to install/run
> > > subset of tests across multiple git repos), which would allow you to
> > > optimize git repo layout to get 1 and 4...
> > > 
> > > Or am I missing something ?
> > > 
> > 
> > Right. The point of having the trademark tests "in tempest" was not
> > to have them "in the tempest repo", that was just an implementation
> > detail of the policy of "put them in a repository managed by people
> > who understand the expanded review rules".
> 
> There was more to it than this, a big part was duplication of effort as
> well.
> Tempest itself is almost a perfect fit for the scope of the testing
> defcore is
> doing. While tempest does additional testing that defcore doesn't use, a
> large
> subset is exactly what they want.
> 
> > 
> > There were a lot of unexpected issues when we started treating the
> > test suite as a production tool for validating a cloud.  We have
> > to be careful about how we change the behavior of tests, for example,
> > even if the API responses are expected to be the same.  It's not
> > fair to vendors or operators who get trademark approval with one
> > release to have significant changes in behavior in the exact same
> > tests for the next release.
> 
> I actually find this to be kinda misleading. Tempest has always had
> running on any cloud as part of it's mission. I think you're referring
> to the monster defcore thread from last summer about proprietary nova
> extensions
> adding on to API responses. This is honestly a completely separate
> problem
> which is not something I want to dive into again, because that was a much
> more
> nuanced problem that involved much more than just code review.
> 
> > 
> > At the early stage, when the DefCore team was still figuring out
> > these issues, it made sense to put all of the tests in one place
> > with a review team that was actively participating in establishing
> > the process. If we better understand the "rules" for these tests
> > now, we can document them and distribute 

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Matthew Treinish]

On Thu, Jun 01, 2017 at 11:57:00AM -0400, Doug Hellmann wrote:
> Excerpts from Thierry Carrez's message of 2017-06-01 11:51:50 +0200:
> > Graham Hayes wrote:
> > > On 01/06/17 01:30, Matthew Treinish wrote:
> > >> TBH, it's a bit premature to have the discussion. These additional 
> > >> programs do
> > >> not exist yet, and there is a governance road block around this. Right 
> > >> now the
> > >> set of projects that can be used defcore/interopWG is limited to the set 
> > >> of 
> > >> projects in:
> > >>
> > >> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html
> > > 
> > > Sure - but that is a solved problem, when the interop committee is
> > > ready to propose them, they can add projects into that tag. Or am I
> > > misunderstanding [1] (again)?
> > 
> > I think you understand it well. The Board/InteropWG should propose
> > additions/removals of this tag, which will then be approved by the TC:
> > 
> > https://governance.openstack.org/tc/reference/tags/tc_approved-release.html#tag-application-process
> > 
> > > [...]
> > >> We had a forum session on it (I can't find the etherpad for the session) 
> > >> which
> > >> was pretty speculative because it was about planning the new programs. 
> > >> Part of
> > >> that discussion was around the feasibility of using tests in plugins and 
> > >> whether
> > >> that would be desirable. Personally, I was in favor of doing that for 
> > >> some of
> > >> the proposed programs because of the way they were organized it was a 
> > >> good fit.
> > >> This is because the proposed new programs were extra additions on top of 
> > >> the
> > >> base existing interop program. But it was hardly a definitive discussion.
> > > 
> > > Which will create 2 classes of testing for interop programs.
> > 
> > FWIW I would rather have a single way of doing "tests used in trademark
> > programs" without differentiating between old and new trademark programs.
> > 
> > I fear that we are discussing solutions before defining the problem. We
> > want:
> > 
> > 1- Decentralize test maintenance, through more tempest plugins, to
> > account for limited QA resources
> > 2- Additional codereview constraints and approval rules for tests that
> > happen to be used in trademark programs
> > 3- Discoverability/ease-of-install of the set of tests that happen to be
> > used in trademark programs
> > 4- A git repo layout that can be simply explained, for new teams to
> > understand
> > 
> > It feels like the current git repo layout (result of that 2016-05-04
> > resolution) optimizes for 2 and 3, which kind of works until you add
> > more trademark programs, at which point it breaks 1 and 4.
> > 
> > I feel like you could get 2 and 3 without necessarily using git repo
> > boundaries (using Gerrit approval rules and some tooling to install/run
> > subset of tests across multiple git repos), which would allow you to
> > optimize git repo layout to get 1 and 4...
> > 
> > Or am I missing something ?
> > 
> 
> Right. The point of having the trademark tests "in tempest" was not
> to have them "in the tempest repo", that was just an implementation
> detail of the policy of "put them in a repository managed by people
> who understand the expanded review rules".

There was more to it than this, a big part was duplication of effort as well.
Tempest itself is almost a perfect fit for the scope of the testing defcore is
doing. While tempest does additional testing that defcore doesn't use, a large
subset is exactly what they want.

> 
> There were a lot of unexpected issues when we started treating the
> test suite as a production tool for validating a cloud.  We have
> to be careful about how we change the behavior of tests, for example,
> even if the API responses are expected to be the same.  It's not
> fair to vendors or operators who get trademark approval with one
> release to have significant changes in behavior in the exact same
> tests for the next release.

I actually find this to be kinda misleading. Tempest has always had
running on any cloud as part of it's mission. I think you're referring
to the monster defcore thread from last summer about proprietary nova extensions
adding on to API responses. This is honestly a completely separate problem
which is not something I want to dive into again, because that was a much more
nuanced problem that involved much more than just code review.

> 
> At the early stage, when the DefCore team was still figuring out
> these issues, it made sense to put all of the tests in one place
> with a review team that was actively participating in establishing
> the process. If we better understand the "rules" for these tests
> now, we can document them and distribute the work of maintaining the
> test suites.

I think you're overestimating how much work is actually being done
bidirectionally here. The interaction with defcore is more straight consumption
then you might think. They tend to just pick and choose from what tempest has
and don't 

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Sean McGinnis]

> 
> And yes, I agree with the argument that we should be fair and treat
> all projects the same way. If we're going to move tests out of the
> tempest repository, we should move all of them. The QA team can
> still help maintain the test suites for whatever projects they want,
> even if those tests are in plugins.
> 
> Doug
> 

+1

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Doug Hellmann]

Excerpts from Thierry Carrez's message of 2017-06-01 11:51:50 +0200:
> Graham Hayes wrote:
> > On 01/06/17 01:30, Matthew Treinish wrote:
> >> TBH, it's a bit premature to have the discussion. These additional 
> >> programs do
> >> not exist yet, and there is a governance road block around this. Right now 
> >> the
> >> set of projects that can be used defcore/interopWG is limited to the set 
> >> of 
> >> projects in:
> >>
> >> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html
> > 
> > Sure - but that is a solved problem, when the interop committee is
> > ready to propose them, they can add projects into that tag. Or am I
> > misunderstanding [1] (again)?
> 
> I think you understand it well. The Board/InteropWG should propose
> additions/removals of this tag, which will then be approved by the TC:
> 
> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html#tag-application-process
> 
> > [...]
> >> We had a forum session on it (I can't find the etherpad for the session) 
> >> which
> >> was pretty speculative because it was about planning the new programs. 
> >> Part of
> >> that discussion was around the feasibility of using tests in plugins and 
> >> whether
> >> that would be desirable. Personally, I was in favor of doing that for some 
> >> of
> >> the proposed programs because of the way they were organized it was a good 
> >> fit.
> >> This is because the proposed new programs were extra additions on top of 
> >> the
> >> base existing interop program. But it was hardly a definitive discussion.
> > 
> > Which will create 2 classes of testing for interop programs.
> 
> FWIW I would rather have a single way of doing "tests used in trademark
> programs" without differentiating between old and new trademark programs.
> 
> I fear that we are discussing solutions before defining the problem. We
> want:
> 
> 1- Decentralize test maintenance, through more tempest plugins, to
> account for limited QA resources
> 2- Additional codereview constraints and approval rules for tests that
> happen to be used in trademark programs
> 3- Discoverability/ease-of-install of the set of tests that happen to be
> used in trademark programs
> 4- A git repo layout that can be simply explained, for new teams to
> understand
> 
> It feels like the current git repo layout (result of that 2016-05-04
> resolution) optimizes for 2 and 3, which kind of works until you add
> more trademark programs, at which point it breaks 1 and 4.
> 
> I feel like you could get 2 and 3 without necessarily using git repo
> boundaries (using Gerrit approval rules and some tooling to install/run
> subset of tests across multiple git repos), which would allow you to
> optimize git repo layout to get 1 and 4...
> 
> Or am I missing something ?
> 

Right. The point of having the trademark tests "in tempest" was not
to have them "in the tempest repo", that was just an implementation
detail of the policy of "put them in a repository managed by people
who understand the expanded review rules".

There were a lot of unexpected issues when we started treating the
test suite as a production tool for validating a cloud.  We have
to be careful about how we change the behavior of tests, for example,
even if the API responses are expected to be the same.  It's not
fair to vendors or operators who get trademark approval with one
release to have significant changes in behavior in the exact same
tests for the next release.

At the early stage, when the DefCore team was still figuring out
these issues, it made sense to put all of the tests in one place
with a review team that was actively participating in establishing
the process. If we better understand the "rules" for these tests
now, we can document them and distribute the work of maintaining the
test suites.

And yes, I agree with the argument that we should be fair and treat
all projects the same way. If we're going to move tests out of the
tempest repository, we should move all of them. The QA team can
still help maintain the test suites for whatever projects they want,
even if those tests are in plugins.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Thierry Carrez]

Graham Hayes wrote:
> On 01/06/17 01:30, Matthew Treinish wrote:
>> TBH, it's a bit premature to have the discussion. These additional programs 
>> do
>> not exist yet, and there is a governance road block around this. Right now 
>> the
>> set of projects that can be used defcore/interopWG is limited to the set of 
>> projects in:
>>
>> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html
> 
> Sure - but that is a solved problem, when the interop committee is
> ready to propose them, they can add projects into that tag. Or am I
> misunderstanding [1] (again)?

I think you understand it well. The Board/InteropWG should propose
additions/removals of this tag, which will then be approved by the TC:

https://governance.openstack.org/tc/reference/tags/tc_approved-release.html#tag-application-process

> [...]
>> We had a forum session on it (I can't find the etherpad for the session) 
>> which
>> was pretty speculative because it was about planning the new programs. Part 
>> of
>> that discussion was around the feasibility of using tests in plugins and 
>> whether
>> that would be desirable. Personally, I was in favor of doing that for some of
>> the proposed programs because of the way they were organized it was a good 
>> fit.
>> This is because the proposed new programs were extra additions on top of the
>> base existing interop program. But it was hardly a definitive discussion.
> 
> Which will create 2 classes of testing for interop programs.

FWIW I would rather have a single way of doing "tests used in trademark
programs" without differentiating between old and new trademark programs.

I fear that we are discussing solutions before defining the problem. We
want:

1- Decentralize test maintenance, through more tempest plugins, to
account for limited QA resources
2- Additional codereview constraints and approval rules for tests that
happen to be used in trademark programs
3- Discoverability/ease-of-install of the set of tests that happen to be
used in trademark programs
4- A git repo layout that can be simply explained, for new teams to
understand

It feels like the current git repo layout (result of that 2016-05-04
resolution) optimizes for 2 and 3, which kind of works until you add
more trademark programs, at which point it breaks 1 and 4.

I feel like you could get 2 and 3 without necessarily using git repo
boundaries (using Gerrit approval rules and some tooling to install/run
subset of tests across multiple git repos), which would allow you to
optimize git repo layout to get 1 and 4...

Or am I missing something ?

-- 
Thierry Carrez (ttx)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Graham Hayes]

On 01/06/17 01:30, Matthew Treinish wrote:
> On Wed, May 31, 2017 at 03:45:52PM +, Jeremy Stanley wrote:
>> On 2017-05-31 15:22:59 + (+), Jeremy Stanley wrote:
>>> On 2017-05-31 09:43:11 -0400 (-0400), Doug Hellmann wrote:
>>> [...]
 it's news to me that they're considering reversing course. If the
 QA team isn't going to continue, we'll need to figure out what
 that means and potentially find another group to do it.
>>>
>>> I wasn't there for the discussion, but it sounds likely to be a
>>> mischaracterization. I'm going to assume it's not true (or much more
>>> nuanced) at least until someone responds on behalf of the QA team.
>>> This particular subthread is only going to go further into the weeds
>>> until it is grounded in some authoritative details.
>>
>> Apologies for replying to myself, but per discussion[*] with Chris
>> in #openstack-dev I'm adjusting the subject header to make it more
>> clear which particular line of speculation I consider weedy.
>>
>> Also in that brief discussion, Graham made it slightly clearer that
>> he was talking about pushback on the tempest repo getting tests for
>> new trademark programs beyond "OpenStack Powered Platform,"
>> "OpenStack Powered Compute" and "OpenStack Powered Object Storage."
> 
> TBH, it's a bit premature to have the discussion. These additional programs do
> not exist yet, and there is a governance road block around this. Right now the
> set of projects that can be used defcore/interopWG is limited to the set of 
> projects in:
> 
> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html

Sure - but that is a solved problem, when the interop committee is
ready to propose them, they can add projects into that tag. Or am I
misunderstanding [1] (again)?

This *is* the time to discuss it, as these programs are aimed as
advisory for the 2018 spec - which means we need to solve these
problems, and soon.

> We had a forum session on it (I can't find the etherpad for the session) which
> was pretty speculative because it was about planning the new programs. Part of
> that discussion was around the feasibility of using tests in plugins and 
> whether
> that would be desirable. Personally, I was in favor of doing that for some of
> the proposed programs because of the way they were organized it was a good 
> fit.
> This is because the proposed new programs were extra additions on top of the
> base existing interop program. But it was hardly a definitive discussion.

Which will create 2 classes of testing for interop programs.

> We will have to have discussions about how we're going to actually implement
> the additional programs when we start to create them, but that's not happening
> yet.
> 

Except it is - at least one team has submitted capabilities, and others
are doing so at the moment.

1 - https://review.openstack.org/#/c/368240/

> -Matt Treinish
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 




signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Matthew Treinish]

On Thu, Jun 01, 2017 at 12:32:03PM +0900, Ghanshyam Mann wrote:
> On Thu, Jun 1, 2017 at 9:46 AM, Matthew Treinish  wrote:
> > On Wed, May 31, 2017 at 04:24:14PM +, Jeremy Stanley wrote:
> >> On 2017-05-31 17:18:54 +0100 (+0100), Graham Hayes wrote:
> >> [...]
> >> > Trademark programs are trademark programs - we should have a unified
> >> > process for all of them. Let's not make the same mistakes again by
> >> > creating classes of projects / programs. I do not want this to be
> >> > a distinction as we move forward.
> >>
> >> This I agree with. However I'll be surprised if a majority of the QA
> >> team disagree on this point (logistic concerns with how to curate
> >> this over time I can understand, but that just means they need to
> >> interest some people in working on a manageable solution).
> >
> > +1 I don't think anyone disagrees with this. There is a logistical concern
> > with the way the new proposed programs are going to be introduced. Quite
> > frankly it's too varied and broad and I don't think we'll have enough people
> > working on this space to help maintain it in the same manner.
> >
> > It's the same reason we worked on the plugin decomposition in the first 
> > place.
> > You can easily look at the numbers of tests to see this:
> >
> > https://raw.githubusercontent.com/mtreinish/qa-in-the-open/lca2017/tests_per_proj.png
> >
> > Which shows things before the plugin decomposition (and before the big 
> > tent) Just
> > because we said we'd support all the incubated and integrated projects in 
> > tempest
> > didn't mean people were contributing and/or the tests were well maintained.
> >
> > But, as I said elsewhere in this thread this is a bit too early to have the
> > conversation because the new interop programs don't actually exist yet.
> 
> Yes, there is no question on goal to have a unified process for all.
> As Jeremy, Matthew mentioned, key things here is manageability issues.
> 
> We know contributors in QA are reducing cycle by cycle. I might be
> thinking over but I thought about QA team situation when we have
> around 30-40 trademark projects and all tests on Tempest
> repo.Personally I am ok to have tests in Tempest repo or a dedicated
> interop plugin repo which can be controlled by QA at some level But we

I actually don't think a dedicated interop plugin is a good idea. It doesn't
actually solve anything, because the tests are going to be the same and the
same people are going to be maintaining them. All you did was move it into a
different repo which solves none of the problems. What I was referring to was
exploring a more distributed approach to handling the tests (like what we did
for plugin decomposition for higher level services) That is the only way I see
us addressing the work overload problem. But, as I said before this is still
too early to talk about because there aren't defined new programs yet, just
the idea for them and a rough plan. We're still talking very much in the
abstract about everything...

-Matt Treinish

> need dedicated participation from interop + projects liason (I am not
> sure that worked well in pass but if with TC help it might work :)).
> 
> I can recall that, QA team has many patches on plugin side to improve
> them or fix them but may of them has no active reviews or much
> attentions from project team. I am afraid about same case for
> trademark projects also.
> 
> May be broad direction on trademark program and scope of it can help
> to imagine the quantity of programs and tests which QA teams need to
> maintain.



signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Ghanshyam Mann]

On Thu, Jun 1, 2017 at 9:46 AM, Matthew Treinish  wrote:
> On Wed, May 31, 2017 at 04:24:14PM +, Jeremy Stanley wrote:
>> On 2017-05-31 17:18:54 +0100 (+0100), Graham Hayes wrote:
>> [...]
>> > Trademark programs are trademark programs - we should have a unified
>> > process for all of them. Let's not make the same mistakes again by
>> > creating classes of projects / programs. I do not want this to be
>> > a distinction as we move forward.
>>
>> This I agree with. However I'll be surprised if a majority of the QA
>> team disagree on this point (logistic concerns with how to curate
>> this over time I can understand, but that just means they need to
>> interest some people in working on a manageable solution).
>
> +1 I don't think anyone disagrees with this. There is a logistical concern
> with the way the new proposed programs are going to be introduced. Quite
> frankly it's too varied and broad and I don't think we'll have enough people
> working on this space to help maintain it in the same manner.
>
> It's the same reason we worked on the plugin decomposition in the first place.
> You can easily look at the numbers of tests to see this:
>
> https://raw.githubusercontent.com/mtreinish/qa-in-the-open/lca2017/tests_per_proj.png
>
> Which shows things before the plugin decomposition (and before the big tent) 
> Just
> because we said we'd support all the incubated and integrated projects in 
> tempest
> didn't mean people were contributing and/or the tests were well maintained.
>
> But, as I said elsewhere in this thread this is a bit too early to have the
> conversation because the new interop programs don't actually exist yet.

Yes, there is no question on goal to have a unified process for all.
As Jeremy, Matthew mentioned, key things here is manageability issues.

We know contributors in QA are reducing cycle by cycle. I might be
thinking over but I thought about QA team situation when we have
around 30-40 trademark projects and all tests on Tempest
repo.Personally I am ok to have tests in Tempest repo or a dedicated
interop plugin repo which can be controlled by QA at some level But we
need dedicated participation from interop + projects liason (I am not
sure that worked well in pass but if with TC help it might work :)).

I can recall that, QA team has many patches on plugin side to improve
them or fix them but may of them has no active reviews or much
attentions from project team. I am afraid about same case for
trademark projects also.

May be broad direction on trademark program and scope of it can help
to imagine the quantity of programs and tests which QA teams need to
maintain.

-gmann

>
> -Matt Treinish
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Matthew Treinish]

On Wed, May 31, 2017 at 04:24:14PM +, Jeremy Stanley wrote:
> On 2017-05-31 17:18:54 +0100 (+0100), Graham Hayes wrote:
> [...]
> > Trademark programs are trademark programs - we should have a unified
> > process for all of them. Let's not make the same mistakes again by
> > creating classes of projects / programs. I do not want this to be
> > a distinction as we move forward.
> 
> This I agree with. However I'll be surprised if a majority of the QA
> team disagree on this point (logistic concerns with how to curate
> this over time I can understand, but that just means they need to
> interest some people in working on a manageable solution).

+1 I don't think anyone disagrees with this. There is a logistical concern
with the way the new proposed programs are going to be introduced. Quite
frankly it's too varied and broad and I don't think we'll have enough people
working on this space to help maintain it in the same manner.

It's the same reason we worked on the plugin decomposition in the first place.
You can easily look at the numbers of tests to see this:

https://raw.githubusercontent.com/mtreinish/qa-in-the-open/lca2017/tests_per_proj.png

Which shows things before the plugin decomposition (and before the big tent) 
Just
because we said we'd support all the incubated and integrated projects in 
tempest
didn't mean people were contributing and/or the tests were well maintained.

But, as I said elsewhere in this thread this is a bit too early to have the
conversation because the new interop programs don't actually exist yet.

-Matt Treinish


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests (was: more tempest plugins)

[Matthew Treinish]

On Wed, May 31, 2017 at 03:45:52PM +, Jeremy Stanley wrote:
> On 2017-05-31 15:22:59 + (+), Jeremy Stanley wrote:
> > On 2017-05-31 09:43:11 -0400 (-0400), Doug Hellmann wrote:
> > [...]
> > > it's news to me that they're considering reversing course. If the
> > > QA team isn't going to continue, we'll need to figure out what
> > > that means and potentially find another group to do it.
> > 
> > I wasn't there for the discussion, but it sounds likely to be a
> > mischaracterization. I'm going to assume it's not true (or much more
> > nuanced) at least until someone responds on behalf of the QA team.
> > This particular subthread is only going to go further into the weeds
> > until it is grounded in some authoritative details.
> 
> Apologies for replying to myself, but per discussion[*] with Chris
> in #openstack-dev I'm adjusting the subject header to make it more
> clear which particular line of speculation I consider weedy.
> 
> Also in that brief discussion, Graham made it slightly clearer that
> he was talking about pushback on the tempest repo getting tests for
> new trademark programs beyond "OpenStack Powered Platform,"
> "OpenStack Powered Compute" and "OpenStack Powered Object Storage."

TBH, it's a bit premature to have the discussion. These additional programs do
not exist yet, and there is a governance road block around this. Right now the
set of projects that can be used defcore/interopWG is limited to the set of 
projects in:

https://governance.openstack.org/tc/reference/tags/tc_approved-release.html

We had a forum session on it (I can't find the etherpad for the session) which
was pretty speculative because it was about planning the new programs. Part of
that discussion was around the feasibility of using tests in plugins and whether
that would be desirable. Personally, I was in favor of doing that for some of
the proposed programs because of the way they were organized it was a good fit.
This is because the proposed new programs were extra additions on top of the
base existing interop program. But it was hardly a definitive discussion.

We will have to have discussions about how we're going to actually implement
the additional programs when we start to create them, but that's not happening
yet.

-Matt Treinish


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Amrith Kumar]

I've been following this discussion from a safe distance on the mailing
list, and I just caught up on the IRC conversation as well.

It was my understanding that if a project had tests that were going to be
run for the purposes of determination whether something met the standards of
"OpenStack Powered Databases" (TBD), then those tests would reside in the
tempest repository. Any other tempest tests that the project would have for
any purpose (or purposes) should, I understand, be moved into an independent
repository per the Queens community goal[1].

I'm particularly interested in having this matter decided in an email thread
such as this one because of the comment in IRC about a verbal discussion at
the forum not being considered to be a 'binding decision' [2]. It had been
my understanding that the forum and the PTG were places where 'decisions'
were made, but if that is not the case, I'm hoping that one will be
documented as part of this thread and formalized (maybe) with a resolution
by the TC or the Interop/DefCore body?

-amrith


[1]
https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html
[2]
http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.201
7-05-31.log.html#t2017-05-31T15:39:33

--
Amrith Kumar
amrith.ku...@gmail.com


> -Original Message-
> From: Jeremy Stanley [mailto:fu...@yuggoth.org]
> Sent: Wednesday, May 31, 2017 12:24 PM
> To: OpenStack Development Mailing List (not for usage questions)
 d...@lists.openstack.org>
> Subject: Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark
tests
> 
> On 2017-05-31 17:18:54 +0100 (+0100), Graham Hayes wrote:
> [...]
> > Trademark programs are trademark programs - we should have a unified
> > process for all of them. Let's not make the same mistakes again by
> > creating classes of projects / programs. I do not want this to be a
> > distinction as we move forward.
> 
> This I agree with. However I'll be surprised if a majority of the QA team
> disagree on this point (logistic concerns with how to curate this over
time I
> can understand, but that just means they need to interest some people in
> working on a manageable solution).
> --
> Jeremy Stanley


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Jeremy Stanley]

On 2017-05-31 17:18:54 +0100 (+0100), Graham Hayes wrote:
[...]
> Trademark programs are trademark programs - we should have a unified
> process for all of them. Let's not make the same mistakes again by
> creating classes of projects / programs. I do not want this to be
> a distinction as we move forward.

This I agree with. However I'll be surprised if a majority of the QA
team disagree on this point (logistic concerns with how to curate
this over time I can understand, but that just means they need to
interest some people in working on a manageable solution).
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][tc][all] Tempest to reject trademark tests

[Graham Hayes]

On 31/05/17 16:45, Jeremy Stanley wrote:
> On 2017-05-31 15:22:59 + (+), Jeremy Stanley wrote:
>> On 2017-05-31 09:43:11 -0400 (-0400), Doug Hellmann wrote:
>> [...]
>>> it's news to me that they're considering reversing course. If the
>>> QA team isn't going to continue, we'll need to figure out what
>>> that means and potentially find another group to do it.
>>
>> I wasn't there for the discussion, but it sounds likely to be a
>> mischaracterization. I'm going to assume it's not true (or much more
>> nuanced) at least until someone responds on behalf of the QA team.
>> This particular subthread is only going to go further into the weeds
>> until it is grounded in some authoritative details.
> 
> Apologies for replying to myself, but per discussion[*] with Chris
> in #openstack-dev I'm adjusting the subject header to make it more
> clear which particular line of speculation I consider weedy.
> 
> Also in that brief discussion, Graham made it slightly clearer that
> he was talking about pushback on the tempest repo getting tests for
> new trademark programs beyond "OpenStack Powered Platform,"
> "OpenStack Powered Compute" and "OpenStack Powered Object Storage."

Trademark programs are trademark programs - we should have a unified
process for all of them. Let's not make the same mistakes again by
creating classes of projects / programs. I do not want this to be
a distinction as we move forward.

> [*]  http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2017-05-31.log.html#t2017-05-31T15:28:07
>  >
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 




signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa][tc][all] Tempest to reject trademark tests (was: more tempest plugins)

[Jeremy Stanley]

On 2017-05-31 15:22:59 + (+), Jeremy Stanley wrote:
> On 2017-05-31 09:43:11 -0400 (-0400), Doug Hellmann wrote:
> [...]
> > it's news to me that they're considering reversing course. If the
> > QA team isn't going to continue, we'll need to figure out what
> > that means and potentially find another group to do it.
> 
> I wasn't there for the discussion, but it sounds likely to be a
> mischaracterization. I'm going to assume it's not true (or much more
> nuanced) at least until someone responds on behalf of the QA team.
> This particular subthread is only going to go further into the weeds
> until it is grounded in some authoritative details.

Apologies for replying to myself, but per discussion[*] with Chris
in #openstack-dev I'm adjusting the subject header to make it more
clear which particular line of speculation I consider weedy.

Also in that brief discussion, Graham made it slightly clearer that
he was talking about pushback on the tempest repo getting tests for
new trademark programs beyond "OpenStack Powered Platform,"
"OpenStack Powered Compute" and "OpenStack Powered Object Storage."

[*] http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2017-05-31.log.html#t2017-05-31T15:28:07
 >
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev