On 16/06/2016 00:30, Matthew Treinish wrote: > On Wed, Jun 15, 2016 at 09:10:30AM -0400, Doug Hellmann wrote: >> Excerpts from Chris Hoge's message of 2016-06-14 16:37:06 -0700: >>> Top posting one note and direct comments inline, I’m proposing >>> this as a member of the DefCore working group, but this >>> proposal itself has not been accepted as the forward course of >>> action by the working group. These are my own views as the >>> administrator of the program and not that of the working group >>> itself, which may independently reject the idea outside of the >>> response from the upstream devs. >>> >>> I posted a link to this thread to the DefCore mailing list to make >>> that working group aware of the outstanding issues. >>> >>>> On Jun 14, 2016, at 3:50 PM, Matthew Treinish <[email protected]> wrote: >>>> >>>> On Tue, Jun 14, 2016 at 05:42:16PM -0400, Doug Hellmann wrote: >>>>> Excerpts from Matthew Treinish's message of 2016-06-14 15:12:45 -0400: >>>>>> On Tue, Jun 14, 2016 at 02:41:10PM -0400, Doug Hellmann wrote: >>>>>>> Excerpts from Matthew Treinish's message of 2016-06-14 14:21:27 -0400: >>>>>>>> On Tue, Jun 14, 2016 at 10:57:05AM -0700, Chris Hoge wrote:
<snip> >>> The current active guidelines cover icehouse through mitaka. The release >>> of 2016.08 will change that to cover juno through mitaka (with newton >>> as an add-on to 2016.08 when it’s released). There’s overlap between >>> the guidelines, so 2016.01 covers juno through mitaka while 2016.08 >>> will cover kilo through newton. Essentially two years of releases. >>> >>>>> We may also need to consider that test implementation details may >>>>> change, and have a review process within DefCore to help expose >>>>> those changes to make them clearer to deployers. >>>>> >>>>> Fixing the process issue may also mean changing the way we implement >>>>> things in Tempest. In this case, adding a flag helps move ahead >>>>> more smoothly. Perhaps we adopt that as a general policy in the >>>>> future when we make underlying behavioral changes like this to >>>>> existing tests. Perhaps instead we have a policy that we do not >>>>> change the behavior of existing tests in such significant ways, at >>>>> least if they're tagged as being used by DefCore. I don't know -- >>>>> those are things we need to discuss. >>>> >>>> Sure I agree, this thread raises larger issues which need to be figured >>>> out. >>>> But, that is probably an independent discussion. >>> >>> I’m beginning to wonder if we need to make DefCore use release >>> branches then back-port bug-fixes and relevant features additions >>> as necessary. What I suggested when the TC decided on keeping the tests in tempest was to keep them as a tempest plugin. This would allow the tests to progress overtime, and be versioned, so that def-core 201x.y would be equal to def-core-tempest-plugin version 201x.y . This allows the project developers to continue to develop tests that match their vision, without causing unforeseen breakages in def-core. It also allows the def-core tests to target a wider range - tests that are run currently have no guarantee that they will run against Kilo, but the def-core plugin could be tested against Kilo known good clouds in its gate. Is there any major blocker for moving them? >> We should definitely have that conversation, to understand what >> effect it would have both on Tempest and on DefCore. >> > > While from a quick glance this would seem like it would solve some of the > problems when you start to dig into it you'll see that it actually wouldn't, > and would just end up causing more issues in the long run. Branchless tempest > was originally started back at the icehouse release and was implemented to > actually enforce the API is the same across release boundaries. We had hit > many > issues where incompatibilities inadvertently were introduced into projects' > APIs > which weren't caught because of divergence between tempest master and tempest > stable/*. If you decide to branch you'll end up having to do the same thing or > you'll risk the same types of regressions. Testing stable branches against > master puts you in a weird place where upstream changes could break things for > your branch and you'd never know until you tried to land something. From a > tempest dev standpoint branching quite frankly doesn't make any sense. > > Also, the other thing to consider is that there wouldn't actually be anything > to > branch on. Tempest always supports whatever releases the community is > supporting. Every commit is tested against all the branches of OpenStack that > still exist. When we EOL a stable branch there is no longer anything to run > tests against. Assuming you're primarily motivated by the fact defcore > attempts > to support branches that no longer have upstream support you wouldn't actually > be able to do this by branching. When a branch is removed there isn't anything > for you to test tempest changes against, and merging code without testing it > first is a terrible idea and the opposite of how we do things in the openstack > community. > > > -Matt Treinish > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
