On 08/10/2016 01:57 PM, Matthew Treinish wrote:
On Wed, Aug 10, 2016 at 09:52:55AM -0700, Clay Gerrard wrote:
On Wed, Aug 10, 2016 at 7:42 AM, Ben Swartzlander <b...@swartzlander.org>
wrote:
A big source of problems IMO is that tempest doesn't have stable branches.
We use the master branch of tempest to test stable branches of other
projects, and tempest regularly adds new features.
How come not this +1000 just fix this?
Well, mostly because it's actually not a problem and ignores the history on why
tempest is branchless. We actually used to do this pre-icehouse and it actually
made things much worse. What was happening back then was we didn't have enough
activity to keep the stable branches working at all. So we'd go very long
periods where nothing actually could land. We also often wedged ourselves where
master tempest changed to a point where we couldn't sanely backport a fix to
the stable branch. This would often mean that up until right before a stable
release things just couldn't land until someone was actually motivated to try
and dig us out. But, what more often happened was we had to just disable tempest
on the branch, because we didn't have another option. It also turns out that
having different tests across a release boundary meant we weren't actually
validating that the OpenStack APIs were consistent and worked the same. We had
many instances where a projects API just changed between release boundaries,
which violates our API consistency and backwards compatibility guidelines.
Tempest is about verifying the API and just like an other API client it should
work against any OpenStack release.
Doing this has been a huge benefit for making things actually work on the stable
branches. (in fact just thinking back about how broken everything was all the
time back then makes me appreciate it even more) We also test every incoming
tempest change on all the stable branches, and nothing can land unless it works
on all supported branches. It means we have a consistent and stable api across
releases. We do have occasional bugs where a new test or change in tempest
triggers a new race in a project's stable branch. But, that's a real bug and
normally a fix can be backported.(which is the whole point of doing stable
branches) If it can't and the race is bad enough to actively interfere with
things, we have a mechanism to skip the test. (but that's normally a last
resort) Although, these issues tend to come up pretty infrequently in practice,
especially as we slowly ramp up the stability of things over time.
FWIW, a lot of these details are covered in the original spec for implementing
this: (although it's definitely assumes a bit of prior knowledge about the
state of things going on when it was written)
http://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html
I still don't agree with this stance. Code doesn't just magically stop
working. Code breaks when things change which aren't version controlled
properly or when you have undeclared dependencies.
Yes, managing dependencies and doing version control is a lot of work. I
understand that the tempest team is strapped for resources. However
simply declaring that the solution is to stop doing version control is
an epic failure. I read the spec above and it sounds like cry for help
rather than a well thought out idea.
Personally I would be interested in helping improve this situation, but
I think the proper way to improve it is to actually do version control
of everything that matters and use dependency management. If you do this
correctly, then maintaining the old stuff stops being a chore because it
never breaks. By definition, if a stable branch breaks without a change
going into it, you failed at doing dependency management.
I'm not sure how I even could help with the current situation. It feels
like we've dug ourselves into a hole and the plan of record is to get
used to living underground. Does anyone have a will to make thing better
and get to a place where stable branches could reasonably expected to
remain stable for months or years without constant fixing?
-Ben Swartzlander
-Matt Treinish
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
