On 03/11/2014 07:35 AM, Sean Dague wrote:
On 03/11/2014 10:15 AM, Steven Dake wrote:
On 03/11/2014 04:04 AM, Sean Dague wrote:
On 03/04/2014 12:39 PM, Steven Hardy wrote:
Hi all,
As some of you know, I've been working on the instance-users blueprint[1].
This blueprint implementation requires three new items to be added to the
heat.conf, or some resources (those which create keystone users) will not
work:
https://review.openstack.org/#/c/73978/
https://review.openstack.org/#/c/76035/
So on upgrade, the deployer must create a keystone domain and domain-admin
user, add the details to heat.conf, as already been done in devstack[2].
The changes requried for this to work have already landed in devstack, but
it was discussed to day and Clint suggested this may be unacceptable
upgrade behavior - I'm not sure so looking for guidance/comments.
My plan was/is:
- Make devstack work
- Talk to tripleo folks to assist in any transition (what prompted this
discussion)
- Document the upgrade requirements in the Icehouse release notes so the
wider community can upgrade from Havana.
- Try to give a heads-up to those maintaining downstream heat deployment
tools (e.g stackforge/puppet-heat) that some tweaks will be required for
Icehouse.
However some have suggested there may be an openstack-wide policy which
requires peoples old config files to continue working indefinitely on
upgrade between versions - is this right? If so where is it documented?
This is basically enforced in code in grenade, the language for this
actually got lost in the project requirements discussion in the TC, I'll
bring that back in the post graduation requirements discussion we're
having again.
The issue is - Heat still doesn't materially participate in grenade.
Heat is substantially far behind the other integrated projects in it's
integration with the upstream testing. Only monday did we finally start
gating on a real unit of work for Heat (the heat-slow jobs). If I was
letter grading projects right now on upstream testing I'd give Nova an
A, Neutron a C (still no full run, no working grenade), and Heat a D.
Sean,
I agree the Heat community hasn't done a bang-up job of getting
integrated with Tempest. We only have 50 functional tests implemented.
The community clearly needs to do more and provide better functional
coverage with Heat.
It is inappropriate to say "Only monday did we finally start gating"
because that was a huge move in the right direction. It took alot of
effort and should not be so easily dismissed. Clearly the community,
and especially the core developers, are making an effort. Keep in mind
we have to balance upstream development work, answering user questions,
staying on top of a 5 page review queue, keeping relationships and track
of the various integrated projects which are consuming Heat as a
building block, plus all of the demands of our day jobs.
I agree it was a huge step in the right direction. It's not clear to me
why expressing that this was very recent was inappropriate.
Recent conversations have made me realize that a lot of the Heat core
team doesn't realize that Heat's participation in upstream gating is
below average, so I decided to be blunt about it. Because it was only
after being blunt about that with the Neutron team in Hong Kong did we
get any real motion on it (Neutron has seen huge gains this cycle).
All the integrated projects have the same challenges.
Upstream QA is really important. It not only protects heat from itself,
it protects it from changes in other projects.
We just don't have enough bandwidth on the core team to tackle writing
all of the tempest test cases ourselves. We have made an effort to
distribute this work to the overall heat community via wishlist bugs in
Heat which several new folks have picked up. I hope to see our coverage
improve over time, especially with more advanced scenario tests through
this effort.
Bandwidth is a problem for everyone. It's a matter of priorities. The
fact that realistic upstream gating is considered wishlist priority in
from a Heat perspective is something I find troubling.
Sean,
Unfortunately the root of the problem is there is no way to track in one
place the suggested test cases for projects. The Tempest community
doesn't want test cases in the tempest launchpad tracker. At one point
we were told to track the work using etherpads, which is absolutely
ridiculous.
So we must resort to using wishlist priority. In all cases, a user bug
that has a negative impact on operation of Heat is higher priority then
implementing functional testing. I get that if we had functional
testing, maybe that bug wouldn't have been filed in the first case.
However, we are in a situation where we already have the bugs, and they
already need to be addressed.
If the test cases were stored in tempest launchpad, they could be
properly prioritized from a "upstream-testing POV". The purpose of the
Heat launchpad tracker is to identify "Does Heat function properly".
Upstream gating doesn't really fit into that limited scope, unless the
problem is somehow negatively effecting the correct operation of the
check/gate. I believe this was the same objection the tempest folks had
with tracking tempest scenarios for Heat in the tempest tracker.
Putting the investment into realistic scenarios in Tempest / gate is
going to be a huge timesaving for the Heat team. It will ensure Heat is
functioning at every commit (not just releases), it will protect Heat
from chasing breaking issues in Keystone or Nova, and it will mean that
we'll expose more subtle issues that only come with being able to do
data analysis on 10k runs.
I get it's never fun to hear that a project is below average on a metric
that's important to the OpenStack community. But if we aren't honest and
open about these things they never change.
I understand the value of a proper check/gate for functional testing. I
also understand that a global check/gate for heat tempest scenario
testing would be a huge improvement. I am also willing to accept
criticism that the current state of things needs attention. I agree
that the current situation is not ideal.
I took offense at the implication that we aren't doing anything about
our lack of tempest integration. We clearly are.
Steps heat-core have taken:
1. Identify the entire feature matrix of Heat across the project
2. Break down the feature matrix of Heat into unique scenarios that need
testing and file workable bugs for people to take on
3. Get a heat-cfntools enabled image in the gate, so it can be run with
tempest
4. Add a gating job for Heat for the current coverage in tempest
This work didn't do itself. If you wan't to grade us on our work,
grading on our commitment to improving the situation would be more
productive.
Regards,
-steve
-Sean
--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
