On 08/11/2014 04:21 PM, Matthew Treinish wrote:
I apologize for the delay in my response to this thread, between
and having a stuck 'a' key on my laptop this is the earliest I could
I opted for a separate branch on this thread to summarize my views and
respond inline later on some of the previous discussion.
On Wed, Aug 06, 2014 at 12:30:35PM +0200, Thierry Carrez wrote:
> Hi everyone,
> At the TC meeting yesterday we discussed Rally program request and
> incubation request. We quickly dismissed the incubation request, as
> Rally appears to be able to live happily on top of OpenStack and would
> benefit from having a release cycle decoupled from the OpenStack
> "integrated release".
> That leaves the question of the program. OpenStack programs are created
> by the Technical Committee, to bless existing efforts and teams that are
> considered *essential* to the production of the "OpenStack" integrated
> release and the completion of the OpenStack project mission. There are 3
> ways to look at Rally and official programs at this point:
> 1. Rally as an essential QA tool
> Performance testing (and especially performance regression testing) is
> an essential QA function, and a feature that Rally provides. If the QA
> team is happy to use Rally to fill that function, then Rally can
> obviously be adopted by the (already-existing) QA program. That said,
> that would put Rally under the authority of the QA PTL, and that raises
> a few questions due to the current architecture of Rally, which is more
> product-oriented. There needs to be further discussion between the QA
> core team and the Rally team to see how that could work and if that
> option would be acceptable for both sides.
So ideally this is where Rally would belong, the scope of what Rally is
attempting to do is definitely inside the scope of the QA program. I
any reason why that isn't the case. The issue is with what Rally is in
current form. It's scope is too large and monolithic, and it
duplicates much of
the functionality we either already have or need in current QA or Infra
projects. But, nothing in Rally is designed to be used outside of it.
feel pretty strongly that in it's current form Rally should *not* be a
any OpenStack program.
All of the points Sean was making in the other branch on this thread
probably respond to later) are a huge concerns I share with Rally. He
summarized most of my views on the topic, so I'll try not to rewrite
But, the fact that all of this duplicate functionality was implemented
completely separate manner which is Rally specific and can't really be
unless all of Rally is used is of a large concern. What I think the path
forward here is to have both QA and Rally work together on getting common
functionality that is re-usable and shareable. Additionally, I have some
concerns over the methodology that Rally uses for it's performance
But, I'll table that discussion because I think it would partially
So one open question is long-term where would this leave Rally if we
bring it in under the QA program. (after splitting up the
functionality to more
conducive with all our existing tools and projects) The one thing
here which we don't have an analogous solution for is, for lack of
the post processing layer. The part that generates the performs the
the collected data and generates the graphs. That is something that
an eventually need for and that is something that we can work on
into as we migrate everything to actually work together.
There are probably also other parts of Rally which don't fit into an
QA program project, (or the QA program in general) and in those cases we
probably should split them off as smaller projects to implement that
example, the SLA stuff Rally has that probably should be a separate
well, but I'm unsure if that fits under QA program.
My primary concern is the timing for doing all of this work. We're
J-3 and honestly this feels like it would take the better part of an
cycle to analyze, plan, and then implement. Starting an analysis of
how to do
all of the work at this point I feel would just distract everyone from
completing our dev goals for the cycle. Probably the Rally team, if
to move forward here, should start the analysis of this structural
split and we
can all pick this up together post-juno.
> 2. Rally as an essential operator tool
> Regular benchmarking of OpenStack deployments is a best practice for
> cloud operators, and a feature that Rally provides. With a bit of a
> stretch, we could consider that benchmarking is essential to the
> completion of the OpenStack project mission. That program could one day
> evolve to include more such "operations best practices" tools. In
> addition to the slight stretch already mentioned, one concern here is
> that we still want to have performance testing in QA (which is clearly
> essential to the production of "OpenStack"). Letting Rally primarily be
> an operational tool might make that outcome more difficult.
So I'm opposed to this option. It feels to me like this is only on the
because the Rally team has not done a great job of communicating or
anyone else except for when it comes to either push using Rally, or this
conversation about adopting Rally.
That being said, looking at a separate "operator tool" program for
make much sense to me. There is nothing in Rally that is more or less
tooling specific compared to Tempest or some of the infra tooling. I
fail to see
what in Rally warrants a separate program. To be a bit sardonic, my
if Tempest had a REST API  then should we move it to the proposed
operators program too? The other thing, which came out of the summit,
tempest is often used by operators in a loop to get a heartbeat on
My point is that just because a tool is part of the QA program doesn't
it's not useful for operators. I think that's something that seems to
during this discussion. (or just brushed over) Sure, our first
priority is going
to be on making things work in dev environment and the gate, but that
necessarily preclude using things against a production environment.
at least, that's something we actually explicitly support. 
We were a little slow out of the gate (so to speak) on this but are
making progress by eliminating all devstack-specific stuff from tempest
configuration, adding support for non-admin parallel tempest with
multiple users, and in general getting rid of discovered roadblocks to
real use. As has been pointed out before, many folks use tempest against
real clouds, including many members of the tempest core team. IMO this
should be considered an equal priority with gating a dev environment.
The biggest problem with that goal is that tempest gate jobs do not run
in most of the vast number of actual configurations that most real
clouds can use and so it is hard to keep it working with all
configurations. But we should support these cases as best we can.
Maybe, one day there will be a need for a program like this, but I'm
seeing it here with Rally.
> 3. Let Rally be a product on top of OpenStack
> The last option is to not have Rally in any program, and not consider it
> *essential* to the production of the "OpenStack" integrated release or
> the completion of the OpenStack project mission. Rally can happily exist
> as an operator tool on top of OpenStack. It is built as a monolithic
> product: that approach works very well for external complementary
> solutions... Also be more integrated in OpenStack or part of the
> OpenStack programs might come at a cost (slicing some functionality out
> of rally to make it more a framework and less a product) that might not
> be what its authors want.
Honestly, if the Rally team wants the project to remain in it's
current form and
scope then I agree that it belongs outside of OpenStack. It definitely
like a product to me, and there is nothing stopping them from
operate as they do now on top of OpenStack. I'm sorry, but the fact
docs in the rally tree has a section for user testimonials  I feel
lot about the intent of the project.
> Let's explore each option to see which ones are viable, and the pros and
> cons of each.
I apologize if any of this is somewhat incoherent, I'm still a bit
so I'm not sure that I'm making much sense.
OpenStack-dev mailing list
OpenStack-dev mailing list