I agree with Duncan and John here, I am not as old contributor in OpneStack as most of the people commenting here are, but I see we have done this right throughout the OpenStack lifecycle, at the start we only had Nova and we could have always said "hey lets have everything in Nova" but we went ahead to a modularized approach having specific projects concentrate on specific needs for OpenStack as whole. If we have a project that concentrates on Performance and Benchmarking with the help of other tools, we should encourage to have such tool.
Regarding having Rally integrated in OpenStack Release cycles, I think its better to have this as a integrated tool in OpenStack which Operators can use at their deployments for benchmarking and performance analysis. It could be very similar to what we have with branchless tempest is integrated in OpenStack release. Best Regards, Swapnil Kulkarni irc : coolsvap cools...@gmail.com +91-87960 10622(c) http://in.linkedin.com/in/coolsvap *"It's better to SHARE"* On Thu, Aug 7, 2014 at 6:38 AM, Yingjun Li <liyingjun1...@gmail.com> wrote: > From a user’s aspect i do think Rally is more suitable for a product-ready > cloud, and seems like it is where it focused on. It’s very easy to > evaluate that if the performance of the cloud is better after we adjust > some configs or some other tuning. It also provides SLA which maybe not > so powerful currently but it’s a good start. So I think Rally is good > enough to be in separated program. > > I totally agree that tempest shouldn’t try to cover everything, simple > makes a thing better. > > > On Aug 7, 2014, at 5:48, John Griffith <john.griff...@solidfire.com> > wrote: > > I have to agree with Duncan here. I also don't know if I fully understand > the limit in options. Stress test seems like it could/should be different > (again overlap isn't a horrible thing) and I don't see it as siphoning off > resources so not sure of the issue. We've become quite wrapped up in > projects, programs and the like lately and it seems to hinder forward > progress more than anything else. > > I'm also not convinced that Tempest is where all things belong, in fact > I've been thinking more and more that a good bit of what Tempest does today > should fall more on the responsibility of the projects themselves. For > example functional testing of features etc, ideally I'd love to have more > of that fall on the projects and their respective teams. That might even > be something as simple to start as saying "if you contribute a new feature, > you have to also provide a link to a contribution to the Tempest test-suite > that checks it". Sort of like we do for unit tests, cross-project tracking > is difficult of course, but it's a start. The other idea is maybe > functional test harnesses live in their respective projects. > > Honestly I think who better to write tests for a project than the folks > building and contributing to the project. At some point IMO the QA team > isn't going to scale. I wonder if maybe we should be thinking about > proposals for delineating responsibility and goals in terms of functional > testing? > > > > > On Wed, Aug 6, 2014 at 12:25 PM, Duncan Thomas <duncan.tho...@gmail.com> > wrote: > >> I'm not following here - you complain about rally being monolithic, >> then suggest that parts of it should be baked into tempest - a tool >> that is already huge and difficult to get into. I'd rather see tools >> that do one thing well and some overlap than one tool to rule them >> all. >> >> On 6 August 2014 14:44, Sean Dague <s...@dague.net> wrote: >> > On 08/06/2014 09:11 AM, Russell Bryant wrote: >> >> On 08/06/2014 06:30 AM, Thierry Carrez wrote: >> >>> Hi everyone, >> >>> >> >>> At the TC meeting yesterday we discussed Rally program request and >> >>> incubation request. We quickly dismissed the incubation request, as >> >>> Rally appears to be able to live happily on top of OpenStack and would >> >>> benefit from having a release cycle decoupled from the OpenStack >> >>> "integrated release". >> >>> >> >>> That leaves the question of the program. OpenStack programs are >> created >> >>> by the Technical Committee, to bless existing efforts and teams that >> are >> >>> considered *essential* to the production of the "OpenStack" integrated >> >>> release and the completion of the OpenStack project mission. There >> are 3 >> >>> ways to look at Rally and official programs at this point: >> >>> >> >>> 1. Rally as an essential QA tool >> >>> Performance testing (and especially performance regression testing) is >> >>> an essential QA function, and a feature that Rally provides. If the QA >> >>> team is happy to use Rally to fill that function, then Rally can >> >>> obviously be adopted by the (already-existing) QA program. That said, >> >>> that would put Rally under the authority of the QA PTL, and that >> raises >> >>> a few questions due to the current architecture of Rally, which is >> more >> >>> product-oriented. There needs to be further discussion between the QA >> >>> core team and the Rally team to see how that could work and if that >> >>> option would be acceptable for both sides. >> >>> >> >>> 2. Rally as an essential operator tool >> >>> Regular benchmarking of OpenStack deployments is a best practice for >> >>> cloud operators, and a feature that Rally provides. With a bit of a >> >>> stretch, we could consider that benchmarking is essential to the >> >>> completion of the OpenStack project mission. That program could one >> day >> >>> evolve to include more such "operations best practices" tools. In >> >>> addition to the slight stretch already mentioned, one concern here is >> >>> that we still want to have performance testing in QA (which is clearly >> >>> essential to the production of "OpenStack"). Letting Rally primarily >> be >> >>> an operational tool might make that outcome more difficult. >> >>> >> >>> 3. Let Rally be a product on top of OpenStack >> >>> The last option is to not have Rally in any program, and not consider >> it >> >>> *essential* to the production of the "OpenStack" integrated release or >> >>> the completion of the OpenStack project mission. Rally can happily >> exist >> >>> as an operator tool on top of OpenStack. It is built as a monolithic >> >>> product: that approach works very well for external complementary >> >>> solutions... Also be more integrated in OpenStack or part of the >> >>> OpenStack programs might come at a cost (slicing some functionality >> out >> >>> of rally to make it more a framework and less a product) that might >> not >> >>> be what its authors want. >> >>> >> >>> Let's explore each option to see which ones are viable, and the pros >> and >> >>> cons of each. >> >> >> >> My feeling right now is that Rally is trying to accomplish too much at >> >> the start (both #1 and #2). I would rather see the project focus on >> >> doing one of them as best as it can before increasing scope. >> >> >> >> It's my opinion that #1 is the most important thing that Rally can be >> >> doing to help ensure the success of OpenStack, so I'd like to explore >> >> the "Rally as a QA tool" in more detail to start with. >> > >> > I want to clarify some things. I don't think that rally in it's current >> > form belongs in any OpenStack project. It's a giant monolythic tool, >> > which is apparently a design point. That's the wrong design point for an >> > OpenStack project. >> > >> > For instance: >> > >> > >> https://github.com/stackforge/rally/tree/master/rally/benchmark/scenarios >> should >> > all be tests in Tempest (and actually today mostly are via API tests). >> > There is an existing stress framework in Tempest which does the >> > repetitive looping that rally does on these already. This fact has been >> > brought up before. >> > >> > >> https://github.com/stackforge/rally/tree/master/rally/verification/verifiers >> > - should be baked back into Tempest (at least on the results side, >> > though diving in there now it looks largely duplicative from existing >> > subunit to html code). >> > >> > https://github.com/stackforge/rally/blob/master/rally/db/api.py - is >> > largely (not entirely) what we'd like from a long term trending piece >> > that subunit2sql is working on. Again this was just all thrown into the >> > Rally db instead of thinking about how to split it off. Also, notable >> > here is there are some fundamental testr bugs (like worker >> > misallocation) which mean the data is massively dirty today. It would be >> > good for people to actually work on fixing those things. >> > >> > The parts that should stay outside of Tempest are the setup tool >> > (separation of concerns is that Tempest is the load runner, not the >> > setup environment) and any of the SLA portions. >> > >> > I think rally brings forward a good point about making Tempest easier to >> > run. But I think that shouldn't be done outside Tempest. Making the test >> > tool easier to use should be done in the tool itself. If that means >> > adding a tempest cmd or such, so be it. Note this was a topic for >> > discussion at last summit: >> > https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA >> > >> >> From the TC meeting, it seems that the QA group (via sdague, at least) >> >> has provided some feedback to Rally over the last several months. I >> >> would really like to see an analysis and write-up from the QA group on >> >> the current state of Rally and how it may (or may not) be able to serve >> >> the performance QA needs. >> > >> > Something that we need to figure out is given where we are in the >> > release cycle do we want to ask the QA team to go off and do Rally deep >> > dive now to try to pull it apart into the parts that make sense for >> > other programs to take in. There are always trade offs. >> > >> > Like the fact that right now the rally team is proposing gate jobs which >> > have some overlap to the existing largeops jobs. Did they start a >> > conversation about it? Nope. They just went off to do their thing >> > instead. https://review.openstack.org/#/c/112251/ >> > >> > So now we're going to run 2 jobs that do very similar things, with >> > different teams adjusting the test loads. Which I think is basically >> > madness. >> > >> > -Sean >> > >> > -- >> > Sean Dague >> > http://dague.net >> > >> > _______________________________________________ >> > OpenStack-dev mailing list >> > OpenStack-dev@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> -- >> Duncan Thomas >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
