Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Hello Boris, see below. Zitat von Boris Pavlovic bo...@pavlovic.me: Jay, Thanks for review of proposal. Some my comments below.. I think this is one of the roots of the problem that folks like David and Sean keep coming around to. If Rally were less monolithic, it would be easier to say OK, bring this piece into Tempest, have this piece be a separate library and live in the QA program, and have the service endpoint that allows operators to store and periodically measure SLA performance indicators against their cloud. Actually Rally was designed to be a glue service (and cli tool) that will bind everything together and present service endpoint for Operators. I really do not understand what can be split? and put to tempest? and actually why? Could you elaborate pointing on current Rally code, maybe there is some misleading here. I think this should be discussed in more details.. A good example for that is Rally's Tempest configuration module. Currently Rally has all the logic to configure Tempest and for that you have your own way to build the tempest conf out of a template [1]. If the QA team decides to rework the configuration Rally is broken. [1]: https://github.com/stackforge/rally/blob/master/rally/verification/verifiers/tempest/config.ini [snip] I found the Scalr incubation discussion: http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack-meeting.2011-06-14-20.03.log.html The reasons of reject were next: *) OpenStack shouldn't put PaaS in OpenStack core # rally is not PaaS *) Duplication of functionality (actually dashboard) # Rally doesn't duplicate anything IMHO rally duplicates at least some pieces. So you can find parts of Tempest scenarios tests in the benchmarks area, Tempest stress tests and Tempest config. Regards Marc *) Development is done behind closed doors # Not about Rally http://stackalytics.com/?release=junometric=commitsproject_type=Allmodule=rally Seems like Rally is quite different case and this comparison is misleading irrelevant to current case. , that is why I think Rally should be a separated program (i.e. Rally scope is just different from QA scope). As well, It's not clear for me, why collaboration is possible only in case of one program? In my opinion collaboration programs are irrelevant things. Sure, it's certainly possible for collaboration to happen across programs. I think what Sean is alluding to is the fact that the Tempest and Rally communities have done little collaboration to date, and that is worrying to him. Could you please explain this paragraph. What do you mean by have done little collaboration We integrated Tempest in Rally: http://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/ We are working on spec in Tempest about tempest conf generation: https://review.openstack.org/#/c/94473/ # probably not so fast as we would like We had design session: http://junodesignsummit.sched.org/event/2815ca60f70466197d3a81d62e1ee7e4#.U9_ugYCSz1g I am going to work on integration OSprofiler in tempest, as soon as I get it in core projects. By the way, I am really not sure how being one Program will help us to collaborate? What it actually changes? About collaboration between Rally Tempest teams... Major goal of integration Tempest in Rally is to make it simpler to use tempest on production clouds via OpenStack API. Plenty of folks run Tempest without Rally against production clouds as an acceptance test platform. I see no real benefit to arguing that Rally is for running against production clouds and Tempest is for non-production clouds. There just isn't much of a difference there. Hm, I didn't say anything about Tempest is for non-prduction clouds... I said that Rally team is working on making it simpler to use on production clouds.. The problem I see is that Rally is not *yet* exposing the REST service endpoint that would make it a full-fledged Operator Tool outside the scope of its current QA focus. Once Rally does indeed publish a REST API that exposes resource endpoints for an operator to store a set of KPIs associated with an SLA, and allows the operator to store the run schedule that Rally would use to go and test such metrics, *then* would be the appropriate time to suggest that Rally be the pilot project in this new Operator Tools program, IMO. It's really almost done.. It is all about 2 weeks of work... I'm sure all of those things would be welcome additions to Tempest. At the same time, Rally contributors would do well to work on an initial REST API endpoint that would expose the resources I denoted above. As I said before it's almost finished.. Best regards, Boris Pavlovic On Mon, Aug 4, 2014 at 8:25 PM, Jay Pipes jaypi...@gmail.com wrote: On 08/04/2014 11:21 AM, Boris Pavlovic wrote: Rally is quite monolithic and can't be split I think this is one of the roots of the problem that folks like David and Sean keep coming around to. If Rally
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Boris Pavlovic wrote: By the way, I am really not sure how being one Program will help us to collaborate? What it actually changes? Being in one program means you're in the same team, the same meetings, with ultimate decisions taken by the same one PTL. It obviously makes it easier to avoid duplication of effort and make stronger architectural or placement decisions. Being in two separate programs means we need to arbitrate conflicts between the two programs at the TC level, or accept some amount of duplication of effort or technical debt increase. This is why the TC is so focused on non-overlapping scopes when considering new programs, and is willing to defer blessing applications until teams work in a complementary fashion. Here it seems that we are combining several things: an instrumentation library (which sounds more like Oslo or QA), a performance testing system (which sounds more like QA), and future operator tools like a SLA management platform, LogaaS (which sounds more like their own program, but those tools are not really there yet). The combination is what creates the tension. I'm not saying there is no solution to this puzzle, just explaining where the tension comes from. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Marc, A good example for that is Rally's Tempest configuration module. Currently Rally has all the logic to configure Tempest and for that you have your own way to build the tempest conf out of a template [1]. If the QA team decides to rework the configuration Rally is broken. Absolutely agree with this point. That is why we are working on: https://review.openstack.org/#/c/94473/ that adds this feature to Tempest, when this will be implemented we will remove tempest configuration from Rally. IMHO rally duplicates at least some pieces. So you can find parts of Tempest scenarios tests in the benchmarks area, Tempest stress tests and Tempest config. Yep agree there is similar functionality in Rally Tempest about load generation: 1) tempest: https://github.com/openstack/tempest/tree/master/tempest/stress 2) rally: https://github.com/stackforge/rally/tree/master/rally/benchmark/scenarios/tempest But seems like Rally has more common load generator that can use both Tempest Rally scenarios. Maybe it makes sense to keep only one solution for this? Best regards, Boris Pavlovic On Tue, Aug 5, 2014 at 10:38 AM, m...@koderer.com wrote: Hello Boris, see below. Zitat von Boris Pavlovic bo...@pavlovic.me: Jay, Thanks for review of proposal. Some my comments below.. I think this is one of the roots of the problem that folks like David and Sean keep coming around to. If Rally were less monolithic, it would be easier to say OK, bring this piece into Tempest, have this piece be a separate library and live in the QA program, and have the service endpoint that allows operators to store and periodically measure SLA performance indicators against their cloud. Actually Rally was designed to be a glue service (and cli tool) that will bind everything together and present service endpoint for Operators. I really do not understand what can be split? and put to tempest? and actually why? Could you elaborate pointing on current Rally code, maybe there is some misleading here. I think this should be discussed in more details.. A good example for that is Rally's Tempest configuration module. Currently Rally has all the logic to configure Tempest and for that you have your own way to build the tempest conf out of a template [1]. If the QA team decides to rework the configuration Rally is broken. [1]: https://github.com/stackforge/rally/blob/master/rally/ verification/verifiers/tempest/config.ini [snip] I found the Scalr incubation discussion: http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack- meeting.2011-06-14-20.03.log.html The reasons of reject were next: *) OpenStack shouldn't put PaaS in OpenStack core # rally is not PaaS *) Duplication of functionality (actually dashboard) # Rally doesn't duplicate anything IMHO rally duplicates at least some pieces. So you can find parts of Tempest scenarios tests in the benchmarks area, Tempest stress tests and Tempest config. Regards Marc *) Development is done behind closed doors # Not about Rally http://stackalytics.com/?release=junometric=commits; project_type=Allmodule=rally Seems like Rally is quite different case and this comparison is misleading irrelevant to current case. , that is why I think Rally should be a separated program (i.e. Rally scope is just different from QA scope). As well, It's not clear for me, why collaboration is possible only in case of one program? In my opinion collaboration programs are irrelevant things. Sure, it's certainly possible for collaboration to happen across programs. I think what Sean is alluding to is the fact that the Tempest and Rally communities have done little collaboration to date, and that is worrying to him. Could you please explain this paragraph. What do you mean by have done little collaboration We integrated Tempest in Rally: http://www.mirantis.com/blog/rally-openstack-tempest- testing-made-simpler/ We are working on spec in Tempest about tempest conf generation: https://review.openstack.org/#/c/94473/ # probably not so fast as we would like We had design session: http://junodesignsummit.sched.org/event/2815ca60f70466197d3a81d62e1ee7 e4#.U9_ugYCSz1g I am going to work on integration OSprofiler in tempest, as soon as I get it in core projects. By the way, I am really not sure how being one Program will help us to collaborate? What it actually changes? About collaboration between Rally Tempest teams... Major goal of integration Tempest in Rally is to make it simpler to use tempest on production clouds via OpenStack API. Plenty of folks run Tempest without Rally against production clouds as an acceptance test platform. I see no real benefit to arguing that Rally is for running against production clouds and Tempest is for non-production clouds. There just isn't much of a difference there. Hm, I didn't say anything about Tempest is for non-prduction clouds... I
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Alex Freedland wrote: [...] I can envision other tools building a community around them and they too should become part of OpenStack operations tooling. Maybe Operator Tools program would be a better name? +1: Operator Tools is less ambiguous than SLA management or even Operations imho. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Le 02/08/2014 04:31, Alex Freedland a écrit : Angus, Rally is designed as an operations tool. Its purpose is to run a production cloud and give an operator tools and data to profile a production cloud. It is intended as a first of many such tools. There is a strong support in the community that operations tools should be developed as part of OpenStack and Rally is the first such successful community effort. I can envision other tools building a community around them and they too should become part of OpenStack operations tooling. Maybe Operator Tools program would be a better name? Some tooling exists already for development purposes : Devstack, Grenade, Tempest for the one most known. All of them are part of the QA program, except Devstack which is probably soon to be integrated as well in that QA program (see [1]) IMHO, there are 2 distinct concerns : - either we consider that Rally is another great tool for qualifying OpenStack releases, and then IMHO, the QA Program mission statement can cover this. - or, we consider that Rally is for operations only (IMHO we would loose some benefits then) and then possibly a new program could make sense. That said, by looking at the Deployment Program mission statement, I'm thinking that Rally could be part of, as it would be a gread addition for TripleO deployments. -Sylvain [1] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041731.html Alex Freedland Co-Founder Mirantis, Inc. On Thu, Jul 31, 2014 at 3:55 AM, Angus Salkeld angus.salk...@rackspace.com mailto:angus.salk...@rackspace.com wrote: On Sun, 2014-07-27 at 07:57 -0700, Sean Dague wrote: On 07/26/2014 05:51 PM, Hayes, Graham wrote: On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. I think that those discussions will still take place, it will just be on a per repository basis, instead of a per program one. [snip] Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean Is the concept of a separate review team not the point of a program? In the the thread from Designate's Incubation request Thierry said [1]: Programs just let us bless goals and teams and let them organize code however they want, with contribution to any code repo under that umbrella being considered official and ATC-status-granting. I do think that this is something that needs to be clarified by the TC - Rally could not get a PTL if they were part of the QA project, but every time we get a program request, the same discussion happens. I think that mission statements can be edited to fit new programs as they occur, and that it is more important to let teams that have been working closely together to stay as a distinct group. My big concern here is that many of the things that these efforts have been doing are things we actually want much closer to the base. For instance, metrics on Tempest runs. When Rally was first created it had it's own load generator. It took a ton of effort to keep the team from duplicating that and instead just use some subset of Tempest. Then when measuring showed up, we actually said that is something that would be great in Tempest, so whoever ran it, be it for Testing, Monitoring, or Performance gathering, would have access to that data. But the Rally team went off in a corner and did it otherwise. That's caused the QA team to have to go and redo this work from scratch with subunit2sql, in a way that can be consumed by multiple efforts. So I'm generally -1 to this being a separate effort on the basis that so far the team has decided to stay in their own sandbox instead of participating actively where many of us thing the functions should be added. I also think this isn't like Designate, because this isn't intended to be part of the integrated release. From reading Boris's email it seems like rally will provide a horizon panel and api to back it (for the operator to kick of performance runs and
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Thierry, I like Operations name for this program. It totally makes sense as we can keep this name and extend just mission of this program to include such projects like satori, rubick, logaas and others in future. Sylvian, Thank you for your input. In my opinion, having 2 separated programs doesn't mean that these 2 programs shouldn't collaborate and work together. Imho we are all one big community and should help each other. Actually the goal of Operations program is to be able to organize centralized work on OpenStack API that will provide for OpenStack Operators everything required to analyze how live production OpenStack cloud perform. And in case of issues, to be able fast to detect and debug them. If we present this via OpenStack APi, it will just make a life of QA simpler, because they won't need to do all this via scripts in gates (which is much harder task) Best regards, Boris Pavlovic On Mon, Aug 4, 2014 at 1:04 PM, Sylvain Bauza sba...@redhat.com wrote: Le 02/08/2014 04:31, Alex Freedland a écrit : Angus, Rally is designed as an operations tool. Its purpose is to run a production cloud and give an operator tools and data to profile a production cloud. It is intended as a first of many such tools. There is a strong support in the community that operations tools should be developed as part of OpenStack and Rally is the first such successful community effort. I can envision other tools building a community around them and they too should become part of OpenStack operations tooling. Maybe Operator Tools program would be a better name? Some tooling exists already for development purposes : Devstack, Grenade, Tempest for the one most known. All of them are part of the QA program, except Devstack which is probably soon to be integrated as well in that QA program (see [1]) IMHO, there are 2 distinct concerns : - either we consider that Rally is another great tool for qualifying OpenStack releases, and then IMHO, the QA Program mission statement can cover this. - or, we consider that Rally is for operations only (IMHO we would loose some benefits then) and then possibly a new program could make sense. That said, by looking at the Deployment Program mission statement, I'm thinking that Rally could be part of, as it would be a gread addition for TripleO deployments. -Sylvain [1] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041731.html Alex Freedland Co-Founder Mirantis, Inc. On Thu, Jul 31, 2014 at 3:55 AM, Angus Salkeld angus.salk...@rackspace.com wrote: On Sun, 2014-07-27 at 07:57 -0700, Sean Dague wrote: On 07/26/2014 05:51 PM, Hayes, Graham wrote: On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. I think that those discussions will still take place, it will just be on a per repository basis, instead of a per program one. [snip] Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean Is the concept of a separate review team not the point of a program? In the the thread from Designate's Incubation request Thierry said [1]: Programs just let us bless goals and teams and let them organize code however they want, with contribution to any code repo under that umbrella being considered official and ATC-status-granting. I do think that this is something that needs to be clarified by the TC - Rally could not get a PTL if they were part of the QA project, but every time we get a program request, the same discussion happens. I think that mission statements can be edited to fit new programs as they occur, and that it is more important to let teams that have been working closely together to stay as a distinct group. My big concern here is that many of the things that these efforts have been doing are things we actually want much closer to the base. For instance, metrics on Tempest runs. When Rally was first created it had it's own load generator. It took a ton of effort to keep the team from duplicating that and instead just use some subset of Tempest. Then when measuring showed up, we actually said that is something that would be great in Tempest, so whoever ran it, be it for Testing, Monitoring, or Performance gathering, would have access to that data. But the Rally
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
On 07/31/2014 06:55 AM, Angus Salkeld wrote: On Sun, 2014-07-27 at 07:57 -0700, Sean Dague wrote: On 07/26/2014 05:51 PM, Hayes, Graham wrote: On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. I think that those discussions will still take place, it will just be on a per repository basis, instead of a per program one. [snip] Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean Is the concept of a separate review team not the point of a program? In the the thread from Designate's Incubation request Thierry said [1]: Programs just let us bless goals and teams and let them organize code however they want, with contribution to any code repo under that umbrella being considered official and ATC-status-granting. I do think that this is something that needs to be clarified by the TC - Rally could not get a PTL if they were part of the QA project, but every time we get a program request, the same discussion happens. I think that mission statements can be edited to fit new programs as they occur, and that it is more important to let teams that have been working closely together to stay as a distinct group. My big concern here is that many of the things that these efforts have been doing are things we actually want much closer to the base. For instance, metrics on Tempest runs. When Rally was first created it had it's own load generator. It took a ton of effort to keep the team from duplicating that and instead just use some subset of Tempest. Then when measuring showed up, we actually said that is something that would be great in Tempest, so whoever ran it, be it for Testing, Monitoring, or Performance gathering, would have access to that data. But the Rally team went off in a corner and did it otherwise. That's caused the QA team to have to go and redo this work from scratch with subunit2sql, in a way that can be consumed by multiple efforts. So I'm generally -1 to this being a separate effort on the basis that so far the team has decided to stay in their own sandbox instead of participating actively where many of us thing the functions should be added. I also think this isn't like Designate, because this isn't intended to be part of the integrated release. From reading Boris's email it seems like rally will provide a horizon panel and api to back it (for the operator to kick of performance runs and view stats). So this does seem like something that would be a part of the integrated release (if I am reading things correctly). Is the QA program happy to extend their scope to include that? QA could become Quality Assurance of upstream code and running OpenStack installations. If not we need to find some other program for rally. I think that's realistically already the scope of the QA program, we might just need to change the governance wording. Tempest has always been intended to be run on production clouds (public or private) to ensure proper function. Many operators are doing this today as part of normal health management. And we continue to evolve it to be something which works well in that environment. All the statistics collection / analysis parts in Rally today I think are basically things that should be part of any Tempest installation / run. It's cool that Rally did a bunch of work there, but having that code outside of Tempest is sort of problematic, especially as there are huge issues with the collection of that data because of missing timing information in subunit. So realistically to get accurate results there needs to be additional events added into Tempest tests to build this correctly. If you stare at the raw results here today they have such huge accuracy problems (due to unaccounted for time in setupClass, which is a known problem) to the point of being misleading, and possibly actually harmful. These are things that are fixable, but hard to do outside of the Tempest project itself. Exporting accurate timing / stats should be a feature close to the test load, not something that's done externally with guessing and fudge factors. So every time I look at the docs in Rally - https://github.com/stackforge/rally I see largely features that should be coming out of the test runners themselves. -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Hi Sean, I would consider collaboration between different programs and their missions as a 2 different topics. To clear up the situation I made diagram that shows how new Operations program is aligned with OpenStack Integrated release and OpenStack QA. [image: Inline image 1] This means that I agree that part of Rally belongs to QA, but from other side it is going to present OpenStack API, so it's as well part of integrated release. Rally is quite monolithic and can't be split, that is why I think Rally should be a separated program (i.e. Rally scope is just different from QA scope). As well, It's not clear for me, why collaboration is possible only in case of one program? In my opinion collaboration programs are irrelevant things. About collaboration between Rally Tempest teams... Major goal of integration Tempest in Rally is to make it simpler to use tempest on production clouds via OpenStack API. This work requires a lot of collaboration between teams, as you already mention we should work on improving measuring durations and tempest.conf generation. I fully agree that this belongs to Tempest. By the way, Rally team is already helping with this part. In my opinion, end result should be something like: Rally just calls Tempest (or couple of scripts from tempest) and store results to its DB, presenting to end user tempest functionality via OpenStack API. To get this done, we should implement next features in tempest: 1) Auto tempest.conf generation 2) Production ready cleanup - tempest should be absolutely safe for run against cloud 3) Improvements related to time measurement. 4) Integration of OSprofiler Tempest. So in any case I would prefer to continue collaboration.. Thoughts? Best regards, Boris Pavlovic On Mon, Aug 4, 2014 at 4:24 PM, Sean Dague s...@dague.net wrote: On 07/31/2014 06:55 AM, Angus Salkeld wrote: On Sun, 2014-07-27 at 07:57 -0700, Sean Dague wrote: On 07/26/2014 05:51 PM, Hayes, Graham wrote: On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. I think that those discussions will still take place, it will just be on a per repository basis, instead of a per program one. [snip] Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean Is the concept of a separate review team not the point of a program? In the the thread from Designate's Incubation request Thierry said [1]: Programs just let us bless goals and teams and let them organize code however they want, with contribution to any code repo under that umbrella being considered official and ATC-status-granting. I do think that this is something that needs to be clarified by the TC - Rally could not get a PTL if they were part of the QA project, but every time we get a program request, the same discussion happens. I think that mission statements can be edited to fit new programs as they occur, and that it is more important to let teams that have been working closely together to stay as a distinct group. My big concern here is that many of the things that these efforts have been doing are things we actually want much closer to the base. For instance, metrics on Tempest runs. When Rally was first created it had it's own load generator. It took a ton of effort to keep the team from duplicating that and instead just use some subset of Tempest. Then when measuring showed up, we actually said that is something that would be great in Tempest, so whoever ran it, be it for Testing, Monitoring, or Performance gathering, would have access to that data. But the Rally team went off in a corner and did it otherwise. That's caused the QA team to have to go and redo this work from scratch with subunit2sql, in a way that can be consumed by multiple efforts. So I'm generally -1 to this being a separate effort on the basis that so far the team has decided to stay in their own sandbox instead of participating actively where many of us thing the functions should be added. I also think this isn't like Designate, because this isn't intended to be part of the integrated release. From reading Boris's email it seems like rally will provide a horizon panel and api to back it (for the operator to kick of performance runs and view stats). So this does seem like something that would be a part of the
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
On 08/04/2014 11:21 AM, Boris Pavlovic wrote: Rally is quite monolithic and can't be split I think this is one of the roots of the problem that folks like David and Sean keep coming around to. If Rally were less monolithic, it would be easier to say OK, bring this piece into Tempest, have this piece be a separate library and live in the QA program, and have the service endpoint that allows operators to store and periodically measure SLA performance indicators against their cloud. Incidentally, this is one of the problems that Scalr faced when applying for incubation, years ago, and one of the reasons the PPB at the time voted not to incubate Scalr: it had a monolithic design that crossed too many different lines in terms of duplicating functionality that already existed in a number of other projects. , that is why I think Rally should be a separated program (i.e. Rally scope is just different from QA scope). As well, It's not clear for me, why collaboration is possible only in case of one program? In my opinion collaboration programs are irrelevant things. Sure, it's certainly possible for collaboration to happen across programs. I think what Sean is alluding to is the fact that the Tempest and Rally communities have done little collaboration to date, and that is worrying to him. About collaboration between Rally Tempest teams... Major goal of integration Tempest in Rally is to make it simpler to use tempest on production clouds via OpenStack API. Plenty of folks run Tempest without Rally against production clouds as an acceptance test platform. I see no real benefit to arguing that Rally is for running against production clouds and Tempest is for non-production clouds. There just isn't much of a difference there. That said, an Operator Tools program is actually an entirely different concept -- with a different audience and mission from the QA program. I think you've seen here some initial support for such a proposed Operator Tools program. The problem I see is that Rally is not *yet* exposing the REST service endpoint that would make it a full-fledged Operator Tool outside the scope of its current QA focus. Once Rally does indeed publish a REST API that exposes resource endpoints for an operator to store a set of KPIs associated with an SLA, and allows the operator to store the run schedule that Rally would use to go and test such metrics, *then* would be the appropriate time to suggest that Rally be the pilot project in this new Operator Tools program, IMO. This work requires a lot of collaboration between teams, as you already mention we should work on improving measuring durations and tempest.conf generation. I fully agree that this belongs to Tempest. By the way, Rally team is already helping with this part. In my opinion, end result should be something like: Rally just calls Tempest (or couple of scripts from tempest) and store results to its DB, presenting to end user tempest functionality via OpenStack API. To get this done, we should implement next features in tempest: 1) Auto tempest.conf generation 2) Production ready cleanup - tempest should be absolutely safe for run against cloud 3) Improvements related to time measurement. 4) Integration of OSprofiler Tempest. I'm sure all of those things would be welcome additions to Tempest. At the same time, Rally contributors would do well to work on an initial REST API endpoint that would expose the resources I denoted above. Best, -jay So in any case I would prefer to continue collaboration.. Thoughts? Best regards, Boris Pavlovic On Mon, Aug 4, 2014 at 4:24 PM, Sean Dague s...@dague.net mailto:s...@dague.net wrote: On 07/31/2014 06:55 AM, Angus Salkeld wrote: On Sun, 2014-07-27 at 07:57 -0700, Sean Dague wrote: On 07/26/2014 05:51 PM, Hayes, Graham wrote: On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. I think that those discussions will still take place, it will just be on a per repository basis, instead of a per program one. [snip] Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean Is the concept of a separate review team not the point of a program? In the the thread from Designate's Incubation request Thierry said [1]: Programs just let us bless goals and teams and let them organize code however they want, with contribution to any code repo under that umbrella
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Jay, Thanks for review of proposal. Some my comments below.. I think this is one of the roots of the problem that folks like David and Sean keep coming around to. If Rally were less monolithic, it would be easier to say OK, bring this piece into Tempest, have this piece be a separate library and live in the QA program, and have the service endpoint that allows operators to store and periodically measure SLA performance indicators against their cloud. Actually Rally was designed to be a glue service (and cli tool) that will bind everything together and present service endpoint for Operators. I really do not understand what can be split? and put to tempest? and actually why? Could you elaborate pointing on current Rally code, maybe there is some misleading here. I think this should be discussed in more details.. By the way I believe that monolithic architecture is the best one, like Linus does.=) http://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_debate Incidentally, this is one of the problems that Scalr faced when applying for incubation, years ago, and one of the reasons the PPB at the time voted not to incubate Scalr: it had a monolithic design that crossed too many different lines in terms of duplicating functionality that already existed in a number of other projects. I found the Scalr incubation discussion: http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack-meeting.2011-06-14-20.03.log.html The reasons of reject were next: *) OpenStack shouldn't put PaaS in OpenStack core # rally is not PaaS *) Duplication of functionality (actually dashboard) # Rally doesn't duplicate anything *) Development is done behind closed doors # Not about Rally http://stackalytics.com/?release=junometric=commitsproject_type=Allmodule=rally Seems like Rally is quite different case and this comparison is misleading irrelevant to current case. , that is why I think Rally should be a separated program (i.e. Rally scope is just different from QA scope). As well, It's not clear for me, why collaboration is possible only in case of one program? In my opinion collaboration programs are irrelevant things. Sure, it's certainly possible for collaboration to happen across programs. I think what Sean is alluding to is the fact that the Tempest and Rally communities have done little collaboration to date, and that is worrying to him. Could you please explain this paragraph. What do you mean by have done little collaboration We integrated Tempest in Rally: http://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/ We are working on spec in Tempest about tempest conf generation: https://review.openstack.org/#/c/94473/ # probably not so fast as we would like We had design session: http://junodesignsummit.sched.org/event/2815ca60f70466197d3a81d62e1ee7e4#.U9_ugYCSz1g I am going to work on integration OSprofiler in tempest, as soon as I get it in core projects. By the way, I am really not sure how being one Program will help us to collaborate? What it actually changes? About collaboration between Rally Tempest teams... Major goal of integration Tempest in Rally is to make it simpler to use tempest on production clouds via OpenStack API. Plenty of folks run Tempest without Rally against production clouds as an acceptance test platform. I see no real benefit to arguing that Rally is for running against production clouds and Tempest is for non-production clouds. There just isn't much of a difference there. Hm, I didn't say anything about Tempest is for non-prduction clouds... I said that Rally team is working on making it simpler to use on production clouds.. The problem I see is that Rally is not *yet* exposing the REST service endpoint that would make it a full-fledged Operator Tool outside the scope of its current QA focus. Once Rally does indeed publish a REST API that exposes resource endpoints for an operator to store a set of KPIs associated with an SLA, and allows the operator to store the run schedule that Rally would use to go and test such metrics, *then* would be the appropriate time to suggest that Rally be the pilot project in this new Operator Tools program, IMO. It's really almost done.. It is all about 2 weeks of work... I'm sure all of those things would be welcome additions to Tempest. At the same time, Rally contributors would do well to work on an initial REST API endpoint that would expose the resources I denoted above. As I said before it's almost finished.. Best regards, Boris Pavlovic On Mon, Aug 4, 2014 at 8:25 PM, Jay Pipes jaypi...@gmail.com wrote: On 08/04/2014 11:21 AM, Boris Pavlovic wrote: Rally is quite monolithic and can't be split I think this is one of the roots of the problem that folks like David and Sean keep coming around to. If Rally were less monolithic, it would be easier to say OK, bring this piece into Tempest, have this piece be a separate library and live in the QA
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Angus, Rally is designed as an operations tool. Its purpose is to run a production cloud and give an operator tools and data to profile a production cloud. It is intended as a first of many such tools. There is a strong support in the community that operations tools should be developed as part of OpenStack and Rally is the first such successful community effort. I can envision other tools building a community around them and they too should become part of OpenStack operations tooling. Maybe Operator Tools program would be a better name? Alex Freedland Co-Founder Mirantis, Inc. On Thu, Jul 31, 2014 at 3:55 AM, Angus Salkeld angus.salk...@rackspace.com wrote: On Sun, 2014-07-27 at 07:57 -0700, Sean Dague wrote: On 07/26/2014 05:51 PM, Hayes, Graham wrote: On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. I think that those discussions will still take place, it will just be on a per repository basis, instead of a per program one. [snip] Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean Is the concept of a separate review team not the point of a program? In the the thread from Designate's Incubation request Thierry said [1]: Programs just let us bless goals and teams and let them organize code however they want, with contribution to any code repo under that umbrella being considered official and ATC-status-granting. I do think that this is something that needs to be clarified by the TC - Rally could not get a PTL if they were part of the QA project, but every time we get a program request, the same discussion happens. I think that mission statements can be edited to fit new programs as they occur, and that it is more important to let teams that have been working closely together to stay as a distinct group. My big concern here is that many of the things that these efforts have been doing are things we actually want much closer to the base. For instance, metrics on Tempest runs. When Rally was first created it had it's own load generator. It took a ton of effort to keep the team from duplicating that and instead just use some subset of Tempest. Then when measuring showed up, we actually said that is something that would be great in Tempest, so whoever ran it, be it for Testing, Monitoring, or Performance gathering, would have access to that data. But the Rally team went off in a corner and did it otherwise. That's caused the QA team to have to go and redo this work from scratch with subunit2sql, in a way that can be consumed by multiple efforts. So I'm generally -1 to this being a separate effort on the basis that so far the team has decided to stay in their own sandbox instead of participating actively where many of us thing the functions should be added. I also think this isn't like Designate, because this isn't intended to be part of the integrated release. From reading Boris's email it seems like rally will provide a horizon panel and api to back it (for the operator to kick of performance runs and view stats). So this does seem like something that would be a part of the integrated release (if I am reading things correctly). Is the QA program happy to extend their scope to include that? QA could become Quality Assurance of upstream code and running OpenStack installations. If not we need to find some other program for rally. -Angus Of course you could decide to slice up the universe in a completely different way, but we have toolchains today, which I think the focus should be on participating there. -Sean ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Zitat von Sylvain Bauza sba...@redhat.com: Le 30/07/2014 22:13, Boris Pavlovic a écrit : Hi all, This thread is very useful. We've detect issue related to the mission statement and name of proposed program on early steps. Seems like mission statement and name are totally unclear and don't present in the right perspective goals of this program. I updated name and mission statement: name: SLA Management mission: Provide SLA Management for production OpenStack clouds. This includes measuring and tracking performance of OpenStack Services, key API methods and cloud applications, performance and functional tests on demand, and everything that is required to detect and debug issues in live production clouds. As well, I updated patch to governance: https://review.openstack.org/#/c/108502/3 I hope now it's more clear, what is the goal of this program and why we should add new program. Thoughts? -1 -1 to it. SLA means that you create a contract in between a provider and an user. Here, you don't create a contract (ie. you don't create ratios and engagement) but you monitor these contracts. So I agree with Sylvain, SLA Management would imply to have a tool set that monitors agreements of a service. I don't see how this fits into the existing projects you are referring to. IMHO SLA Management would never consist of debugging or tracing anything. It would be always focused on a service and not on a root-cause analysis. Regards Marc As there are no SLAs now in OpenStack upstream (that's something for operators), we can't say if the KPIs are OK or not. How can you ensure that the code will be 9 nines if you don't look at how OpenStack will be deployed ? If you say the mission statement is to provide measurement tools for OpenStack, then it possibly goes either in QA or in Telemetry programs. If you say that the goal is to detect and debug issues in clouds, then it clearly goes into QA program. IMHO, both your name and your mission statement are confusing. Long story short, I'm pro Rally as a separate project but in the QA program. -Sylvain Best regards, Boris Pavlovic On Tue, Jul 29, 2014 at 12:39 AM, Boris Pavlovic bo...@pavlovic.me mailto:bo...@pavlovic.me wrote: Hi Sean, I appreciate you valuing Rally so highly as to suggesting it should join the QA program. It is a great vote of confidence for me. While I believe that Rally and Tempest will always work closely together, the intended utility and the direction of where we are planing to take Rally will not be compatible with the direction of where I think the QA program is going. Please let me explain in more details below. Tempest is a collection of Functional and Performance Tests which is used by the developers to improve the quality of the OpenStack code. Rally on the other hand, is envisioned as a Tool that is going to be run by the cloud operators in order to measure, tune and continuously improve the performance of an OpenStack cloud. Moreover, we have an SLA module that allows the Operator to define what constitutes an acceptable level of performance and a profiler that would provide both the user and the developer the diagnostic set of performance data. Finally, Rally is designed to run on production clouds and to be integrated as a Horizon plugin. In the future, we envision integrating Rally with other services (e.g. Logging as a Service, Satori, Rubick, and other operator-targeted services). I believe that this is not the direction compatible with the mission of the the QA program . Before applying for a new Performance and Scalability program, we have thought that the best existing program that Rally could be a part of now and in the future is the Telemetry program. We have discussed with Eoghan Glynn the idea of extending the scope of its mission to include other operator related projects and include Rally to it. Eoghan liked the idea in general but felt that Ceilometer currently has too much on its plate and was not in a position to merge in a new project. However, I can still see the two programs maturing and potentially becoming one down the road. Now, regarding the point that you make of Rally and Tempest doing some duplicate work. I completely agree with you that we should avoid it as much as possible and we should stay in close communication to make sure that duplicate requirements are only implemented once. Following our earlier discussion, Rally is now using Tempest for those benchmarks that do not require special complex environments, we also encapsulated and automated Tempest usage to make it more accessible for the Operators (here is the Blog documenting it -- http://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/). We would like to further continue to de-duplicate the work inside
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
On Sun, 2014-07-27 at 07:57 -0700, Sean Dague wrote: On 07/26/2014 05:51 PM, Hayes, Graham wrote: On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. I think that those discussions will still take place, it will just be on a per repository basis, instead of a per program one. [snip] Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean Is the concept of a separate review team not the point of a program? In the the thread from Designate's Incubation request Thierry said [1]: Programs just let us bless goals and teams and let them organize code however they want, with contribution to any code repo under that umbrella being considered official and ATC-status-granting. I do think that this is something that needs to be clarified by the TC - Rally could not get a PTL if they were part of the QA project, but every time we get a program request, the same discussion happens. I think that mission statements can be edited to fit new programs as they occur, and that it is more important to let teams that have been working closely together to stay as a distinct group. My big concern here is that many of the things that these efforts have been doing are things we actually want much closer to the base. For instance, metrics on Tempest runs. When Rally was first created it had it's own load generator. It took a ton of effort to keep the team from duplicating that and instead just use some subset of Tempest. Then when measuring showed up, we actually said that is something that would be great in Tempest, so whoever ran it, be it for Testing, Monitoring, or Performance gathering, would have access to that data. But the Rally team went off in a corner and did it otherwise. That's caused the QA team to have to go and redo this work from scratch with subunit2sql, in a way that can be consumed by multiple efforts. So I'm generally -1 to this being a separate effort on the basis that so far the team has decided to stay in their own sandbox instead of participating actively where many of us thing the functions should be added. I also think this isn't like Designate, because this isn't intended to be part of the integrated release. From reading Boris's email it seems like rally will provide a horizon panel and api to back it (for the operator to kick of performance runs and view stats). So this does seem like something that would be a part of the integrated release (if I am reading things correctly). Is the QA program happy to extend their scope to include that? QA could become Quality Assurance of upstream code and running OpenStack installations. If not we need to find some other program for rally. -Angus Of course you could decide to slice up the universe in a completely different way, but we have toolchains today, which I think the focus should be on participating there. -Sean ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: This is a digitally signed message part ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Hi all, This thread is very useful. We've detect issue related to the mission statement and name of proposed program on early steps. Seems like mission statement and name are totally unclear and don't present in the right perspective goals of this program. I updated name and mission statement: name: SLA Management mission: Provide SLA Management for production OpenStack clouds. This includes measuring and tracking performance of OpenStack Services, key API methods and cloud applications, performance and functional tests on demand, and everything that is required to detect and debug issues in live production clouds. As well, I updated patch to governance: https://review.openstack.org/#/c/108502/3 I hope now it's more clear, what is the goal of this program and why we should add new program. Thoughts? Best regards, Boris Pavlovic On Tue, Jul 29, 2014 at 12:39 AM, Boris Pavlovic bo...@pavlovic.me wrote: Hi Sean, I appreciate you valuing Rally so highly as to suggesting it should join the QA program. It is a great vote of confidence for me. While I believe that Rally and Tempest will always work closely together, the intended utility and the direction of where we are planing to take Rally will not be compatible with the direction of where I think the QA program is going. Please let me explain in more details below. Tempest is a collection of Functional and Performance Tests which is used by the developers to improve the quality of the OpenStack code. Rally on the other hand, is envisioned as a Tool that is going to be run by the cloud operators in order to measure, tune and continuously improve the performance of an OpenStack cloud. Moreover, we have an SLA module that allows the Operator to define what constitutes an acceptable level of performance and a profiler that would provide both the user and the developer the diagnostic set of performance data. Finally, Rally is designed to run on production clouds and to be integrated as a Horizon plugin. In the future, we envision integrating Rally with other services (e.g. Logging as a Service, Satori, Rubick, and other operator-targeted services). I believe that this is not the direction compatible with the mission of the the QA program . Before applying for a new Performance and Scalability program, we have thought that the best existing program that Rally could be a part of now and in the future is the Telemetry program. We have discussed with Eoghan Glynn the idea of extending the scope of its mission to include other operator related projects and include Rally to it. Eoghan liked the idea in general but felt that Ceilometer currently has too much on its plate and was not in a position to merge in a new project. However, I can still see the two programs maturing and potentially becoming one down the road. Now, regarding the point that you make of Rally and Tempest doing some duplicate work. I completely agree with you that we should avoid it as much as possible and we should stay in close communication to make sure that duplicate requirements are only implemented once. Following our earlier discussion, Rally is now using Tempest for those benchmarks that do not require special complex environments, we also encapsulated and automated Tempest usage to make it more accessible for the Operators (here is the Blog documenting it -- http://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/ ). We would like to further continue to de-duplicate the work inside Tempest and Rally. We made some joint design decisions in Atlanta to transfer some of the Integration code from Rally to Tempest, resulting in the work performed by Andrew Kurilin (https://review.openstack.org/#/c/94473/). I would encourage and welcome more of such cooperation in the future. I trust that this addresses most of your concerns and please do not hesitate to bring up more questions and suggestions. Sincerely, Boris On Sun, Jul 27, 2014 at 6:57 PM, Sean Dague s...@dague.net wrote: On 07/26/2014 05:51 PM, Hayes, Graham wrote: On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. I think that those discussions will still take place, it will just be on a per repository basis, instead of a per program one. [snip] Right, 100% agreed. Rally would remain with it's own repo + review team, just like
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Le 30/07/2014 22:13, Boris Pavlovic a écrit : Hi all, This thread is very useful. We've detect issue related to the mission statement and name of proposed program on early steps. Seems like mission statement and name are totally unclear and don't present in the right perspective goals of this program. I updated name and mission statement: name: SLA Management mission: Provide SLA Management for production OpenStack clouds. This includes measuring and tracking performance of OpenStack Services, key API methods and cloud applications, performance and functional tests on demand, and everything that is required to detect and debug issues in live production clouds. As well, I updated patch to governance: https://review.openstack.org/#/c/108502/3 I hope now it's more clear, what is the goal of this program and why we should add new program. Thoughts? -1 to it. SLA means that you create a contract in between a provider and an user. Here, you don't create a contract (ie. you don't create ratios and engagement) but you monitor these contracts. As there are no SLAs now in OpenStack upstream (that's something for operators), we can't say if the KPIs are OK or not. How can you ensure that the code will be 9 nines if you don't look at how OpenStack will be deployed ? If you say the mission statement is to provide measurement tools for OpenStack, then it possibly goes either in QA or in Telemetry programs. If you say that the goal is to detect and debug issues in clouds, then it clearly goes into QA program. IMHO, both your name and your mission statement are confusing. Long story short, I'm pro Rally as a separate project but in the QA program. -Sylvain Best regards, Boris Pavlovic On Tue, Jul 29, 2014 at 12:39 AM, Boris Pavlovic bo...@pavlovic.me mailto:bo...@pavlovic.me wrote: Hi Sean, I appreciate you valuing Rally so highly as to suggesting it should join the QA program. It is a great vote of confidence for me. While I believe that Rally and Tempest will always work closely together, the intended utility and the direction of where we are planing to take Rally will not be compatible with the direction of where I think the QA program is going. Please let me explain in more details below. Tempest is a collection of Functional and Performance Tests which is used by the developers to improve the quality of the OpenStack code. Rally on the other hand, is envisioned as a Tool that is going to be run by the cloud operators in order to measure, tune and continuously improve the performance of an OpenStack cloud. Moreover, we have an SLA module that allows the Operator to define what constitutes an acceptable level of performance and a profiler that would provide both the user and the developer the diagnostic set of performance data. Finally, Rally is designed to run on production clouds and to be integrated as a Horizon plugin. In the future, we envision integrating Rally with other services (e.g. Logging as a Service, Satori, Rubick, and other operator-targeted services). I believe that this is not the direction compatible with the mission of the the QA program . Before applying for a new Performance and Scalability program, we have thought that the best existing program that Rally could be a part of now and in the future is the Telemetry program. We have discussed with Eoghan Glynn the idea of extending the scope of its mission to include other operator related projects and include Rally to it. Eoghan liked the idea in general but felt that Ceilometer currently has too much on its plate and was not in a position to merge in a new project. However, I can still see the two programs maturing and potentially becoming one down the road. Now, regarding the point that you make of Rally and Tempest doing some duplicate work. I completely agree with you that we should avoid it as much as possible and we should stay in close communication to make sure that duplicate requirements are only implemented once. Following our earlier discussion, Rally is now using Tempest for those benchmarks that do not require special complex environments, we also encapsulated and automated Tempest usage to make it more accessible for the Operators (here is the Blog documenting it -- http://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/). We would like to further continue to de-duplicate the work inside Tempest and Rally. We made some joint design decisions in Atlanta to transfer some of the Integration code from Rally to Tempest, resulting in the work performed by Andrew Kurilin (https://review.openstack.org/#/c/94473/). I would encourage and welcome more of such cooperation in the future. I trust
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Hi Sean, I appreciate you valuing Rally so highly as to suggesting it should join the QA program. It is a great vote of confidence for me. While I believe that Rally and Tempest will always work closely together, the intended utility and the direction of where we are planing to take Rally will not be compatible with the direction of where I think the QA program is going. Please let me explain in more details below. Tempest is a collection of Functional and Performance Tests which is used by the developers to improve the quality of the OpenStack code. Rally on the other hand, is envisioned as a Tool that is going to be run by the cloud operators in order to measure, tune and continuously improve the performance of an OpenStack cloud. Moreover, we have an SLA module that allows the Operator to define what constitutes an acceptable level of performance and a profiler that would provide both the user and the developer the diagnostic set of performance data. Finally, Rally is designed to run on production clouds and to be integrated as a Horizon plugin. In the future, we envision integrating Rally with other services (e.g. Logging as a Service, Satori, Rubick, and other operator-targeted services). I believe that this is not the direction compatible with the mission of the the QA program . Before applying for a new Performance and Scalability program, we have thought that the best existing program that Rally could be a part of now and in the future is the Telemetry program. We have discussed with Eoghan Glynn the idea of extending the scope of its mission to include other operator related projects and include Rally to it. Eoghan liked the idea in general but felt that Ceilometer currently has too much on its plate and was not in a position to merge in a new project. However, I can still see the two programs maturing and potentially becoming one down the road. Now, regarding the point that you make of Rally and Tempest doing some duplicate work. I completely agree with you that we should avoid it as much as possible and we should stay in close communication to make sure that duplicate requirements are only implemented once. Following our earlier discussion, Rally is now using Tempest for those benchmarks that do not require special complex environments, we also encapsulated and automated Tempest usage to make it more accessible for the Operators (here is the Blog documenting it -- http://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/ ). We would like to further continue to de-duplicate the work inside Tempest and Rally. We made some joint design decisions in Atlanta to transfer some of the Integration code from Rally to Tempest, resulting in the work performed by Andrew Kurilin (https://review.openstack.org/#/c/94473/). I would encourage and welcome more of such cooperation in the future. I trust that this addresses most of your concerns and please do not hesitate to bring up more questions and suggestions. Sincerely, Boris On Sun, Jul 27, 2014 at 6:57 PM, Sean Dague s...@dague.net wrote: On 07/26/2014 05:51 PM, Hayes, Graham wrote: On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. I think that those discussions will still take place, it will just be on a per repository basis, instead of a per program one. [snip] Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean Is the concept of a separate review team not the point of a program? In the the thread from Designate's Incubation request Thierry said [1]: Programs just let us bless goals and teams and let them organize code however they want, with contribution to any code repo under that umbrella being considered official and ATC-status-granting. I do think that this is something that needs to be clarified by the TC - Rally could not get a PTL if they were part of the QA project, but every time we get a program request, the same discussion happens. I think that mission statements can be edited to fit new programs as they occur, and that it is more important to let teams that have been working closely together to stay as a distinct group. My big concern here is that many of the things that these efforts have been doing are things we actually want much closer to the base. For instance, metrics on Tempest runs. When Rally was first
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
On 07/26/2014 05:51 PM, Hayes, Graham wrote: On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. I think that those discussions will still take place, it will just be on a per repository basis, instead of a per program one. [snip] Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean Is the concept of a separate review team not the point of a program? In the the thread from Designate's Incubation request Thierry said [1]: Programs just let us bless goals and teams and let them organize code however they want, with contribution to any code repo under that umbrella being considered official and ATC-status-granting. I do think that this is something that needs to be clarified by the TC - Rally could not get a PTL if they were part of the QA project, but every time we get a program request, the same discussion happens. I think that mission statements can be edited to fit new programs as they occur, and that it is more important to let teams that have been working closely together to stay as a distinct group. My big concern here is that many of the things that these efforts have been doing are things we actually want much closer to the base. For instance, metrics on Tempest runs. When Rally was first created it had it's own load generator. It took a ton of effort to keep the team from duplicating that and instead just use some subset of Tempest. Then when measuring showed up, we actually said that is something that would be great in Tempest, so whoever ran it, be it for Testing, Monitoring, or Performance gathering, would have access to that data. But the Rally team went off in a corner and did it otherwise. That's caused the QA team to have to go and redo this work from scratch with subunit2sql, in a way that can be consumed by multiple efforts. So I'm generally -1 to this being a separate effort on the basis that so far the team has decided to stay in their own sandbox instead of participating actively where many of us thing the functions should be added. I also think this isn't like Designate, because this isn't intended to be part of the integrated release. Of course you could decide to slice up the universe in a completely different way, but we have toolchains today, which I think the focus should be on participating there. -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. I think that those discussions will still take place, it will just be on a per repository basis, instead of a per program one. [snip] Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean Is the concept of a separate review team not the point of a program? In the the thread from Designate's Incubation request Thierry said [1]: Programs just let us bless goals and teams and let them organize code however they want, with contribution to any code repo under that umbrella being considered official and ATC-status-granting. I do think that this is something that needs to be clarified by the TC - Rally could not get a PTL if they were part of the QA project, but every time we get a program request, the same discussion happens. I think that mission statements can be edited to fit new programs as they occur, and that it is more important to let teams that have been working closely together to stay as a distinct group. Graham 1 - http://lists.openstack.org/pipermail/openstack-dev/2014-May/036213.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. QA Program mission: Develop, maintain, and initiate tools and plans to ensure the upstream stability and quality of OpenStack, and its release readiness at any point during the release cycle. It is hard to see how $subj falls outside of that mission. Of course rally would continue to have its own repo, review team, etc. as do tempest and grenade. -David On 07/21/2014 05:53 PM, Boris Pavlovic wrote: Hi Stackers and TC, The Rally contributor team would like to propose a new OpenStack program with a mission to provide scalability and performance benchmarking, and code profiling tools for OpenStack components. We feel we've achieved a critical mass in the Rally project, with an active, diverse contributor team. The Rally project will be the initial project in a new proposed Performance and Scalability program. Below, the details on our proposed new program. Thanks for your consideration, Boris [1] https://review.openstack.org/#/c/108502/ Official Name = Performance and Scalability Codename Rally Scope = Scalability benchmarking, performance analysis, and profiling of OpenStack components and workloads Mission === To increase the scalability and performance of OpenStack clouds by: * defining standard benchmarks * sharing performance data between operators and developers * providing transparency of code paths through profiling tools Maturity * Meeting logs http://eavesdrop.openstack.org/meetings/rally/2014/ * IRC channel: #openstack-rally * Rally performance jobs are in (Cinder, Glance, Keystone Neutron) check pipelines. * 950 commits over last 10 months * Large, diverse contributor community * http://stackalytics.com/?release=junometric=commitsproject_type=Allmodule=rally * http://stackalytics.com/report/contribution/rally/180 * Non official lead of project is Boris Pavlovic * Official election In progress. Deliverables Critical deliverables in the Juno cycle are: * extending Rally Benchmark framework to cover all use cases that are required by all OpenStack projects * integrating OSprofiler in all core projects * increasing functional unit testing coverage of Rally. Discussion == One of the major goals of Rally is to make it simple to share results of standardized benchmarks and experiments between operators and developers. When an operator needs to verify certain performance indicators meet some service level agreement, he will be able to run benchmarks (from Rally) and share with the developer community the results along with his OpenStack configuration. These benchmark results will assist developers in diagnosing particular performance and scalability problems experienced with the operator's configuration. Another interesting area is Rally the OpenStack CI process. Currently, working on performance issues upstream tends to be a more social than technical process. We can use Rally in the upstream gates to identify performance regressions and measure improvement in scalability over time. The use of Rally in the upstream gates will allow a more rigorous, scientific approach to performance analysis. In the case of an integrated OSprofiler, it will be possible to get detailed information about API call flows (e.g. duration of API calls in different services). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. QA Program mission: Develop, maintain, and initiate tools and plans to ensure the upstream stability and quality of OpenStack, and its release readiness at any point during the release cycle. It is hard to see how $subj falls outside of that mission. Of course rally would continue to have its own repo, review team, etc. as do tempest and grenade. Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally][qa] Application for a new OpenStack Program: Performance and Scalability
Sean, David So seems like I am better in writing code then english, sorry for that=) Let me try to explain my position and try to make this situation clear. We have the great program QA that helps to keep OpenStack working (automation of testing, unit/function/integration tests, log analyze, elastic recheck, dsvm jobs and so on). I really appreciate what you guys are doing, and thank you for that! But the scope of what Rally team is working on is quite different. It's more about Operations cases e.g. making it simple to understand what is happening inside production OpenStack clouds (especially under load). To do that, it's not enough just to write tools (like Rally), it requires extending OpenStack API, to make it simple to retrieve via OpenStack API audit information (profiling data - OSprofiler, querying logs - LogaaS, configuration discovery - Satori), which is really out of scope of QA (it should be done inside OpenStack projects, not on top of them). And to coordinate this job we should have Program Operations for that. The reason why this program is called Performance Scalability and not Operations is that current Rally team scope is Performance Scalability. But as I see Rally team really would like to extend it's scope to Operations. So Performance Scalability should be the first piece of big Operation program (that includes such projects like: rally, osprofiler, logaas, satori, rubick and probably some others) So let's move to the Operation program with baby steps. Best regards, Boris Pavlovic On Tue, Jul 22, 2014 at 8:18 PM, Sean Dague s...@dague.net wrote: On 07/22/2014 11:58 AM, David Kranz wrote: On 07/22/2014 10:44 AM, Sean Dague wrote: Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean +1 It will also lead to pointless discussions/arguments about which activities are part of QA and which are part of Performance and Scalability Testing. QA Program mission: Develop, maintain, and initiate tools and plans to ensure the upstream stability and quality of OpenStack, and its release readiness at any point during the release cycle. It is hard to see how $subj falls outside of that mission. Of course rally would continue to have its own repo, review team, etc. as do tempest and grenade. Right, 100% agreed. Rally would remain with it's own repo + review team, just like grenade. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
