Re: [openstack-dev] [tc][rally] Application for a new OpenStack Program: Performance and Scalability
It's interesting and practical! There're some early efforts to make openstack more robust and efficient, however, we're still lacking one such framework. Just a little question. On the wiki, it says to consider the scalability problem. How can we measure the performance at large scale? By real deployment or just simulating the input? Through the descriptions, seems the method is to try testing every single component. Also, suggest adding necessary perf checks into the test/integration jobs, too. Thanks! On Tue, Jul 22, 2014 at 5:53 AM, Boris Pavlovic bo...@pavlovic.me wrote: Hi Stackers and TC, The Rally contributor team would like to propose a new OpenStack program with a mission to provide scalability and performance benchmarking, and code profiling tools for OpenStack components. We feel we've achieved a critical mass in the Rally project, with an active, diverse contributor team. The Rally project will be the initial project in a new proposed Performance and Scalability program. Below, the details on our proposed new program. Thanks for your consideration, Boris [1] https://review.openstack.org/#/c/108502/ Official Name = Performance and Scalability Codename Rally Scope = Scalability benchmarking, performance analysis, and profiling of OpenStack components and workloads Mission === To increase the scalability and performance of OpenStack clouds by: * defining standard benchmarks * sharing performance data between operators and developers * providing transparency of code paths through profiling tools Maturity * Meeting logs http://eavesdrop.openstack.org/meetings/rally/2014/ * IRC channel: #openstack-rally * Rally performance jobs are in (Cinder, Glance, Keystone Neutron) check pipelines. * 950 commits over last 10 months * Large, diverse contributor community * http://stackalytics.com/?release=junometric=commitsproject_type=Allmodule=rally * http://stackalytics.com/report/contribution/rally/180 * Non official lead of project is Boris Pavlovic * Official election In progress. Deliverables Critical deliverables in the Juno cycle are: * extending Rally Benchmark framework to cover all use cases that are required by all OpenStack projects * integrating OSprofiler in all core projects * increasing functional unit testing coverage of Rally. Discussion == One of the major goals of Rally is to make it simple to share results of standardized benchmarks and experiments between operators and developers. When an operator needs to verify certain performance indicators meet some service level agreement, he will be able to run benchmarks (from Rally) and share with the developer community the results along with his OpenStack configuration. These benchmark results will assist developers in diagnosing particular performance and scalability problems experienced with the operator's configuration. Another interesting area is Rally the OpenStack CI process. Currently, working on performance issues upstream tends to be a more social than technical process. We can use Rally in the upstream gates to identify performance regressions and measure improvement in scalability over time. The use of Rally in the upstream gates will allow a more rigorous, scientific approach to performance analysis. In the case of an integrated OSprofiler, it will be possible to get detailed information about API call flows (e.g. duration of API calls in different services). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best wishes! Baohua ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally] Application for a new OpenStack Program: Performance and Scalability
As I mentioned on the review, the title could cause confusion. Rally tests performance and scalability but the title could lead people to think that if you install Rally, you will get Performance and Scale from your OpenStack instance. Adding Benchmark or Testing in to the description would clarify this. Tim From: Baohua Yang [mailto:yangbao...@gmail.com] Sent: 22 July 2014 10:59 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [tc][rally] Application for a new OpenStack Program: Performance and Scalability It's interesting and practical! There're some early efforts to make openstack more robust and efficient, however, we're still lacking one such framework. Just a little question. On the wiki, it says to consider the scalability problem. How can we measure the performance at large scale? By real deployment or just simulating the input? Through the descriptions, seems the method is to try testing every single component. Also, suggest adding necessary perf checks into the test/integration jobs, too. Thanks! On Tue, Jul 22, 2014 at 5:53 AM, Boris Pavlovic bo...@pavlovic.memailto:bo...@pavlovic.me wrote: Hi Stackers and TC, The Rally contributor team would like to propose a new OpenStack program with a mission to provide scalability and performance benchmarking, and code profiling tools for OpenStack components. We feel we've achieved a critical mass in the Rally project, with an active, diverse contributor team. The Rally project will be the initial project in a new proposed Performance and Scalability program. Below, the details on our proposed new program. Thanks for your consideration, Boris [1] https://review.openstack.org/#/c/108502/ Official Name = Performance and Scalability Codename Rally Scope = Scalability benchmarking, performance analysis, and profiling of OpenStack components and workloads Mission === To increase the scalability and performance of OpenStack clouds by: * defining standard benchmarks * sharing performance data between operators and developers * providing transparency of code paths through profiling tools Maturity * Meeting logs http://eavesdrop.openstack.org/meetings/rally/2014/ * IRC channel: #openstack-rally * Rally performance jobs are in (Cinder, Glance, Keystone Neutron) check pipelines. * 950 commits over last 10 months * Large, diverse contributor community * http://stackalytics.com/?release=junometric=commitsproject_type=Allmodule=rally * http://stackalytics.com/report/contribution/rally/180 * Non official lead of project is Boris Pavlovic * Official election In progress. Deliverables Critical deliverables in the Juno cycle are: * extending Rally Benchmark framework to cover all use cases that are required by all OpenStack projects * integrating OSprofiler in all core projects * increasing functional unit testing coverage of Rally. Discussion == One of the major goals of Rally is to make it simple to share results of standardized benchmarks and experiments between operators and developers. When an operator needs to verify certain performance indicators meet some service level agreement, he will be able to run benchmarks (from Rally) and share with the developer community the results along with his OpenStack configuration. These benchmark results will assist developers in diagnosing particular performance and scalability problems experienced with the operator's configuration. Another interesting area is Rally the OpenStack CI process. Currently, working on performance issues upstream tends to be a more social than technical process. We can use Rally in the upstream gates to identify performance regressions and measure improvement in scalability over time. The use of Rally in the upstream gates will allow a more rigorous, scientific approach to performance analysis. In the case of an integrated OSprofiler, it will be possible to get detailed information about API call flows (e.g. duration of API calls in different services). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best wishes! Baohua ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally] Application for a new OpenStack Program: Performance and Scalability
Honestly, I'm really not sure I see this as a different program, but is really something that should be folded into the QA program. I feel like a top level effort like this is going to lead to a lot of duplication in the data analysis that's currently going on, as well as functionality for better load driver UX. -Sean On 07/21/2014 05:53 PM, Boris Pavlovic wrote: Hi Stackers and TC, The Rally contributor team would like to propose a new OpenStack program with a mission to provide scalability and performance benchmarking, and code profiling tools for OpenStack components. We feel we've achieved a critical mass in the Rally project, with an active, diverse contributor team. The Rally project will be the initial project in a new proposed Performance and Scalability program. Below, the details on our proposed new program. Thanks for your consideration, Boris [1] https://review.openstack.org/#/c/108502/ Official Name = Performance and Scalability Codename Rally Scope = Scalability benchmarking, performance analysis, and profiling of OpenStack components and workloads Mission === To increase the scalability and performance of OpenStack clouds by: * defining standard benchmarks * sharing performance data between operators and developers * providing transparency of code paths through profiling tools Maturity * Meeting logs http://eavesdrop.openstack.org/meetings/rally/2014/ * IRC channel: #openstack-rally * Rally performance jobs are in (Cinder, Glance, Keystone Neutron) check pipelines. * 950 commits over last 10 months * Large, diverse contributor community * http://stackalytics.com/?release=junometric=commitsproject_type=Allmodule=rally * http://stackalytics.com/report/contribution/rally/180 * Non official lead of project is Boris Pavlovic * Official election In progress. Deliverables Critical deliverables in the Juno cycle are: * extending Rally Benchmark framework to cover all use cases that are required by all OpenStack projects * integrating OSprofiler in all core projects * increasing functional unit testing coverage of Rally. Discussion == One of the major goals of Rally is to make it simple to share results of standardized benchmarks and experiments between operators and developers. When an operator needs to verify certain performance indicators meet some service level agreement, he will be able to run benchmarks (from Rally) and share with the developer community the results along with his OpenStack configuration. These benchmark results will assist developers in diagnosing particular performance and scalability problems experienced with the operator's configuration. Another interesting area is Rally the OpenStack CI process. Currently, working on performance issues upstream tends to be a more social than technical process. We can use Rally in the upstream gates to identify performance regressions and measure improvement in scalability over time. The use of Rally in the upstream gates will allow a more rigorous, scientific approach to performance analysis. In the case of an integrated OSprofiler, it will be possible to get detailed information about API call flows (e.g. duration of API calls in different services). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally] Application for a new OpenStack Program: Performance and Scalability
Thanks, boris, really big +1! we at huawei are using Rally as our performance testing tool for OpenStack API, but I still hope Rally will be more mature with more and more talented folks within the OpenStack community. I am really excited to be able to participate. 2014-07-22 5:53 GMT+08:00 Boris Pavlovic bo...@pavlovic.me: Hi Stackers and TC, The Rally contributor team would like to propose a new OpenStack program with a mission to provide scalability and performance benchmarking, and code profiling tools for OpenStack components. We feel we've achieved a critical mass in the Rally project, with an active, diverse contributor team. The Rally project will be the initial project in a new proposed Performance and Scalability program. Below, the details on our proposed new program. Thanks for your consideration, Boris [1] https://review.openstack.org/#/c/108502/ Official Name = Performance and Scalability Codename Rally Scope = Scalability benchmarking, performance analysis, and profiling of OpenStack components and workloads Mission === To increase the scalability and performance of OpenStack clouds by: * defining standard benchmarks * sharing performance data between operators and developers * providing transparency of code paths through profiling tools Maturity * Meeting logs http://eavesdrop.openstack.org/meetings/rally/2014/ * IRC channel: #openstack-rally * Rally performance jobs are in (Cinder, Glance, Keystone Neutron) check pipelines. * 950 commits over last 10 months * Large, diverse contributor community * http://stackalytics.com/?release=junometric=commitsproject_type=Allmodule=rally * http://stackalytics.com/report/contribution/rally/180 * Non official lead of project is Boris Pavlovic * Official election In progress. Deliverables Critical deliverables in the Juno cycle are: * extending Rally Benchmark framework to cover all use cases that are required by all OpenStack projects * integrating OSprofiler in all core projects * increasing functional unit testing coverage of Rally. Discussion == One of the major goals of Rally is to make it simple to share results of standardized benchmarks and experiments between operators and developers. When an operator needs to verify certain performance indicators meet some service level agreement, he will be able to run benchmarks (from Rally) and share with the developer community the results along with his OpenStack configuration. These benchmark results will assist developers in diagnosing particular performance and scalability problems experienced with the operator's configuration. Another interesting area is Rally the OpenStack CI process. Currently, working on performance issues upstream tends to be a more social than technical process. We can use Rally in the upstream gates to identify performance regressions and measure improvement in scalability over time. The use of Rally in the upstream gates will allow a more rigorous, scientific approach to performance analysis. In the case of an integrated OSprofiler, it will be possible to get detailed information about API call flows (e.g. duration of API calls in different services). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Regards! --- Lingxian Kong ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally] Application for a new OpenStack Program: Performance and Scalability
+1 At yahoo we are using and contributing to Rally. Within our company performance engineering and QE are distinct job groups with different skill sets and focus. As a result, I think that it makes perfect sense to have a dedicated program for performance and scalability within OpenStack. I also think it makes sense to start growing that capability from active community of developers already working on Rally and OSprofiler. -- Regards, Marco From: Boris Pavlovic bo...@pavlovic.memailto:bo...@pavlovic.me Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, July 21, 2014 at 2:53 PM To: OpenStack Development Mailing List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org, openstack...@lists.openstack.orgmailto:openstack...@lists.openstack.org openstack...@lists.openstack.orgmailto:openstack...@lists.openstack.org Subject: [openstack-dev] [tc][rally] Application for a new OpenStack Program: Performance and Scalability Hi Stackers and TC, The Rally contributor team would like to propose a new OpenStack program with a mission to provide scalability and performance benchmarking, and code profiling tools for OpenStack components. We feel we've achieved a critical mass in the Rally project, with an active, diverse contributor team. The Rally project will be the initial project in a new proposed Performance and Scalability program. Below, the details on our proposed new program. Thanks for your consideration, Boris [1] https://review.openstack.org/#/c/108502/ Official Name = Performance and Scalability Codename Rally Scope = Scalability benchmarking, performance analysis, and profiling of OpenStack components and workloads Mission === To increase the scalability and performance of OpenStack clouds by: * defining standard benchmarks * sharing performance data between operators and developers * providing transparency of code paths through profiling tools Maturity * Meeting logs http://eavesdrop.openstack.org/meetings/rally/2014/ * IRC channel: #openstack-rally * Rally performance jobs are in (Cinder, Glance, Keystone Neutron) check pipelines. * 950 commits over last 10 months * Large, diverse contributor community * http://stackalytics.com/?release=junometric=commitsproject_type=Allmodule=rally * http://stackalytics.com/report/contribution/rally/180 * Non official lead of project is Boris Pavlovic * Official election In progress. Deliverables Critical deliverables in the Juno cycle are: * extending Rally Benchmark framework to cover all use cases that are required by all OpenStack projects * integrating OSprofiler in all core projects * increasing functional unit testing coverage of Rally. Discussion == One of the major goals of Rally is to make it simple to share results of standardized benchmarks and experiments between operators and developers. When an operator needs to verify certain performance indicators meet some service level agreement, he will be able to run benchmarks (from Rally) and share with the developer community the results along with his OpenStack configuration. These benchmark results will assist developers in diagnosing particular performance and scalability problems experienced with the operator's configuration. Another interesting area is Rally the OpenStack CI process. Currently, working on performance issues upstream tends to be a more social than technical process. We can use Rally in the upstream gates to identify performance regressions and measure improvement in scalability over time. The use of Rally in the upstream gates will allow a more rigorous, scientific approach to performance analysis. In the case of an integrated OSprofiler, it will be possible to get detailed information about API call flows (e.g. duration of API calls in different services). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tc][rally] Application for a new OpenStack Program: Performance and Scalability
Hi Stackers and TC, The Rally contributor team would like to propose a new OpenStack program with a mission to provide scalability and performance benchmarking, and code profiling tools for OpenStack components. We feel we've achieved a critical mass in the Rally project, with an active, diverse contributor team. The Rally project will be the initial project in a new proposed Performance and Scalability program. Below, the details on our proposed new program. Thanks for your consideration, Boris [1] https://review.openstack.org/#/c/108502/ Official Name = Performance and Scalability Codename Rally Scope = Scalability benchmarking, performance analysis, and profiling of OpenStack components and workloads Mission === To increase the scalability and performance of OpenStack clouds by: * defining standard benchmarks * sharing performance data between operators and developers * providing transparency of code paths through profiling tools Maturity * Meeting logs http://eavesdrop.openstack.org/meetings/rally/2014/ * IRC channel: #openstack-rally * Rally performance jobs are in (Cinder, Glance, Keystone Neutron) check pipelines. * 950 commits over last 10 months * Large, diverse contributor community * http://stackalytics.com/?release=junometric=commitsproject_type=Allmodule=rally * http://stackalytics.com/report/contribution/rally/180 * Non official lead of project is Boris Pavlovic * Official election In progress. Deliverables Critical deliverables in the Juno cycle are: * extending Rally Benchmark framework to cover all use cases that are required by all OpenStack projects * integrating OSprofiler in all core projects * increasing functional unit testing coverage of Rally. Discussion == One of the major goals of Rally is to make it simple to share results of standardized benchmarks and experiments between operators and developers. When an operator needs to verify certain performance indicators meet some service level agreement, he will be able to run benchmarks (from Rally) and share with the developer community the results along with his OpenStack configuration. These benchmark results will assist developers in diagnosing particular performance and scalability problems experienced with the operator's configuration. Another interesting area is Rally the OpenStack CI process. Currently, working on performance issues upstream tends to be a more social than technical process. We can use Rally in the upstream gates to identify performance regressions and measure improvement in scalability over time. The use of Rally in the upstream gates will allow a more rigorous, scientific approach to performance analysis. In the case of an integrated OSprofiler, it will be possible to get detailed information about API call flows (e.g. duration of API calls in different services). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][rally] Application for a new OpenStack Program: Performance and Scalability
Cool, Rally is really helpful for performance benchmarking and optimizing for our openstack cloud. On Jul 22, 2014, at 5:53, Boris Pavlovic bo...@pavlovic.me wrote: Hi Stackers and TC, The Rally contributor team would like to propose a new OpenStack program with a mission to provide scalability and performance benchmarking, and code profiling tools for OpenStack components. We feel we've achieved a critical mass in the Rally project, with an active, diverse contributor team. The Rally project will be the initial project in a new proposed Performance and Scalability program. Below, the details on our proposed new program. Thanks for your consideration, Boris [1] https://review.openstack.org/#/c/108502/ Official Name = Performance and Scalability Codename Rally Scope = Scalability benchmarking, performance analysis, and profiling of OpenStack components and workloads Mission === To increase the scalability and performance of OpenStack clouds by: * defining standard benchmarks * sharing performance data between operators and developers * providing transparency of code paths through profiling tools Maturity * Meeting logs http://eavesdrop.openstack.org/meetings/rally/2014/ * IRC channel: #openstack-rally * Rally performance jobs are in (Cinder, Glance, Keystone Neutron) check pipelines. * 950 commits over last 10 months * Large, diverse contributor community * http://stackalytics.com/?release=junometric=commitsproject_type=Allmodule=rally * http://stackalytics.com/report/contribution/rally/180 * Non official lead of project is Boris Pavlovic * Official election In progress. Deliverables Critical deliverables in the Juno cycle are: * extending Rally Benchmark framework to cover all use cases that are required by all OpenStack projects * integrating OSprofiler in all core projects * increasing functional unit testing coverage of Rally. Discussion == One of the major goals of Rally is to make it simple to share results of standardized benchmarks and experiments between operators and developers. When an operator needs to verify certain performance indicators meet some service level agreement, he will be able to run benchmarks (from Rally) and share with the developer community the results along with his OpenStack configuration. These benchmark results will assist developers in diagnosing particular performance and scalability problems experienced with the operator's configuration. Another interesting area is Rally the OpenStack CI process. Currently, working on performance issues upstream tends to be a more social than technical process. We can use Rally in the upstream gates to identify performance regressions and measure improvement in scalability over time. The use of Rally in the upstream gates will allow a more rigorous, scientific approach to performance analysis. In the case of an integrated OSprofiler, it will be possible to get detailed information about API call flows (e.g. duration of API calls in different services). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
