Boris, can You take a look on HA scenarios [0] to give a quick feedback on how it should be implemented with Rally in right way? Plugins/templates/how yaml should look like. To make correct comparison of ostf and rally-based version.
Thanks! [0] http://paste.openstack.org/show/523983/ On Tue, Jun 28, 2016 at 3:34 AM, Boris Pavlovic <bpavlo...@mirantis.com> wrote: > Igor, > > > I wonder what are the benefits of using Rally then? We can run whatever >> we want by means of MCollective or Ansible. Is there something that could >> help us? I don't know, maybe some dashboard integration with per test >> results or running by tests by tag? > > > Benefits are in the Rally framework, engine and integrated tooling, that > are doing very hard things to provide simple interfaces for writing simple > plugins that are emulating complicated test cases. > > The major benefits are next: > > *1) Generalization* > 1.1) One tool with one reporting system and one output format for all > kinds of testing strategies (functional, load, perf, scale, ...) > 1.2) One set of plugins (code) that can be used to generate all kinds of > testing strategies > 1.3.) One API for all kinds of testing strategies > > *2) Simplicity * > 2.1) Plugins are really simple to write, usually requires one method to be > implemented > 2.2) Auto discovery: adding plugins == add code in special (or specified) > directory > > *3) Reusability & Data Driven approach: * > 3.1) Split code (plugins) & tests cases (yaml files) > 3.2) Test cases is mixture of different plugins > 3.3) Plugins accept arguments > > *4) Integrated tooling* > 4.1) All results are persisted in Rally DB and you can access it in any > moment > 4.2) Results can be exported in different formats (you can write even own > plugins for simplifying integration) > 4.3) Detailed HMTL reports with task results overview and trends > > > > > Before switching Fuel from ostf to rally, I would like to see feature >> parity comparison. It's very necessary to understand how much work we need >> to spend to rewrite all our tests in rally way. > > > Totally agree, let's do it. > > > Best regards, > Boris Pavlovic > > > > On Mon, Jun 27, 2016 at 8:58 AM, Vladimir Kuklin <vkuk...@mirantis.com> > wrote: > >> +1 to initial suggestion, but I guess we need to have a full feature >> equality (e.g. HA tests for mysql and rabbitmq replication) before >> switching to Rally. >> >> On Mon, Jun 27, 2016 at 6:17 PM, Sergii Golovatiuk < >> sgolovat...@mirantis.com> wrote: >> >>> Hi, >>> >>> Before switching Fuel from ostf to rally, I would like to see feature >>> parity comparison. It's very necessary to understand how much work we need >>> to spend to rewrite all our tests in rally way. >>> >>> >>> >>> -- >>> Best regards, >>> Sergii Golovatiuk, >>> Skype #golserge >>> IRC #holser >>> >>> On Mon, Jun 27, 2016 at 4:32 PM, Alexander Kostrikov < >>> akostri...@mirantis.com> wrote: >>> >>>> Hello, everybody! >>>> Hello, Alex! >>>> >I thought Rally was more for benchmarking. Wouldn't Tempest make >>>> more sense? >>>> Rally is a good tool with nice api/usage/extensibility. >>>> I really liked "up and running tests in 5 minutes" in Rally with clear >>>> picture of what I am doing. >>>> So, I 100% for a Rally as a QA. >>>> >>>> Another note: >>>> We will need to implement some HA tests, probably not in Rally. >>>> >>>> On Mon, Jun 27, 2016 at 4:57 PM, Andrey Kurilin <akuri...@mirantis.com> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Mon, Jun 27, 2016 at 4:46 PM, Igor Kalnitsky < >>>>> ikalnit...@mirantis.com> wrote: >>>>> >>>>>> >>>>>> > On Jun 27, 2016, at 16:23, Alex Schultz <aschu...@mirantis.com> >>>>>> wrote: >>>>>> > >>>>>> > I thought Rally was more for benchmarking. Wouldn't Tempest make >>>>>> more sense? >>>>>> >>>>>> According to Rally wiki page [1], it seems they have a verification >>>>>> layer (Tempest so far). Hm, I wonder does it mean we will need to rewrite >>>>>> our scenarios for Tempest? >>>>>> >>>>>> >>>>> Rally consists of two main components: Rally Task and Rally >>>>> Verification. They are totally separated. >>>>> Task component is fully pluggable and you can run there whatever you >>>>> want in whatever you want way. >>>>> Verification component is hardcoded now. It was designed for >>>>> managing(install, configure) and launching(execute and store results) >>>>> Tempest. But we have a spec to make this component pluggable too. >>>>> >>>>> >>>>>> - igor >>>>>> >>>>>> >>>>>> [1] https://wiki.openstack.org/wiki/Rally >>>>>> >>>>>> __________________________________________________________________________ >>>>>> OpenStack Development Mailing List (not for usage questions) >>>>>> Unsubscribe: >>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> Andrey Kurilin. >>>>> >>>>> >>>>> __________________________________________________________________________ >>>>> OpenStack Development Mailing List (not for usage questions) >>>>> Unsubscribe: >>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Kind Regards, >>>> >>>> Alexandr Kostrikov, >>>> >>>> Mirantis, Inc. >>>> >>>> 35b/3, Vorontsovskaya St., 109147, Moscow, Russia >>>> >>>> >>>> Tel.: +7 (495) 640-49-04 >>>> Tel.: +7 (925) 716-64-52 <%2B7%20%28906%29%20740-64-79> >>>> >>>> Skype: akostrikov_mirantis >>>> >>>> E-mail: akostri...@mirantis.com <elogut...@mirantis.com> >>>> >>>> *www.mirantis.com <http://www.mirantis.ru/>* >>>> *www.mirantis.ru <http://www.mirantis.ru/>* >>>> >>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> >> -- >> Yours Faithfully, >> Vladimir Kuklin, >> Fuel Library Tech Lead, >> Mirantis, Inc. >> +7 (495) 640-49-04 >> +7 (926) 702-39-68 >> Skype kuklinvv >> 35bk3, Vorontsovskaya Str. >> Moscow, Russia, >> www.mirantis.com <http://www.mirantis.ru/> >> www.mirantis.ru >> vkuk...@mirantis.com >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Kind Regards, Alexandr Kostrikov, Mirantis, Inc. 35b/3, Vorontsovskaya St., 109147, Moscow, Russia Tel.: +7 (495) 640-49-04 Tel.: +7 (925) 716-64-52 <%2B7%20%28906%29%20740-64-79> Skype: akostrikov_mirantis E-mail: akostri...@mirantis.com <elogut...@mirantis.com> *www.mirantis.com <http://www.mirantis.ru/>* *www.mirantis.ru <http://www.mirantis.ru/>*
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev