On Thu, Mar 20, 2014 at 1:19 PM, Rochelle.RochelleGrober < rochelle.gro...@huawei.com> wrote:
> > > > -----Original Message----- > > From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] > > Sent: Thursday, March 20, 2014 12:13 PM > > > > 'project specific functional testing' in the Marconi context is > > treating > > Marconi as a complete system, making Marconi API calls & verifying the > > response - just like an end user would, but without keystone. If one of > > these tests fail, it is because there is a bug in the Marconi code , > > and > > not because its interaction with Keystone caused it to fail. > > > > "That being said there are certain cases where having a project > > specific > > functional test makes sense. For example swift has a functional test > > job > > that > > starts swift in devstack. But, those things are normally handled on a > > per > > case > > basis. In general if the project is meant to be part of the larger > > OpenStack > > ecosystem then Tempest is the place to put functional testing. That way > > you know > > it works with all of the other components. The thing is in openstack > > what > > seems > > like a project isolated functional test almost always involves another > > project > > in real use cases. (for example keystone auth with api requests) > > > > " > > > > One of the concerns we heard in the review was 'having the functional > > tests elsewhere (I.e within the project itself) does not count and they > > have to be in Tempest'. > > This has made us as a team wonder if we should migrate all our > > functional > > tests to Tempest. > > But from Matt's response, I think it is reasonable to continue in our > > current path & have the functional tests in Marconi coexist along with > > the tests in Tempest. > > > > While there always exceptions to this rule, in general I think functional tests belong in tempest for all of the reasons discussed above. > I think that what is being asked, really is that the functional tests > could be a single set of tests that would become a part of the tempest > repository and that these tests would have an ENV variable as part of the > configuration that would allow either "no Keystone" or "Keystone" or some > such, if that is the only configuration issue that separates running the > tests isolated vs. integrated. The functional tests need to be as much as > possible a single set of tests to reduce duplication and remove the > likelihood of two sets getting out of sync with each other/development. If > they only run in the integrated environment, that's ok, but if you want to > run them isolated to make debugging easier, then it should be a > configuration option and a separate test job. > > So, if my assumptions are correct, QA only requires functional tests for > integrated runs, but if the project QAs/Devs want to run isolated for dev > and devtest purposes, more power to them. Just keep it a single set of > functional tests and put them in the Tempest repository so that if a > failure happens, anyone can find the test and do the debug work without > digging into a separate project repository. > > Hopefully, the tests as designed could easily take a new configuration > directive and a short bit of work with OS QA will get the integrated FTs > working as well as the isolated ones. > > --Rocky > > > On 3/20/14 1:59 PM, "Matthew Treinish" <mtrein...@kortar.org> wrote: > > > > >On Thu, Mar 20, 2014 at 11:35:15AM +0000, Malini Kamalambal wrote: > > >> Hello all, > > >> > > >> I have been working on adding tests in Tempest for Marconi, for the > > >>last few months. > > >> While there are many amazing people to work with, the process has > > been > > >>more difficult than I expected. > > >> > > >> Couple of pain-points and suggestions to make the process easier for > > >>myself & future contributors. > > >> > > >> 1. The QA requirements for a project to graduate needs details > > beyond > > >>the "Project must have a *basic* devstack-gate job set up" > > >> 2. The scope of Tempest needs clarification - what tests should be > > in > > >>Tempest vs. in the individual projects? Or should they be in both > > >>tempest and the project? > > >> > > >> See details below. > > >> > > >> 1. There is little documentation on graduation requirement from a QA > > >>perspective beyond 'Project must have a basic devstack-gate job set > > up'. > > >> > > >> As a result, I hear different interpretations on what a basic > > devstack > > >>gate job is. > > >> This topic was discussed in one of the QA meetings a few weeks back > > [1]. > > >> Based on the discussion there, having a basic job - such as one that > > >>will let us know 'if a keystone change broke marconi' was good > > enough. > > >> My efforts in getting Marconi meet graduation requirements w.r.t > > >>Tempest was based on the above discussion. > > >> > > >> However, my conversations with the TC during Marconi's graduation > > >>review lead me to believe that these requirements aren't yet > > formalized. > > >> We were told that we needed to have more test coverage in tempest, & > > >>having them elsewhere (i.e. functional tests in the Marconi project > > >>itself) was not good enough. > > > > > >So having only looked at the Marconi ML thread and not the actual TC > > >meeting > > >minutes I might be missing the whole picture. But, from what I saw > > when I > > >looked > > >at both a marconi commit and a tempest commit is that there is no > > gating > > >marconi > > >devstack-gate job on marconi commits. It's only non-voting in the > > check > > >pipeline. > > >Additionally, there isn't a non-voting job on tempest or devstack- > > gate. > > >For > > >example, look at how savanna has it's tempest jobs setup and this is > > what > > >marconi > > >needs to have. > > > > > >> > > >> I will never debate the value of having good test coverage - after > > all > > >>I define myself professionally as a QA ;) > > >> I am proud of the unit and functional test suites & the test > > coverage > > >>we have in Marconi [2]. > > >> Marconi team is continuing its efforts in this direction. > > >> We are looking forward to adding more tests in Tempest and making > > sure > > >>Marconi is in par with the community standards. > > >> > > >> But what frustrates me is that the test requirements seem to evolve, > > >>catching new contributors by surprise. > > >> > > >> It will really help to have these requirements documented in detail > > - > > >>answering at least the following questions, > > >> a. What tests are needed to graduate - API, Scenario, CLI? > > >> b. How much coverage is good enough to graduate? > > >> > > >> That will make sure that contributors focus their time & energy in > > the > > >>right direction. > > >> I am willing to lead the effort to document the QA-level graduation > > >>requirements for a project and help solidify them. > > > > > >Testing contributions will always be an iterative process. The actual > > test > > >coverage doesn't matter as much up front. The graduation requirement > > as I > > >understood it was just to have the glue in place and to verify that > > >everything > > >runs. As long as there is steady contribution and interaction from the > > >marconi > > >community with tempest IMO that matters far more then actually having > > >complete > > >coverage upfront. > > > > > >> > > >> 2. Clarify the scope of Tempest - what tests should be in Tempest > > vs > > >>in the individual projects ? > > >> > > >> It sounds like the scope of tempest is to make sure that, > > >> a. Projects are functionally tested (AND) > > >> b. Openstack components (a.k.a projects) do not have integration > > issues. > > >> > > >> Assuming my understanding is correct, does it make sense to have the > > >>project specific functional tests in Tempest? > > >> Troubleshooting failures related to project specific functionality > > >>requires deep understanding of the individual projects. > > >> Isn't it better to leave it to the individual projects to make sure > > >>that they are functional? > > >> That will help the contributors to Tempest spend their time on what > > >>only Tempest can do -i.e. identify integration issues. > > > > > >What do you mean by project specific functional testing? What makes > > >debugging > > >a marconi failure in a tempest gate job any more involved than > > debugging a > > >nova or neutron failure? Part of the point of having an integrated > > gate is > > >saying that the project works well with all the others in OpenStack. > > IMO > > >that's > > >not just in project functionality but also in community. When there is > > an > > >issue > > >with a gate job everyone comes together to work on it. For example if > > you > > >have > > >a keystone patch that breaks a marconi test in check there is open > > >communication > > >about what happened and how to fix it. > > > > > >That being said there are certain cases where having a project > > specific > > >functional test makes sense. For example swift has a functional test > > job > > >that > > >starts swift in devstack. But, those things are normally handled on a > > per > > >case > > >basis. In general if the project is meant to be part of the larger > > >OpenStack > > >ecosystem then Tempest is the place to put functional testing. That > > way > > >you know > > >it works with all of the other components. The thing is in openstack > > what > > >seems > > >like a project isolated functional test almost always involves another > > >project > > >in real use cases. (for example keystone auth with api requests) > > > > > >Now for the boundary between what kind of test belongs where isn't > > always > > >clear > > >and every project has a different rule of thumb on that. There really > > >isn't a > > >hard and fast rule for tempest on what's allowed as long as it fits > > >within the > > >criteria for the different test categories. It's handled on a per case > > >basis. > > >Honestly, though I normally recommend putting as much in tempest as > > you > > >can > > >because a big advantage of having an external test suite is that it > > keeps > > >you > > >honest because the tests are in a separate repo. > > > > > >-Matt Treinish > > > > > >_______________________________________________ > > >OpenStack-dev mailing list > > >OpenStack-dev@lists.openstack.org > > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev