----- Original Message ----- > From: "Chris Evich" <[email protected]> > To: "Lucas Meneghel Rodrigues" <[email protected]> > Cc: "Vimal Patel" <[email protected]>, [email protected], "Virt > Test Development Mailing List" > <[email protected]> > Sent: Tuesday, November 6, 2012 3:36:50 PM > Subject: Re: [Autotest] RFC : Adding a TestCaseID into the Results of Testing > > On 11/06/2012 05:57 AM, Lucas Meneghel Rodrigues wrote: > > On 11/05/2012 02:36 PM, Vimal Patel wrote: > >> Requesting a Change to report a TestCase ID from ScanResults.py or > >> the > >> Results in General > > > > No problem, but I don't fully understand what the workflow would > > be. I'm > > guessing: > > > > * There's a Test Case Management System (TCMS), say, Testopia, with > > an > > XMLRPC API interface that you can use to notify things like 'test > > 1234 > > PASS'. > > * After the autotest job you need to parse results and report to > > the TCMS. > > > > Provided that this is my correct understanding, we have options: > > > > 1) Formal API to report to such systems. May take a while to model > > and > > get 'right'. > > 2) Do it Ad-Hoc, in which case it's fairly straight forward and > > doesn't > > require any change on our side. > > > >> We (spice-qa team) require a way to map our TestCaseIDs in TCMS > >> with > >> the Tests in KVM-Autotest. > >> > >> After a conversation with Chris Evich, a good way to get this > >> accomplished is through using a test_case_id parameter and > >> reporting > >> this in the results including scanresults.py. > > > > Maybe generate a dict with test case names and test case ids. > > Something > > like: > > > > t_case_mapping = {'kvm.spice.foo1': 1234 > > 'kvm.spice.bar2': 1235} > > > > And during results parsing, if kvm.spice.foo1 PASS, call the XMLRPC > > API > > in the test case management. I'm guessing your script does that. > > > > As for creating a formal autotest API, I'd say, why not? But it > > might > > take a while to work on that, and given the current TODO list, this > > would have reduced priority. > > > >> Currently, the spiceqa team created a script using the results > >> from > >> scanresults.py to parse through to get the ID & results to update > >> TCMS. > > > > Could you guys share the script? > > As we discussed on IRC, I'm a very little bit worried about having > all > those test_case_id parameters all over the upstream Cartesian config. > Eventhough I suggested this option :S Really it would only be > meaningful to us, and just "noise" for everyone else. > > However, I do think others may appreciate some kind of standardized > parameter / API for the purpose of tracking virt. autotest case > association with test case management system. So this is a good > problem > for us to be solving, I'll bet others have this need as well (or will > eventually). > > I like the translation-dictionary idea, maybe we could incorporate > that > into some kind of official script (similar to scanresults) that would > interface with your update script? (i.e. an option in between hard > and > easy) :D
I am planning on investigating the test case mapping option later this week, I think that's the way we should go with it. Thanks for the suggestion Chris/Lucas. Of course we will share our scripts. our scripts right now work, but are in the very early stages of development, it doesn't handle any negative cases yet or try to tell the user what is wrong if it's incorrect. We were planning on pushing the scripts onto the spiceqa fork of virt-test, I will let you know when this happens. I don't think there is a need to have it upstream as this will just be used for internal use. Let me know if you disagree. Thanks, Vimal > > -- > Chris Evich, RHCA, RHCE, RHCDS, RHCSS > Quality Assurance Engineer > e-mail: cevich + `@' + redhat.com o: 1-888-RED-HAT1 x44214 > _______________________________________________ Autotest-kernel mailing list [email protected] https://www.redhat.com/mailman/listinfo/autotest-kernel
