Hi, There has been separate discussions about test case result verdicts and how QA gives them. When case is pass, when fail, when N/A etc. These verdict rules should be aligned trough QA and rules should be simple for everyone to understand (test execution and audience looking reports). So, wanted to open a discussion about this.
To unify and simplify the verdict rules, I propose following: Pass = When QA can verify that tested feature works as expected. Fail = When QA can verify that tested feature does not work as expected. (ex. feature is not implemented.) N/A = Feature is implemented but QA can't give pass or fail verdict. NOTES: QA-Reports gives us a good way to add more information to test reports and individual test cases. Updating feature info, updating bug info, comments etc. describing what is the cause of something. - PASS is simple. When something works like describing, it works. Exception is Dataflow testing. When speaking about Dataflow testing, it's enough that tested Data Flow works, ex. in handset icon color or localization does not need to be correct, cause "data flows" are working properly trough SW stacks. But this is on Dataflow testing context only. - FAIL, this should also be simple. When something does not work as described, it fails. There shall always be explanation about this. If there is a bug that causes test case to fail, bug ID shall be linked. If feature is not implemented, naturally it does not work, causes test case to fail and feature ID will be linked. If there are fail due unknown reason, root cause shall be analyzed and bug filed, temporarily "under investication etc." marked. So from QA-Reports, it should be really easy to see the root cause of the fail. - N/A points mainly to QA, whenever QA cant give verdict, test case is not available = N/A. There needs to be explanation why this has been happened and if root cause is test asset, bug to be filed. Reference: QA-Reports: http://qa-reports.meego.com/1.2 Core OS N900 report with these rules: http://qa-reports.meego.com/1.2/Core/Basic%20feature%20testing/N900/478 Bug about missing "blocked" status: http://bugs.meego.com/show_bug.cgi?id=10388 BR, .niko --- Niko Rantalainen MeeGo Core OS QA Owner +358 40 838 1891 http://wiki.meego.com/Quality _______________________________________________ MeeGo-qa mailing list [email protected] http://lists.meego.com/listinfo/meego-qa
