[Dev] Automation framework's inconsistent behavior

2013-08-07 Thread Vijitha Kumara
Hi All, Apparently there's some serious ordering/execution issue when the bunch of tests run with the framework. For example if you run a single class LCStateIdNameTestCase all test passes but fails when running only with the package (org.wso2.carbon.registry.lifecycle.test in this case).

Re: [Dev] Automation framework's inconsistent behavior

2013-08-07 Thread Nuwan Wimalasekara
This happens when the resources used for a test is not cleared properly after the test. Those clearance should be called under AfterClass or After Test annotated methods. Then Those methods will call even if the test is failed or passed. Then there are no effects of other test classes. Some test

Re: [Dev] Automation framework's inconsistent behavior

2013-08-07 Thread Vijitha Kumara
Hi Nuwan, I see most of the test cases have cleanup code to do the necessary stuff but we need to review all of them as well if there's any doubt. But this is not about resource cleanup rather the issue with the inferences and the related consequences I explained. The test in focus is not done

Re: [Dev] Automation framework's inconsistent behavior

2013-08-07 Thread Krishantha Samaraweera
First of all this is not framework issue. The issue has caused by the way we have done test design. The same behavior was observed when using priority and dependOn annotation together (could be caused by testNG issue) I have explained the reason for the using of duplicate create service/LC