Hi, Here is a suggestion when writing integration tests for our products. - Build a separate p2 profile (test profile) for integration tests which consists of features that you'd include in a normal distribution + integration test features - When running the test cases, query the test bundle (via a REST API perhaps) when doing assertions
Unavailability of runtime context information within test cases has been a headache (at least for me) when writing integration tests. Which sometimes leads to doing assertions via console log output and doing thread sleeps. This could cause intermittent test failures that is hard to troubleshoot. Just out of curiosity...what exactly is the value addition that we are looking for by using PAX Exam? As I understood it can be used to create the product OSGi runtime from within the test case itself and that will give the developer/tester exclusive access to any runtime information. I guess we could do the same thing by injecting a test bundle and expose whatever we need through that. Would really like to know how we can write better test cases using PAX Exam. Thanks. On Thu, Dec 17, 2015 at 11:10 AM, Sameera Jayasoma <[email protected]> wrote: > Hi Azeez, > > Pax Exam is not a replacement for our integration tests. It is a way to > run unit tests which require an OSGi environment. > > Pax Exam is the most suitable solution to test Carbon Kernel and our > components (repos). We need to evaluate Pax Exam for testing our middleware > products. Carbon Kernel does not have transports, hence its bit hard to > execute integration tests. Carbon Kernel provides different OSGI utilities > for products. Therefore we need to run unit tests in an OSGI environment. > This is a must for Carbon kernel. > > On Thu, Dec 17, 2015 at 12:12 AM, Afkham Azeez <[email protected]> wrote: > >> 1. PAX exam tests being successful does not imply that the product pack >> will properly start >> This is because the way in which bundles are added to the PAX exam >> runtime is completely different from how the features are installed into >> product packs >> > > I don't agree with this point. :) Let me explain. > > We load all the bundles from the bundles.info file into a clean OSGI > environments . In PAX Exam we load all the bundles programmatically into a > clean environment plus Pax Exam bundles. Only difference is the Pax Exam > bundles. > > >> >> 2. Installing 100s of bundles programatically to test a product is not >> practical and a very cumbersome error prone approach >> I don't believe PAX exam was designed to do this. IMO, PAX exam should be >> used to test a bundle or may be just a few bundles, and also instead of a >> fully fledged runtime like a product, it is designed to have mock OSGi >> services & bundles along with the bundles under test. >> > > One option is to load all bundles from the bundles.info file. This will > make sure Pax Exam tests are up to date with product changes.. > > > >> 3. Correct approach for product integration testing is to use the product >> pack >> This is because of 1 & 2 above. In Carbon 4, we use the pack, extract it >> and run tests against it. I still believe that is the correct approach >> which needs to be followed in Carbon 5 based products as well. >> > > Yes this is true for products. We have to go with integration test > approach for our products. Pax Exam can be used to test functionalities > which requires OSGi in Carbon kernel and components. > > >> >> 4. PAX exam along with mock bundles & mock OSGi services for testing OSGi >> bundles >> I think PAX exam will be useful for unit testing individual bundles >> > > This is true. > > > Thanks, > Sameera. > > > >> >> Thanks >> Azeez >> >> >> On Wed, Dec 16, 2015 at 9:49 PM, Aruna Karunarathna <[email protected]> >> wrote: >> >>> >>> >>> On Wed, Dec 16, 2015 at 8:22 PM, Afkham Azeez <[email protected]> wrote: >>> >>>> I have been trying to develop PAX exam based tests for MSS and it seems >>>> very hard to use because there is little help for troubleshooting. MSS has >>>> just one feature and it has been a struggle to get the test setup. With >>>> much more complex products, it would be a total nightmare. A lot of time is >>>> wasted due to trial and error method of troubleshooting. >>>> >>>> Hi Azeez, >>> >>> Yes, agree on you that, it's not very easy to configure the test setup >>> at the beginning, when we were integrating it to kernel it took some time >>> to understand the anatomy of PAX-Exam test cases and the first test case to >>> up and running. >>> First what we did was, make the base test case solid, and make sure the >>> server is up and running, For example, when the test container starts it >>> does not start as we run the carbon.sh file, since the PAX-Test container >>> doesn't use the org.wso2.carbon.launcher.jar to boot up the server, so we >>> had to simulate that before starting the PAX-Exam container [1]. But after >>> that >>> people wrote test cases without much hassle. >>> >>> After those efforts, I think we make use of Pax to solve most of our in >>> container testing scenarios, We were able to make sure each OSGi services >>> are running solidly. So after code changes, it made very easy to identify >>> the broken OSGi services etc, without manual testing. also it helped to >>> simulate complex cases such as start up coordination. >>> >>> >>>> Either we are using PAX exam incorrectly or PAX exam is not best >>>> framework to be adopted. >>>> >>>> I'm not sure that PAX exam complicates things when it comes to >>> products, but for kernel I think we need in container OSGi test cases. So >>> we can verify the services are working accordingly. >>> >>> From what I have experienced so far, in most of our current products, >>> missing OSGi service or some service component issues can only be >>> identified after starting the product. We need to identify these issues at >>> the build time. Either using Pax or some other alternative. >>> >>> [1]. >>> https://github.com/wso2/carbon-kernel/blob/master/tests/osgi-tests/src/test/java/org/wso2/carbon/osgi/utils/OSGiTestUtils.java#L43 >>> >>> Regards, >>> Aruna >>> >>>> Thoughts welcome. >>>> >>>> -- >>>> *Afkham Azeez* >>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>> Member; Apache Software Foundation; http://www.apache.org/ >>>> * <http://www.apache.org/>* >>>> *email: **[email protected]* <[email protected]> >>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * >>>> *http://blog.afkham.org* <http://blog.afkham.org> >>>> *twitter: **http://twitter.com/afkham_azeez* >>>> <http://twitter.com/afkham_azeez> >>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez >>>> <http://lk.linkedin.com/in/afkhamazeez>* >>>> >>>> *Lean . Enterprise . Middleware* >>>> >>> >>> >>> >>> -- >>> >>> *Aruna Sujith Karunarathna *| Software Engineer >>> WSO2, Inc | lean. enterprise. middleware. >>> #20, Palm Grove, Colombo 03, Sri Lanka >>> Mobile: +94 71 9040362 | Work: +94 112145345 >>> Email: [email protected] | Web: www.wso2.com >>> >>> >> >> >> >> -- >> *Afkham Azeez* >> Director of Architecture; WSO2, Inc.; http://wso2.com >> Member; Apache Software Foundation; http://www.apache.org/ >> * <http://www.apache.org/>* >> *email: **[email protected]* <[email protected]> >> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * >> *http://blog.afkham.org* <http://blog.afkham.org> >> *twitter: **http://twitter.com/afkham_azeez* >> <http://twitter.com/afkham_azeez> >> *linked-in: **http://lk.linkedin.com/in/afkhamazeez >> <http://lk.linkedin.com/in/afkhamazeez>* >> >> *Lean . Enterprise . Middleware* >> > > > > -- > Sameera Jayasoma, > Software Architect, > > WSO2, Inc. (http://wso2.com) > email: [email protected] > blog: http://blog.sameera.org > twitter: https://twitter.com/sameerajayasoma > flickr: http://www.flickr.com/photos/sameera-jayasoma/collections > Mobile: 0094776364456 > > Lean . Enterprise . Middleware > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Akila Ravihansa Perera WSO2 Inc.; http://wso2.com/ Blog: http://ravihansa3000.blogspot.com
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
