Prasanna: 2) More and more requests from contributors on how to write unit-tests show up on our lists every week.
I agree we need more unit tests. At this moment there isn’t much detailed guideline or material about how to write a unit test with unified style. For example, before 4.1.0, mocks are injected with MockComponentLocator and these mock objects must be implemented manually. Later we bring mockito back and simplify the test codes. Now with spring being introduced, should we also use spring test framework? And for different layers, the way to implement unit tests might be different, to test a Resource class, we need a lot of mocks/stubs of external library; to test a DAO, should we prepare the database with test data? I think we need some consensus on ' what should be unit tested' and 'the recommend way to write a unit test'. 4. Easing test writing is my primary concern at the moment and I've been refactoring marvin to accomadate a easier dsl style test composition. I hope to have a working test case sometime next week. I haven’t followed this for a while, and I don’t use devCloud . Are there any documents describing how to run Marvin test with simulator/hypervisors? Regards Mice -----Original Message----- From: prasanna [mailto:srivatsav.prasa...@gmail.com] On Behalf Of Prasanna Santhanam Sent: Wednesday, January 30, 2013 2:02 AM To: cloudstack-dev@incubator.apache.org Subject: [DISCUSS] Concerns on testing and improvements On Tue, Jan 29, 2013 at 10:43:33AM -0500, Alex Huang wrote: > > > It is also true CloudStack does not have a good automated > > > regression test suite to make sure a check-in like this does not > > > break some other features in CloudStack. But lack of a thorough > > > automated regression suite a problem with CloudStack in general. We've > > > let in other big changes in this release. > > > I know developers who wrote this check-in have thoroughly > > > regression > > tested > > > to the best of their abilities. > > > > > > > > This is a point I tried to make to a few folks before CloudStack > > came here. If you want to provide a smooth path to committership you > > need a solid regression testing framework (with near total coverage) > > and an environment. Feeling blind and vulnerable to big changes like > > this limits trust and we can't have a lack of trust. If you want > > this project growing faster and healthier regression testing is key: it has > > a social impact. > > > > +1 Again Agree on this. CloudStack has been lacking a good > automated regression tests. We need to start doing something about > it. This warrants its own thread (which unfortunately has become a common feature on our lists) IMO - cloudstack is and remains largely untestable! I've often seen these requests for more tests spring (no pun intended) up when a large code merge is called for. Happened with api_refactoring as well. But the truth of the matter is "tests are an afterthought". If you don't believe me: 1) One only needs look at the list of features waiting on ipclearance without much to call in terms of tests. Feature is developed - testing can come later. 2) More and more requests from contributors on how to write unit-tests show up on our lists every week. 3) Tests cannot magically appear unless there is a concerted effort in testing right from feature conception which is lacking. There's another fundamental problem. Traditionally cloudstack development has happened separate from testing processes. Code is 'thrown-over-the-wall' to be tested which is not very different from how things are today. With that attitude I only see test playing 'catch-up' to the almost *maddening* pace of development. And now for some solutions: 1. Start test development alongside feature development. Attach a test engineer if possible to every feature branch. Make unit-test writing mandatory for both committers and non-committers. 2. I've been working with some qa folk within my company to address the issue more proactively. They (being non-committers) look forward to start working on github [1] along with some of our engineers who wrote the original integration tests. The test activity will proceed in the same feature branch as being used by the developer. And will be connected to appropriate infrastructure. But likely this will happen only after 4.1 3. Cloudstack needs massive infrastructure, build and packaging pieces to come together for doing live provider testing. While I got this up for 4.0 with a live environment, this hasn't received much love on master. Over the past couple days I've been testing the replicability of this infrastructure and have been moving the pieces to builds.a.o 4. Easing test writing is my primary concern at the moment and I've been refactoring marvin to accomadate a easier dsl style test composition. I hope to have a working test case sometime next week. Much of this has come too little, too late. However I beseech others to pitch in with ideas and contributions on improving testability on the whole to ease this pain for the forthcoming releases. [1] https://github.com/cloudstack-qa -- Prasanna., PS: Forgot to break the thread earlier in haste.
