Hi, I have to agree 100% on the effectiveness of the integration tests, specially the ones with security and I know in many cases we ended up finding bugs in non-security code due to the security tests. In effect it means that these tests need to be there somewhere to run constantly to figure out whether there is a problem.
One thing I wanted to point out was that this is true for all sub projects - including sandesha , rampart and any other part of Axis2 we think of moving out to a seperate project. Also there can be issues with certain combinations of these modules (Say Sandesha and Rampart together makes some weird problem but Sandesha alone and rampart alone runs fine). Where would this kind of 'mixed up' tests reside ? Here is my suggestion. We move all the integration 'cross project' tests into a seperate project , something like 'Axis2 Integration'. This project will contain only unit tests and depend on Axis2 and all its sub projects. Since we are running continuum it can detect any of the test failures in the integration project. Eran has a point about many commits being done upto an integration failure (which makes it hard to detect the erroneous commit). What I think is by increasing the frequency of continuum (say 4 hours or 6 hours instead of 24 hours) we can sufficiently mitigate the risk. IMHO this would decrease the build time of Axis2 and also gives us a clear place to put all the cross project tests. Ajith On 1/7/07, Eran Chinthaka <[EMAIL PROTECTED]> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Illsley wrote: > I'm dead against moving security out of the axis2 build and leaving a > substantial amount of security tests in the axis2 builds. The converse > to an earlier argument is also true, namely that axis2 becomes > dependant on rampart to build and that people working on rampart won't > see the axis2 build failures. +1. And +1 for setting up the continuum build. But this is different from the point I wanna make. Forgive me if my explanation made you all think some thing else. What I was continuously pointing out was lack of integration tests Axis2 has on its own. The security tests Ruchith had written, which are in integration module, were testing the real Axis2 integration stuff together with his security scenarios. What I was asking is to understand the exact integration scenarios those security tests are testing, in security tests, and write new set of tests Axis2, without depending on security or any other tests. Let me give you an example. IIRC, Security scenario 1 test fails, if the addressing handlers are not engaged. This tests, the module deployment functionality, handler integration mechanism, execution chains and the addressing module. If we move this scenario 1 test out from Axis2, then what I ask is some one to write a new tests to test the above functionalities. Please understand that I never wanted to keep security tests inside Axis2, when the rampart is moved out. Yes I also against it. One could argue that continuum will fix the problem. But IMHO, it won't. The reason is none will check with continuum for each and every commit. So the only option is to run continuum as a cron job, once a day. By the time continuum fails and generates a mail, there can be numerous commits done to the commit. So everyone is having trouble fixing the problem. Since I believe in prevention than cure :), let's add some more integration tests in to Axis2, to fill the vacuum created due to the removal of security tests. Since we always run all the tests in Axis2, before committing anything, you will get to know the problem before hand, rather than waiting for continuum. But yes, continuum will point out the effect of a change in Axis2 to some other dependent project. But the changes I am referring are critical for Axis2. When I was doing changes to Axis2, getting the integration tests passed, especially the security tests, was the most challenging part, at least for me. I assume it is the same for everyone out there. Ruchith, you can remember we talked about this some time back, meaning the importance of security integration tests. If what I am talking is unclear, please add more to this discussion. Thanks, Chinthaka -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFoRuojON2uBzUhh8RAr23AJsF/4oxi6n/PJOPNisdZlKYtKqheACgmSDv W7zOCQWxprF0yVzXL/w88zw= =x9pa -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
-- Ajith Ranabahu --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
