To see how it might go I created a simple spreadsheet [1] to record test results. "0" for untested, positve value for passed tests and negative value for failed tests.
I realized it would be overwhelming for a single developer to carryout all the tests so I think its easier to just go on with the usual tests we do and mark them in the spreadsheet what we covered and later RM (or someone) can figureout a way to carryout tests which was not covered by anyone. Let me know if anyone needs edit privileges to the spreadsheet. Saminda 1. https://docs.google.com/spreadsheet/ccc?key=0AqYI3-ZrFz-EdFI0S3htaWJ1MDV4RE1WM19Ga0lhbEE&usp=sharing#gid=0 On Mon, Dec 16, 2013 at 1:12 PM, Amila Jayasekara <[email protected]>wrote: > +1, please. > > - Thejaka Amila > > > On Mon, Dec 16, 2013 at 2:05 PM, Suresh Marru <[email protected]> wrote: > >> On Dec 16, 2013, at 1:51 PM, Saminda Wijeratne <[email protected]> >> wrote: >> >> > I was thinking of an actual checklist where we can check-off/vote-off >> once each test is done. Perhaps we can start with a simple spreadsheet with >> the Tests specified by Raman added. >> >> + 1. Here is an example from Rave. Template for Quality Assurance [1] and >> an example [2]. >> >> Bottom line, for atleast few days during the release process, we all >> should become the QA Team. >> >> Currently, we are doing scripted testing like 5, 10 minute tutorials and >> grid job submissions and lot of code still does not get touched. As an >> example, provenance aware search became nonfunctional and until Sanjaya >> pointed it out, we did not notice it. It will be useful, if randomly (or by >> co-ordination) we all test an RC against various features and then post >> them to DISCUSS thread. Otherwise, the releases just become pointing to a >> tag. We need to move from releases being a formality to every release >> robusting the code. We have so much active development and if we turn some >> energy to testing and bug fixing, I think our users will be happy with the >> outcome. >> >> Suresh >> [1] - http://wiki.apache.org/rave/ReleaseManagement/QualityAssurance >> [2] - >> http://wiki.apache.org/rave/ReleaseManagement/ReleaseSchedule/VerificationResults-0.11 >> > >> > >> > On Mon, Dec 16, 2013 at 10:24 AM, Chathuri Wimalasena < >> [email protected]> wrote: >> > There is a general checklist added by Raman [1], which covers basic >> functionalities. >> > >> > Thanks.. >> > Chathuri >> > >> > [1] >> https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Release+Testing >> > >> > >> > On Mon, Dec 16, 2013 at 12:56 PM, Saminda Wijeratne <[email protected]> >> wrote: >> > >> > >> > >> > On Mon, Dec 16, 2013 at 9:28 AM, Suresh Marru <[email protected]> >> wrote: >> > Thanks Amila for weighing in. Comments inline: >> > >> > On Dec 16, 2013, at 11:29 AM, Amila Jayasekara <[email protected]> >> wrote: >> > >> > > Hi Suresh, >> > > >> > > I have some comments inline. >> > > >> > > >> > > On Mon, Dec 16, 2013 at 10:53 AM, Suresh Marru <[email protected]> >> wrote: >> > > Hi All, >> > > >> > > This is a very good question. Lets discuss these options so we are >> consistent across releases. >> > > >> > > If we look at the way we are doing releases, we are calling a feature >> freeze and code freeze and cutting a release. Most of the time, our build >> is broken. Jenkins statistics for Airavata is not looking good at all [1]. >> > > >> > > There is something wrong with the Jenkins configurations. I tried to >> figure out sometime back I was unable to do so. Even though builds are >> successful in our local machines they are failing intermittently in Jenkins. >> > > >> > > We are barely fixing the build a day before the release, putting out >> an RC and testing on it and releasing it in a quick succession. >> > > >> > > This is not entirely true. For the past few months I only experienced >> one or two build breaks (maybe less). I build couple of times per week. I >> believe usually build is stable and with integration tests passing, we >> always get a workable version. I know its not a good practice not to rely >> on the build server. But commiters have personal discipline to keep the >> build stable. Nevertheless we must fix Jenkins configuration issue. >> > >> > May be we should put focus on Jenkins configuration? Any volunteers? >> > >> > > >> > > As we are seeing on user lists, we have users upgrading with every >> release. I think we should increase the release quality. >> > > >> > > +1 for this. >> > > >> > > I would vote for atleast 3 RC’s per release. If we are not finding >> issues in first RC, I would say, either the software has magically become >> too too good or we are not doing through testing. I suspect the later. >> > How about we keep a checklist of release tests? I know we already send >> a mail on dev on what needs to be tested for each RC, but I need that is >> too abstract. For core developers of Airavata I think there should be test >> cases predefined (a test document if you may). Since we have several core >> developers in the list we can atleast decide upon what must be tested and >> make sure that each test case is covered by atleast one developer for a RC. >> > > >> > > I guess you mentioned this under assumption that build is not stable. >> > >> > Half of my assumption is on Jenkins, so if builds are ok and Jenkins is >> thinking wrong, then we can alleviate it by fixing it. >> > >> > > I will propose the following, please counter it and lets agree on a >> process: >> > > >> > > * Lets post a RC1 as is (which means it will have a snapshot). This >> pack, we should all test as much as possible, so its more of a test >> candidate then a release candidate. If it helps, we can use the name TC1. I >> am not particular on the naming but trying to emphasize the need for having >> atleast more RC's per release. >> > > >> > > I am not sure whether we really need a TC. The release manager should >> be doing some verifications on the RC before putting it out. Therefore it >> should be a RC. Anyhow i am fine having TC concept and trying it out. >> > >> > We probably should stick to RC, but I think the onus should not be on >> the RM to test it. They should coordinate and mobilize every one to do the >> testing including doing a testing bit more than others. But my point is, we >> should test and the only way to do that is to put a series of RC’s and have >> focused testing. >> > A TC should be something internal IMO. But when we are going for a >> release it should be alpha, beta and then RC releases. I think it need not >> be mandatory for the RMs to do pre-evaluation of the builds other than >> making sure all the unit tests and integration tests pass. Once an RC is >> confirmed of release quality I think we can follow the actual release cycle >> from the trunk itself with since its in a code freeze anyway. >> > >> > Suresh >> > >> > > >> > > What we really need is set of verifiable test cases. >> > > >> > > Thank you >> > > Regards >> > > Amila >> > > >> > > >> > > * If we do not expose significant issues in RC/TC 1 then we proceed >> with RC2 which will follow the proper release process. But if we have a >> reasonable issues bought out, we need a RC2/TC2 also without following the >> release process. >> > > >> > > * The key thing I am proposing is, we keep doing RC/TC’s until we all >> are sure the quality is good enough with documented known issues. When we >> are sure, then we proceed to have RC with proper release process. >> > > >> > > So this will mean more testing and twice (or more) the times every >> one has to test, but I think it is worth it. This might also get over the 6 >> week release cycle, but I think we need to trade for some quality releases >> as we march towards 1.0. >> > > >> > > Suresh >> > > [1] - https://builds.apache.org/job/Apache%20Airavata/ >> > > >> > > >> > > On Dec 15, 2013, at 4:28 PM, Lahiru Gunathilake <[email protected]> >> wrote: >> > > >> > > > >> > > > Hi Chathuri, >> > > > >> > > > I think having snapshot as the version in RC is wrong. Every RC has >> to be like a release and if it pass we just call a vote/discussion thread >> and do the release. If we do with snapshot and if things go right, then >> have to change versions and test again. But we can do the release just by >> changing snapshot without testing but that wrong AFAIT. >> > > > >> > > > I remember doing this mistake in earlier release with RC1 build. I >> think we can stick to the release management instructions in airavata.org >> . >> > > > >> > > > Regards >> > > > Lahiru >> > > > >> > > > >> > > > On Fri, Dec 13, 2013 at 3:43 PM, Chathuri Wimalasena < >> [email protected]> wrote: >> > > > Hi All, >> > > > >> > > > Airavata 0.11 RC1[1] is ready for testing. >> > > > >> > > > Here are some pointers for testing >> > > > • Verify the fixed issue for this release [2] >> > > > • Verify the basic workflow composition/execution/monitoring >> scenarios from >> > > > • Airavata 5 & 10 min tutorials [3],[4] >> > > > • Verify airavata client samples >> > > > • Verify the stability with derby & mysql backend databases >> > > > • Verify that the XBaya JNLP distribution works >> > > > • Verify deploying Airavata server in a tomcat distribution >> > > > Please report any issues[5] if you encounter while testing. Thank >> you for your time in validating the release. >> > > > >> > > > Regards, >> > > > Chathuri (On behalf of Airavata PMC) >> > > > >> > > > [1] https://dist.apache.org/repos/dist/dev/airavata/0.11/RC1/ >> > > > [2] >> https://issues.apache.org/jira/browse/AIRAVATA-278?jql=project%20%3D%20AIRAVATA%20AND%20fixVersion%20%3D%20%220.11%22%20ORDER%20BY%20status%20DESC%2C%20priority%20DESC >> > > > [3] >> http://airavata.apache.org/documentation/tutorials/airavata-in-5-minutes.html >> > > > [4] >> http://airavata.apache.org/documentation/tutorials/airavata-in-10-minutes.html >> > > > [5] https://issues.apache.org/jira/browse/AIRAVATA >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > System Analyst Programmer >> > > > PTI Lab >> > > > Indiana University >> > > >> > > >> > >> > >> > >> > >> >> >
