On Dec 16, 2013, at 1:51 PM, Saminda Wijeratne <[email protected]> wrote:
> I was thinking of an actual checklist where we can check-off/vote-off once > each test is done. Perhaps we can start with a simple spreadsheet with the > Tests specified by Raman added. + 1. Here is an example from Rave. Template for Quality Assurance [1] and an example [2]. Bottom line, for atleast few days during the release process, we all should become the QA Team. Currently, we are doing scripted testing like 5, 10 minute tutorials and grid job submissions and lot of code still does not get touched. As an example, provenance aware search became nonfunctional and until Sanjaya pointed it out, we did not notice it. It will be useful, if randomly (or by co-ordination) we all test an RC against various features and then post them to DISCUSS thread. Otherwise, the releases just become pointing to a tag. We need to move from releases being a formality to every release robusting the code. We have so much active development and if we turn some energy to testing and bug fixing, I think our users will be happy with the outcome. Suresh [1] - http://wiki.apache.org/rave/ReleaseManagement/QualityAssurance [2] - http://wiki.apache.org/rave/ReleaseManagement/ReleaseSchedule/VerificationResults-0.11 > > > On Mon, Dec 16, 2013 at 10:24 AM, Chathuri Wimalasena <[email protected]> > wrote: > There is a general checklist added by Raman [1], which covers basic > functionalities. > > Thanks.. > Chathuri > > [1] > https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Release+Testing > > > On Mon, Dec 16, 2013 at 12:56 PM, Saminda Wijeratne <[email protected]> > wrote: > > > > On Mon, Dec 16, 2013 at 9:28 AM, Suresh Marru <[email protected]> wrote: > Thanks Amila for weighing in. Comments inline: > > On Dec 16, 2013, at 11:29 AM, Amila Jayasekara <[email protected]> > wrote: > > > Hi Suresh, > > > > I have some comments inline. > > > > > > On Mon, Dec 16, 2013 at 10:53 AM, Suresh Marru <[email protected]> wrote: > > Hi All, > > > > This is a very good question. Lets discuss these options so we are > > consistent across releases. > > > > If we look at the way we are doing releases, we are calling a feature > > freeze and code freeze and cutting a release. Most of the time, our build > > is broken. Jenkins statistics for Airavata is not looking good at all [1]. > > > > There is something wrong with the Jenkins configurations. I tried to figure > > out sometime back I was unable to do so. Even though builds are successful > > in our local machines they are failing intermittently in Jenkins. > > > > We are barely fixing the build a day before the release, putting out an RC > > and testing on it and releasing it in a quick succession. > > > > This is not entirely true. For the past few months I only experienced one > > or two build breaks (maybe less). I build couple of times per week. I > > believe usually build is stable and with integration tests passing, we > > always get a workable version. I know its not a good practice not to rely > > on the build server. But commiters have personal discipline to keep the > > build stable. Nevertheless we must fix Jenkins configuration issue. > > May be we should put focus on Jenkins configuration? Any volunteers? > > > > > As we are seeing on user lists, we have users upgrading with every release. > > I think we should increase the release quality. > > > > +1 for this. > > > > I would vote for atleast 3 RC’s per release. If we are not finding issues > > in first RC, I would say, either the software has magically become too too > > good or we are not doing through testing. I suspect the later. > How about we keep a checklist of release tests? I know we already send a mail > on dev on what needs to be tested for each RC, but I need that is too > abstract. For core developers of Airavata I think there should be test cases > predefined (a test document if you may). Since we have several core > developers in the list we can atleast decide upon what must be tested and > make sure that each test case is covered by atleast one developer for a RC. > > > > I guess you mentioned this under assumption that build is not stable. > > Half of my assumption is on Jenkins, so if builds are ok and Jenkins is > thinking wrong, then we can alleviate it by fixing it. > > > I will propose the following, please counter it and lets agree on a process: > > > > * Lets post a RC1 as is (which means it will have a snapshot). This pack, > > we should all test as much as possible, so its more of a test candidate > > then a release candidate. If it helps, we can use the name TC1. I am not > > particular on the naming but trying to emphasize the need for having > > atleast more RC's per release. > > > > I am not sure whether we really need a TC. The release manager should be > > doing some verifications on the RC before putting it out. Therefore it > > should be a RC. Anyhow i am fine having TC concept and trying it out. > > We probably should stick to RC, but I think the onus should not be on the RM > to test it. They should coordinate and mobilize every one to do the testing > including doing a testing bit more than others. But my point is, we should > test and the only way to do that is to put a series of RC’s and have focused > testing. > A TC should be something internal IMO. But when we are going for a release it > should be alpha, beta and then RC releases. I think it need not be mandatory > for the RMs to do pre-evaluation of the builds other than making sure all the > unit tests and integration tests pass. Once an RC is confirmed of release > quality I think we can follow the actual release cycle from the trunk itself > with since its in a code freeze anyway. > > Suresh > > > > > What we really need is set of verifiable test cases. > > > > Thank you > > Regards > > Amila > > > > > > * If we do not expose significant issues in RC/TC 1 then we proceed with > > RC2 which will follow the proper release process. But if we have a > > reasonable issues bought out, we need a RC2/TC2 also without following the > > release process. > > > > * The key thing I am proposing is, we keep doing RC/TC’s until we all are > > sure the quality is good enough with documented known issues. When we are > > sure, then we proceed to have RC with proper release process. > > > > So this will mean more testing and twice (or more) the times every one has > > to test, but I think it is worth it. This might also get over the 6 week > > release cycle, but I think we need to trade for some quality releases as we > > march towards 1.0. > > > > Suresh > > [1] - https://builds.apache.org/job/Apache%20Airavata/ > > > > > > On Dec 15, 2013, at 4:28 PM, Lahiru Gunathilake <[email protected]> wrote: > > > > > > > > Hi Chathuri, > > > > > > I think having snapshot as the version in RC is wrong. Every RC has to be > > > like a release and if it pass we just call a vote/discussion thread and > > > do the release. If we do with snapshot and if things go right, then have > > > to change versions and test again. But we can do the release just by > > > changing snapshot without testing but that wrong AFAIT. > > > > > > I remember doing this mistake in earlier release with RC1 build. I think > > > we can stick to the release management instructions in airavata.org. > > > > > > Regards > > > Lahiru > > > > > > > > > On Fri, Dec 13, 2013 at 3:43 PM, Chathuri Wimalasena > > > <[email protected]> wrote: > > > Hi All, > > > > > > Airavata 0.11 RC1[1] is ready for testing. > > > > > > Here are some pointers for testing > > > • Verify the fixed issue for this release [2] > > > • Verify the basic workflow composition/execution/monitoring > > > scenarios from > > > • Airavata 5 & 10 min tutorials [3],[4] > > > • Verify airavata client samples > > > • Verify the stability with derby & mysql backend databases > > > • Verify that the XBaya JNLP distribution works > > > • Verify deploying Airavata server in a tomcat distribution > > > Please report any issues[5] if you encounter while testing. Thank you for > > > your time in validating the release. > > > > > > Regards, > > > Chathuri (On behalf of Airavata PMC) > > > > > > [1] https://dist.apache.org/repos/dist/dev/airavata/0.11/RC1/ > > > [2] > > > https://issues.apache.org/jira/browse/AIRAVATA-278?jql=project%20%3D%20AIRAVATA%20AND%20fixVersion%20%3D%20%220.11%22%20ORDER%20BY%20status%20DESC%2C%20priority%20DESC > > > [3] > > > http://airavata.apache.org/documentation/tutorials/airavata-in-5-minutes.html > > > [4] > > > http://airavata.apache.org/documentation/tutorials/airavata-in-10-minutes.html > > > [5] https://issues.apache.org/jira/browse/AIRAVATA > > > > > > > > > > > > > > > > > > > > > -- > > > System Analyst Programmer > > > PTI Lab > > > Indiana University > > > > > > > >
