Just my late response on this thread: 1) If I include a test plan but should forget steps/test areas etc., and a non-developer tests the patch: aren’t we decreasing the value of signoff? In that case we could still end up with not signing patches in the long run.. Still, something is perhaps better than nothing at all.
2) A patch should not be too big. 3) Much depends on how the QA role is filled in [Ian’s plans sounds good]. If the QA could give more guidance on which patches should be tested next, testers are not just cherry picking patches.. Why should QA not mail once a week or so the first 10 or 20 patches that he wants to be signed? Could prevent older/larger/less interesting sounding patches waiting and newer (smaller/..) patches getting signed and pushed. Would make the process even more ‘fair’ to everyone. Marcel Van: [email protected] [mailto:[email protected]] Namens Ian Walls Verzonden: donderdag 27 januari 2011 17:50 Aan: Paul Poulain CC: [email protected] Onderwerp: Re: [Koha-devel] Development workflow Paul, I hear you. There are lots of really great features that are in limbo, waiting to be tested. The unfortunate situation of life is that we're all incredibly busy, and it's hard to find time to test things. This is compounded when testing certain features requires coding or systems experience, since that limits the pool of potential testers. As a result, lots of good stuff sits, waits, and diverges. So what can be done? I don't think that changing the signoff procedure will have the desired effect. That'll just let more unreviewed code slip in, potentially introducing bugs we won't find for weeks/months/years. I think a better solution is make it easier to test and signoff on work. There are several components to this: * Testing plans for larger developments/bugfixes * A robust testing data set made readily available * Teaching people how to test and signoff on code By including testing plans with developments or complex bugfixes, the developer is communicating to everyone how they can prove their code works. It lays out the intention of the development (it should do x, y and z), and a series of tests to show how to get x, y and z without losing a, b and c. Combine this testing plan (written in language librarians can understand, not coder jargon) with the necessary data set to do the testing (an SQL dump you just load into a DB), and you've lowered the barrier for testing so that anyone who can afford a little time to run through a series of listed procedures can answer the question "does this work?". The third step to this is to lay out the procedure for running through the test plan in a clear, simple manner, and distribute that information far and wide. Make it something that librarians can do by following a list of steps. Lowering the threshold of experience required to test things will allow us to harness the Long Tail. To this end, I'm throwing my hat in the ring for Quality Assurance Manager for Koha 3.6. My proposal can be found on the wiki (http://wiki.koha-community.org/wiki/QA_Manager_for_3.6_Proposal), and much of it is explained above. In addition to this, I would also serve as a coordinator for testing work submitted, and provide regular reports to the community on the status of these developments. Branches that are not receiving active testing feedback would receive attention towards creating a simpler, easier to follow testing plan. I would very much like to discuss this at the next IRC meeting. It'll be pretty early for me (5am, I believe), but I'll caffeinate heavily beforehand :) Cheers, -Ian On Thu, Jan 27, 2011 at 11:03 AM, Paul Poulain <[email protected]<mailto:[email protected]>> wrote: Hello koha-devel, The more time passes, the more I think we have a workflow for Koha development that does not work correctly/efficiently/fluently. The rule we have is: * create a branch for the feature * file a bug for it * submit your branch to koha-patches ML * the RM checks that everything apply smoothly on master & ask for QA * someone signs-off the branch/patch * the RM merge the branch. For the maintenance branch, this workflow is perfect and must be secured more and more to avoid any regression. As I see it, there is always someone that can find time to sign-off a patch or 2. As ppl have system live with it i'm sure it's cricital for many ppl to check & double check ! But for the next release/unstable branch this workflow does not work well. Here is a single line of something owen said on the chat today: <owen> paul_p: I would much prefer to be testing your stuff today, but other stuff is taking precedence. God knows i'll be forever thankfull to owen for the work he does. But this sentence reveals something: for most of us, there is always "other stuff taking precedence" :\ BibLibre has submitted many new features 3 months ago. I tried to motivate ppl to get feedback & sign-off. I even have organised an IRC meeting ... where no one came. Those patches includes new features, and requires a lot of work to be tested. 2 weeks ago, I did the same for a bunch of small improvements (and I have many many more pending). Once again, no feedback, except for one very easy-to-test (new images) : signed-off by Nicole & pushed by chris in less than 24H! Those 2 minor events (with owen & nicole) made me realize that it's technically and functionnaly hard & long to test: rebase, drop database, install, file test cases, check the new features,... At KohaCon, liz tested some of those features and said "wow, great, I want this feature for my library !" (I don't remember what it was exactly). The problem is that liz is not a techie woman, and can't test patches by her own. For our images, it was easy for Nicole to test, no need to updatedatabase or anything complex, just reach the itemtypes.pl<http://itemtypes.pl> page check the images are here. Easy for a librarian ! Other point: dependancies between new features. Let me take an example: Templates on those devs relies on html::template. Chris is working on moving everything to Template::Toolkit. Once he consider it's ready, he should submit for QA, as for every new features. Suppose no-one sign-off. And suppose someone sign-off, there are 2 options: * move master to T::T when it's signed-off. What happends with other branches not merged ? they can't be merged anymore, they have to be rewritten. wow, I can't imagine that something we've submitted for QA 3 months ago has to be rewritten because no-one took care of it ! * wait until those branches are merged. Two major problems here: the release date won't be respected, and we have switched again to feature based release. Or 3.4 will be released on time, but won't contain many things all those great features that are live in all our libraries since months. The second problem is that other devs are on the way. They still rely on H::T as master is based on that. Switching to T::T will cause problems for those branches later. well, I think that none of those solutions is a correct one. I can take an other example: the analytics record feature may/will probably overlap functionnaly with the new features we have submitted for cataloguing (not sure, but i think so). Suppose our cataloguing branch is integrated in 2 weeks. it has been submitted 3 months ago. It means our indian colleagues will have to rebase & face problems to get their analytical record branch working. If our branch had been merged quickly, they would have had 3 more months to deal with it. Our actual workflow, causes a lot of overhead ! And -worst- I don't think it improves the stability & security: there are merge problems that are difficults to detect. It means that any test done on analytical records would have to be done again if our cataloguing branch is merged : more work, more pain. And the later a branch is merged, the smaller the time to find a problem. My main conclusion is that we are not a large enough community to deal with testing/validating/merging new features in a short timeline. So I think we must change the way we integrate new features. The general idea being: remove the sign-off lock on the workflow. Here is a proposal: Let's go back to the timeline: we plan to release a version every 6 months. We could have a window of, say 4 months. During those 4 months a feature can be included if : 1- the submitter provides something that applies on master (ie: it's technically valid) 2- no-one object in 2 weeks (or 3, or 4, or defined by the RM when the branch is submitted) It put more pressure on others to test quickly, and don't put pain on parallels/parallelizing developments. Once a branch is merged if there is a problem, then everybody should see it, and it will be fixed ! Then, for 1 month, any new feature can be applied only if a newly submitted branch is approved/signed-off (less than 2 months to detect a problem in a new feature is a must-have) Then, feature freeze, 1 month debugging and translating. I know it is a big change in the workflow, but the actual one is not working well, so, we have to find another one. I humbly request to have this question being put on the next irc agenda (which does not mean you should not also start a thread here, of course) PS: pls don't say i'm wrong and the workflow works well. Having branches submitted 3 months ago, getting no feedback, and planning to have a release in less than 3 months, can't be considered as something working well. Friendly. -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08 _______________________________________________ Koha-devel mailing list [email protected]<mailto:[email protected]> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/ -- Ian Walls Lead Development Specialist ByWater Solutions Phone # (888) 900-8944 http://bywatersolutions.com [email protected]<mailto:[email protected]> Twitter: @sekjal
_______________________________________________ Koha-devel mailing list [email protected] http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
