Re: [Freeipa-devel] [PROPOSAL] FreeIPA Test Plan Workflow
On 19.3.2015 12:33, Martin Kosek wrote: > On 03/19/2015 11:45 AM, Petr Spacek wrote: >> On 19.3.2015 10:11, Martin Kosek wrote: >>> On 03/19/2015 09:25 AM, Petr Spacek wrote: Hello, I do not much to add to the process itself. After first reading it seems pretty heavyweight but let's try it, it can be refined at any time :-) >>> >>> Right, but then we would need to migrate the data about test completion and >>> so >>> on - which is more work. So it is much better to define some working now, >>> than >>> to change it couple months later. >>> >>> We were already trying to invent something as much lightweight as possible, >>> this was the minimum new fields we come for to be able to track the test >>> coverage and plans. If you have another proposal how to track it better, I >>> would love to hear it, really :-) >> >> Sure. For me the main question is when *designing of tests* should start and >> how it is synchronized with feature design. Is it done in parallel? Or >> sequentially? When the feedback from test designers flows back? Isn't it too >> late? >> >> Let's discuss ticket workflow like this: >> new -> design functionality&tests -> write code&tests -> test run -> closed >> >> IMHO we should have tests *designed* before we start to implement the final >> version of the functionality. It may be too late to find out that interface >> design is flawed (e.g. from user's point of view) when the feature is fully >> implemented and test phase is reached. >> >> Designing/writing tests early could discover things like poor interface >> design >> sooner, when it is still easy to change interfaces. Currently we have >> 'design' >> reviews before the implementation starts but actually designing tests at the >> same time would attract more eyes/brains to the feature design phase. We may >> call it 'first usability review' if we wish :-) >> >> In my mind, test designers should be first feature users (even virtually) so >> the early feedback is crucial. >> >> Note that this approach does not preclude experimental/quick&dirty >> prototyping >> as part of the design phase but it has to be clear that prototype might (and >> should!) be thrown away if the first idea wasn't the best one. > > Yes! This is exactly why this QE team was created - to be able to test as > early > as possible, review designs with QE eyes as early as possible. Great, in that case we can ignore the next section completely (it was meant as fallback). >> If this is too radical: /snip/ >> Then there is the question if we actually need to separate field for QE state >> and Test case field. Test case could behave in the same way as Bugzilla link >> field: >> - empty field - undecided >> - 0 (or string "not necessary" or something) - test case is deemed >> unnecessary >> - non-zero link - apparently, a test case exists >> >> It would be more consistent with what we have for Bugzilla links. > > The metadata we come up should be able to supply at least following queries: > - which tickets (RFEs/bugs) are covered with tests in a specific milestone, > what are the test cases > - who, from QE team, is working on which tickets > - list of tickets where we want the tests and which are for grabs by QE > engineer > > I am not sure if this can be covered just with the extra QE phase and Test > Case > link. Okay, it might be easier with more explicit fields as proposed. -- Petr^2 Spacek -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PROPOSAL] FreeIPA Test Plan Workflow
On 03/19/2015 11:45 AM, Petr Spacek wrote: > On 19.3.2015 10:11, Martin Kosek wrote: >> On 03/19/2015 09:25 AM, Petr Spacek wrote: >>> Hello, >>> >>> I do not much to add to the process itself. After first reading it seems >>> pretty heavyweight but let's try it, it can be refined at any time :-) >> >> Right, but then we would need to migrate the data about test completion and >> so >> on - which is more work. So it is much better to define some working now, >> than >> to change it couple months later. >> >> We were already trying to invent something as much lightweight as possible, >> this was the minimum new fields we come for to be able to track the test >> coverage and plans. If you have another proposal how to track it better, I >> would love to hear it, really :-) > > Sure. For me the main question is when *designing of tests* should start and > how it is synchronized with feature design. Is it done in parallel? Or > sequentially? When the feedback from test designers flows back? Isn't it too > late? > > Let's discuss ticket workflow like this: > new -> design functionality&tests -> write code&tests -> test run -> closed > > IMHO we should have tests *designed* before we start to implement the final > version of the functionality. It may be too late to find out that interface > design is flawed (e.g. from user's point of view) when the feature is fully > implemented and test phase is reached. > > Designing/writing tests early could discover things like poor interface design > sooner, when it is still easy to change interfaces. Currently we have 'design' > reviews before the implementation starts but actually designing tests at the > same time would attract more eyes/brains to the feature design phase. We may > call it 'first usability review' if we wish :-) > > In my mind, test designers should be first feature users (even virtually) so > the early feedback is crucial. > > Note that this approach does not preclude experimental/quick&dirty prototyping > as part of the design phase but it has to be clear that prototype might (and > should!) be thrown away if the first idea wasn't the best one. Yes! This is exactly why this QE team was created - to be able to test as early as possible, review designs with QE eyes as early as possible. > If this is too radical: > > To me it seems kind on unnatural to separate testing from overall bug state. > Equivalent of ON_QA state in Bugzilla seems more natural to me as it is kind > of weird to claim that ticket is closed/finished before full testing cycle is > finished. > > I.e. the ticket could have states like: > new -> assigned -> qe -> closed > "qe" state can be easily skipped if no testing is (deemed to be) necessary. This is an alternate approach yes. Trac has workflow plugin that should be able to add it. But wouldn't that workflow actually support the classic waterfall approach and not the more agile approach with testing more or less in parallel with the work on the code? The point is that a RFE may be still in development and also in QE state in parallel - thus the field. > > Then there is the question if we actually need to separate field for QE state > and Test case field. Test case could behave in the same way as Bugzilla link > field: > - empty field - undecided > - 0 (or string "not necessary" or something) - test case is deemed unnecessary > - non-zero link - apparently, a test case exists > > It would be more consistent with what we have for Bugzilla links. The metadata we come up should be able to supply at least following queries: - which tickets (RFEs/bugs) are covered with tests in a specific milestone, what are the test cases - who, from QE team, is working on which tickets - list of tickets where we want the tests and which are for grabs by QE engineer I am not sure if this can be covered just with the extra QE phase and Test Case link. -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PROPOSAL] FreeIPA Test Plan Workflow
On 19.3.2015 10:11, Martin Kosek wrote: > On 03/19/2015 09:25 AM, Petr Spacek wrote: >> Hello, >> >> I do not much to add to the process itself. After first reading it seems >> pretty heavyweight but let's try it, it can be refined at any time :-) > > Right, but then we would need to migrate the data about test completion and so > on - which is more work. So it is much better to define some working now, than > to change it couple months later. > > We were already trying to invent something as much lightweight as possible, > this was the minimum new fields we come for to be able to track the test > coverage and plans. If you have another proposal how to track it better, I > would love to hear it, really :-) Sure. For me the main question is when *designing of tests* should start and how it is synchronized with feature design. Is it done in parallel? Or sequentially? When the feedback from test designers flows back? Isn't it too late? Let's discuss ticket workflow like this: new -> design functionality&tests -> write code&tests -> test run -> closed IMHO we should have tests *designed* before we start to implement the final version of the functionality. It may be too late to find out that interface design is flawed (e.g. from user's point of view) when the feature is fully implemented and test phase is reached. Designing/writing tests early could discover things like poor interface design sooner, when it is still easy to change interfaces. Currently we have 'design' reviews before the implementation starts but actually designing tests at the same time would attract more eyes/brains to the feature design phase. We may call it 'first usability review' if we wish :-) In my mind, test designers should be first feature users (even virtually) so the early feedback is crucial. Note that this approach does not preclude experimental/quick&dirty prototyping as part of the design phase but it has to be clear that prototype might (and should!) be thrown away if the first idea wasn't the best one. If this is too radical: To me it seems kind on unnatural to separate testing from overall bug state. Equivalent of ON_QA state in Bugzilla seems more natural to me as it is kind of weird to claim that ticket is closed/finished before full testing cycle is finished. I.e. the ticket could have states like: new -> assigned -> qe -> closed "qe" state can be easily skipped if no testing is (deemed to be) necessary. Then there is the question if we actually need to separate field for QE state and Test case field. Test case could behave in the same way as Bugzilla link field: - empty field - undecided - 0 (or string "not necessary" or something) - test case is deemed unnecessary - non-zero link - apparently, a test case exists It would be more consistent with what we have for Bugzilla links. -- Petr^2 Spacek -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PROPOSAL] FreeIPA Test Plan Workflow
On 03/19/2015 09:25 AM, Petr Spacek wrote: > Hello, > > I do not much to add to the process itself. After first reading it seems > pretty heavyweight but let's try it, it can be refined at any time :-) Right, but then we would need to migrate the data about test completion and so on - which is more work. So it is much better to define some working now, than to change it couple months later. We were already trying to invent something as much lightweight as possible, this was the minimum new fields we come for to be able to track the test coverage and plans. If you have another proposal how to track it better, I would love to hear it, really :-) -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PROPOSAL] FreeIPA Test Plan Workflow
Hello, I do not much to add to the process itself. After first reading it seems pretty heavyweight but let's try it, it can be refined at any time :-) On 19.3.2015 08:57, Martin Kosek wrote: > Test Coverage: yes/no/in progress/ / I'm very much in favor of more descriptive names like: covered/not necessary/test in works/undecided It would allow random walkers by (like me :-) to understand what is going on without necessity to study some internal processes. -- Petr^2 Spacek -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PROPOSAL] FreeIPA Test Plan Workflow
On 03/18/2015 08:18 PM, Martin Koci wrote:
> Hi,
> working with Test Plans for 4.2 features I'd like to outline workflow
> for test plans. The main aim is to have something documented and more
> clear.
> So I'd like to start with track ticket options. For better tracking and
> managing tickets we could consider 3 new fields in the track ticket.
track --> Trac
>
> - First field for tracking link for test plan/test case which we can
> refer to. We would call this field just "Test".
I would name it "Test Case", if it would link to Test Case on wiki/other tool.
With "Test", I would rather understand it to be a link to the actual tests.
> - The second field for tracking tester (QE). Let's called this field "QA
> Contact". This field should inform about contributor (QE). That means -
> you can track unassigned bugs (tracks), assigned but not reviewed (the
> work hasn't started yet), and finished ("-", "+" - see below). According
> to this you can also see who is overloaded and who can help to the
> others.
Right now, we have following person-related fields (in form name - label):
reporter - Reported by
reviewer - Patch review by
So for consistency sake, what about:
tester - Test by
> - The third one for tracking states or flags for test coverage.
> Something like "QE Test Coverage" with four states/flags:
>
> * Review is needed - " " (review is required for this issue)
> Empty "QE Test Coverage" should mean "Hey, I need to be reviewed and
> considered whether some test is needed or not".
> * Test is not needed - "-" (Test is not required for this issue)
> * Test exists - "+" (Test already exists - QE done)
> * Test in progress - "?" (Test is required and QE {will} works on
> test{s})
>
> I can imagine some naming instead of flags +,-,?, ,. Any ideas?
Right, this is the most important one as we will use it which RFEs/tickets we
want to cover with tests and which not. The states make sense (we come up with
them together anyway), +/-/?/ / makes sense, alternatively we can do:
Test Coverage: yes/no/in progress/ /
> In the track ticket should be described the issue or link to design page
> to get all necessary information for coverage. QE changes state "QE Test
> Coverage" (Test in progress - "?") and creates test plan on wiki [1] and
> add this link to "Test" field in the track. Then QE informs
> freeipa-devel list about test plan if it's OK providing the link to test
> plan. If it's OK QE will start work on particular test cases. When QE is
> done then changes state "QE Test Coverage" to "+" (test exists).
Makes sense for big features, yes. What about smaller features or bug fixes? Is
test plan also required or would it be sufficient to just add that one or two
tests to XMLRPC tests?
> As well I'd like to propose the possibility that ticket will not be
> closed until QE is done with test?
Still not sure, the requirement is that the test would need to be finished
before GA, as the ticket must be in the milestone where the code is released. I
personally do not see a problem with closing the ticket and leaving it with
"Test Coverage: in progress" which would then be changed to "yes", when test is
done. It is easy to filter tickets with unfinished tests, even if they are
closed.
> Hope it makes sense to you.
> Can I get your thoughts on this, please?
>
> Thanks,
> /koca
> *[1] - http://www.freeipa.org/page/Main_Page
>
>
--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
