Hi all,
Referring to the attached discussion thread initialed by Jean-Baptiste (great
thanks!), we are trying to make testcase management tool Litmus
(https://tcm.documentfoundation.org/run_tests.cgi) better managed and
organized. The following explained all current states. As usual, comments are
welcome! Thanks for all the support!
[For QA people running cases]
The run created by new method for 3.4.1:
https://tcm.documentfoundation.org/run_tests.cgi?test_run_id=13
Basically no practical changes from before. Meanwhile please
notice:
a. make sure the build version you are using is suitable for
the corresponding test run. For example, for the test run
- LibreOffice 3.4.0 RC Regression test
libreoffice build 3.4.0.x is needed.
b. Please make sure the platform, operating system and build
id information, which should be filled before the testing,
exactly reflecting the testing environment.
c. Attaching a bug id to a failed case result would be very
nice!
[For Litmus admins]
We are trying to make every thing much more clear and easy to
update. The current test case organization changes:
a. Divide test case set to Regression test and New Feature
test ([1])
b. Regression Test
A branch *Master Regression TC* was created
specifically. So that we can all work on a consistent
test case base, from which new branches can be always
*cloned*.
Keep all new/updated cases in this branch would be
necessary to make everything inside reusable.
That is to say, we use the Master branch to keep a
reusable and continuously maintanable test cases storage,
but NOT for creating test runs.
When a new build (big version as x.x) will come, we
1. clone (Recursively) a new regression test branch
for the build from Master branch, like
3.4 Regresstion Test (branch)
3.5 Regresstion Test (branch)
...
2. through the branches, create a test run for the corresponding
builds:
libreoffice 3.4.0 rc regresstion test (run created from 3.4
branch)
libreoffice 3.4.1 rc regresstion test (run created from 3.4
branch)
libreoffice 3.4.2 rc regresstion test (run created from 3.4
branch)
libreoffice 3.5.0 rc regresstion test (run created from 3.5
branch)
libreoffice 3.5.1 rc regresstion test (run created from 3.5
branch)
libreoffice 3.5.2 rc regresstion test (run created from 3.5
branch)
...
*NOTICE*
If some new cases needed, please create them in the
Master Testcase branch. Thus we can use the new
regression test case set next time creating
branches([2]).
I have noticed a new set of pt-BR test cases
created in '3.4 Regression test' branch today, I will
'sync' them to the Master branch later.
c. New Feature Test
A branch *Master Feature TC* was created
specifically. This is more like an outline without cases
(but with group subgroup info) filled in. So for each new
big version (x.x),
1. clone from the Master Feature branch:
3.4 Feature Test (branch)
3.5 Feature Test (branch)
3.6 Feature Test (branch)
2. we create a single run for each big version:
libreoffice 3.4 feature test (run created from 3.4 branch)
libreoffice 3.5 feature test (run created from 3.5 branch)
libreoffice 3.6 feature test (run created from 3.6 branch)
...
3. we 'seek' and write new feature cases in the
corresponding subgroup according to git log,
mailing list, release notes or so.
4. we update important feature case to Master
Regression TC branch
[Footnotes]
[1] Regression and Feature test are there for quite different
purpose:
The Regression test would hold relatively stable set of
test cases, which should be ran in each of the release
build.
The New Feature test would be dynamically changed based
on a big version update (3.3->3.4). Meanwhile New feature
test might be run from even early development build
phase.
[2] As you may notice, by this process we didn't really
change the running branch. However I didn't find an easy way
to sync test cases between branches :( A workaround method to
'link' new test cases is:
1. Add a new test cases in the Master branch, and
assign the cases to a proper subgroup
2. Edit current running branch's subgroup to link a new
case from the Master branch to the current running
one. (manage subgroups -> select Product as libreoffice
and Banch as currently running -> edit subgroups ->
manage testcases for this subgroup)
Please also let me know if things could be better.
--
Yifan Jiang
Libreoffice
Contact: yifan - irc.freenode.net/libreoffice
=============================================
http://www.libreoffice.org/
http://www.documentfoundation.org/
--- Begin Message ---
Hi Jean-Baptiste,
Thanks for the nice ideas and suggestions! My comments as below.
Hi All,
This discussion is probably leading the next step we will adjust test case
management (Litmus) structure of Libreoffice, ideas and experience sharing
are welcome :) And I will summarize where we will go finally.
On Mon, Jun 13, 2011 at 01:54:46PM +0200, Jean-Baptiste Faure wrote:
> Le 13/06/2011 12:47, Yifan Jiang a écrit :
> Hi Yifan,
>
> I think we should do some simplifications in Litmus.
Yes! we need to adjust it to a more explicit way :)
> In fact I do not understand well how branches are supposed to work in
> Litmus. When you manage testcases and testgroups, you can see that there
> is no one in all branches but only in branch 3.3.1 (in Litmus meaning).
You are right, the branch information is from an relatively old test case set
(3.3.1). What I did for current runs is simply branch from an old test run
without touching test cases. So currently we have a simple set of testcases
reused everywhere. This actually needs to be improved.
> On the other hand when you manage the testruns you can see that each one
> contains the three testgroups defined in branch 3.3.1 (EN, DE and FR
> testgroups).
Ye, a bit confusing :)
> It seems to me that it would be more clear and easy to manage if we
> defined things as follows:
> - branches corresponding to x.y versions of LibreOffice: 3.3, 3.4, 3.5,
> and so on.
> - the branch x.y+1 would be defined by cloning the x.y branch and next
> adding new testcases needed by QA of the new functions implemented in
> the x.y+1 version.
My thinking is to:
1. have a relatively stable regression test case set which cover most
important function areas such as installation, launch, file open, save
etc.
2. have a changing feature cases (new functions) with the
evolution of Libreoffice builds. These cases can be various among
different Libreoffice build. And we can start testing features from beta
phase.
> - we manage RC and bugfix versions (x.y.1, x.y.2, etc.) only by creating
> dedicated testruns: there is no reason to change the testcases set when
> we pass from version x.y.0 to rc x.y.1 rc1 to bugfix x.y.1 to ... etc.
> So there is no reason to have a new branch for a new rc or bugfix
> - we need to define the buildID for each testrun, which does not seem to
> be the case at this moment.
Great idea! Actually we may not really want to have big change of regression
test cases set even between big versions as x.y and x.y+n.
A problem is if it is worth to repeatedly run regression cases in diffrent rc
builds unders the same big version? My initial thinking about this is to
define a rc test run for each minor version build, QA could run the tests
always on the corresponding latest rc build they have. This is based on the
assumptions:
1. code change between rc release would be safe
2. currently we have not enough resource to repeatedly run all regression
tests on each rc build.
But I guess it is not a big problem since when can always clone between rc
runs, the matter is how to define the run name. How about let us use 'rc' only
at very beginning and see if they are going well. Then we can expand the test
dedicating to specific rc builds if needed.
> According to that, we would have:
> - 2 branches 3.3 and 3.4
> - for the branch 3.3 :
> + one testrun 3.3.3-rc1
> + one testrun 3.3.3-rc2
> - for the branch 3.4 :
> + one testrun 3.4.0
> + one testrun 3.4.1-rc1
> + one testrun 3.4.1-rc2
> + etc.
> + one testrun 3.4.2-rc1
> + etc.
> - then a branch 3.5
> + etc.
Here is my thinking based on your proposal:
Take into consideration libreoffice build 3.3 and 3.4:
Branches:
- 3.3 Regression test branch (assuming it contains stable regression cases
written from scratch)
+ Test run - 3.3.0-rc regression test
+ Test run - 3.3.1-rc regression test
+ Test run - 3.3.2-rc regression test
...
- 3.3 New Feature test branch
+ Test run - 3.3 feature test ( new features in 3.3 build )
- 3.4 Regression test branch (containing stable regression cases, could be
cloned from 3.3 and did some improvement)
+ Test run - 3.4.0-rc regression test
+ Test run - 3.4.1-rc regression test
+ Test run - 3.4.2-rc regression test
...
- 3.4 New Feature test branch
+ Test run - 3.4 feature test ( new features in 3.4 build )
> I think we need to agree on the general organization before
> that the system has become too big and too widely used.
> I can do these modifications if we have a general agreement.
> As that assume to remove several branches and to rebuild testruns, it
> would be prudent to not be several to do changes at the same time.
That'd be great and thanks for your offer of help! Meanwhile we won't want to
lose the existing testing results data :)
> What do think about this proposal? Did I overlook some point which make
> my beautiful organization completely stupid?
They are indeed beautiful, I appreciate your ideas! :) Thank you!
Best wishes,
Yifan
--
Yifan Jiang
Libreoffice
Contact: yifan - irc.freenode.net/libreoffice
=============================================
http://www.libreoffice.org/
http://www.documentfoundation.org/
--- End Message ---
_______________________________________________
LibreOffice mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
