Re: [Libreoffice-qa] QA ideas for 3.5 WAS Re: [Libreoffice] Improving the QA and Release for 3.5
Hi Yifan, Yifan Jiang wrote (14-12-11 07:46) 1. Thanks for Rimas and Petr's actively participating, what we did in the past two month to improve the structure of Litmus and now the new stuff is online and a 3.5.0 beta regression test run is created, in which we are encouraging people to run the tests. https://tcm.documentfoundation.org/ There are a few important changes need to notify people, especially for l10n testers that: [...] OK, good news! 4. Is it possible to emphasize the regression strategy in the bug hunting session you are planning? :) I think it's good to clearly advertise the strategy and explain whatever is needed for the improved structure, indeed. Not sure if some of the points answered your original question :) Looks as if you perfectly managed to provide the needed additional information ;-) Please let me know if anyone needs more regression test and litmus related information! People my feel invited directly here and now - if any needs comes to my ears/eyes, I'll make sure it will be further communicated. Thanks for you help work, kind regards, Cor -- - Cor - http://nl.libreoffice.org ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] QA ideas for 3.5 WAS Re: [Libreoffice] Improving the QA and Release for 3.5
Hi Cor, I hope I didn't interrupt anything :) But here's some of my comments. 1. Thanks for Rimas and Petr's actively participating, what we did in the past two month to improve the structure of Litmus and now the new stuff is online and a 3.5.0 beta regression test run is created, in which we are encouraging people to run the tests. https://tcm.documentfoundation.org/ There are a few important changes need to notify people, especially for l10n testers that: - we now have the localization test cases merged together with English cases, both in content and structure. Then we add locale-dependent property for each test case. https://tcm.documentfoundation.org/show_test.cgi?id=3105 Specifically, a test result submitted for a locale dependent cases will not influence the progress statistics of the test cases running in another locale condition. For a locale independent case,the executing result will be shared among all tests whatever locales were seleted. For locale selection I meant the bottom field here in test configuration details set: http://wiki.documentfoundation.org/File:Litmus_chose-de.png So it is important to make people informed correct locale selection is necessary before submitting test results. Also it is important to let l10n people know the translation version of test cases are not disappeared, all languages versions of test cases were just put together to save our effort depend on how much they related with languages. - we abandoned the build id selection to avoid duplicating test effort. So it is important to make people informed to actively use correct build submitting a result in the corresponding test run. For build id selection I also meant the field in configuration details set (the screenshot needs to update for this...): http://wiki.documentfoundation.org/File:Litmus_chose-de.png - we grouped test cases with priorities, and it is preferred the higher priority test cases are executed before lower ones. This can be seen when it comes to test group selection after the above configuration set. 2. We also got the wiki page updated and cleaned up. The important start pages to understand all related stuff can be: http://wiki.documentfoundation.org/Litmus http://wiki.documentfoundation.org/QA/Testing/Regression_Tests 3. As for the regression test plan, we got a full regression test related section in the QA processes page typically to describe what a Litmus admin should do when there comes a build: http://wiki.documentfoundation.org/QA/Processes#Full_regression_test_management Briefly speaking, we create a single test run for every beta and rc phases, during whose time slot the test cases inside test runs are supposed to be executed. For example, from the release plan, we will have 3 beta builds from 5/Dec to 9/Jan. In this phase, we do NOT create three test runs, instead the only test run 3.5.0 beta regression test will be recommended there for nearly a month, when continuously submitting test result in the latest 3.5 beta build is expected. 4. Is it possible to emphasize the regression strategy in the bug hunting session you are planning? :) Not sure if some of the points answered your original question :) Please let me know if anyone needs more regression test and litmus related information! Thank you! Best wishes, Yifan On Tue, Dec 13, 2011 at 11:35:44PM +0100, Cor Nouws wrote: Hi Petr, Cor Nouws wrote (09-12-11 13:44) On the wiki, I added a section with the draft for the later to create separate wiki page: http://wiki.documentfoundation.org/QA/Improving_QA-Release-3.5#DRAFT_for_page_with_info_for_bug-hunting_session Is it possible for you - without wanting to stress you - to tell smthg more about the expected planning? tjanks a lot ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] QA ideas for 3.5 WAS Re: [Libreoffice] Improving the QA and Release for 3.5
Hi Cor, I have added QA and devel mailing lists into CC. We should not cook such an important event in background. Cor Nouws píše v Út 29. 11. 2011 v 00:06 +0100: Hi Yifan Petr, Is it OK for you if I proceed more or less as outlined below? I have some additional remarks that I'll take into account. I think I'll explain clearly to the people the way to Litmus for if they want to do the structural tests. I saw some work on changed naming. Is that already in place? Heh, we are going to do one more restructuralization in Litmus. Rimas implemented a nice feature yesterday. We are now able to mark some test cases as language and/or OS dependent. This option affects when the test will be marked as done in test run. This way we could get easier structure and optimize the testing efforts. Yi Fan is creating documentation at http://wiki.documentfoundation.org/QA/Testing/Test_Cases_Organization Also he is going to update the existing test cases accordingly. He wants to finish this until Dec 6 or so. Another problem is that we still have only few testcases. We need to create more. We still could do it during the test phase. Any helping hands are welcome. Cor Nouws wrote (21-11-11 10:59) Cor Nouws wrote (19-11-11 16:56) Goal is (as discussed before) get more people's focus on early testing, rather then with the first RC, and also more people testing :-) I have draft the following schedule with special events: - With the first beta, two day's organised bug-hunting: - Friday 9 and Saturday 10 December - the beta-0, makes sure that reliable builds are there for all - mail all people on l10n lists and ask to spread the news - mail to discuss users - people available on irc for guidance It sounds good. Rainer is just afraid that we do not get enough people attention. He tried to do such events in the past and nobody appeared. Well, I am not sure how Rainer announced it. You might know better channels. - during the months, a two day's competition: - Saturday 14 and Sunday 15 January - a prize for the three that post most bug-weight in those two day's - bug weight: points/issue: 1 for normal, 1,5 for major, 2,5 for every blocker issue - severity needs to be confirmed by QA specialists (in max one week) - prizes are e.g. sweater and mug. - QA specialists cannot win a prize (they should just get the sweater and mug without that) I am not sure about this action. I am afraid that we could get too many bug reports and it will be hard to sort them, ... Note that we currently have only very few people doing the bug triage. Another problem is who would judge the results :-) - with the first RC, two day's organised bug-hunting - Friday 20 and Saturday 21 January - ... (see above for more) Same as for the first bug hunting session. When I think about it, we have a chicken and egg problem. I am afraid that we do not have enough active QA people that would help to organize such event. On the other hand, such action could bring new people or get them from the hiding corner. I would try a well announced bug-hunting session. If we get enough people there, I would skip the competition because it is hard to organize. Another question is how to instruct people during the bug-hunting session. We do not have enough test cases. One possibility would be to ask them to create them as described at http://wiki.documentfoundation.org/QA/Testing/Test_Case Otherwise, it would end in a chaotic testing. Well, it would be better than nothing. Best Regards, Petr ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] QA ideas for 3.5 WAS Re: [Libreoffice] Improving the QA and Release for 3.5
Hi Petr, Petr Mladek wrote (29-11-11 11:54) I have added QA and devel mailing lists into CC. We should not cook such an important event in background. Agreed - that is why I started at the list initially of course ;-) Cor Nouws píše v Út 29. 11. 2011 v 00:06 +0100: Is it OK for you if I proceed more or less as outlined below? I have some additional remarks that I'll take into account. I think I'll explain clearly to the people the way to Litmus for if they want to do the structural tests. I saw some work on changed naming. Is that already in place? Heh, we are going to do one more restructuralization in Litmus. Rimas implemented a nice feature yesterday. We are now able to mark some test cases as language and/or OS dependent. This option affects when the test will be marked as done in test run. This way we could get easier structure and optimize the testing efforts. Ah cool. That reads to me as a feature that will prevent more people doing the same test without good reason. Yi Fan is creating documentation at http://wiki.documentfoundation.org/QA/Testing/Test_Cases_Organization Also he is going to update the existing test cases accordingly. He wants to finish this until Dec 6 or so. Another problem is that we still have only few testcases. We need to create more. We still could do it during the test phase. Any helping hands are welcome. I saw your note in an earlier mail yes. But did not really think about it, sorry. Is it a good idea if for the next 7/8 weeks we focus on as much testing as possible and that we after that try to encourage people to help with writing tests cases? Cor Nouws wrote (19-11-11 16:56) Goal is (as discussed before) get more people's focus on early testing, rather then with the first RC, and also more people testing :-) I have draft the following schedule with special events: - With the first beta, two day's organised bug-hunting: - Friday 9 and Saturday 10 December - the beta-0, makes sure that reliable builds are there for all - mail all people on l10n lists and ask to spread the news - mail to discuss users - people available on irc for guidance It sounds good. Rainer is just afraid that we do not get enough people attention. He tried to do such events in the past and nobody appeared. Well, I am not sure how Rainer announced it. You might know better channels. I share his experience from years past that it is not easy. On the other hand: - all we get on testing early is important; - we might help l10n people/groups to present it. Announcing the first beta of the next great LibreOffice (- features..) and a bug hunt session, sure will get some media attention. Which is good for the overall project and good for the l10n teams... Of course, needs some preparation, but should not be too difficult. I have some hours available this and next week to help. - during the months, a two day's competition: - Saturday 14 and Sunday 15 January - a prize for the three that post most bug-weight in those two day's - bug weight: points/issue: 1 for normal, 1,5 for major, 2,5 for every blocker issue - severity needs to be confirmed by QA specialists (in max one week) - prizes are e.g. sweater and mug. - QA specialists cannot win a prize (they should just get the sweater and mug without that) I am not sure about this action. I am afraid that we could get too many bug reports and it will be hard to sort them, ... Note that we currently have only very few people doing the bug triage. That is a possible trap, yes. Maybe there are some simple way's to make it better (will think about that later, but of course, hints welcome..) So best is the we start with the first step, and in the mean while think about if a competition can be managed in a pretty fair way. Another problem is who would judge the results :-) I thought you would like to volunteer ;-) - with the first RC, two day's organised bug-hunting - Friday 20 and Saturday 21 January - ... (see above for more) Same as for the first bug hunting session. Same for me ;-) When I think about it, we have a chicken and egg problem. I am afraid that we do not have enough active QA people that would help to organize such event. On the other hand, such action could bring new people or get them from the hiding corner. Agree. So what might help is find some support in organising, I think I can do that. I would try a well announced bug-hunting session. If we get enough people there, I would skip the competition because it is hard to organize. Might be a good choice - as written before, we can decide later. Another question is how to instruct people during the bug-hunting session. We do not have enough test cases. One possibility would be to ask them to create them as described at http://wiki.documentfoundation.org/QA/Testing/Test_Case Otherwise, it would end in a chaotic testing. Well, it would be better than nothing. Chaotic tests give great results - if people can write down the results
Re: [Libreoffice-qa] QA ideas for 3.5 WAS Re: [Libreoffice] Improving the QA and Release for 3.5
Cor Nouws píše v Út 29. 11. 2011 v 15:34 +0100: Petr Mladek wrote (29-11-11 11:54) Rimas implemented a nice feature yesterday. We are now able to mark some test cases as language and/or OS dependent. This option affects when the test will be marked as done in test run. This way we could get easier structure and optimize the testing efforts. Ah cool. That reads to me as a feature that will prevent more people doing the same test without good reason. Exactly. Another problem is that we still have only few testcases. We need to create more. We still could do it during the test phase. Any helping hands are welcome. I saw your note in an earlier mail yes. But did not really think about it, sorry. Is it a good idea if for the next 7/8 weeks we focus on as much testing as possible and that we after that try to encourage people to help with writing tests cases? I would encourage people to do both. Testing is important. On the other hand, the written tests help to avoid duplication. Also they are inspiration for people that have less experience. Note most of the test make sense to duplicate on Windows, Linux, and MAC. I would write it the way that we are looking for testers. Any test make sense. We want to do the existing tests from Litmus. They are also inspiration for people that are in doubts. On the other hand, if anyone does an interesting test that is missing in Litmus, we kindly ask them to document it there. The description need not be perfect, we could always improve it later. I think that people do not need to do this only during the bug-hunting session. They could do it at any time. We should mention links to the documentation, .e.g. http://wiki.documentfoundation.org/QA/Testing/Regression_Tests http://wiki.documentfoundation.org/QA/Testing/Test_Case http://wiki.documentfoundation.org/QA/Testing/Test_Cases_Contribution Cor Nouws wrote (19-11-11 16:56) - With the first beta, two day's organised bug-hunting: - Friday 9 and Saturday 10 December - the beta-0, makes sure that reliable builds are there for all - mail all people on l10n lists and ask to spread the news - mail to discuss users - people available on irc for guidance It sounds good. Rainer is just afraid that we do not get enough people attention. He tried to do such events in the past and nobody appeared. Well, I am not sure how Rainer announced it. You might know better channels. I share his experience from years past that it is not easy. On the other hand: - all we get on testing early is important; - we might help l10n people/groups to present it. Announcing the first beta of the next great LibreOffice (- features..) and a bug hunt session, sure will get some media attention. Which is good for the overall project and good for the l10n teams... Of course, needs some preparation, but should not be too difficult. I have some hours available this and next week to help. Sounds good to me. - during the months, a two day's competition: - Saturday 14 and Sunday 15 January - a prize for the three that post most bug-weight in those two day's - bug weight: points/issue: 1 for normal, 1,5 for major, 2,5 for every blocker issue - severity needs to be confirmed by QA specialists (in max one week) - prizes are e.g. sweater and mug. - QA specialists cannot win a prize (they should just get the sweater and mug without that) I am not sure about this action. I am afraid that we could get too many bug reports and it will be hard to sort them, ... Note that we currently have only very few people doing the bug triage. That is a possible trap, yes. Maybe there are some simple way's to make it better (will think about that later, but of course, hints welcome..) So best is the we start with the first step, and in the mean while think about if a competition can be managed in a pretty fair way. Yup, I like this step by step approach. Another problem is who would judge the results :-) I thought you would like to volunteer ;-) Please not! It would be my nightmare. I would be stressed that I would not be objective :-) Another question is how to instruct people during the bug-hunting session. We do not have enough test cases. One possibility would be to ask them to create them as described at http://wiki.documentfoundation.org/QA/Testing/Test_Case Otherwise, it would end in a chaotic testing. Well, it would be better than nothing. Chaotic tests give great results - if people can write down the results in a structure way of course ;-) I can draft a page to get just the people in the bug-hunt sessions on the right track. That would be great if you could draft something. Best Regards, Petr ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems?
[Libreoffice-qa] QA ideas for 3.5 WAS Re: [Libreoffice] Improving the QA and Release for 3.5
Hi all, Cor Nouws wrote (19-11-11 16:56) A test-event, was one of the ideas from the presentation (1), slide #24 Time is running fast, and since the developers (reading the most recent report of the ESC) did their bits to help us with better QA, I would like to take a more comprehensive approach then in my previous mail. Goal is (as discussed before) get more people's focus on early testing, rather then with the first RC, and also more people testing :-) I have draft the following schedule with special events: - With the first beta, two day's organised bug-hunting: - Friday 9 and Saturday 10 December - the beta-0, makes sure that reliable builds are there for all - mail all people on l10n lists and ask to spread the news - mail to discuss users - people available on irc for guidance - during the months, a two day's competition: - Saturday 14 and Sunday 15 January - a prize for the three that post most bug-weight in those two day's - bug weight: points/issue: 1 for normal, 1,5 for major, 2,5 for every blocker issue - severity needs to be confirmed by QA specialists (in max one week) - prizes are e.g. sweater and mug. - QA specialists cannot win a prize (they should just get the sweater and mug without that) - with the first RC, two day's organised bug-hunting - Friday 20 and Saturday 21 January - ... (see above for more) Ideas? Of course, people @marketing will be able to help spreading the information (in steps ...). Cheers, -- - Cor - http://nl.libreoffice.org ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
