subject:"Re\: Some thoughts on quality"

Re: Some thoughts on quality

2013-08-16 Thread Andrea Pescetti


On 15/08/2013 Rob Weir wrote:

A thought experiment:   If we ran the existing test automation on
4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think
would be detected?


I would expect this 4.0.1 blocker issue (wrong results in certain Calc 
functions) to be detectable by automated testing:

https://issues.apache.org/ooo/show_bug.cgi?id=122997

Regards,
  Andrea.

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

2013-08-16 Thread Kay Schenk

On Wed, Aug 14, 2013 at 3:35 PM, sebb seb...@gmail.com wrote:

 On 14 August 2013 23:10, Rob Weir robw...@apache.org wrote:
  On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) marcus.m...@wtnet.de
 wrote:
  Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
 
  Le 14/08/2013 21:39, Rob Weir a écrit :
 
  Maybe we need to call an earlier build the RC so it will get more
  attention? We had a complete test build that we were testing for
  over a month. But maybe it is ignored unless we call it an RC? In
  other words, there were many opportunities for users to help try 4.0
  before it was released, but maybe there opportunities were not well
  known.
 
 
  I think that it was too difficult to get to the dev versions and it was
  too much complicated. There was no clear link to download a dev version
  (I had to rely on the url in the messages from the dev list).
 
 
  This was intended.
 
  We hadn't published the dev builds via Apache or SF mirrors but only on
 2
  people accounts. Apache policy says it's not allowed to publish them
 for a
  wider audience to save the servers from a high traffic load. It's the
 job of
  the mirrors to handle this.
 
 
  What I see (from a standard user point of view) for a RC:
  - When a dev version is almost done, rename it RC and make it known
  (blog and we would relay the announcement in the forums of course)
  - Have a link visible under the main download button of the download
 
 
  Both can be done, depending where the install files are located.
 
 
  page (perhaps a similar button as a dedicated entry)
  - Make that RC installable in parallel with a stable version
  - No file association allowed for that RC by design
 
 
  IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a
 RC
  anymore. One of the RC attributes is to change it into the final release
  with, e.g., just a file name change. But this has to be done without any
  code changes. Otherwise you have to change code parts, build again, test
  again, ... ;-)
 
  But a Beta release could go this separated way.
 
 
  Right.  A release is a release is a release.  The basic requirements
  for every release still apply:
 
  1) 3 PMC +1 votes
  2) Must include source files
  3) Digital signatures, hash files, etc.
 
  But we can have a beta release, that follows these rules, and it
  would be acceptable.  We can then host on the mirrors, publicize, etc.

 There would likely be some restrictions on how many extra downloads
 are permitted.
 For example, the ASF mirrors probably could not cope with a set of
 betas of all the languages for all the OSes in addition to the current
 GA release.


Looking again at this thread, and knowing the amount of time and resources
it takes to actually mount an approved release, I wonder if some of the
issues we had with the 4.0 release could have been addressed by a longer RC
vote timeframe -- like 2 weeks or so. The vote this time was relatively
short as I recall on both the original RC and RC2.  We have over 400 people
on this list. I would hope given a longer test period, we might receive
better testing and feedback if we took a bit longer for approval.

I realize this is not the same as a beta -- we would still be using
development snapshots, but the RC could be provided in all languages we
intend to use, etc.




 -
 To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
 For additional commands, e-mail: dev-h...@openoffice.apache.org




-- 
-
MzK

When in doubt, cop an attitude.
-- Cat laws

Re: Some thoughts on quality

2013-08-16 Thread Rob Weir

On Fri, Aug 16, 2013 at 12:30 PM, Kay Schenk kay.sch...@gmail.com wrote:
 On Wed, Aug 14, 2013 at 3:35 PM, sebb seb...@gmail.com wrote:

 On 14 August 2013 23:10, Rob Weir robw...@apache.org wrote:
  On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) marcus.m...@wtnet.de
 wrote:
  Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
 
  Le 14/08/2013 21:39, Rob Weir a écrit :
 
  Maybe we need to call an earlier build the RC so it will get more
  attention? We had a complete test build that we were testing for
  over a month. But maybe it is ignored unless we call it an RC? In
  other words, there were many opportunities for users to help try 4.0
  before it was released, but maybe there opportunities were not well
  known.
 
 
  I think that it was too difficult to get to the dev versions and it was
  too much complicated. There was no clear link to download a dev version
  (I had to rely on the url in the messages from the dev list).
 
 
  This was intended.
 
  We hadn't published the dev builds via Apache or SF mirrors but only on
 2
  people accounts. Apache policy says it's not allowed to publish them
 for a
  wider audience to save the servers from a high traffic load. It's the
 job of
  the mirrors to handle this.
 
 
  What I see (from a standard user point of view) for a RC:
  - When a dev version is almost done, rename it RC and make it known
  (blog and we would relay the announcement in the forums of course)
  - Have a link visible under the main download button of the download
 
 
  Both can be done, depending where the install files are located.
 
 
  page (perhaps a similar button as a dedicated entry)
  - Make that RC installable in parallel with a stable version
  - No file association allowed for that RC by design
 
 
  IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a
 RC
  anymore. One of the RC attributes is to change it into the final release
  with, e.g., just a file name change. But this has to be done without any
  code changes. Otherwise you have to change code parts, build again, test
  again, ... ;-)
 
  But a Beta release could go this separated way.
 
 
  Right.  A release is a release is a release.  The basic requirements
  for every release still apply:
 
  1) 3 PMC +1 votes
  2) Must include source files
  3) Digital signatures, hash files, etc.
 
  But we can have a beta release, that follows these rules, and it
  would be acceptable.  We can then host on the mirrors, publicize, etc.

 There would likely be some restrictions on how many extra downloads
 are permitted.
 For example, the ASF mirrors probably could not cope with a set of
 betas of all the languages for all the OSes in addition to the current
 GA release.


 Looking again at this thread, and knowing the amount of time and resources
 it takes to actually mount an approved release, I wonder if some of the
 issues we had with the 4.0 release could have been addressed by a longer RC
 vote timeframe -- like 2 weeks or so. The vote this time was relatively
 short as I recall on both the original RC and RC2.  We have over 400 people
 on this list. I would hope given a longer test period, we might receive
 better testing and feedback if we took a bit longer for approval.


You can answer that question yourself.  Look at the bugs that we're
fixing in 4.0.1.  Which of those would have been found if the same
project members spent two more weeks reviewing the same code?

I doubt it would have made much of a difference.  The slow Excel
saving bug, for example, was in the code since November 2012.  So this
is not question of time.   I'm hoping we can all look closely at the
facts and come to a similar conclusion.

The basic concept here is defect yield, which is a measure of what %
of the defects in the code are found by a given testing technique.
We can consider the formal QA test cases as one technique.  We can
consider the PMC voting period as another testing technique,
although an informal one.  We could consider beta testing, automated
testing, etc., all as different techniques.

Hopefully it is clear that the defect yield of a technique does not
increase greatly by spending more time doing it.  For example, suppose
our test cases were capable of finding 25% of the bugs in the code.
So we run the test cases once and find 23% of the bugs.  We don't get
25% because of human error in testing.  We could double the amount of
time spent testing and run all the tests again.  That might get us to
24%.   Not a very efficient use of incremental time.

Ditto for PMC voting.  The informal testing is going to just poke at
the surface.  It will have some bug yield, less than the formal
testing did.  But we all have our usage patterns, the subset of
features that we use.   It is very likely that in two weeks we
exercise only slightly more of the features than we did in a one week
voting period.   Diminishing returns.  In any case, we need to escape
the incorrect notion that a RC vote is an invitation to start testing.
 It

Re: Some thoughts on quality

2013-08-16 Thread Rob Weir

On Fri, Aug 16, 2013 at 6:55 AM, Andrea Pescetti pesce...@apache.org wrote:
 On 15/08/2013 Rob Weir wrote:

 A thought experiment:   If we ran the existing test automation on
 4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think
 would be detected?


 I would expect this 4.0.1 blocker issue (wrong results in certain Calc
 functions) to be detectable by automated testing:
 https://issues.apache.org/ooo/show_bug.cgi?id=122997


I don't believe this is actually covered in our test automation.  So
maybe the actual root causes is that there were no test cases for this
function.  If there were test cases then the defect could have been
detected by manual or automated testing.  I agree with you that
spreadsheets functions lend themselves to automated testing.  But the
problem here was not lack of automation.  It was lack of a test case,
period.

-Rob

 Regards,
   Andrea.


 -
 To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
 For additional commands, e-mail: dev-h...@openoffice.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

2013-08-16 Thread Rob Weir

On Fri, Aug 16, 2013 at 1:06 PM, Kay Schenk kay.sch...@gmail.com wrote:
 On Fri, Aug 16, 2013 at 9:47 AM, Rob Weir robw...@apache.org wrote:

 On Fri, Aug 16, 2013 at 12:30 PM, Kay Schenk kay.sch...@gmail.com wrote:
  On Wed, Aug 14, 2013 at 3:35 PM, sebb seb...@gmail.com wrote:
 
  On 14 August 2013 23:10, Rob Weir robw...@apache.org wrote:
   On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) marcus.m...@wtnet.de
  wrote:
   Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
  
   Le 14/08/2013 21:39, Rob Weir a écrit :
  
   Maybe we need to call an earlier build the RC so it will get more
   attention? We had a complete test build that we were testing for
   over a month. But maybe it is ignored unless we call it an RC? In
   other words, there were many opportunities for users to help try
 4.0
   before it was released, but maybe there opportunities were not well
   known.
  
  
   I think that it was too difficult to get to the dev versions and it
 was
   too much complicated. There was no clear link to download a dev
 version
   (I had to rely on the url in the messages from the dev list).
  
  
   This was intended.
  
   We hadn't published the dev builds via Apache or SF mirrors but only
 on
  2
   people accounts. Apache policy says it's not allowed to publish them
  for a
   wider audience to save the servers from a high traffic load. It's the
  job of
   the mirrors to handle this.
  
  
   What I see (from a standard user point of view) for a RC:
   - When a dev version is almost done, rename it RC and make it known
   (blog and we would relay the announcement in the forums of course)
   - Have a link visible under the main download button of the download
  
  
   Both can be done, depending where the install files are located.
  
  
   page (perhaps a similar button as a dedicated entry)
   - Make that RC installable in parallel with a stable version
   - No file association allowed for that RC by design
  
  
   IMHO the last both points doesn't apply to a RC [1] as it wouldn't
 be a
  RC
   anymore. One of the RC attributes is to change it into the final
 release
   with, e.g., just a file name change. But this has to be done without
 any
   code changes. Otherwise you have to change code parts, build again,
 test
   again, ... ;-)
  
   But a Beta release could go this separated way.
  
  
   Right.  A release is a release is a release.  The basic requirements
   for every release still apply:
  
   1) 3 PMC +1 votes
   2) Must include source files
   3) Digital signatures, hash files, etc.
  
   But we can have a beta release, that follows these rules, and it
   would be acceptable.  We can then host on the mirrors, publicize, etc.
 
  There would likely be some restrictions on how many extra downloads
  are permitted.
  For example, the ASF mirrors probably could not cope with a set of
  betas of all the languages for all the OSes in addition to the current
  GA release.
 
 
  Looking again at this thread, and knowing the amount of time and
 resources
  it takes to actually mount an approved release, I wonder if some of the
  issues we had with the 4.0 release could have been addressed by a longer
 RC
  vote timeframe -- like 2 weeks or so. The vote this time was relatively
  short as I recall on both the original RC and RC2.  We have over 400
 people
  on this list. I would hope given a longer test period, we might receive
  better testing and feedback if we took a bit longer for approval.
 

 You can answer that question yourself.  Look at the bugs that we're
 fixing in 4.0.1.  Which of those would have been found if the same
 project members spent two more weeks reviewing the same code?


 Well this was my point. Not the same members, but different members on this
 list who, due to the timeframe we had for voting on the RC, were not
 available for more extensive testing. It is the psychological difference
 between a developer snapshot and an RC candidate.


But then you have the fact that one of the more serious bugs was in
the code since November.  So time alone is not the solution.   It is
how we use the time that counts.

-Rob



 I doubt it would have made much of a difference.  The slow Excel
 saving bug, for example, was in the code since November 2012.  So this
 is not question of time.   I'm hoping we can all look closely at the
 facts and come to a similar conclusion.

 The basic concept here is defect yield, which is a measure of what %
 of the defects in the code are found by a given testing technique.
 We can consider the formal QA test cases as one technique.  We can
 consider the PMC voting period as another testing technique,
 although an informal one.  We could consider beta testing, automated
 testing, etc., all as different techniques.

 Hopefully it is clear that the defect yield of a technique does not
 increase greatly by spending more time doing it.  For example, suppose
 our test cases were capable of finding 25% of the bugs in the code.
 So we run the test cases once

Re: Some thoughts on quality

2013-08-15 Thread Oliver-Rainer Wittmann

Hi,

On 14.08.2013 18:59, Rob Weir wrote:

We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
that we're doing this, and their are no arguments against it, shows
that we value quality. I'd like to take this a step further, and see
what we can learn from the defects in AOO 4.0.0 and what we can do
going forward to improve.

Quality, in the end, is a process, not a state of grace. We improve
by working smarter, not working harder. The goal should be to learn
and improve, as individuals and as a community.

Every regression that made it into 4.0.0 was added there by a
programmer. And the defect went undetected by testers. This is not
to blame. It just means that we're all human. We know that. We all
make mistakes. I make mistakes. A quality process is not about
becoming perfect, but about acknowledging that we make mistakes and
that certain formal and informal practices are needed to prevent and
detect these mistakes.

But enough about generalities. I'm hoping you'll join with me in
examining the 32 confirmed 4.0.0 regression defects and answering a
few questions:

1) What caused the bug? What was the root cause? Note:
programmer error is not really a cause. We should ask what caused
the error.

2) What can we do to prevent bugs like this from being checked in?

3) Why wasn't the bug found during testing? Was it not covered by any
existing test case? Was a test case run but the defect was not
recognized? Was the defect introduced into the software after the
tests had already been executed?

4) What can we do to ensure that bugs like this are caught during testing?

So 2 basic questions -- what went wrong and how can we prevent it in
the future, looked at from perspective of programmers and testers. If
we can keep these questions in mind, and try to answer them, we may be
able to find some patterns that can lead to some process changes for
AOO 4.1.

You can find the 4.0.0 regressions in Bugzilla here:

https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834

Please include also issue with status ACCEPTED.

Best regards, Oliver.

Regards,

-Rob

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

2013-08-15 Thread Jürgen Schmidt

On 8/14/13 8:30 PM, Rob Weir wrote:
On Wed, Aug 14, 2013 at 1:55 PM, janI j...@apache.org wrote:
On 14 August 2013 19:36, Edwin Sharp el...@mail-page.com wrote:

Dear Rob
The 4.0 release was too ambitious - we should advance in smaller steps.
Nothing compares to general public testing - betas and release candidates
should not be avoided.
TestLink cases should be less comprehesive (in terms of feature coverage)
and more stress testing oriented.
Regards,
Edwin

On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
that we're doing this, and their are no arguments against it, shows
that we value quality. I'd like to take this a step further, and see
what we can learn from the defects in AOO 4.0.0 and what we can do
going forward to improve.

Quality, in the end, is a process, not a state of grace. We improve
by working smarter, not working harder. The goal should be to learn
and improve, as individuals and as a community.

But enough about generalities. I'm hoping you'll join with me in
examining the 32 confirmed 4.0.0 regression defects and answering a
few questions:

1) What caused the bug? What was the root cause? Note:
programmer error is not really a cause. We should ask what caused
the error.

2) What can we do to prevent bugs like this from being checked in?

4) What can we do to ensure that bugs like this are caught during
testing?

You can find the 4.0.0 regressions in Bugzilla here:

https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834

Regards,

-Rob

I strongly believe that one of the things that went wrong is our limited
possibility to retest (due to resources), when I look at our current manual

I wonder about that as well. That's one reason it would be good to
know how many of the confirmed regressions were introduced late in the
release process, and thus missed coverage in our full test pass.

testcases, a lot of those could be automated, e.g. with a simple UI macro,
that would enable us to run these test cases with every build. It may sound
like a dream but where I come from, we did that every night, and it caught
a lot of regression bugs and sideeffects.

This begs the question: Is the functionality of the regressions
covered by our test cases? Or are they covered but we didn't execute
them? Or we executed them but didn't recognize the defect? I don't
know (yet).

A simple start, if to request that every bug fix, is issued with at least
one test case (automated or manual).

Often there is, though this information lives in Bugzilla. One thing
we did on another (non open source) project is to mark defects in our
bugtracking system that should become test cases. Not every bug did
that. For example, a defect report to update a mispelling in the UI
would not lead to a new test case. But many would.

we have the automated test framework that needs some more attention and
polishing. And of course the tests have to improved to get satisfying
result.

We have

BVT - build verification test
FVT - functional verification test
PVT - performance verification test
SVT - system verification test

But I have to confess that I have limited knowledge about it yet

Juergen

Regards,

-Rob

rgds
jan I.

-
To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org
For additional commands, e-mail: qa-h...@openoffice.apache.org

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

2013-08-15 Thread janI

On Aug 15, 2013 11:14 AM, Jürgen Schmidt jogischm...@gmail.com wrote:

On 8/14/13 8:30 PM, Rob Weir wrote:
On Wed, Aug 14, 2013 at 1:55 PM, janI j...@apache.org wrote:
On 14 August 2013 19:36, Edwin Sharp el...@mail-page.com wrote:

Dear Rob
The 4.0 release was too ambitious - we should advance in smaller
steps.
Nothing compares to general public testing - betas and release
candidates
should not be avoided.
TestLink cases should be less comprehesive (in terms of feature
coverage)
and more stress testing oriented.
Regards,
Edwin

On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The
fact
that we're doing this, and their are no arguments against it, shows
that we value quality. I'd like to take this a step further, and
see
what we can learn from the defects in AOO 4.0.0 and what we can do
going forward to improve.

Quality, in the end, is a process, not a state of grace. We improve
by working smarter, not working harder. The goal should be to learn
and improve, as individuals and as a community.

But enough about generalities. I'm hoping you'll join with me in
examining the 32 confirmed 4.0.0 regression defects and answering a
few questions:

1) What caused the bug? What was the root cause? Note:
programmer error is not really a cause. We should ask what caused
the error.

2) What can we do to prevent bugs like this from being checked in?

3) Why wasn't the bug found during testing? Was it not covered by
any
existing test case? Was a test case run but the defect was not
recognized? Was the defect introduced into the software after the
tests had already been executed?

4) What can we do to ensure that bugs like this are caught during
testing?

So 2 basic questions -- what went wrong and how can we prevent it in
the future, looked at from perspective of programmers and testers.
If
we can keep these questions in mind, and try to answer them, we may
be
able to find some patterns that can lead to some process changes for
AOO 4.1.

You can find the 4.0.0 regressions in Bugzilla here:

https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834

Regards,

-Rob

I strongly believe that one of the things that went wrong is our
limited
possibility to retest (due to resources), when I look at our current
manual

testcases, a lot of those could be automated, e.g. with a simple UI
macro,
that would enable us to run these test cases with every build. It may
sound
like a dream but where I come from, we did that every night, and it
caught
a lot of regression bugs and sideeffects.

A simple start, if to request that every bug fix, is issued with at
least
one test case (automated or manual).

we have the automated test framework that needs some more attention and
polishing. And of course the tests have to improved to get satisfying
result.

We have

BVT - build verification test
FVT - functional verification test
PVT - performance verification test
SVT - system verification test

But I have to confess that I have limited knowledge about it yet

I aware that we ha a limited automated framework, at least thats what I
found and played with.

but, it is not integrated into our build, or our buildbot. Especially
testing in buildbot gives better qa. An manually controlled automated test
is not really an ideal solution.

Take a look at the manual test cases, a lot could be automated, and free qa
resources for more complex testing.

rgds
jan I

Juergen

Regards,

-Rob

rgds
jan I.

Re: Some thoughts on quality

2013-08-15 Thread Jürgen Schmidt

On 8/15/13 12:19 PM, janI wrote:
On Aug 15, 2013 11:14 AM, Jürgen Schmidt jogischm...@gmail.com wrote: