Re: [sqlite] MC/DC coverage explained wrong in the home page?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/25/2011 10:40 AM, Richard Hipp wrote: > 100% MC/DC has worked very, very well at finding and preventing bugs in > SQLite. The one place it doesn't help is when error returns of functions are ignored. For example this happens when xCurrentTime

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Sun, Sep 25, 2011 at 9:03 AM, Sami Liedes wrote: > Note that I'm decidedly not trying to claim MC/DC coverage as a metric > > * makes any sense > * is reasonable in the case of SQLite > * is ever reasonable for any program > > Actually in my opinion it's probably too

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Sat, Sep 24, 2011 at 10:05 PM, Pavel Ivanov wrote: > > > Dead code is indicative of a bug. Usually programmers write code only > > in the expectation that it be run. The same can be said of conditions > > in an if statement, and what the compiler does to it doesn't change

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Sat, Sep 24, 2011 at 10:05:36PM -0400, Pavel Ivanov wrote: > No, that's exactly the point that Richard tries to explain to you and > you don't seem to understand. SQLite's testing coverage is a coverage > of the resulting object code, not some source code lines. And so it Yes, I now realize

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Sat, Sep 24, 2011 at 7:35 AM, Sami Liedes wrote: > On Fri, Sep 23, 2011 at 10:26:43PM -0400, Pavel Ivanov wrote: >> >   if (A || 1) ... >> > >> > You can get (e) by giving test cases for A and !A, but most certainly >> > flipping A does not "independently affect the outcome"

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Sat, Sep 24, 2011 at 9:50 AM, Stephan Beal wrote: > On Sat, Sep 24, 2011 at 3:17 AM, Richard Hipp wrote: > > > The object-code branch coverage testing of SQLite clearly fulfills > > guidelines a, b, c, d, and e. The question at hand is does 100%

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Sat, Sep 24, 2011 at 3:17 AM, Richard Hipp wrote: > The object-code branch coverage testing of SQLite clearly fulfills > guidelines a, b, c, d, and e. The question at hand is does 100% branch > coverage fulfill f in a short-circuit language such as C/C++. Note that > the >

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Sat, Sep 24, 2011 at 7:49 AM, Sami Liedes wrote: > > The example is a four-input AND gate with short-circuit logic (_ = > don't care/not evaluated). This would be a sufficient set of tests: > > expression = (A && B && C && D) > > #A B C D Result > 11 1 1 1 1

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Sat, Sep 24, 2011 at 04:58:35AM +0300, Sami Liedes wrote: > On Fri, Sep 23, 2011 at 09:17:43PM -0400, Richard Hipp wrote: > > -- > > 1. Structural coverage guidelines are: > > a) Every statement in the program has been invoked at least once; > > b) Every point of entry

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Fri, Sep 23, 2011 at 10:26:43PM -0400, Pavel Ivanov wrote: > >   if (A || 1) ... > > > > You can get (e) by giving test cases for A and !A, but most certainly > > flipping A does not "independently affect the outcome" as required by > > the plain reading of (f). > > I'm pretty sure that the

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Sep 23, 2011, at 9:17 PM, Richard Hipp wrote: > paper above completely ignores this issue. It is as if the authors had > never heard of short-circuit evaluation. Or, perhaps they are familiar with > the problem but could not reach agreement on its solution so simply didn't > bring it up.

Re: [sqlite] MC/DC coverage explained wrong in the home page?

>   if (A || 1) ... > > You can get (e) by giving test cases for A and !A, but most certainly > flipping A does not "independently affect the outcome" as required by > the plain reading of (f). I'm pretty sure that the latest versions of modern compilers will optimize the above if statement to

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Fri, Sep 23, 2011 at 09:17:43PM -0400, Richard Hipp wrote: > -- > 1. Structural coverage guidelines are: > a) Every statement in the program has been invoked at least once; > b) Every point of entry and exit in the program has been invoked at least > once; > c) Every

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Fri, Sep 23, 2011 at 5:57 PM, Sami Liedes wrote: > [Note: In case my explanations are not clear enough, there's a fairly > formal position paper by Certification Authorities Software Team > (CAST-10) clarifying MC/DC here: > > >

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On 23 Sep 2011, at 11:30pm, Darren Duncan wrote: > This is why I think it is valuable for a programming language to provide > multiple versions of some operations such as boolean "and","or" where one > variant doesn't short-circuit and the other does. > > The primary purpose, then, of

Re: [sqlite] MC/DC coverage explained wrong in the home page?

Richard Hipp wrote: Opinions vary on the exact meaning of MC/DC for a language (such as C) that has short-circuit boolean operators. There are problems with this view, though. In many instances, B is undefined when A is false. In other words, if A is false, any attempt to calculate B will

Re: [sqlite] MC/DC coverage explained wrong in the home page?

[Note: In case my explanations are not clear enough, there's a fairly formal position paper by Certification Authorities Software Team (CAST-10) clarifying MC/DC here: http://www.faa.gov/aircraft/air_cert/design_approvals/air_software/cast/cast_papers/media/cast-10.pdf ] On Fri, Sep 23,

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Fri, Sep 23, 2011 at 1:51 PM, Sami Liedes wrote: > > > For a decision of the form: > > > > if( A && B )... > > > > The test suite for SQLite tries at least three cases: > > > >(1) A false > >(2) A true and B false > >(3) A true and B true > > > > You

Re: [sqlite] MC/DC coverage explained wrong in the home page?

On Fri, Sep 23, 2011 at 11:16:37AM -0400, Richard Hipp wrote: > Opinions vary on the exact meaning of MC/DC for a language (such as C) that > has short-circuit boolean operators. You are advocating a more rigorous > view of MC/DC that what I have heard before. This is not to say it is > wrong,

Re: [sqlite] MC/DC coverage explained wrong in the home page?

Opinions vary on the exact meaning of MC/DC for a language (such as C) that has short-circuit boolean operators. You are advocating a more rigorous view of MC/DC that what I have heard before. This is not to say it is wrong, only different. For a decision of the form: if( A && B )... The

[sqlite] MC/DC coverage explained wrong in the home page?

Hi! Looking at http://www.sqlite.org/testing.html it seems to me that MC/DC coverage is explained wrong there. This in turn makes me wonder if SQLite tests really have 100% MC/DC coverage or if this claim is just based on mistaken understanding of MC/DC. The page explains: