> On Jul 24, 2018, at 9:13 PM, Rob Tompkins <chtom...@gmail.com> wrote:
>
>
>
>> On Jul 24, 2018, at 7:04 PM, Gilles <gil...@harfang.homelinux.org> wrote:
>>
>> Hi Rob.
>>
>> On Tue, 24 Jul 2018 18:33:40 -0400, Rob Tompkins wrote:
>>> I know that the tests will be necessarily non-deterministic, but we
>>> can at least get closer to having determinism by running the same test
>>> 1000 times and expecting some reasonable number of passes right? Could
>>> we use the underlying distribution that we are testing to sort out
>>> this value?
>>
>> This *is* what the test is doing, although it repeats 50 times
>> (takes quite some time already) instead of 1000.
>> As I've reported on this list, it is quite possible that the
>> failure probabilities are underestimated; (first) review welcome:
>> the tests are fairly well documented as to what they are doing
>> but I might have committed some bugs wrt the statistics involved.
>
> Once I get the release out, I’ll have a look.
So the curiosity here is a standard probability problem. It seems that we have
N tests each with some probability of failing P_N. For some arbitrary test T,
P_T is fairly inconsequential, but when aggregated together with in with P_1,
P_2, … , P_{T-1}, P_{T}, …., P_N the probability of failure of test approaches
something between 10% and 50% which is indeed consequential. I’m going to have
to think about this some. If I recall correctly, we could use the central limit
theorem here about overall test failure, right? Could we apply the same
characteristic to the over all number of tests in the project? I don’t think we
can avoid it. Does surefire accommodate a percentage of test failures for
passing the build?
-Rob
>
> Cheers,
> -Rob
>
>>
>> Regards,
>> Gilles
>>
>>>
>>> -Rob
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>> For additional commands, e-mail: dev-h...@commons.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org
