On Wed, 25 Jul 2018 21:08:57 -0400, Rob Tompkins wrote:
On Jul 24, 2018, at 9:13 PM, Rob Tompkins <chtom...@gmail.com>
wrote:
On Jul 24, 2018, at 7:04 PM, Gilles <gil...@harfang.homelinux.org>
wrote:
Hi Rob.
On Tue, 24 Jul 2018 18:33:40 -0400, Rob Tompkins wrote:
I know that the tests will be necessarily non-deterministic, but
we
can at least get closer to having determinism by running the same
test
1000 times and expecting some reasonable number of passes right?
Could
we use the underlying distribution that we are testing to sort out
this value?
This *is* what the test is doing, although it repeats 50 times
(takes quite some time already) instead of 1000.
As I've reported on this list, it is quite possible that the
failure probabilities are underestimated; (first) review welcome:
the tests are fairly well documented as to what they are doing
but I might have committed some bugs wrt the statistics involved.
Once I get the release out, I’ll have a look.
So the curiosity here is a standard probability problem. It seems
that we have N tests each with some probability of failing P_N. For
some arbitrary test T, P_T is fairly inconsequential, but when
aggregated together with in with P_1, P_2, … , P_{T-1}, P_{T}, ….,
P_N
the probability of failure of test approaches something between 10%
and 50% which is indeed consequential.
If p is the probability that the test will fail, 1-p is
the probability that it'll succeed. The probability that
all N tests succeed is (1-p)^N.
Example from empirical runs: Overall failure is ~25% (3/12 as
per previous post); there ~35 such tests, thus p is ~1%.
We'd have to look for how to reduce this latter value.
Gilles
I’m going to have to think
about this some. If I recall correctly, we could use the central
limit
theorem here about overall test failure, right? Could we apply the
same characteristic to the over all number of tests in the project? I
don’t think we can avoid it. Does surefire accommodate a percentage
of
test failures for passing the build?
-Rob
Cheers,
-Rob
Regards,
Gilles
-Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org
