I opened a PR after changing the expected failure probability to 1e-5.
I had an old version of the GH build file when I estimated it used 4
runs. The latest CI runs 4 JDKs on 3 platforms plus CodeQL and
coverage. So this is 14 runs. We should see failures with a
probability of:
(1 - (1 -
Hi Alex,
I'd prefer if you could give a shot at adjusting this test when you can take
the time.
TY,
Gary
On 2023/10/20 18:17:35 Alex Herbert wrote:
> On Fri, 20 Oct 2023 at 18:55, Alex Herbert wrote:
> >
> > The chi-square critical value (13.82) is correct:
> >
> > >>> from scipy.stats import
On Fri, 20 Oct 2023 at 18:55, Alex Herbert wrote:
>
> The chi-square critical value (13.82) is correct:
>
> >>> from scipy.stats import chi2
> >>> chi2(2).isf(0.001)
> 13.815510557964274
>
> The test seems to fail with the expected frequency when run locally. I
> annotated with:
>
>
The chi-square critical value (13.82) is correct:
>>> from scipy.stats import chi2
>>> chi2(2).isf(0.001)
13.815510557964274
The test seems to fail with the expected frequency when run locally. I
annotated with:
@RepeatedTest(value = 10)
I observe 93 failures (just under 1 in 1000). So it
It's possible the chi square test is miscalculated. Perhaps some stats
expert can check that. It's also possible the chi square test isn't
the right one to use here. Again, consult a stats expert.
It's also very possible that the randomness is not nearly as random as
it's supposed to be. That's
Despite the failure comment:
RandomStringUtilsTest.testRandomStringUtilsHomog:474 test homogeneity -- will
fail about 1 in 1000 times ==> expected: but was:
This test fails a LOT more than once every 1000 times, based on how many GitHub
builds I need to restart every week.
What can be done
