Hi.
(1)
I suspect there might be a bug at line 207 in file "RandomDataTest.java"
(package "o.a.c.m.random"), indicate by "HERE" in the excerpt below:
---CUT---
private void checkNextLongUniform(int min, int max) throws Exception {
final Frequency freq = new Frequency();
for (int i = 0; i < smallSampleSize; i++) {
final long value = randomData.nextLong(min, max);
Assert.assertTrue("nextLong range", (value >= min) && (value <=
max));
freq.addValue(value);
}
final int len = max - min + 1;
final long[] observed = new long[len];
for (int i = 0; i < len; i++) {
observed[i] = freq.getCount(min + i);
}
final double[] expected = new double[len];
for (int i = 0; i < len; i++) {
expected[i] = 1d / len; // <---- HERE
// expected[i] = ((double) smallSampleSize) / len;
}
TestUtils.assertChiSquareAccept(expected, observed, 0.01);
}
---CUT---
When I run the "ISAACTest" (using a modified version of "ISAACRandom"), I
get one failure. The 2 attached files show the output of the junit run, the
first with the original line, the second with the modified line (commented
out in the above). The "expected" and "observed" arrays are somehow
compared, in the last statement, but their content is not of the same nature
(frequencies vs counts).
If this bug is confirmed, there are probably other similar ones in that same
file.
===Update===
Digging further, it turns out that the "ChiSquareTest.chiSquareTest" method
performs a rescaling of the "expected" data.
Thus, it is not a bug here; but nevertheless it would be clearer to not let
people think that we are comparing oranges and apples...
[It would also avoid the confusion when the output af failed test appears on
the console (cf. attachments).]
(2)
Then, I have a hard time understanding the "TestUtils.assertChiSquareAccept"
statement.
In "TestUtils":
---CUT---
// Fail if we can reject null hypothesis that distributions are the same
if (chiSquareTest.chiSquareTest(expected, observed, alpha)) {
---CUT---
In "ChiSquareTest" (excerpt of Javadoc for method "chiSquareTest"):
---CUT---
* Chi-square goodness of fit test</a> evaluating the null hypothesis
* that the observed counts conform to the frequency distribution described
by the expected
* counts, with significance level <code>alpha</code>. Returns true iff
the null
* hypothesis can be rejected with 100 * (1 - alpha) percent confidence.
---CUT---
If "alpha" is set to "0.01", the unit test fails; if set to "0.001", the
unit test succeeds.
In the latter case,
We cannot reject with 99.9 % confidence that the distributions are the same.
In the former case,
We can reject with 99 % confidence that the distributions are the same.
Can someone explain how this proves the assumption of the unit test (i.e.
that the sequence of numbers produced is distributed uniformly)?
My worry is that the test fails for many choices of seeds. And I don't know
how to figure out whether the unit test is wrong, or the code is wrong, or
it is normal that the choice of seed has such an influence on the test (in
which case it should be documented that not all seeds are equal...).
Thanks,
Gilles
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Commons Math
[INFO] task-segment: [test]
[INFO] ------------------------------------------------------------------------
[INFO] [antrun:run {execution: javadoc.resources}]
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO] Setting property: classpath.resource.loader.class =>
'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
[INFO] Setting property: velocimacro.messages.on => 'false'.
[INFO] Setting property: resource.loader => 'classpath'.
[INFO] Setting property: resource.manager.logwhenfound => 'false'.
[INFO] [remote-resources:process {execution: default}]
[INFO] [resources:resources {execution: default-resources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources to META-INF
[INFO] Copying 5 resources
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Compiling 46 source files to
/home/gilles/devel/SVN/commons-math/trunk/target/classes
[INFO] [bundle:manifest {execution: bundle-manifest}]
[INFO] [resources:testResources {execution: default-testResources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 13 resources
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [surefire:test {execution: default-test}]
[INFO] Surefire report directory:
/home/gilles/devel/SVN/commons-math/trunk/target/surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.commons.math.random.ISAACTest
Tests run: 60, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.18 sec <<<
FAILURE!
Results :
Failed tests:
testNextLongNegativeRange(org.apache.commons.math.random.ISAACTest):
Chisquare test failed p-value = 0.0015147105753583467 chisquare statistic =
33.33999999999999.
value expected observed
1 0.07 67
2 0.07 75
3 0.07 72
4 0.07 77
5 0.07 72
6 0.07 57
7 0.07 60
8 0.07 62
9 0.07 82
10 0.07 72
11 0.07 106
12 0.07 55
13 0.07 84
14 0.07 59
This test can fail randomly due to sampling error with probability 0.01.
Tests run: 60, Failures: 1, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] There are test failures.
Please refer to
/home/gilles/devel/SVN/commons-math/trunk/target/surefire-reports for the
individual test results.
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10 seconds
[INFO] Finished at: Fri Jan 27 14:49:20 CET 2012
[INFO] Final Memory: 35M/300M
[INFO] ------------------------------------------------------------------------
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Commons Math
[INFO] task-segment: [test]
[INFO] ------------------------------------------------------------------------
[INFO] [antrun:run {execution: javadoc.resources}]
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO] Setting property: classpath.resource.loader.class =>
'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
[INFO] Setting property: velocimacro.messages.on => 'false'.
[INFO] Setting property: resource.loader => 'classpath'.
[INFO] Setting property: resource.manager.logwhenfound => 'false'.
[INFO] [remote-resources:process {execution: default}]
[INFO] [resources:resources {execution: default-resources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources to META-INF
[INFO] Copying 5 resources
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Compiling 46 source files to
/home/gilles/devel/SVN/commons-math/trunk/target/classes
[INFO] [bundle:manifest {execution: bundle-manifest}]
[INFO] [resources:testResources {execution: default-testResources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 13 resources
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [surefire:test {execution: default-test}]
[INFO] Surefire report directory:
/home/gilles/devel/SVN/commons-math/trunk/target/surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.commons.math.random.ISAACTest
Tests run: 60, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.075 sec <<<
FAILURE!
Results :
Failed tests:
testNextLongNegativeRange(org.apache.commons.math.random.ISAACTest):
Chisquare test failed p-value = 0.0015147105753583467 chisquare statistic =
33.34.
value expected observed
1 71.43 67
2 71.43 75
3 71.43 72
4 71.43 77
5 71.43 72
6 71.43 57
7 71.43 60
8 71.43 62
9 71.43 82
10 71.43 72
11 71.43 106
12 71.43 55
13 71.43 84
14 71.43 59
This test can fail randomly due to sampling error with probability 0.01.
Tests run: 60, Failures: 1, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] There are test failures.
Please refer to
/home/gilles/devel/SVN/commons-math/trunk/target/surefire-reports for the
individual test results.
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10 seconds
[INFO] Finished at: Fri Jan 27 14:48:36 CET 2012
[INFO] Final Memory: 34M/300M
[INFO] ------------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org
