Alex Herbert created STATISTICS-99:
--------------------------------------
Summary: The MannWhitneyUTest can allocate very large memory to
compute the exact p-value
Key: STATISTICS-99
URL: https://issues.apache.org/jira/browse/STATISTICS-99
Project: Commons Statistics
Issue Type: Bug
Components: inference
Affects Versions: 1.3
Reporter: Alex Herbert
The MannWhitneyUTest can compute an exact p-value using a method that requires
tabulation of double values. The default limit for the AUTO configuration will
consume a maximum of approximately 2.86MiB of memory before switching to the
asymptotic approximation when max(n, m) >= 50.
A user can configure the test to use the exact p-value computation for any size
(n, m). The binomial coefficient value binom(n + m, m) limits the size that can
be used to compute the p-value. The largest value of n+m for which all
coefficients can fit into a double is 1029. Larger n+m may result in infinity
depending on the value of m. Any min(m, n) >= 515 is infinite and will not call
the exact computation.
For a m=n=514 the computation is possible and the maximum memory is
approximately 32.6 GiB.
A suggested fix it to have a default value for the maximum space to allocate
for the exact p-value computation. The test suite uses allocations up to 389MiB
or 5.1e7 doubles. A limit of 512MiB allows reasonable usage of the exact
p-value computation but will require a user to opt in to large allocations.
Issue identified using a security scan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
