[
https://issues.apache.org/jira/browse/STATISTICS-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109517#comment-18109517
]
Alex Herbert commented on STATISTICS-99:
----------------------------------------
The exact p-value computation tabulates a large amount of data. This is stored
in a system wide
[SoftReference|https://docs.oracle.com/en/java/javase/25/docs/api/java.base/java/lang/ref/SoftReference.html].
This allows repeat calls to the MannWhitneyUTest to compute the exact p-value
with reduced overhead as the computation is reused.
The memory can be cleared by the JVM if the allocation is softly reachable
before throwing an out-of-memory (OOM) error. However clearing the memory may
not always be possible, and large allocations can create OOM errors.
Since the memory allocation is global to the JVM, a configurable property
cannot be added to an instance of the MannWhitneyUTest. It must be added at a
static level.
There a few options to fix this:
# Set a hard limit that is not configurable (e.g. 512 MiB).
# Use a system property that sets the limit for the JVM when the
MannWhitneyUTest is initialised.
# Add static methods to control the limit.
Option 3 requires tracking the current allocation size, and managing it
whenever updates to the limit are made via the methods. I prefer the reasonable
hard limit and an option to increase it via a system property.
Note that precedence for use of an environment variable in Commons Math
libraries is the control of the backend for the
o.a.c.math4.core.jdkmath.JdkMath API. This uses an environment variable
org.apache.commons.math.jdkmath to switch the implementation at class
initialisation between Commons Math AccurateMath and the JDK Math methods.
> The MannWhitneyUTest can allocate very large memory to compute the exact
> p-value
> --------------------------------------------------------------------------------
>
> Key: STATISTICS-99
> URL: https://issues.apache.org/jira/browse/STATISTICS-99
> Project: Commons Statistics
> Issue Type: Bug
> Components: inference
> Affects Versions: 1.3
> Reporter: Alex Herbert
> Priority: Minor
>
> The MannWhitneyUTest can compute an exact p-value using a method that
> requires tabulation of double values. The default limit for the AUTO
> configuration will consume a maximum of approximately 2.86MiB of memory
> before switching to the asymptotic approximation when max(n, m) >= 50.
> A user can configure the test to use the exact p-value computation for any
> size (n, m). The binomial coefficient value binom(n + m, m) limits the size
> that can be used to compute the p-value. The largest value of n+m for which
> all coefficients can fit into a double is 1029. Larger n+m may result in
> infinity depending on the value of m. Any min(m, n) >= 515 is infinite and
> will not call the exact computation.
> For a m=n=514 the computation is possible and the maximum memory is
> approximately 32.6 GiB.
> A suggested fix it to have a default value for the maximum space to allocate
> for the exact p-value computation. The test suite uses allocations up to
> 389MiB or 5.1e7 doubles. A limit of 512MiB allows reasonable usage of the
> exact p-value computation but will require a user to opt in to large
> allocations.
> Issue identified using a security scan.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
