Alex Herbert created STATISTICS-59:
--------------------------------------
Summary: Correct Pareto distribution sampling with extreme shape
parameter
Key: STATISTICS-59
URL: https://issues.apache.org/jira/browse/STATISTICS-59
Project: Commons Statistics
Issue Type: Improvement
Components: distribution
Affects Versions: 1.0
Reporter: Alex Herbert
Fix For: 1.0
The Pareto distribution has CDF:
{noformat}
( scale )^shape
CDF(x) = 1 - ( ----- )
( x ){noformat}
This is inverted using high precision Math functions to support very small p
values:
{noformat}
x = scale / exp(log(1 - p) / shape)
= scale / Math.exp(Math.log1p(-p) / shape);{noformat}
This is sampled using inverse transform sampling as:
{noformat}
x = scale / (1 - p)^(1 / shape)
= scale / Math.pow(1 - p, 1 / shape){noformat}
This is fast as it requires a single call to Math.pow. It must only handle
p-values down to 2^-53 as sampling generates p as one of the 2^53 dyadic
rationals in [0, 1).
However it has some issues when the shape parameter is extreme: either shape is
infinite or 1 / shape is infinite.
Here is a table of the inverse CDF and the sample value for scale = 1 and an
extreme shape. p has been set using the most extreme values from the dyadic
rationals (0, 2^-53, 1 - 2^-53, 1):
||Shape||p||icdf(p)||sample||
|Infinity|0.0|1.0|1.0|
|Infinity|1.1102230246251565E-16|1.0|1.0|
|Infinity|0.9999999999999999|1.0|1.0|
|Infinity|1.0|Infinity|1.0|
|4.9E-324|0.0|1.0|NaN|
|4.9E-324|1.1102230246251565E-16|Infinity|Infinity|
|4.9E-324|0.9999999999999999|Infinity|Infinity|
|4.9E-324|1.0|Infinity|Infinity|
When 1 / shape is infinite the NaN occurs when Math.pow(1, Infinity) == NaN. In
this case sampling inversion is an error.
When shape is infinite the mismatch occurs when Math.pow(0, 0) == 1 and the
shape is returned rather than the distribution upper bound. This is because the
inverse CDF detects this edge case when the input p=1. In this case pure
inversion of the CDF is creating an outlier and the sampling inversion is more
suitable.
The sampling should be updated to avoid the possibility of NaN generation and
ensure samples are returned without outliers from the main region of the CDF.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
