[
https://issues.apache.org/jira/browse/STATISTICS-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803538#comment-17803538
]
Alex Herbert commented on STATISTICS-84:
----------------------------------------
I created a test using the following methods:
||Method ||Description||
|ES|Compute the mean from an extended precision sum|
|ESC|Use ES then a second pass correction of the mean|
|R|Use the rolling mean algorithm|
|RC|Use R then a second pass correction of the mean|
|S|Compute the mean from a standard sum|
|SC|Use S then a second pass correction of the mean|
The test generates random data from a standard normal distribution centred on a
specified mean. The mean was created using a linear space for an exponent of 0
to 16 as 10^-exp. Thus the mean ranges from 1 to 1e-16. As the mean approaches
zero the condition number increases. 200 repeats of 4096 samples were made at
each mean.
I coded all algorithms in float precision. The following shows the mean
relative error of each algorithm against the condition number:
!mean_float_4096-1.png|width=810,height=600!
Notes:
* The ES method is not visible. It always has an error of zero.
* Use of the correction makes the ES mean worse.
* The correction helps the S and R methods at a low condition number (up to
about 150).
* At high condition number the correction is not reducing error. Since it does
not make the result less accurate it is adding random noise to the result.
The same test using algorithms coded in double precision with a mean exponent
in 0 to 32:
!mean_double_4096.png|width=810,height=600!
The same behaviour is seen, only with a different relative error reflecting the
double precision.
h2. Conclusion
The two-pass correction method does not help a mean computation based on an
extended precision sum. The two-pass method can be used to improve the result
computed from a simple sum or the rolling mean algorithm. When the data is
badly conditioned then the second pass does not change the precision.
The Mean should be updated to create the mean from an array using an extended
precision sum. The rolling algorithm with a second pass correction can be used
when the sum is not finite.
> Create the Mean from array input using a summation
> --------------------------------------------------
>
> Key: STATISTICS-84
> URL: https://issues.apache.org/jira/browse/STATISTICS-84
> Project: Commons Statistics
> Issue Type: Improvement
> Components: descriptive
> Reporter: Alex Herbert
> Priority: Trivial
> Attachments: mean_double_4096.png, mean_float_4096-1.png,
> mean_float_4096.png
>
>
> The creation of the Mean from an array uses the rolling mean algorithm. This
> requires a divide for each input value. The algorithm is slow when compared
> to a simple sum of the values and a divide of the sum. The advantage is that
> the algorithm is overflow safe and computes the mean in a single pass (useful
> for streams).
> In the majority of use cases a sum will not overflow. Change the array
> construction to attempt a sum to compute the mean, reverting to the rolling
> mean if this fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
