Dan Smith created GEODE-6203:
--------------------------------
Summary: Include uncertainity values to anaylzeRun results
Key: GEODE-6203
URL: https://issues.apache.org/jira/browse/GEODE-6203
Project: Geode
Issue Type: Task
Components: benchmarks
Reporter: Dan Smith
It's hard to tell from the results of analyzeRun whether differences between
runs are statistically significant or not.
As first cut, we should try to include a confidence interval with each of our
measurements.
We can use 3 * the standard error (99.7% confidence interval) for the
throughput and mean latency measurements, and propagate the error to the ratio
between two runs. That should give us a confidence interval for the ratio as
well which will help us determine if the changes have a statistically
significant difference.
After making these changes, we should investigate to see if multiple runs on
the same hardware remain within this confidence interval. This methodology may
not work, since all of the measurements we take within a single run aren't
really independent. If the multiple runs don't stay within the interval, we may
need to consider running each benchmark multiple times like JMH to produce a
more accurate error estimate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
