Raymie Stata created AVRO-2269:
----------------------------------
Summary: Improve variances seen across Perf.java runs
Key: AVRO-2269
URL: https://issues.apache.org/jira/browse/AVRO-2269
Project: Apache Avro
Issue Type: Test
Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
In attempting to use Perf.java to show that proposed performance changes
actually improved performance, different runs of Perf.java using the exact same
code base resulted variances of 5% or greater – and often 10% or greater – for
about half the test cases. With variance this high within a code base, it's
impossible to tell if a proposed "improved" code base indeed improves
performance. I will post to the wiki and elsewhere some documents and scripts I
developed to reduce this variance. This JIRA is for changes to Perf.java that
reduce the variance. Specifically:
* Access the {{reader}} and {{writer}} instance variables directly in the
inner-loop for {{SpecificTest}}, as well as switched to a "reuse" object for
reading records, rather than constructing fresh objects for each read. Both
helped to significantly reduce variance for {{FooBarSpecificRecordTestWrite}},
a major target of recent performance-improvement efforts.
* Switched to {{DirectBinaryEncoder}} instead of {{BufferedBinaryEncoder}} for
write tests. Although this slowed writer-tests a bit, it reduced variance a
lot, especially for performance tests of primitives like booleans, making it a
better choice for measuring the performance-impact of code changes.
* Started the timer of a test after the encoder/decoder for the test is
constructed, rather than before. Helps a little.
* Added the ability to output the _minimum_ runtime of a test case across
multiple cycles (vs the total runtime across all cycles). This was inspired by
JVMSpec, which used to use a minimum. I was able to reduce the variance of
total runtime enough to obviate the need for this metric, but since it's
helpful diagnostically, I left it in.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
