[ https://issues.apache.org/jira/browse/AVRO-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16698096#comment-16698096 ]
Raymie Stata commented on AVRO-2269: ------------------------------------ This work has changed direction. The focus shifted away from variance and towards usability. I've updated the subject and description accordingly. > Improve usability of Perf.java > ------------------------------ > > Key: AVRO-2269 > URL: https://issues.apache.org/jira/browse/AVRO-2269 > Project: Apache Avro > Issue Type: Test > Components: java > Reporter: Raymie Stata > Assignee: Raymie Stata > Priority: Major > > In attempting to use Perf.java to show that proposed performance changes > actually improved performance, different runs of Perf.java using the exact > same code base resulted variances of 5% or greater – and often 10% or greater > – for about half the test cases. With variance this high within a code base, > it's impossible to tell if a proposed "improved" code base indeed improves > performance. I will post to the wiki and elsewhere some documents and scripts > I developed to reduce this variance. This JIRA is for changes to Perf.java > that reduce the variance. Specifically: > * Access the {{reader}} and {{writer}} instance variables directly in the > inner-loop for {{SpecificTest}}, as well as switched to a "reuse" object for > reading records, rather than constructing fresh objects for each read. Both > helped to significantly reduce variance for > {{FooBarSpecificRecordTestWrite}}, a major target of recent > performance-improvement efforts. > * Switched to {{DirectBinaryEncoder}} instead of {{BufferedBinaryEncoder}} > for write tests. Although this slowed writer-tests a bit, it reduced variance > a lot, especially for performance tests of primitives like booleans, making > it a better choice for measuring the performance-impact of code changes. > * Started the timer of a test after the encoder/decoder for the test is > constructed, rather than before. Helps a little. > * Added the ability to output the _minimum_ runtime of a test case across > multiple cycles (vs the total runtime across all cycles). This was inspired > by JVMSpec, which used to use a minimum. I was able to reduce the variance > of total runtime enough to obviate the need for this metric, but since it's > helpful diagnostically, I left it in. -- This message was sent by Atlassian JIRA (v7.6.3#76005)