[
https://issues.apache.org/jira/browse/AVRO-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymie Stata updated AVRO-2269:
-------------------------------
Description:
The class {{org.apache.avro.ipc.io.Perf}} is Avro's performance test suite.
This JIRA aims to make it easier to use. Specifically:
* Added a file {{performance-testing.html}} with guidance on how to use the
suite
* Added script {{run-script.sh}} that uses {{Perf}} to run structured
experiments.
* Added tests for performance of resolution of unchanged unions and
enumerations, which will be subject to future optimizations.
* Tweaks to {{Perf}} for better experimentation (e.g., support for minimum as
well as average aggregation).
was:
In attempting to use Perf.java to show that proposed performance changes
actually improved performance, different runs of Perf.java using the exact same
code base resulted variances of 5% or greater – and often 10% or greater – for
about half the test cases. With variance this high within a code base, it's
impossible to tell if a proposed "improved" code base indeed improves
performance. I will post to the wiki and elsewhere some documents and scripts I
developed to reduce this variance. This JIRA is for changes to Perf.java that
reduce the variance. Specifically:
* Access the {{reader}} and {{writer}} instance variables directly in the
inner-loop for {{SpecificTest}}, as well as switched to a "reuse" object for
reading records, rather than constructing fresh objects for each read. Both
helped to significantly reduce variance for {{FooBarSpecificRecordTestWrite}},
a major target of recent performance-improvement efforts.
* Switched to {{DirectBinaryEncoder}} instead of {{BufferedBinaryEncoder}} for
write tests. Although this slowed writer-tests a bit, it reduced variance a
lot, especially for performance tests of primitives like booleans, making it a
better choice for measuring the performance-impact of code changes.
* Started the timer of a test after the encoder/decoder for the test is
constructed, rather than before. Helps a little.
* Added the ability to output the _minimum_ runtime of a test case across
multiple cycles (vs the total runtime across all cycles). This was inspired by
JVMSpec, which used to use a minimum. I was able to reduce the variance of
total runtime enough to obviate the need for this metric, but since it's
helpful diagnostically, I left it in.
> Improve usability of Perf.java
> ------------------------------
>
> Key: AVRO-2269
> URL: https://issues.apache.org/jira/browse/AVRO-2269
> Project: Apache Avro
> Issue Type: Test
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Priority: Major
>
> The class {{org.apache.avro.ipc.io.Perf}} is Avro's performance test suite.
> This JIRA aims to make it easier to use. Specifically:
> * Added a file {{performance-testing.html}} with guidance on how to use the
> suite
> * Added script {{run-script.sh}} that uses {{Perf}} to run structured
> experiments.
> * Added tests for performance of resolution of unchanged unions and
> enumerations, which will be subject to future optimizations.
> * Tweaks to {{Perf}} for better experimentation (e.g., support for minimum as
> well as average aggregation).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
