[
https://issues.apache.org/jira/browse/FLINK-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber closed FLINK-12983.
-------------------------------
Fix Version/s: 1.10.0
Resolution: Fixed
merged to master via f57a615
> Replace descriptive histogram's storage back-end
> ------------------------------------------------
>
> Key: FLINK-12983
> URL: https://issues.apache.org/jira/browse/FLINK-12983
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Metrics
> Reporter: Nico Kruber
> Assignee: Nico Kruber
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{DescriptiveStatistics}} relies on their {{ResizableDoubleArray}} for
> storing double values for their histograms. However, this is constantly
> resizing an internal array and seems to have quite some overhead.
> Additionally, we're not using {{SynchronizedDescriptiveStatistics}} which,
> according to its docs, we should. Currently, we seem to be somewhat safe
> because {{ResizableDoubleArray}} has some synchronized parts but these are
> scheduled to go away with commons.math version 4.
> Internal tests with the current implementation, one based on a linear array
> of twice the histogram size (and moving values back to the start once the
> window reaches the end), and one using a circular array (wrapping around with
> flexible start position) has shown these numbers using the optimised code
> from FLINK-10236, FLINK-12981, and FLINK-12982:
> # only adding values to the histogram
> {code}
> Benchmark Mode Cnt Score
> Error Units
> HistogramBenchmarks.dropwizardHistogramAdd thrpt 30 47985.359 ±
> 25.847 ops/ms
> HistogramBenchmarks.descriptiveHistogramAdd thrpt 30 70158.792 ±
> 276.858 ops/ms
> --- with FLINK-10236, FLINK-12981, and FLINK-12982 ---
> HistogramBenchmarks.descriptiveHistogramAdd thrpt 30 75303.040 ±
> 475.355 ops/ms
> HistogramBenchmarks.descrHistogramCircularAdd thrpt 30 200906.902 ±
> 384.483 ops/ms
> HistogramBenchmarks.descrHistogramLinearAdd thrpt 30 189788.728 ±
> 233.283 ops/ms
> {code}
> # after adding each value, also retrieving a common set of metrics:
> {code}
> Benchmark Mode Cnt Score
> Error Units
> HistogramBenchmarks.dropwizardHistogram thrpt 30 400.274 ±
> 4.930 ops/ms
> HistogramBenchmarks.descriptiveHistogram thrpt 30 124.533 ±
> 1.060 ops/ms
> --- with FLINK-10236, FLINK-12981, and FLINK-12982 ---
> HistogramBenchmarks.descriptiveHistogram thrpt 30 251.895 ±
> 1.809 ops/ms
> HistogramBenchmarks.descrHistogramCircular thrpt 30 301.068 ±
> 2.077 ops/ms
> HistogramBenchmarks.descrHistogramLinear thrpt 30 234.050 ±
> 5.485 ops/ms
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
