[
https://issues.apache.org/jira/browse/FLINK-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159033#comment-17159033
]
Roman Khachatryan edited comment on FLINK-18614 at 7/16/20, 9:06 AM:
---------------------------------------------------------------------
I [submitted|http://codespeed.dak8s.net:8080/job/flink-benchmark-request/194/]
an older version (before the regression - f730e16) to benchmark again.
Compared to the
[previous|http://http//codespeed.dak8s.net:8080/job/flink-master-benchmarks/6187/]
benchmark of this f730e16 commit, it shows significant regression in all
benchmarks:
|Benchmark|f730e16 Jul 06|f730e16 Jul 15|delta|params|
|TwoInputBenchmark.twoInputMapSink|19554.543834|12433.798288|-36.41%| |
|SerializationFrameworkMiniBenchmarks.serializerPojo|697.942095|504.415277|-27.73%|
|
|KeyByBenchmarks.tupleKeyBy|6326.143904|4638.543555|-26.68%| |
|BlockingPartitionBenchmark.uncompressedMmapPartition|9291.513297|6898.695382|-25.75%|
|
|SerializationFrameworkMiniBenchmarks.serializerRow|862.468117|671.346967|-22.16%|
|
|KeyByBenchmarks.arrayKeyBy|3549.952163|2797.727893|-21.19%| |
|SerializationFrameworkMiniBenchmarks.serializerTuple|993.588036|792.453082|-20.24%|
|
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|75836.784163|61426.172823|-19.00%|100,100ms|
|RemoteChannelThroughputBenchmark.remoteRebalance|18300.374713|14980.789678|-18.14%|
|
|TwoInputBenchmark.twoInputOneIdleMapSink|11798.691534|9793.882213|-16.99%| |
|SerializationFrameworkMiniBenchmarks.serializerAvro|568.687328|481.019554|-15.42%|
|
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|14136.11755|12088.743814|-14.48%|1000,100ms,SSL|
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|45936.696655|39393.861324|-14.24%|1000,1ms|
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|17105.577957|14726.522754|-13.91%|100,100ms,SSL|
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|51594.0401|44693.704593|-13.37%|1000,100ms|
|MemoryStateBackendBenchmark.stateBackends|4682.529423|4073.686595|-13.00%|MEMORY|
|SerializationFrameworkMiniBenchmarks.serializerKryo|244.001776|212.473621|-12.92%|
|
|MemoryStateBackendBenchmark.stateBackends|4335.240886|3787.143054|-12.64%|FS|
|BlockingPartitionBenchmark.uncompressedFilePartition|8988.964525|7930.936505|-11.77%|
|
|InputBenchmark.mapRebalanceMapSink|10833.173473|9584.143495|-11.53%| |
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|37876.283084|33631.728543|-11.21%|1000,100ms,OpenSSL|
|WindowBenchmarks.globalWindow|7398.471169|6570.142998|-11.20%| |
|BlockingPartitionBenchmark.compressedFilePartition|8269.729104|7400.71175|-10.51%|
|
|MemoryStateBackendBenchmark.stateBackends|4726.982779|4278.369844|-9.49%|FS_ASYNC|
|SerializationFrameworkMiniBenchmarks.serializerHeavyString|242.494046|222.855766|-8.10%|
|
|StreamNetworkBroadcastThroughputBenchmarkExecutor.networkBroadcastThroughput|1085.52807|998.249968|-8.04%|
|
|InputBenchmark.mapSink|16594.379276|15328.197699|-7.63%| |
|WindowBenchmarks.sessionWindow|891.5685|829.102496|-7.01%| |
|RocksStateBackendBenchmark.stateBackends|325.958457|305.387748|-6.31%|ROCKS|
|AsyncWaitOperatorBenchmark.asyncWait|1654.341203|1550.847558|-6.26%|UNORDERED|
|WindowBenchmarks.tumblingWindow|4703.042299|4566.554823|-2.90%| |
|DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput|32376.829723|31532.9253|-2.61%|
|
|WindowBenchmarks.slidingWindow|749.309761|736.68323|-1.69%| |
|AsyncWaitOperatorBenchmark.asyncWait|1634.387558|1608.773111|-1.57%|ORDERED|
|RocksStateBackendBenchmark.stateBackends|319.227313|315.683627|-1.11%|ROCKS_INC|
|StreamNetworkLatencyBenchmarkExecutor.networkLatency1to1|10.213493|10.213394|0.00%|
|
|ContinuousFileReaderOperatorBenchmark.readFileSplit|42153.469978|42963.95343|1.92%|
|
So I suspect there is a problem with a benchmarking stand.
I also can't confirm regression reliably when benchmarking locally.
was (Author: roman_khachatryan):
I [submitted|http://codespeed.dak8s.net:8080/job/flink-benchmark-request/194/]
an older version (before the regression - f730e16) to benchmark again.
Compared to the
[previous|http://http//codespeed.dak8s.net:8080/job/flink-master-benchmarks/6187/]
benchmark of this f730e16 commit, it shows significant regression in all
benchmarks:
|Benchmark|f730e16 Jul 06|f730e16 Jul 15|delta|params|
|TwoInputBenchmark.twoInputMapSink|19554.543834|12433.798288|-36.41%| |
|SerializationFrameworkMiniBenchmarks.serializerPojo|697.942095|504.415277|-27.73%|
|
|KeyByBenchmarks.tupleKeyBy|6326.143904|4638.543555|-26.68%| |
|BlockingPartitionBenchmark.uncompressedMmapPartition|9291.513297|6898.695382|-25.75%|
|
|SerializationFrameworkMiniBenchmarks.serializerRow|862.468117|671.346967|-22.16%|
|
|KeyByBenchmarks.arrayKeyBy|3549.952163|2797.727893|-21.19%| |
|SerializationFrameworkMiniBenchmarks.serializerTuple|993.588036|792.453082|-20.24%|
|
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|75836.784163|61426.172823|-19.00%|100,100ms|
|RemoteChannelThroughputBenchmark.remoteRebalance|18300.374713|14980.789678|-18.14%|
|
|TwoInputBenchmark.twoInputOneIdleMapSink|11798.691534|9793.882213|-16.99%| |
|SerializationFrameworkMiniBenchmarks.serializerAvro|568.687328|481.019554|-15.42%|
|
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|14136.11755|12088.743814|-14.48%|1000,100ms,SSL|
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|45936.696655|39393.861324|-14.24%|1000,1ms|
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|17105.577957|14726.522754|-13.91%|100,100ms,SSL|
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|51594.0401|44693.704593|-13.37%|1000,100ms|
|MemoryStateBackendBenchmark.stateBackends|4682.529423|4073.686595|-13.00%|MEMORY|
|SerializationFrameworkMiniBenchmarks.serializerKryo|244.001776|212.473621|-12.92%|
|
|MemoryStateBackendBenchmark.stateBackends|4335.240886|3787.143054|-12.64%|FS|
|BlockingPartitionBenchmark.uncompressedFilePartition|8988.964525|7930.936505|-11.77%|
|
|InputBenchmark.mapRebalanceMapSink|10833.173473|9584.143495|-11.53%| |
|StreamNetworkThroughputBenchmarkExecutor.networkThroughput|37876.283084|33631.728543|-11.21%|1000,100ms,OpenSSL|
|WindowBenchmarks.globalWindow|7398.471169|6570.142998|-11.20%| |
|BlockingPartitionBenchmark.compressedFilePartition|8269.729104|7400.71175|-10.51%|
|
|MemoryStateBackendBenchmark.stateBackends|4726.982779|4278.369844|-9.49%|FS_ASYNC|
|SerializationFrameworkMiniBenchmarks.serializerHeavyString|242.494046|222.855766|-8.10%|
|
|StreamNetworkBroadcastThroughputBenchmarkExecutor.networkBroadcastThroughput|1085.52807|998.249968|-8.04%|
|
|InputBenchmark.mapSink|16594.379276|15328.197699|-7.63%| |
|WindowBenchmarks.sessionWindow|891.5685|829.102496|-7.01%| |
|RocksStateBackendBenchmark.stateBackends|325.958457|305.387748|-6.31%|ROCKS|
|AsyncWaitOperatorBenchmark.asyncWait|1654.341203|1550.847558|-6.26%|UNORDERED|
|WindowBenchmarks.tumblingWindow|4703.042299|4566.554823|-2.90%| |
|DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput|32376.829723|31532.9253|-2.61%|
|
|WindowBenchmarks.slidingWindow|749.309761|736.68323|-1.69%| |
|AsyncWaitOperatorBenchmark.asyncWait|1634.387558|1608.773111|-1.57%|ORDERED|
|RocksStateBackendBenchmark.stateBackends|319.227313|315.683627|-1.11%|ROCKS_INC|
|StreamNetworkLatencyBenchmarkExecutor.networkLatency1to1|10.213493|10.213394|0.00%|
|
|ContinuousFileReaderOperatorBenchmark.readFileSplit|42153.469978|42963.95343|1.92%|
|
So I suspect there is a problem with a benchmarking stand.
I also can't confirm regression reliably when benchmarking locally.
> Performance regression 2020.07.13 (most benchmarks)
> ---------------------------------------------------
>
> Key: FLINK-18614
> URL: https://issues.apache.org/jira/browse/FLINK-18614
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks
> Reporter: Roman Khachatryan
> Assignee: Roman Khachatryan
> Priority: Major
>
> [http://codespeed.dak8s.net:8000/timeline/]
>
> Last good commit 0fbea46ac0271dd84fa8acd7f99f449a9a0d458c Jul 12 21:22:22 2020
> FLINK-18552[tests] Update migration tests of
> StatefulJobWBroadcastStateMigrationITCase to cover migration till release-1.11
>
> First bad commit a028ba216a35a70c4f2d337bdede195bf7701192 Jul 8 13:48:35 2020
> FLINK-18528[table] Update UNNEST to new type system
>
> Commits are adjacent.
> Commit a028ba is not included into release-1.11 branch or tags.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
