[
https://issues.apache.org/jira/browse/FLINK-15318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17139243#comment-17139243
]
Yun Tang edited comment on FLINK-15318 at 6/18/20, 9:06 AM:
------------------------------------------------------------
[~sewen] After digging into those tests, I think all of them could be dropped.
{{RocksDBListStatePerformanceTest}} targets for performance of
"stringappendtest" merge operator, which has been covered by
[ListStateBenchmark#add|https://github.com/apache/flink-benchmarks/blob/8b449865cf733dbb3c01e997fe44b1a5b6f82cdc/src/main/java/org/apache/flink/state/benchmark/ListStateBenchmark.java#L118].
{{RocksDBWriteBatchPerformanceTest}} targets for performance of WriteBatch
which should be covered by
[MapStateBenchmark#mapPutAll|https://github.com/apache/flink-benchmarks/blob/8b449865cf733dbb3c01e997fe44b1a5b6f82cdc/src/main/java/org/apache/flink/state/benchmark/MapStateBenchmark.java#L160]
{{RocksDBPerformanceTest}} targets for performance of merge and iterator seek
and next, which have been covered by
[ListStateBenchmark#add|https://github.com/apache/flink-benchmarks/blob/8b449865cf733dbb3c01e997fe44b1a5b6f82cdc/src/main/java/org/apache/flink/state/benchmark/ListStateBenchmark.java#L118]
and
[MapStateBenchmark#mapIterator|https://github.com/apache/flink-benchmarks/blob/8b449865cf733dbb3c01e997fe44b1a5b6f82cdc/src/main/java/org/apache/flink/state/benchmark/MapStateBenchmark.java#L143]
And the most important thing is unit test cannot watch the performance issues
clearly. If the execution time expands from 2 seconds to 2.5 seconds, which
means the performance regression is about 25%. However, a timeout limit of 3
seconds cannot detect such performance regression. Thus, I do not see real
benefits to still keep them in unit tests, not to mention sometimes break the
CI.
was (Author: yunta):
[~sewen] After digging into those tests, I think all of them could be dropped.
{{RocksDBListStatePerformanceTest}} targets for performance of
"stringappendtest" merge operator, which has been covered by
[ListStateBenchmark#add|https://github.com/apache/flink-benchmarks/blob/8b449865cf733dbb3c01e997fe44b1a5b6f82cdc/src/main/java/org/apache/flink/state/benchmark/ListStateBenchmark.java#L118].
{{RocksDBWriteBatchPerformanceTest}} targets for performance of WriteBatch
which should be covered by
[MapStateBenchmark#mapPutAll|https://github.com/apache/flink-benchmarks/blob/8b449865cf733dbb3c01e997fe44b1a5b6f82cdc/src/main/java/org/apache/flink/state/benchmark/MapStateBenchmark.java#L160]
{{RocksDBPerformanceTest}} targets for performance of merge and iterator seek
and next, which have been covered by
[ListStateBenchmark#add|https://github.com/apache/flink-benchmarks/blob/8b449865cf733dbb3c01e997fe44b1a5b6f82cdc/src/main/java/org/apache/flink/state/benchmark/ListStateBenchmark.java#L118]
and
[MapStateBenchmark#mapIterator|https://github.com/apache/flink-benchmarks/blob/8b449865cf733dbb3c01e997fe44b1a5b6f82cdc/src/main/java/org/apache/flink/state/benchmark/MapStateBenchmark.java#L143]
And the most important thing is unit test cannot watch the performance issues
clearly. If the execution time expands from 2 seconds to 2.5 seconds, which
means the performance regression is about 25%. However, a timeout limit of 3
seconds cannot detect such performance regression.
> RocksDBWriteBatchPerformanceTest.benchMark fails on ppc64le
> -----------------------------------------------------------
>
> Key: FLINK-15318
> URL: https://issues.apache.org/jira/browse/FLINK-15318
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks, Runtime / State Backends
> Environment: arch: ppc64le
> os: rhel7.6, ubuntu 18.04
> jdk: 8, 11
> mvn: 3.3.9, 3.6.2
> Reporter: Siddhesh Ghadi
> Priority: Major
> Attachments: surefire-report.txt
>
>
> RocksDBWriteBatchPerformanceTest.benchMark fails due to TestTimedOut, however
> when test-timeout is increased from 2s to 5s in
> org/apache/flink/contrib/streaming/state/benchmark/RocksDBWriteBatchPerformanceTest.java:75,
> it passes. Is this acceptable solution?
> Note: Tests are ran inside a container.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
