[
https://issues.apache.org/jira/browse/GEODE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307269#comment-17307269
]
Donal Evans edited comment on GEODE-8950 at 3/23/21, 5:41 PM:
--------------------------------------------------------------
To expand on Wayne's comment above, while I do believe that there was a
performance degradation in this benchmark due to GEODE-8496: top up dependency
updates (#5772), SHA 217be41f, (see attached charts) the process of determining
that that commit was the likely cause was extremely time-consuming and required
240 runs of the benchmark across 19 separate commits to produce enough data to
be somewhat confident in the result. The level of inconsistency in the
benchmark is such that unless a catastrophic degradation was introduced, it
would be difficult to notice amongst the background noise. Given that such a
degradation would almost certainly be detected in other benchmarks or
performance tests, it's difficult to see what value the
P2pPartitionedPutLongBenchmark is currently providing.
!GEODE-8950-performance degradation vs 1.13.png!
!GEODE-8950-before-after-histogram-chart.png!
was (Author: donalevans):
To expand on Wayne's comment above, while I do believe that there was a
performance degradation in this benchmark due to GEODE-8496: top up dependency
updates (#5772), SHA 217be41f, the process of determining that that commit was
the likely cause was extremely time-consuming and required 240 runs of the
benchmark across 19 separate commits to produce enough data to be somewhat
confident in the result. The level of inconsistency in the benchmark is such
that unless a catastrophic degradation was introduced, it would be difficult to
notice amongst the background noise. Given that such a degradation would almost
certainly be detected in other benchmarks or performance tests, it's difficult
to see what value the P2pPartitionedPutLongBenchmark is currently providing.
> Benchmark failure - P2pPartitionedPutLongBenchmark
> --------------------------------------------------
>
> Key: GEODE-8950
> URL: https://issues.apache.org/jira/browse/GEODE-8950
> Project: Geode
> Issue Type: Bug
> Components: benchmarks
> Affects Versions: 1.15.0
> Reporter: Donal Evans
> Assignee: Donal Evans
> Priority: Major
> Labels: blocks-1.14.0, blocks-1.15.0
> Attachments: GEODE-8950-before-after-histogram-chart.png,
> GEODE-8950-performance degradation vs 1.13.png
>
>
> Multiple benchmark failures due to P2pPartitionedPutLongBenchmark have been
> seen recently.
> This run saw 3 out of the 5 repeats fail due to flagged degradations in
> P2pPartitionedPutLongBenchmark:
> [https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/Benchmark_base/builds/16|https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/Benchmark_base/builds/16#L601ed52d:5552]
> This run saw 1 out of the 5 repeats fail due to flagged degradations in
> P2pPartitionedPutLongBenchmark:
> [https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/Benchmark_base/builds/20]
> This run saw 4 out of the 5 repeats fail due to flagged degradations in
> P2pPartitionedPutLongBenchmark:
> [https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/Benchmark_base/builds/27]
> In all the above benchmarks, the other failed runs were due to EOFExceptions
> rather than flagged degradations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
