Donal Evans created GEODE-9302:
----------------------------------
Summary: Benchmark instability in PartitionedPutStringBenchmark
Key: GEODE-9302
URL: https://issues.apache.org/jira/browse/GEODE-9302
Project: Geode
Issue Type: Bug
Components: benchmarks
Reporter: Donal Evans
A benchmark failure due to the recently-introduced
PartitionedPutStringBenchmark was observed:
{noformat}
This is ITERATION 1 of benchmarking against baseline.
P2pPartitionedGetBenchmark avg ops/sec
Baseline: 853001.60 Test: 867151.67 Difference: +1.7%
avg latency
Baseline: 842007.55 Test: 828545.06 Difference: -1.6%
P2pPartitionedPutBenchmark avg ops/sec
Baseline: 128283.47 Test: 126510.92 Difference: -1.4%
avg latency
Baseline: 5785619.62 Test: 5915913.49 Difference: +2.3%
P2pPartitionedPutBytesBenchmark avg ops/sec
Baseline: 175658.08 Test: 174865.97 Difference: -0.5%
avg latency
Baseline: 4130071.43 Test: 4130753.09 Difference: +0.0%
PartitionedFunctionExecutionBenchmark avg ops/sec
Baseline: 254788.26 Test: 268132.99 Difference: +5.2%
avg latency
Baseline: 846158.41 Test: 804199.42 Difference: -5.0%
PartitionedFunctionExecutionWithArgumentsBenchmark avg ops/sec
Baseline: 278669.87 Test: 281504.58 Difference: +1.0%
avg latency
Baseline: 1031826.82 Test: 1021314.54 Difference: -1.0%
PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec
Baseline: 372204.82 Test: 348815.81 Difference: -6.3%
avg latency
Baseline: 1545217.38 Test: 1649706.37 Difference: +6.8%
PartitionedGetBenchmark avg ops/sec
Baseline: 823740.09 Test: 819044.99 Difference: -0.6%
avg latency
Baseline: 872172.75 Test: 877580.02 Difference: +0.6%
PartitionedGetLongBenchmark avg ops/sec
Baseline: 1047221.43 Test: 1045565.89 Difference: -0.2%
avg latency
Baseline: 685757.55 Test: 687005.43 Difference: +0.2%
PartitionedGetStringBenchmark avg ops/sec
Baseline: 1055904.14 Test: 1045420.73 Difference: -1.0%
avg latency
Baseline: 680031.44 Test: 687045.15 Difference: +1.0%
PartitionedIndexedQueryBenchmark avg ops/sec
Baseline: 31596.35 Test: 31653.48 Difference: +0.2%
avg latency
Baseline: 18221302.10 Test: 18216097.86 Difference: -0.0%
PartitionedNonIndexedQueryBenchmark avg ops/sec
Baseline: 95.78 Test: 100.35 Difference: +4.8%
avg latency
Baseline: 750871203.78 Test: 716853923.95 Difference: -4.5%
PartitionedPutAllBenchmark avg ops/sec
Baseline: 8675.75 Test: 8628.10 Difference: -0.5%
avg latency
Baseline: 16595044.73 Test: 16685258.91 Difference: +0.5%
PartitionedPutAllLongBenchmark avg ops/sec
Baseline: 1382.38 Test: 1380.50 Difference: -0.1%
avg latency
Baseline: 104866853.92 Test: 104775538.34 Difference: -0.1%
PartitionedPutBenchmark avg ops/sec
Baseline: 491790.40 Test: 479926.75 Difference: -2.4%
avg latency
Baseline: 1461947.23 Test: 1497519.77 Difference: +2.4%
PartitionedPutBytesBenchmark avg ops/sec
Baseline: 472520.77 Test: 475046.43 Difference: +0.5%
avg latency
Baseline: 1523521.43 Test: 1515515.20 Difference: -0.5%
PartitionedPutLongBenchmark avg ops/sec
Baseline: 412720.03 Test: 389975.92 Difference: -5.5%
avg latency
Baseline: 1740407.45 Test: 1842985.87 Difference: +5.9%
PartitionedPutStringBenchmark avg ops/sec
Baseline: 430083.15 Test: 402523.17 Difference: -6.4%
avg latency
Baseline: 1671189.90 Test: 1786949.71 Difference: +6.9%
ReplicatedFunctionExecutionBenchmark avg ops/sec
Baseline: 513699.75 Test: 514378.00 Difference: +0.1%
avg latency
Baseline: 418897.75 Test: 418374.49 Difference: -0.1%
ReplicatedFunctionExecutionWithArgumentsBenchmark avg ops/sec
Baseline: 525004.44 Test: 522304.20 Difference: -0.5%
avg latency
Baseline: 2192090.72 Test: 2203176.97 Difference: +0.5%
ReplicatedFunctionExecutionWithFiltersBenchmark avg ops/sec
Baseline: 540464.25 Test: 532795.86 Difference: -1.4%
avg latency
Baseline: 1330260.37 Test: 1349440.25 Difference: +1.4%
ReplicatedGetBenchmark avg ops/sec
Baseline: 827036.49 Test: 817813.56 Difference: -1.1%
avg latency
Baseline: 868818.39 Test: 878720.34 Difference: +1.1%
ReplicatedGetLongBenchmark avg ops/sec
Baseline: 1085165.62 Test: 1071977.64 Difference: -1.2%
avg latency
Baseline: 661809.29 Test: 670064.70 Difference: +1.2%
ReplicatedIndexedQueryBenchmark avg ops/sec
Baseline: 32216.21 Test: 31293.98 Difference: -2.9%
avg latency
Baseline: 17873159.19 Test: 18437420.16 Difference: +3.2%
ReplicatedNonIndexedQueryBenchmark avg ops/sec
Baseline: 51.62 Test: 47.74 Difference: -7.5%
avg latency
Baseline: 1390417763.11 Test: 1505134279.26 Difference: +8.3%
ReplicatedPutAllBenchmark avg ops/sec
Baseline: 788.90 Test: 814.00 Difference: +3.2%
avg latency
Baseline: 182735327.71 Test: 177046841.08 Difference: -3.1%
ReplicatedPutAllLongBenchmark avg ops/sec
Baseline: 832.81 Test: 826.93 Difference: -0.7%
avg latency
Baseline: 173063206.58 Test: 174233069.86 Difference: +0.7%
ReplicatedPutBenchmark avg ops/sec
Baseline: 490054.06 Test: 494297.13 Difference: +0.9%
avg latency
Baseline: 1466451.86 Test: 1454784.46 Difference: -0.8%
ReplicatedPutLongBenchmark avg ops/sec
Baseline: 457689.80 Test: 457973.53 Difference: +0.1%
avg latency
Baseline: 1571708.57 Test: 1569573.85 Difference: -0.1%
This is ITERATION 2 of benchmarking against baseline.
PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec
Baseline: 395993.05 Test: 358748.93 Difference: -9.4%
avg latency
Baseline: 1452604.27 Test: 1603625.58 Difference: +10.4%
PartitionedPutLongBenchmark avg ops/sec
Baseline: 422650.17 Test: 415015.59 Difference: -1.8%
avg latency
Baseline: 1702486.44 Test: 1731996.79 Difference: +1.7%
PartitionedPutStringBenchmark avg ops/sec
Baseline: 429296.09 Test: 391046.61 Difference: -8.9%
avg latency
Baseline: 1675896.07 Test: 1837288.98 Difference: +9.6%
ReplicatedNonIndexedQueryBenchmark avg ops/sec
Baseline: 49.02 Test: 50.55 Difference: +3.1%
avg latency
Baseline: 1464289182.72 Test: 1420848686.90 Difference: -3.0%
This is ITERATION 3 of benchmarking against baseline.
PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec
Baseline: 407931.35 Test: 434871.25 Difference: +6.6%
avg latency
Baseline: 1410246.93 Test: 1322795.29 Difference: -6.2%
PartitionedPutStringBenchmark avg ops/sec
Baseline: 427102.32 Test: 402755.46 Difference: -5.7%
avg latency
Baseline: 1682556.85 Test: 1786603.33 Difference: +6.2%
This is ITERATION 4 of benchmarking against baseline.
PartitionedPutStringBenchmark avg ops/sec
Baseline: 434609.00 Test: 402475.52 Difference: -7.4%
avg latency
Baseline: 1656172.72 Test: 1785229.42 Difference: +7.8%
This is ITERATION 5 of benchmarking against baseline.
PartitionedPutStringBenchmark avg ops/sec
Baseline: 423047.66 Test: 402475.29 Difference: -4.9%
avg latency
Baseline: 1700791.11 Test: 1787257.59 Difference: +5.1%
{noformat}
Looking at some previous runs of the benchmarks since this particular benchmark
was introduced, it appears that there is some instability in this benchmark:
{noformat}
average ops/second Baseline: 384659.34 Test: 375511.88 Difference:
-2.4%
average latency Baseline: 1868722.73 Test: 1918780.45 Difference:
+2.7%
average ops/second Baseline: 434900.93 Test: 391933.67 Difference:
-9.9%
average latency Baseline: 1656702.07 Test: 1830827.53 Difference:
+10.5%
average ops/second Baseline: 419108.79 Test: 401341.06 Difference:
-4.2%
average latency Baseline: 1712897.05 Test: 1791919.35 Difference:
+4.6%
average ops/second Baseline: 421598.58 Test: 399128.86 Difference:
-5.3%
average latency Baseline: 1707314.91 Test: 1801691.41 Difference:
+5.5%
average ops/second Baseline: 424445.58 Test: 403798.53 Difference:
-4.9%
average latency Baseline: 1694539.23 Test: 1781948.40 Difference:
+5.2%
average ops/second Baseline: 417622.25 Test: 388290.32 Difference:
-7.0%
average latency Baseline: 1721608.57 Test: 1850134.14 Difference:
+7.5%
average ops/second Baseline: 413538.43 Test: 396040.63 Difference:
-4.2%
average latency Baseline: 1738858.16 Test: 1820714.96 Difference:
+4.7%
average ops/second Baseline: 363423.19 Test: 394963.41 Difference:
+8.7%
average latency Baseline: 1981399.62 Test: 1822056.94 Difference:
-8.0%
average ops/second Baseline: 369520.84 Test: 394239.61 Difference:
+6.7%
average latency Baseline: 1948697.59 Test: 1824966.98 Difference:
-6.3%
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
