Jinmei Liao created GEODE-7231:
----------------------------------
Summary: CI: BENCHMARK FAILED:
org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark
average latency is 5% worse than baseline.
Key: GEODE-7231
URL: https://issues.apache.org/jira/browse/GEODE-7231
Project: Geode
Issue Type: Bug
Components: benchmarks, ci
Reporter: Jinmei Liao
https://concourse.gemfire-ci.info/teams/main/pipelines/gemfire-develop-main/jobs/Benchmark/builds/106
> Task :harness:analyzeRun FAILED
Running analyzer
Comparing test result at
/tmp/build/2ea77113/results/benchmarks-gemfire-develop-main-Benchmark-106-27727/branch/benchmarks_20190921000552
to baseline at
/tmp/build/2ea77113/results/benchmarks-gemfire-develop-main-Benchmark-106-27727/baseline/benchmarks_20190920215707
org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark
average ops/second Baseline: 171383.24 Test: 159507.38
Difference: -6.9%
ops/second standard error Baseline: 499.38 Test: 473.79
Difference: -5.1%
ops/second standard deviation Baseline: 8635.16 Test: 8192.61
Difference: -5.1%
YS 99th percentile latency Baseline: 1804.00 Test: 1903.00
Difference: +5.5%
median latency Baseline: 637951.00 Test: 685055.00
Difference: +7.4%
90th percentile latency Baseline: 1198079.00 Test: 1320959.00
Difference: +10.3%
99th percentile latency Baseline: 4059135.00 Test: 4159487.00
Difference: +2.5%
99.9th percentile latency Baseline: 13574143.00 Test: 14589951.00
Difference: +7.5%
average latency Baseline: 839055.57 Test: 901530.36
Difference: +7.4%
latency standard deviation Baseline: 929586.39 Test: 965788.36
Difference: +3.9%
latency standard error Baseline: 129.65 Test: 139.61
Difference: +7.7%
org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithFiltersBenchmark
average ops/second Baseline: 442946.14 Test: 442990.08
Difference: +0.0%
ops/second standard error Baseline: 435.26 Test: 418.38
Difference: -3.9%
ops/second standard deviation Baseline: 7526.29 Test: 7246.63
Difference: -3.7%
YS 99th percentile latency Baseline: 1504.00 Test: 1503.00
Difference: -0.1%
median latency Baseline: 482559.00 Test: 483327.00
Difference: +0.2%
90th percentile latency Baseline: 988159.00 Test: 987135.00
Difference: -0.1%
99th percentile latency Baseline: 3557375.00 Test: 3532799.00
Difference: -0.7%
99.9th percentile latency Baseline: 10248191.00 Test: 10231807.00
Difference: -0.2%
average latency Baseline: 649048.91 Test: 648977.69
Difference: -0.0%
latency standard deviation Baseline: 757279.93 Test: 758841.81
Difference: +0.2%
latency standard error Baseline: 65.70 Test: 65.83
Difference: +0.2%
org.apache.geode.benchmark.tests.PartitionedGetBenchmark
average ops/second Baseline: 825000.20 Test: 827836.08
Difference: +0.3%
ops/second standard error Baseline: 462.66 Test: 485.61
Difference: +5.0%
ops/second standard deviation Baseline: 8000.22 Test: 8411.04
Difference: +5.1%
YS 99th percentile latency Baseline: 904.00 Test: 905.00
Difference: +0.1%
median latency Baseline: 357631.00 Test: 356607.00
Difference: -0.3%
90th percentile latency Baseline: 492287.00 Test: 489983.00
Difference: -0.5%
99th percentile latency Baseline: 2369535.00 Test: 2375679.00
Difference: +0.3%
99.9th percentile latency Baseline: 6729727.00 Test: 6565887.00
Difference: -2.4%
average latency Baseline: 435231.89 Test: 433718.19
Difference: -0.3%
latency standard deviation Baseline: 535757.54 Test: 542177.55
Difference: +1.2%
latency standard error Baseline: 34.06 Test: 34.41
Difference: +1.0%
org.apache.geode.benchmark.tests.PartitionedGetLongBenchmark
average ops/second Baseline: 1038960.75 Test: 1039353.89
Difference: +0.0%
ops/second standard error Baseline: 1042.48 Test: 442.50
Difference: -57.6%
ops/second standard deviation Baseline: 18026.21 Test: 7651.57
Difference: -57.6%
YS 99th percentile latency Baseline: 1002.00 Test: 1002.00
Difference: +0.0%
median latency Baseline: 305151.00 Test: 305151.00
Difference: +0.0%
90th percentile latency Baseline: 459519.00 Test: 459007.00
Difference: -0.1%
99th percentile latency Baseline: 1418239.00 Test: 1424383.00
Difference: +0.4%
99.9th percentile latency Baseline: 3729407.00 Test: 3788799.00
Difference: +1.6%
average latency Baseline: 345199.75 Test: 345276.27
Difference: +0.0%
latency standard deviation Baseline: 364741.13 Test: 387501.17
Difference: +6.2%
latency standard error Baseline: 20.66 Test: 21.95
Difference: +6.2%
org.apache.geode.benchmark.tests.PartitionedIndexedQueryBenchmark
average ops/second Baseline: 18282.12 Test: 18247.08
Difference: -0.2%
ops/second standard error Baseline: 7.85 Test: 10.12
Difference: +28.9%
ops/second standard deviation Baseline: 135.73 Test: 174.91
Difference: +28.9%
YS 99th percentile latency Baseline: 20093.18 Test: 20092.77
Difference: -0.0%
median latency Baseline: 14557183.00 Test: 13950975.00
Difference: -4.2%
90th percentile latency Baseline: 20594687.00 Test: 23232511.00
Difference: +12.8%
99th percentile latency Baseline: 46235647.00 Test: 60358655.00
Difference: +30.5%
99.9th percentile latency Baseline: 58621951.00 Test: 70582271.00
Difference: +20.4%
average latency Baseline: 15752630.91 Test: 15779064.59
Difference: +0.2%
latency standard deviation Baseline: 7386025.34 Test: 10516174.80
Difference: +42.4%
latency standard error Baseline: 3154.86 Test: 4495.79
Difference: +42.5%
org.apache.geode.benchmark.tests.PartitionedPutAllBenchmark
average ops/second Baseline: 654.62 Test: 649.58
Difference: -0.8%
ops/second standard error Baseline: 3.82 Test: 4.17
Difference: +9.4%
ops/second standard deviation Baseline: 65.09 Test: 71.08
Difference: +9.2%
YS 99th percentile latency Baseline: 20099.00 Test: 20099.00
Difference: +0.0%
median latency Baseline: 67960831.00 Test: 69468159.00
Difference: +2.2%
90th percentile latency Baseline: 217448447.00 Test: 218890239.00
Difference: +0.7%
99th percentile latency Baseline: 312999935.00 Test: 316407807.00
Difference: +1.1%
99.9th percentile latency Baseline: 395837439.00 Test: 390332415.00
Difference: -1.4%
average latency Baseline: 113043700.31 Test: 114136328.08
Difference: +1.0%
latency standard deviation Baseline: 77281720.43 Test: 77650547.89
Difference: +0.5%
latency standard error Baseline: 176865.03 Test: 178541.92
Difference: +0.9%
org.apache.geode.benchmark.tests.PartitionedPutAllLongBenchmark
average ops/second Baseline: 651.36 Test: 645.87
Difference: -0.8%
ops/second standard error Baseline: 4.14 Test: 4.15
Difference: +0.3%
ops/second standard deviation Baseline: 70.46 Test: 70.77
Difference: +0.4%
YS 99th percentile latency Baseline: 20099.00 Test: 20099.00
Difference: +0.0%
median latency Baseline: 64454655.00 Test: 67895295.00
Difference: +5.3%
90th percentile latency Baseline: 225574911.00 Test: 220069887.00
Difference: -2.4%
99th percentile latency Baseline: 352583679.00 Test: 325844991.00
Difference: -7.6%
99.9th percentile latency Baseline: 427032575.00 Test: 413925375.00
Difference: -3.1%
average latency Baseline: 113828250.36 Test: 114725443.16
Difference: +0.8%
latency standard deviation Baseline: 82646931.03 Test: 79661855.97
Difference: -3.6%
latency standard error Baseline: 189811.47 Test: 183679.48
Difference: -3.2%
org.apache.geode.benchmark.tests.PartitionedPutBenchmark
average ops/second Baseline: 419374.09 Test: 414255.28
Difference: -1.2%
ops/second standard error Baseline: 996.43 Test: 736.95
Difference: -26.0%
ops/second standard deviation Baseline: 17229.96 Test: 12743.12
Difference: -26.0%
YS 99th percentile latency Baseline: 1501.00 Test: 1502.00
Difference: +0.1%
median latency Baseline: 676351.00 Test: 676351.00
Difference: +0.0%
90th percentile latency Baseline: 993279.00 Test: 1026047.00
Difference: +3.3%
99th percentile latency Baseline: 1644543.00 Test: 1753087.00
Difference: +6.6%
99.9th percentile latency Baseline: 66191359.00 Test: 65404927.00
Difference: -1.2%
average latency Baseline: 857166.67 Test: 867866.53
Difference: +1.2%
latency standard deviation Baseline: 2941133.91 Test: 2966121.87
Difference: +0.8%
latency standard error Baseline: 262.23 Test: 266.09
Difference: +1.5%
org.apache.geode.benchmark.tests.PartitionedPutLongBenchmark
average ops/second Baseline: 411434.00 Test: 407384.65
Difference: -1.0%
ops/second standard error Baseline: 698.57 Test: 757.94
Difference: +8.5%
ops/second standard deviation Baseline: 12079.39 Test: 13127.90
Difference: +8.7%
YS 99th percentile latency Baseline: 1501.00 Test: 1502.00
Difference: +0.1%
median latency Baseline: 673279.00 Test: 674815.00
Difference: +0.2%
90th percentile latency Baseline: 1010687.00 Test: 1026559.00
Difference: +1.6%
99th percentile latency Baseline: 1696767.00 Test: 1762303.00
Difference: +3.9%
99.9th percentile latency Baseline: 73400319.00 Test: 74842111.00
Difference: +2.0%
average latency Baseline: 873796.73 Test: 882648.37
Difference: +1.0%
latency standard deviation Baseline: 3343859.26 Test: 3397106.08
Difference: +1.6%
latency standard error Baseline: 301.01 Test: 307.33
Difference: +2.1%
org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionBenchmark
average ops/second Baseline: 385497.82 Test: 383556.25
Difference: -0.5%
ops/second standard error Baseline: 157.81 Test: 187.92
Difference: +19.1%
ops/second standard deviation Baseline: 2728.75 Test: 3249.37
Difference: +19.1%
YS 99th percentile latency Baseline: 575.00 Test: 575.25
Difference: +0.0%
median latency Baseline: 273919.00 Test: 275455.00
Difference: +0.6%
90th percentile latency Baseline: 368127.00 Test: 369663.00
Difference: +0.4%
99th percentile latency Baseline: 461055.00 Test: 471551.00
Difference: +2.3%
99.9th percentile latency Baseline: 886783.00 Test: 671743.00
Difference: -24.2%
average latency Baseline: 279243.44 Test: 280609.57
Difference: +0.5%
latency standard deviation Baseline: 188613.48 Test: 186659.89
Difference: -1.0%
latency standard error Baseline: 17.54 Test: 17.40
Difference: -0.8%
org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionWithArgumentsBenchmark
average ops/second Baseline: 595408.61 Test: 615069.19
Difference: +3.3%
ops/second standard error Baseline: 549.38 Test: 402.08
Difference: -26.8%
ops/second standard deviation Baseline: 9499.75 Test: 6952.69
Difference: -26.8%
YS 99th percentile latency Baseline: 1505.00 Test: 1503.00
Difference: -0.1%
median latency Baseline: 731647.00 Test: 775679.00
Difference: +6.0%
90th percentile latency Baseline: 1094655.00 Test: 1083391.00
Difference: -1.0%
99th percentile latency Baseline: 9043967.00 Test: 6017023.00
Difference: -33.5%
99.9th percentile latency Baseline: 22757375.00 Test: 23691263.00
Difference: +4.1%
average latency Baseline: 965601.30 Test: 934625.03
Difference: -3.2%
latency standard deviation Baseline: 1702556.73 Test: 1590548.30
Difference: -6.6%
latency standard error Baseline: 127.41 Test: 117.11
Difference: -8.1%
org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionWithFiltersBenchmark
average ops/second Baseline: 584622.61 Test: 583232.58
Difference: -0.2%
ops/second standard error Baseline: 369.33 Test: 377.86
Difference: +2.3%
ops/second standard deviation Baseline: 6386.34 Test: 6533.81
Difference: +2.3%
YS 99th percentile latency Baseline: 1103.00 Test: 1103.00
Difference: +0.0%
median latency Baseline: 496639.00 Test: 520191.00
Difference: +4.7%
90th percentile latency Baseline: 731135.00 Test: 736255.00
Difference: +0.7%
99th percentile latency Baseline: 3874815.00 Test: 3444735.00
Difference: -11.1%
99.9th percentile latency Baseline: 11780095.00 Test: 10878975.00
Difference: -7.6%
average latency Baseline: 614285.67 Test: 615818.03
Difference: +0.2%
latency standard deviation Baseline: 821732.66 Test: 723485.75
Difference: -12.0%
latency standard error Baseline: 62.05 Test: 54.70
Difference: -11.9%
org.apache.geode.benchmark.tests.ReplicatedGetBenchmark
average ops/second Baseline: 846698.07 Test: 846876.85
Difference: +0.0%
ops/second standard error Baseline: 455.80 Test: 398.00
Difference: -12.7%
ops/second standard deviation Baseline: 7881.58 Test: 6893.61
Difference: -12.5%
YS 99th percentile latency Baseline: 905.00 Test: 905.00
Difference: +0.0%
median latency Baseline: 349183.00 Test: 348415.00
Difference: -0.2%
90th percentile latency Baseline: 484095.00 Test: 486655.00
Difference: +0.5%
99th percentile latency Baseline: 2435071.00 Test: 2369535.00
Difference: -2.7%
99.9th percentile latency Baseline: 6770687.00 Test: 6574079.00
Difference: -2.9%
average latency Baseline: 424071.21 Test: 423929.31
Difference: -0.0%
latency standard deviation Baseline: 530295.63 Test: 521763.35
Difference: -1.6%
latency standard error Baseline: 33.28 Test: 32.74
Difference: -1.6%
org.apache.geode.benchmark.tests.ReplicatedGetLongBenchmark
average ops/second Baseline: 1056863.08 Test: 1056316.60
Difference: -0.1%
ops/second standard error Baseline: 476.01 Test: 533.16
Difference: +12.0%
ops/second standard deviation Baseline: 8231.02 Test: 9219.21
Difference: +12.0%
YS 99th percentile latency Baseline: 1002.00 Test: 1001.00
Difference: -0.1%
median latency Baseline: 299775.00 Test: 299775.00
Difference: +0.0%
90th percentile latency Baseline: 451071.00 Test: 448255.00
Difference: -0.6%
99th percentile latency Baseline: 1371135.00 Test: 1392639.00
Difference: +1.6%
99.9th percentile latency Baseline: 3262463.00 Test: 3409919.00
Difference: +4.5%
average latency Baseline: 339496.49 Test: 339654.77
Difference: +0.0%
latency standard deviation Baseline: 342396.65 Test: 373899.72
Difference: +9.2%
latency standard error Baseline: 19.23 Test: 21.01
Difference: +9.2%
org.apache.geode.benchmark.tests.ReplicatedIndexedQueryBenchmark
average ops/second Baseline: 18229.31 Test: 18224.52
Difference: -0.0%
ops/second standard error Baseline: 10.16 Test: 8.23
Difference: -19.0%
ops/second standard deviation Baseline: 175.60 Test: 142.52
Difference: -18.8%
YS 99th percentile latency Baseline: 20093.73 Test: 20093.00
Difference: -0.0%
median latency Baseline: 12836863.00 Test: 13041663.00
Difference: +1.6%
90th percentile latency Baseline: 26427391.00 Test: 25903103.00
Difference: -2.0%
99th percentile latency Baseline: 71172095.00 Test: 69992447.00
Difference: -1.7%
99.9th percentile latency Baseline: 83427327.00 Test: 82182143.00
Difference: -1.5%
average latency Baseline: 15800858.04 Test: 15802259.29
Difference: +0.0%
latency standard deviation Baseline: 13064421.06 Test: 12769643.21
Difference: -2.3%
latency standard error Baseline: 5588.98 Test: 5463.10
Difference: -2.3%
org.apache.geode.benchmark.tests.ReplicatedPutAllBenchmark
average ops/second Baseline: 686.64 Test: 699.59
Difference: +1.9%
ops/second standard error Baseline: 4.27 Test: 4.15
Difference: -2.8%
ops/second standard deviation Baseline: 73.34 Test: 71.06
Difference: -3.1%
YS 99th percentile latency Baseline: 20099.00 Test: 20099.00
Difference: +0.0%
median latency Baseline: 77856767.00 Test: 73531391.00
Difference: -5.6%
90th percentile latency Baseline: 204079103.00 Test: 202899455.00
Difference: -0.6%
99th percentile latency Baseline: 330563583.00 Test: 324534271.00
Difference: -1.8%
99.9th percentile latency Baseline: 408682495.00 Test: 369098751.00
Difference: -9.7%
average latency Baseline: 106331049.47 Test: 104780049.58
Difference: -1.5%
latency standard deviation Baseline: 71143931.88 Test: 70631162.82
Difference: -0.7%
latency standard error Baseline: 157915.25 Test: 155630.00
Difference: -1.4%
org.apache.geode.benchmark.tests.ReplicatedPutAllLongBenchmark
average ops/second Baseline: 684.32 Test: 701.87
Difference: +2.6%
ops/second standard error Baseline: 4.09 Test: 4.03
Difference: -1.4%
ops/second standard deviation Baseline: 70.20 Test: 69.19
Difference: -1.4%
YS 99th percentile latency Baseline: 20099.00 Test: 20099.00
Difference: +0.0%
median latency Baseline: 74579967.00 Test: 72941567.00
Difference: -2.2%
90th percentile latency Baseline: 207880191.00 Test: 203554815.00
Difference: -2.1%
99th percentile latency Baseline: 344719359.00 Test: 333447167.00
Difference: -3.3%
99.9th percentile latency Baseline: 392691711.00 Test: 376176639.00
Difference: -4.2%
average latency Baseline: 106559160.43 Test: 104315677.10
Difference: -2.1%
latency standard deviation Baseline: 73550701.86 Test: 71595613.07
Difference: -2.7%
latency standard error Baseline: 163397.60 Test: 157398.24
Difference: -3.7%
org.apache.geode.benchmark.tests.ReplicatedPutBenchmark
average ops/second Baseline: 439483.79 Test: 434437.21
Difference: -1.1%
ops/second standard error Baseline: 819.57 Test: 746.63
Difference: -8.9%
ops/second standard deviation Baseline: 14171.70 Test: 12910.43
Difference: -8.9%
YS 99th percentile latency Baseline: 1302.00 Test: 1304.00
Difference: +0.2%
median latency Baseline: 662015.00 Test: 662015.00
Difference: +0.0%
90th percentile latency Baseline: 894463.00 Test: 925183.00
Difference: +3.4%
99th percentile latency Baseline: 1791999.00 Test: 1894399.00
Difference: +5.7%
99.9th percentile latency Baseline: 69730303.00 Test: 68091903.00
Difference: -2.3%
average latency Baseline: 818083.55 Test: 826988.90
Difference: +1.1%
latency standard deviation Baseline: 2889557.20 Test: 2861827.56
Difference: -1.0%
latency standard error Baseline: 251.67 Test: 250.67
Difference: -0.4%
org.apache.geode.benchmark.tests.ReplicatedPutLongBenchmark
average ops/second Baseline: 430225.52 Test: 422842.19
Difference: -1.7%
ops/second standard error Baseline: 808.44 Test: 736.25
Difference: -8.9%
ops/second standard deviation Baseline: 13979.30 Test: 12730.95
Difference: -8.9%
YS 99th percentile latency Baseline: 1303.00 Test: 1303.00
Difference: +0.0%
median latency Baseline: 662527.00 Test: 664063.00
Difference: +0.2%
90th percentile latency Baseline: 912383.00 Test: 931839.00
Difference: +2.1%
99th percentile latency Baseline: 1898495.00 Test: 1918975.00
Difference: +1.1%
99.9th percentile latency Baseline: 73138175.00 Test: 80674815.00
Difference: +10.3%
average latency Baseline: 835773.93 Test: 850337.27
Difference: +1.7%
latency standard deviation Baseline: 3198061.99 Test: 3392288.63
Difference: +6.1%
latency standard error Baseline: 281.54 Test: 301.23
Difference: +7.0%
BENCHMARK FAILED:
org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark
average latency is 5% worse than baseline.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
