[
https://issues.apache.org/jira/browse/GEODE-7231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinmei Liao reassigned GEODE-7231:
----------------------------------
Assignee: Jacob S. Barrett
> CI: BENCHMARK FAILED:
> org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark
> average latency is 5% worse than baseline.
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-7231
> URL: https://issues.apache.org/jira/browse/GEODE-7231
> Project: Geode
> Issue Type: Bug
> Components: benchmarks, ci
> Reporter: Jinmei Liao
> Assignee: Jacob S. Barrett
> Priority: Major
>
> https://concourse.gemfire-ci.info/teams/main/pipelines/gemfire-develop-main/jobs/Benchmark/builds/106
> > Task :harness:analyzeRun FAILED
> Running analyzer
> Comparing test result at
> /tmp/build/2ea77113/results/benchmarks-gemfire-develop-main-Benchmark-106-27727/branch/benchmarks_20190921000552
> to baseline at
> /tmp/build/2ea77113/results/benchmarks-gemfire-develop-main-Benchmark-106-27727/baseline/benchmarks_20190920215707
> org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark
> average ops/second Baseline: 171383.24 Test: 159507.38
> Difference: -6.9%
> ops/second standard error Baseline: 499.38 Test: 473.79
> Difference: -5.1%
> ops/second standard deviation Baseline: 8635.16 Test: 8192.61
> Difference: -5.1%
> YS 99th percentile latency Baseline: 1804.00 Test: 1903.00
> Difference: +5.5%
> median latency Baseline: 637951.00 Test: 685055.00
> Difference: +7.4%
> 90th percentile latency Baseline: 1198079.00 Test: 1320959.00
> Difference: +10.3%
> 99th percentile latency Baseline: 4059135.00 Test: 4159487.00
> Difference: +2.5%
> 99.9th percentile latency Baseline: 13574143.00 Test: 14589951.00
> Difference: +7.5%
> average latency Baseline: 839055.57 Test: 901530.36
> Difference: +7.4%
> latency standard deviation Baseline: 929586.39 Test: 965788.36
> Difference: +3.9%
> latency standard error Baseline: 129.65 Test: 139.61
> Difference: +7.7%
> org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithFiltersBenchmark
> average ops/second Baseline: 442946.14 Test: 442990.08
> Difference: +0.0%
> ops/second standard error Baseline: 435.26 Test: 418.38
> Difference: -3.9%
> ops/second standard deviation Baseline: 7526.29 Test: 7246.63
> Difference: -3.7%
> YS 99th percentile latency Baseline: 1504.00 Test: 1503.00
> Difference: -0.1%
> median latency Baseline: 482559.00 Test: 483327.00
> Difference: +0.2%
> 90th percentile latency Baseline: 988159.00 Test: 987135.00
> Difference: -0.1%
> 99th percentile latency Baseline: 3557375.00 Test: 3532799.00
> Difference: -0.7%
> 99.9th percentile latency Baseline: 10248191.00 Test: 10231807.00
> Difference: -0.2%
> average latency Baseline: 649048.91 Test: 648977.69
> Difference: -0.0%
> latency standard deviation Baseline: 757279.93 Test: 758841.81
> Difference: +0.2%
> latency standard error Baseline: 65.70 Test: 65.83
> Difference: +0.2%
> org.apache.geode.benchmark.tests.PartitionedGetBenchmark
> average ops/second Baseline: 825000.20 Test: 827836.08
> Difference: +0.3%
> ops/second standard error Baseline: 462.66 Test: 485.61
> Difference: +5.0%
> ops/second standard deviation Baseline: 8000.22 Test: 8411.04
> Difference: +5.1%
> YS 99th percentile latency Baseline: 904.00 Test: 905.00
> Difference: +0.1%
> median latency Baseline: 357631.00 Test: 356607.00
> Difference: -0.3%
> 90th percentile latency Baseline: 492287.00 Test: 489983.00
> Difference: -0.5%
> 99th percentile latency Baseline: 2369535.00 Test: 2375679.00
> Difference: +0.3%
> 99.9th percentile latency Baseline: 6729727.00 Test: 6565887.00
> Difference: -2.4%
> average latency Baseline: 435231.89 Test: 433718.19
> Difference: -0.3%
> latency standard deviation Baseline: 535757.54 Test: 542177.55
> Difference: +1.2%
> latency standard error Baseline: 34.06 Test: 34.41
> Difference: +1.0%
> org.apache.geode.benchmark.tests.PartitionedGetLongBenchmark
> average ops/second Baseline: 1038960.75 Test: 1039353.89
> Difference: +0.0%
> ops/second standard error Baseline: 1042.48 Test: 442.50
> Difference: -57.6%
> ops/second standard deviation Baseline: 18026.21 Test: 7651.57
> Difference: -57.6%
> YS 99th percentile latency Baseline: 1002.00 Test: 1002.00
> Difference: +0.0%
> median latency Baseline: 305151.00 Test: 305151.00
> Difference: +0.0%
> 90th percentile latency Baseline: 459519.00 Test: 459007.00
> Difference: -0.1%
> 99th percentile latency Baseline: 1418239.00 Test: 1424383.00
> Difference: +0.4%
> 99.9th percentile latency Baseline: 3729407.00 Test: 3788799.00
> Difference: +1.6%
> average latency Baseline: 345199.75 Test: 345276.27
> Difference: +0.0%
> latency standard deviation Baseline: 364741.13 Test: 387501.17
> Difference: +6.2%
> latency standard error Baseline: 20.66 Test: 21.95
> Difference: +6.2%
> org.apache.geode.benchmark.tests.PartitionedIndexedQueryBenchmark
> average ops/second Baseline: 18282.12 Test: 18247.08
> Difference: -0.2%
> ops/second standard error Baseline: 7.85 Test: 10.12
> Difference: +28.9%
> ops/second standard deviation Baseline: 135.73 Test: 174.91
> Difference: +28.9%
> YS 99th percentile latency Baseline: 20093.18 Test: 20092.77
> Difference: -0.0%
> median latency Baseline: 14557183.00 Test: 13950975.00
> Difference: -4.2%
> 90th percentile latency Baseline: 20594687.00 Test: 23232511.00
> Difference: +12.8%
> 99th percentile latency Baseline: 46235647.00 Test: 60358655.00
> Difference: +30.5%
> 99.9th percentile latency Baseline: 58621951.00 Test: 70582271.00
> Difference: +20.4%
> average latency Baseline: 15752630.91 Test: 15779064.59
> Difference: +0.2%
> latency standard deviation Baseline: 7386025.34 Test: 10516174.80
> Difference: +42.4%
> latency standard error Baseline: 3154.86 Test: 4495.79
> Difference: +42.5%
> org.apache.geode.benchmark.tests.PartitionedPutAllBenchmark
> average ops/second Baseline: 654.62 Test: 649.58
> Difference: -0.8%
> ops/second standard error Baseline: 3.82 Test: 4.17
> Difference: +9.4%
> ops/second standard deviation Baseline: 65.09 Test: 71.08
> Difference: +9.2%
> YS 99th percentile latency Baseline: 20099.00 Test: 20099.00
> Difference: +0.0%
> median latency Baseline: 67960831.00 Test: 69468159.00
> Difference: +2.2%
> 90th percentile latency Baseline: 217448447.00 Test: 218890239.00
> Difference: +0.7%
> 99th percentile latency Baseline: 312999935.00 Test: 316407807.00
> Difference: +1.1%
> 99.9th percentile latency Baseline: 395837439.00 Test: 390332415.00
> Difference: -1.4%
> average latency Baseline: 113043700.31 Test: 114136328.08
> Difference: +1.0%
> latency standard deviation Baseline: 77281720.43 Test: 77650547.89
> Difference: +0.5%
> latency standard error Baseline: 176865.03 Test: 178541.92
> Difference: +0.9%
> org.apache.geode.benchmark.tests.PartitionedPutAllLongBenchmark
> average ops/second Baseline: 651.36 Test: 645.87
> Difference: -0.8%
> ops/second standard error Baseline: 4.14 Test: 4.15
> Difference: +0.3%
> ops/second standard deviation Baseline: 70.46 Test: 70.77
> Difference: +0.4%
> YS 99th percentile latency Baseline: 20099.00 Test: 20099.00
> Difference: +0.0%
> median latency Baseline: 64454655.00 Test: 67895295.00
> Difference: +5.3%
> 90th percentile latency Baseline: 225574911.00 Test: 220069887.00
> Difference: -2.4%
> 99th percentile latency Baseline: 352583679.00 Test: 325844991.00
> Difference: -7.6%
> 99.9th percentile latency Baseline: 427032575.00 Test: 413925375.00
> Difference: -3.1%
> average latency Baseline: 113828250.36 Test: 114725443.16
> Difference: +0.8%
> latency standard deviation Baseline: 82646931.03 Test: 79661855.97
> Difference: -3.6%
> latency standard error Baseline: 189811.47 Test: 183679.48
> Difference: -3.2%
> org.apache.geode.benchmark.tests.PartitionedPutBenchmark
> average ops/second Baseline: 419374.09 Test: 414255.28
> Difference: -1.2%
> ops/second standard error Baseline: 996.43 Test: 736.95
> Difference: -26.0%
> ops/second standard deviation Baseline: 17229.96 Test: 12743.12
> Difference: -26.0%
> YS 99th percentile latency Baseline: 1501.00 Test: 1502.00
> Difference: +0.1%
> median latency Baseline: 676351.00 Test: 676351.00
> Difference: +0.0%
> 90th percentile latency Baseline: 993279.00 Test: 1026047.00
> Difference: +3.3%
> 99th percentile latency Baseline: 1644543.00 Test: 1753087.00
> Difference: +6.6%
> 99.9th percentile latency Baseline: 66191359.00 Test: 65404927.00
> Difference: -1.2%
> average latency Baseline: 857166.67 Test: 867866.53
> Difference: +1.2%
> latency standard deviation Baseline: 2941133.91 Test: 2966121.87
> Difference: +0.8%
> latency standard error Baseline: 262.23 Test: 266.09
> Difference: +1.5%
> org.apache.geode.benchmark.tests.PartitionedPutLongBenchmark
> average ops/second Baseline: 411434.00 Test: 407384.65
> Difference: -1.0%
> ops/second standard error Baseline: 698.57 Test: 757.94
> Difference: +8.5%
> ops/second standard deviation Baseline: 12079.39 Test: 13127.90
> Difference: +8.7%
> YS 99th percentile latency Baseline: 1501.00 Test: 1502.00
> Difference: +0.1%
> median latency Baseline: 673279.00 Test: 674815.00
> Difference: +0.2%
> 90th percentile latency Baseline: 1010687.00 Test: 1026559.00
> Difference: +1.6%
> 99th percentile latency Baseline: 1696767.00 Test: 1762303.00
> Difference: +3.9%
> 99.9th percentile latency Baseline: 73400319.00 Test: 74842111.00
> Difference: +2.0%
> average latency Baseline: 873796.73 Test: 882648.37
> Difference: +1.0%
> latency standard deviation Baseline: 3343859.26 Test: 3397106.08
> Difference: +1.6%
> latency standard error Baseline: 301.01 Test: 307.33
> Difference: +2.1%
> org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionBenchmark
> average ops/second Baseline: 385497.82 Test: 383556.25
> Difference: -0.5%
> ops/second standard error Baseline: 157.81 Test: 187.92
> Difference: +19.1%
> ops/second standard deviation Baseline: 2728.75 Test: 3249.37
> Difference: +19.1%
> YS 99th percentile latency Baseline: 575.00 Test: 575.25
> Difference: +0.0%
> median latency Baseline: 273919.00 Test: 275455.00
> Difference: +0.6%
> 90th percentile latency Baseline: 368127.00 Test: 369663.00
> Difference: +0.4%
> 99th percentile latency Baseline: 461055.00 Test: 471551.00
> Difference: +2.3%
> 99.9th percentile latency Baseline: 886783.00 Test: 671743.00
> Difference: -24.2%
> average latency Baseline: 279243.44 Test: 280609.57
> Difference: +0.5%
> latency standard deviation Baseline: 188613.48 Test: 186659.89
> Difference: -1.0%
> latency standard error Baseline: 17.54 Test: 17.40
> Difference: -0.8%
> org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionWithArgumentsBenchmark
> average ops/second Baseline: 595408.61 Test: 615069.19
> Difference: +3.3%
> ops/second standard error Baseline: 549.38 Test: 402.08
> Difference: -26.8%
> ops/second standard deviation Baseline: 9499.75 Test: 6952.69
> Difference: -26.8%
> YS 99th percentile latency Baseline: 1505.00 Test: 1503.00
> Difference: -0.1%
> median latency Baseline: 731647.00 Test: 775679.00
> Difference: +6.0%
> 90th percentile latency Baseline: 1094655.00 Test: 1083391.00
> Difference: -1.0%
> 99th percentile latency Baseline: 9043967.00 Test: 6017023.00
> Difference: -33.5%
> 99.9th percentile latency Baseline: 22757375.00 Test: 23691263.00
> Difference: +4.1%
> average latency Baseline: 965601.30 Test: 934625.03
> Difference: -3.2%
> latency standard deviation Baseline: 1702556.73 Test: 1590548.30
> Difference: -6.6%
> latency standard error Baseline: 127.41 Test: 117.11
> Difference: -8.1%
> org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionWithFiltersBenchmark
> average ops/second Baseline: 584622.61 Test: 583232.58
> Difference: -0.2%
> ops/second standard error Baseline: 369.33 Test: 377.86
> Difference: +2.3%
> ops/second standard deviation Baseline: 6386.34 Test: 6533.81
> Difference: +2.3%
> YS 99th percentile latency Baseline: 1103.00 Test: 1103.00
> Difference: +0.0%
> median latency Baseline: 496639.00 Test: 520191.00
> Difference: +4.7%
> 90th percentile latency Baseline: 731135.00 Test: 736255.00
> Difference: +0.7%
> 99th percentile latency Baseline: 3874815.00 Test: 3444735.00
> Difference: -11.1%
> 99.9th percentile latency Baseline: 11780095.00 Test: 10878975.00
> Difference: -7.6%
> average latency Baseline: 614285.67 Test: 615818.03
> Difference: +0.2%
> latency standard deviation Baseline: 821732.66 Test: 723485.75
> Difference: -12.0%
> latency standard error Baseline: 62.05 Test: 54.70
> Difference: -11.9%
> org.apache.geode.benchmark.tests.ReplicatedGetBenchmark
> average ops/second Baseline: 846698.07 Test: 846876.85
> Difference: +0.0%
> ops/second standard error Baseline: 455.80 Test: 398.00
> Difference: -12.7%
> ops/second standard deviation Baseline: 7881.58 Test: 6893.61
> Difference: -12.5%
> YS 99th percentile latency Baseline: 905.00 Test: 905.00
> Difference: +0.0%
> median latency Baseline: 349183.00 Test: 348415.00
> Difference: -0.2%
> 90th percentile latency Baseline: 484095.00 Test: 486655.00
> Difference: +0.5%
> 99th percentile latency Baseline: 2435071.00 Test: 2369535.00
> Difference: -2.7%
> 99.9th percentile latency Baseline: 6770687.00 Test: 6574079.00
> Difference: -2.9%
> average latency Baseline: 424071.21 Test: 423929.31
> Difference: -0.0%
> latency standard deviation Baseline: 530295.63 Test: 521763.35
> Difference: -1.6%
> latency standard error Baseline: 33.28 Test: 32.74
> Difference: -1.6%
> org.apache.geode.benchmark.tests.ReplicatedGetLongBenchmark
> average ops/second Baseline: 1056863.08 Test: 1056316.60
> Difference: -0.1%
> ops/second standard error Baseline: 476.01 Test: 533.16
> Difference: +12.0%
> ops/second standard deviation Baseline: 8231.02 Test: 9219.21
> Difference: +12.0%
> YS 99th percentile latency Baseline: 1002.00 Test: 1001.00
> Difference: -0.1%
> median latency Baseline: 299775.00 Test: 299775.00
> Difference: +0.0%
> 90th percentile latency Baseline: 451071.00 Test: 448255.00
> Difference: -0.6%
> 99th percentile latency Baseline: 1371135.00 Test: 1392639.00
> Difference: +1.6%
> 99.9th percentile latency Baseline: 3262463.00 Test: 3409919.00
> Difference: +4.5%
> average latency Baseline: 339496.49 Test: 339654.77
> Difference: +0.0%
> latency standard deviation Baseline: 342396.65 Test: 373899.72
> Difference: +9.2%
> latency standard error Baseline: 19.23 Test: 21.01
> Difference: +9.2%
> org.apache.geode.benchmark.tests.ReplicatedIndexedQueryBenchmark
> average ops/second Baseline: 18229.31 Test: 18224.52
> Difference: -0.0%
> ops/second standard error Baseline: 10.16 Test: 8.23
> Difference: -19.0%
> ops/second standard deviation Baseline: 175.60 Test: 142.52
> Difference: -18.8%
> YS 99th percentile latency Baseline: 20093.73 Test: 20093.00
> Difference: -0.0%
> median latency Baseline: 12836863.00 Test: 13041663.00
> Difference: +1.6%
> 90th percentile latency Baseline: 26427391.00 Test: 25903103.00
> Difference: -2.0%
> 99th percentile latency Baseline: 71172095.00 Test: 69992447.00
> Difference: -1.7%
> 99.9th percentile latency Baseline: 83427327.00 Test: 82182143.00
> Difference: -1.5%
> average latency Baseline: 15800858.04 Test: 15802259.29
> Difference: +0.0%
> latency standard deviation Baseline: 13064421.06 Test: 12769643.21
> Difference: -2.3%
> latency standard error Baseline: 5588.98 Test: 5463.10
> Difference: -2.3%
> org.apache.geode.benchmark.tests.ReplicatedPutAllBenchmark
> average ops/second Baseline: 686.64 Test: 699.59
> Difference: +1.9%
> ops/second standard error Baseline: 4.27 Test: 4.15
> Difference: -2.8%
> ops/second standard deviation Baseline: 73.34 Test: 71.06
> Difference: -3.1%
> YS 99th percentile latency Baseline: 20099.00 Test: 20099.00
> Difference: +0.0%
> median latency Baseline: 77856767.00 Test: 73531391.00
> Difference: -5.6%
> 90th percentile latency Baseline: 204079103.00 Test: 202899455.00
> Difference: -0.6%
> 99th percentile latency Baseline: 330563583.00 Test: 324534271.00
> Difference: -1.8%
> 99.9th percentile latency Baseline: 408682495.00 Test: 369098751.00
> Difference: -9.7%
> average latency Baseline: 106331049.47 Test: 104780049.58
> Difference: -1.5%
> latency standard deviation Baseline: 71143931.88 Test: 70631162.82
> Difference: -0.7%
> latency standard error Baseline: 157915.25 Test: 155630.00
> Difference: -1.4%
> org.apache.geode.benchmark.tests.ReplicatedPutAllLongBenchmark
> average ops/second Baseline: 684.32 Test: 701.87
> Difference: +2.6%
> ops/second standard error Baseline: 4.09 Test: 4.03
> Difference: -1.4%
> ops/second standard deviation Baseline: 70.20 Test: 69.19
> Difference: -1.4%
> YS 99th percentile latency Baseline: 20099.00 Test: 20099.00
> Difference: +0.0%
> median latency Baseline: 74579967.00 Test: 72941567.00
> Difference: -2.2%
> 90th percentile latency Baseline: 207880191.00 Test: 203554815.00
> Difference: -2.1%
> 99th percentile latency Baseline: 344719359.00 Test: 333447167.00
> Difference: -3.3%
> 99.9th percentile latency Baseline: 392691711.00 Test: 376176639.00
> Difference: -4.2%
> average latency Baseline: 106559160.43 Test: 104315677.10
> Difference: -2.1%
> latency standard deviation Baseline: 73550701.86 Test: 71595613.07
> Difference: -2.7%
> latency standard error Baseline: 163397.60 Test: 157398.24
> Difference: -3.7%
> org.apache.geode.benchmark.tests.ReplicatedPutBenchmark
> average ops/second Baseline: 439483.79 Test: 434437.21
> Difference: -1.1%
> ops/second standard error Baseline: 819.57 Test: 746.63
> Difference: -8.9%
> ops/second standard deviation Baseline: 14171.70 Test: 12910.43
> Difference: -8.9%
> YS 99th percentile latency Baseline: 1302.00 Test: 1304.00
> Difference: +0.2%
> median latency Baseline: 662015.00 Test: 662015.00
> Difference: +0.0%
> 90th percentile latency Baseline: 894463.00 Test: 925183.00
> Difference: +3.4%
> 99th percentile latency Baseline: 1791999.00 Test: 1894399.00
> Difference: +5.7%
> 99.9th percentile latency Baseline: 69730303.00 Test: 68091903.00
> Difference: -2.3%
> average latency Baseline: 818083.55 Test: 826988.90
> Difference: +1.1%
> latency standard deviation Baseline: 2889557.20 Test: 2861827.56
> Difference: -1.0%
> latency standard error Baseline: 251.67 Test: 250.67
> Difference: -0.4%
> org.apache.geode.benchmark.tests.ReplicatedPutLongBenchmark
> average ops/second Baseline: 430225.52 Test: 422842.19
> Difference: -1.7%
> ops/second standard error Baseline: 808.44 Test: 736.25
> Difference: -8.9%
> ops/second standard deviation Baseline: 13979.30 Test: 12730.95
> Difference: -8.9%
> YS 99th percentile latency Baseline: 1303.00 Test: 1303.00
> Difference: +0.0%
> median latency Baseline: 662527.00 Test: 664063.00
> Difference: +0.2%
> 90th percentile latency Baseline: 912383.00 Test: 931839.00
> Difference: +2.1%
> 99th percentile latency Baseline: 1898495.00 Test: 1918975.00
> Difference: +1.1%
> 99.9th percentile latency Baseline: 73138175.00 Test: 80674815.00
> Difference: +10.3%
> average latency Baseline: 835773.93 Test: 850337.27
> Difference: +1.7%
> latency standard deviation Baseline: 3198061.99 Test: 3392288.63
> Difference: +6.1%
> latency standard error Baseline: 281.54 Test: 301.23
> Difference: +7.0%
> BENCHMARK FAILED:
> org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark
> average latency is 5% worse than baseline.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
