[
https://issues.apache.org/jira/browse/CASSANDRA-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369517#comment-16369517
]
Jay Zhuang commented on CASSANDRA-13929:
----------------------------------------
Seems like the performance is better without {{Recycler}}. Here is microbench
test result to build a BTree with and without {{Recycler}}
([13929-3.11-perf|https://github.com/cooldoger/cassandra/tree/13929-3.11-perf])
(The score is operation per ms, higher is better)
{noformat}
[java] Benchmark (dataSize) (treeBuilder)
Mode Cnt Score Error Units
[java] BTreeBuildBench.buildTreeTest 1 treeBuilderRecycleAdd
thrpt 6 23112.102 ? 17522.471 ops/ms
[java] BTreeBuildBench.buildTreeTest 1 treeBuilderAdd
thrpt 6 46275.541 ? 60422.458 ops/ms
[java] BTreeBuildBench.buildTreeTest 2 treeBuilderRecycleAdd
thrpt 6 23588.176 ? 16372.260 ops/ms
[java] BTreeBuildBench.buildTreeTest 2 treeBuilderAdd
thrpt 6 42838.298 ? 25339.870 ops/ms
[java] BTreeBuildBench.buildTreeTest 5 treeBuilderRecycleAdd
thrpt 6 24358.111 ? 24339.382 ops/ms
[java] BTreeBuildBench.buildTreeTest 5 treeBuilderAdd
thrpt 6 60074.551 ? 47329.418 ops/ms
[java] BTreeBuildBench.buildTreeTest 10 treeBuilderRecycleAdd
thrpt 6 21412.578 ? 6072.160 ops/ms
[java] BTreeBuildBench.buildTreeTest 10 treeBuilderAdd
thrpt 6 50862.304 ? 30597.546 ops/ms
[java] BTreeBuildBench.buildTreeTest 20 treeBuilderRecycleAdd
thrpt 6 15871.754 ? 5036.739 ops/ms
[java] BTreeBuildBench.buildTreeTest 20 treeBuilderAdd
thrpt 6 33699.725 ? 10857.366 ops/ms
[java] BTreeBuildBench.buildTreeTest 40 treeBuilderRecycleAdd
thrpt 6 5168.225 ? 1212.571 ops/ms
[java] BTreeBuildBench.buildTreeTest 40 treeBuilderAdd
thrpt 6 8806.838 ? 7736.485 ops/ms
[java] BTreeBuildBench.buildTreeTest 100 treeBuilderRecycleAdd
thrpt 6 2114.218 ? 639.589 ops/ms
[java] BTreeBuildBench.buildTreeTest 100 treeBuilderAdd
thrpt 6 3213.333 ? 486.126 ops/ms
[java] BTreeBuildBench.buildTreeTest 1000 treeBuilderRecycleAdd
thrpt 6 335.523 ? 101.230 ops/ms
[java] BTreeBuildBench.buildTreeTest 1000 treeBuilderAdd
thrpt 6 386.678 ? 333.534 ops/ms
[java] BTreeBuildBench.buildTreeTest 10000 treeBuilderRecycleAdd
thrpt 6 35.644 ? 32.171 ops/ms
[java] BTreeBuildBench.buildTreeTest 10000 treeBuilderAdd
thrpt 6 44.250 ? 8.180 ops/ms
[java] BTreeBuildBench.buildTreeTest 100000 treeBuilderRecycleAdd
thrpt 6 3.073 ? 2.165 ops/ms
[java] BTreeBuildBench.buildTreeTest 100000 treeBuilderAdd
thrpt 6 4.651 ? 4.137 ops/ms
{noformat}
And I think the performance gain for CASSANDRA-9766 is not because of the
{{Recycler}} for BTree builder.
With {{Recycler}}, it does reduce the GC by reserving the memory. But for P95
and sometime P99, {{noRecycler}} is still better. Here is the test result with
percentiles (score is time per operation, so smaller is better):
{noformat}
[java] Benchmark (dataSize)
(treeBuilder) Mode Cnt Score Error Units
[java] BTreeBuildBench.buildTreeTest 1
treeBuilderRecycleAdd sample 9244739 1435.033 ? 111.857 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 1
treeBuilderRecycleAdd sample 92.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 1
treeBuilderRecycleAdd sample 638.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 1
treeBuilderRecycleAdd sample 1688.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 1
treeBuilderRecycleAdd sample 2292.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 1
treeBuilderRecycleAdd sample 4544.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 1
treeBuilderRecycleAdd sample 13792.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 1
treeBuilderRecycleAdd sample 124233.984 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 1
treeBuilderRecycleAdd sample 104202240.000 ns/op
[java] BTreeBuildBench.buildTreeTest 1
treeBuilderAdd sample 9334292 880.361 ? 99.171 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 1
treeBuilderAdd sample 74.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 1
treeBuilderAdd sample 177.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 1
treeBuilderAdd sample 557.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 1
treeBuilderAdd sample 2376.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 1
treeBuilderAdd sample 9248.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 1
treeBuilderAdd sample 19200.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 1
treeBuilderAdd sample 92489.050 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 1
treeBuilderAdd sample 66256896.000 ns/op
[java] BTreeBuildBench.buildTreeTest 2
treeBuilderRecycleAdd sample 9066713 1655.971 ? 127.345 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 2
treeBuilderRecycleAdd sample 101.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 2
treeBuilderRecycleAdd sample 756.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 2
treeBuilderRecycleAdd sample 1952.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 2
treeBuilderRecycleAdd sample 2684.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 2
treeBuilderRecycleAdd sample 5696.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 2
treeBuilderRecycleAdd sample 16272.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 2
treeBuilderRecycleAdd sample 189176.730 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 2
treeBuilderRecycleAdd sample 124387328.000 ns/op
[java] BTreeBuildBench.buildTreeTest 2
treeBuilderAdd sample 9816463 1264.312 ? 161.134 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 2
treeBuilderAdd sample 91.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 2
treeBuilderAdd sample 209.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 2
treeBuilderAdd sample 579.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 2
treeBuilderAdd sample 2096.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 2
treeBuilderAdd sample 8104.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 2
treeBuilderAdd sample 17088.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 2
treeBuilderAdd sample 227162.522 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 2
treeBuilderAdd sample 96731136.000 ns/op
[java] BTreeBuildBench.buildTreeTest 5
treeBuilderRecycleAdd sample 8811507 2044.981 ? 142.626 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 5
treeBuilderRecycleAdd sample 126.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 5
treeBuilderRecycleAdd sample 934.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 5
treeBuilderRecycleAdd sample 2384.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 5
treeBuilderRecycleAdd sample 3120.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 5
treeBuilderRecycleAdd sample 5808.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 5
treeBuilderRecycleAdd sample 17280.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 5
treeBuilderRecycleAdd sample 312320.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 5
treeBuilderRecycleAdd sample 117702656.000 ns/op
[java] BTreeBuildBench.buildTreeTest 5
treeBuilderAdd sample 9992328 1044.948 ? 131.496 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 5
treeBuilderAdd sample 90.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 5
treeBuilderAdd sample 260.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 5
treeBuilderAdd sample 607.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 5
treeBuilderAdd sample 2156.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 5
treeBuilderAdd sample 8384.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 5
treeBuilderAdd sample 17856.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 5
treeBuilderAdd sample 98048.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 5
treeBuilderAdd sample 132382720.000 ns/op
[java] BTreeBuildBench.buildTreeTest 10
treeBuilderRecycleAdd sample 8757941 1896.943 ? 116.106 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 10
treeBuilderRecycleAdd sample 144.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 10
treeBuilderRecycleAdd sample 917.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 10
treeBuilderRecycleAdd sample 2152.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 10
treeBuilderRecycleAdd sample 2980.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 10
treeBuilderRecycleAdd sample 6648.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 10
treeBuilderRecycleAdd sample 17056.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 10
treeBuilderRecycleAdd sample 260050.739 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 10
treeBuilderRecycleAdd sample 60030976.000 ns/op
[java] BTreeBuildBench.buildTreeTest 10
treeBuilderAdd sample 9456081 1066.351 ? 109.690 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 10
treeBuilderAdd sample 127.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 10
treeBuilderAdd sample 333.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 10
treeBuilderAdd sample 534.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 10
treeBuilderAdd sample 1884.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 10
treeBuilderAdd sample 7264.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 10
treeBuilderAdd sample 17760.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 10
treeBuilderAdd sample 123570.150 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 10
treeBuilderAdd sample 92274688.000 ns/op
[java] BTreeBuildBench.buildTreeTest 20
treeBuilderRecycleAdd sample 8754165 2601.277 ? 131.160 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 20
treeBuilderRecycleAdd sample 246.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 20
treeBuilderRecycleAdd sample 1696.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 20
treeBuilderRecycleAdd sample 3680.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 20
treeBuilderRecycleAdd sample 4472.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 20
treeBuilderRecycleAdd sample 6864.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 20
treeBuilderRecycleAdd sample 18720.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 20
treeBuilderRecycleAdd sample 229269.350 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 20
treeBuilderRecycleAdd sample 166461440.000 ns/op
[java] BTreeBuildBench.buildTreeTest 20
treeBuilderAdd sample 10323487 1596.081 ? 150.936 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 20
treeBuilderAdd sample 230.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 20
treeBuilderAdd sample 627.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 20
treeBuilderAdd sample 1019.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 20
treeBuilderAdd sample 2500.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 20
treeBuilderAdd sample 7784.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 20
treeBuilderAdd sample 19584.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 20
treeBuilderAdd sample 176973.414 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 20
treeBuilderAdd sample 103546880.000 ns/op
[java] BTreeBuildBench.buildTreeTest 40
treeBuilderRecycleAdd sample 8413786 6104.243 ? 195.964 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 40
treeBuilderRecycleAdd sample 674.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 40
treeBuilderRecycleAdd sample 4136.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 40
treeBuilderRecycleAdd sample 7528.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 40
treeBuilderRecycleAdd sample 8960.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 40
treeBuilderRecycleAdd sample 12976.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 40
treeBuilderRecycleAdd sample 31008.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 40
treeBuilderRecycleAdd sample 2341360.845 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 40
treeBuilderRecycleAdd sample 79560704.000 ns/op
[java] BTreeBuildBench.buildTreeTest 40
treeBuilderAdd sample 8477065 4486.574 ? 230.760 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 40
treeBuilderAdd sample 699.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 40
treeBuilderAdd sample 2156.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 40
treeBuilderAdd sample 4312.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 40
treeBuilderAdd sample 5376.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 40
treeBuilderAdd sample 9520.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 40
treeBuilderAdd sample 28032.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 40
treeBuilderAdd sample 4583153.664 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 40
treeBuilderAdd sample 163840000.000 ns/op
[java] BTreeBuildBench.buildTreeTest 100
treeBuilderRecycleAdd sample 7535010 11631.027 ? 225.736 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 100
treeBuilderRecycleAdd sample 1826.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 100
treeBuilderRecycleAdd sample 7504.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 100
treeBuilderRecycleAdd sample 16960.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 100
treeBuilderRecycleAdd sample 21408.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 100
treeBuilderRecycleAdd sample 31808.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 100
treeBuilderRecycleAdd sample 88576.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 100
treeBuilderRecycleAdd sample 10534912.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 100
treeBuilderRecycleAdd sample 100270080.000 ns/op
[java] BTreeBuildBench.buildTreeTest 100
treeBuilderAdd sample 7564546 10905.697 ? 235.164 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 100
treeBuilderAdd sample 1636.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 100
treeBuilderAdd sample 7280.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 100
treeBuilderAdd sample 14416.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 100
treeBuilderAdd sample 18976.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 100
treeBuilderAdd sample 32512.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 100
treeBuilderAdd sample 63040.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 100
treeBuilderAdd sample 8708838.195 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 100
treeBuilderAdd sample 60293120.000 ns/op
[java] BTreeBuildBench.buildTreeTest 1000
treeBuilderRecycleAdd sample 3117183 101556.600 ? 688.145 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 1000
treeBuilderRecycleAdd sample 15136.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 1000
treeBuilderRecycleAdd sample 80256.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 1000
treeBuilderRecycleAdd sample 169728.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 1000
treeBuilderRecycleAdd sample 206848.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 1000
treeBuilderRecycleAdd sample 295424.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 1000
treeBuilderRecycleAdd sample 573440.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 1000
treeBuilderRecycleAdd sample 16580608.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 1000
treeBuilderRecycleAdd sample 79953920.000 ns/op
[java] BTreeBuildBench.buildTreeTest 1000
treeBuilderAdd sample 3635046 81844.338 ? 695.009 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 1000
treeBuilderAdd sample 17120.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 1000
treeBuilderAdd sample 55936.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 1000
treeBuilderAdd sample 133120.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 1000
treeBuilderAdd sample 168448.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 1000
treeBuilderAdd sample 227584.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 1000
treeBuilderAdd sample 956367.872 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 1000
treeBuilderAdd sample 20004709.990 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 1000
treeBuilderAdd sample 75235328.000 ns/op
[java] BTreeBuildBench.buildTreeTest 10000
treeBuilderRecycleAdd sample 396416 806300.548 ? 5006.312 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 10000
treeBuilderRecycleAdd sample 178176.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 10000
treeBuilderRecycleAdd sample 578560.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 10000
treeBuilderRecycleAdd sample 1439744.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 10000
treeBuilderRecycleAdd sample 2101248.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 10000
treeBuilderRecycleAdd sample 2367488.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 10000
treeBuilderRecycleAdd sample 16711680.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 10000
treeBuilderRecycleAdd sample 27893461.811 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 10000
treeBuilderRecycleAdd sample 69468160.000 ns/op
[java] BTreeBuildBench.buildTreeTest 10000
treeBuilderAdd sample 544398 586738.267 ? 3652.668 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 10000
treeBuilderAdd sample 180992.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 10000
treeBuilderAdd sample 467456.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 10000
treeBuilderAdd sample 898048.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 10000
treeBuilderAdd sample 1052672.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 10000
treeBuilderAdd sample 1515520.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 10000
treeBuilderAdd sample 16334848.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 10000
treeBuilderAdd sample 28628756.070 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 10000
treeBuilderAdd sample 75759616.000 ns/op
[java] BTreeBuildBench.buildTreeTest 100000
treeBuilderRecycleAdd sample 44146 7258758.048 ? 85405.244 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 100000
treeBuilderRecycleAdd sample 1724416.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 100000
treeBuilderRecycleAdd sample 5701632.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 100000
treeBuilderRecycleAdd sample 11796480.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 100000
treeBuilderRecycleAdd sample 15056896.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 100000
treeBuilderRecycleAdd sample 28868608.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 100000
treeBuilderRecycleAdd sample 61781180.416 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 100000
treeBuilderRecycleAdd sample 72159775.949 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 100000
treeBuilderRecycleAdd sample 153092096.000 ns/op
[java] BTreeBuildBench.buildTreeTest 100000
treeBuilderAdd sample 50573 6333830.725 ? 79209.874 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00 100000
treeBuilderAdd sample 1839104.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50 100000
treeBuilderAdd sample 4587520.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90 100000
treeBuilderAdd sample 10764288.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95 100000
treeBuilderAdd sample 16629760.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99 100000
treeBuilderAdd sample 28737536.000 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999 100000
treeBuilderAdd sample 57427755.008 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999 100000
treeBuilderAdd sample 92259640.934 ns/op
[java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00 100000
treeBuilderAdd sample 117833728.000 ns/op
{noformat}
So I would suggest removing {{Recycler}} (similar to
[dc9ed46|https://github.com/apache/cassandra/commit/dc9ed463417aa8028e77e91718e4f3d6ea563210#diff-6aa10752b68c93ed354d642f9bdbe814L27]).
> BTree$Builder / io.netty.util.Recycler$Stack leaking memory
> -----------------------------------------------------------
>
> Key: CASSANDRA-13929
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13929
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Thomas Steinmaurer
> Assignee: Jay Zhuang
> Priority: Major
> Fix For: 3.11.x
>
> Attachments: cassandra_3.11.0_min_memory_utilization.jpg,
> cassandra_3.11.1_NORECYCLE_memory_utilization.jpg,
> cassandra_3.11.1_mat_dominator_classes.png,
> cassandra_3.11.1_mat_dominator_classes_FIXED.png,
> cassandra_3.11.1_snapshot_heaputilization.png,
> cassandra_3.11.1_vs_3.11.2recyclernullingpatch.png,
> dtest_example_80_request.png, dtest_example_80_request_fix.png,
> dtest_example_heap.png, memleak_heapdump_recyclerstack.png
>
>
> Different to CASSANDRA-13754, there seems to be another memory leak in
> 3.11.0+ in BTree$Builder / io.netty.util.Recycler$Stack.
> * heap utilization increase after upgrading to 3.11.0 =>
> cassandra_3.11.0_min_memory_utilization.jpg
> * No difference after upgrading to 3.11.1 (snapshot build) =>
> cassandra_3.11.1_snapshot_heaputilization.png; thus most likely after fixing
> CASSANDRA-13754, more visible now
> * MAT shows io.netty.util.Recycler$Stack as top contributing class =>
> cassandra_3.11.1_mat_dominator_classes.png
> * With -Xmx8G (CMS) and our load pattern, we have to do a rolling restart
> after ~ 72 hours
> Verified the following fix, namely explicitly unreferencing the
> _recycleHandle_ member (making it non-final). In
> _org.apache.cassandra.utils.btree.BTree.Builder.recycle()_
> {code}
> public void recycle()
> {
> if (recycleHandle != null)
> {
> this.cleanup();
> builderRecycler.recycle(this, recycleHandle);
> recycleHandle = null; // ADDED
> }
> }
> {code}
> Patched a single node in our loadtest cluster with this change and after ~ 10
> hours uptime, no sign of the previously offending class in MAT anymore =>
> cassandra_3.11.1_mat_dominator_classes_FIXED.png
> Can' say if this has any other side effects etc., but I doubt.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
