[ 
https://issues.apache.org/jira/browse/CASSANDRA-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369517#comment-16369517
 ] 

Jay Zhuang commented on CASSANDRA-13929:
----------------------------------------

Seems like the performance is better without {{Recycler}}. Here is microbench 
test result to build a BTree with and without {{Recycler}} 
([13929-3.11-perf|https://github.com/cooldoger/cassandra/tree/13929-3.11-perf]) 
(The score is operation per ms, higher is better)
{noformat}
     [java] Benchmark                      (dataSize)          (treeBuilder)   
Mode  Cnt      Score       Error   Units
     [java] BTreeBuildBench.buildTreeTest           1  treeBuilderRecycleAdd  
thrpt    6  23112.102 ? 17522.471  ops/ms
     [java] BTreeBuildBench.buildTreeTest           1         treeBuilderAdd  
thrpt    6  46275.541 ? 60422.458  ops/ms
     [java] BTreeBuildBench.buildTreeTest           2  treeBuilderRecycleAdd  
thrpt    6  23588.176 ? 16372.260  ops/ms
     [java] BTreeBuildBench.buildTreeTest           2         treeBuilderAdd  
thrpt    6  42838.298 ? 25339.870  ops/ms
     [java] BTreeBuildBench.buildTreeTest           5  treeBuilderRecycleAdd  
thrpt    6  24358.111 ? 24339.382  ops/ms
     [java] BTreeBuildBench.buildTreeTest           5         treeBuilderAdd  
thrpt    6  60074.551 ? 47329.418  ops/ms
     [java] BTreeBuildBench.buildTreeTest          10  treeBuilderRecycleAdd  
thrpt    6  21412.578 ?  6072.160  ops/ms
     [java] BTreeBuildBench.buildTreeTest          10         treeBuilderAdd  
thrpt    6  50862.304 ? 30597.546  ops/ms
     [java] BTreeBuildBench.buildTreeTest          20  treeBuilderRecycleAdd  
thrpt    6  15871.754 ?  5036.739  ops/ms
     [java] BTreeBuildBench.buildTreeTest          20         treeBuilderAdd  
thrpt    6  33699.725 ? 10857.366  ops/ms
     [java] BTreeBuildBench.buildTreeTest          40  treeBuilderRecycleAdd  
thrpt    6   5168.225 ?  1212.571  ops/ms
     [java] BTreeBuildBench.buildTreeTest          40         treeBuilderAdd  
thrpt    6   8806.838 ?  7736.485  ops/ms
     [java] BTreeBuildBench.buildTreeTest         100  treeBuilderRecycleAdd  
thrpt    6   2114.218 ?   639.589  ops/ms
     [java] BTreeBuildBench.buildTreeTest         100         treeBuilderAdd  
thrpt    6   3213.333 ?   486.126  ops/ms
     [java] BTreeBuildBench.buildTreeTest        1000  treeBuilderRecycleAdd  
thrpt    6    335.523 ?   101.230  ops/ms
     [java] BTreeBuildBench.buildTreeTest        1000         treeBuilderAdd  
thrpt    6    386.678 ?   333.534  ops/ms
     [java] BTreeBuildBench.buildTreeTest       10000  treeBuilderRecycleAdd  
thrpt    6     35.644 ?    32.171  ops/ms
     [java] BTreeBuildBench.buildTreeTest       10000         treeBuilderAdd  
thrpt    6     44.250 ?     8.180  ops/ms
     [java] BTreeBuildBench.buildTreeTest      100000  treeBuilderRecycleAdd  
thrpt    6      3.073 ?     2.165  ops/ms
     [java] BTreeBuildBench.buildTreeTest      100000         treeBuilderAdd  
thrpt    6      4.651 ?     4.137  ops/ms
{noformat}
And I think the performance gain for CASSANDRA-9766 is not because of the 
{{Recycler}} for BTree builder.

With {{Recycler}}, it does reduce the GC by reserving the memory. But for P95 
and sometime P99, {{noRecycler}} is still better. Here is the test result with 
percentiles (score is time per operation, so smaller is better):
{noformat}
     [java] Benchmark                                            (dataSize)     
     (treeBuilder)    Mode       Cnt          Score       Error  Units
     [java] BTreeBuildBench.buildTreeTest                                 1  
treeBuilderRecycleAdd  sample   9244739       1435.033 ?   111.857  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00             1  
treeBuilderRecycleAdd  sample                   92.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50             1  
treeBuilderRecycleAdd  sample                  638.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90             1  
treeBuilderRecycleAdd  sample                 1688.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95             1  
treeBuilderRecycleAdd  sample                 2292.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99             1  
treeBuilderRecycleAdd  sample                 4544.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999            1  
treeBuilderRecycleAdd  sample                13792.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999           1  
treeBuilderRecycleAdd  sample               124233.984              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00             1  
treeBuilderRecycleAdd  sample            104202240.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                                 1     
    treeBuilderAdd  sample   9334292        880.361 ?    99.171  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00             1     
    treeBuilderAdd  sample                   74.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50             1     
    treeBuilderAdd  sample                  177.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90             1     
    treeBuilderAdd  sample                  557.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95             1     
    treeBuilderAdd  sample                 2376.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99             1     
    treeBuilderAdd  sample                 9248.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999            1     
    treeBuilderAdd  sample                19200.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999           1     
    treeBuilderAdd  sample                92489.050              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00             1     
    treeBuilderAdd  sample             66256896.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                                 2  
treeBuilderRecycleAdd  sample   9066713       1655.971 ?   127.345  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00             2  
treeBuilderRecycleAdd  sample                  101.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50             2  
treeBuilderRecycleAdd  sample                  756.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90             2  
treeBuilderRecycleAdd  sample                 1952.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95             2  
treeBuilderRecycleAdd  sample                 2684.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99             2  
treeBuilderRecycleAdd  sample                 5696.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999            2  
treeBuilderRecycleAdd  sample                16272.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999           2  
treeBuilderRecycleAdd  sample               189176.730              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00             2  
treeBuilderRecycleAdd  sample            124387328.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                                 2     
    treeBuilderAdd  sample   9816463       1264.312 ?   161.134  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00             2     
    treeBuilderAdd  sample                   91.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50             2     
    treeBuilderAdd  sample                  209.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90             2     
    treeBuilderAdd  sample                  579.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95             2     
    treeBuilderAdd  sample                 2096.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99             2     
    treeBuilderAdd  sample                 8104.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999            2     
    treeBuilderAdd  sample                17088.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999           2     
    treeBuilderAdd  sample               227162.522              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00             2     
    treeBuilderAdd  sample             96731136.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                                 5  
treeBuilderRecycleAdd  sample   8811507       2044.981 ?   142.626  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00             5  
treeBuilderRecycleAdd  sample                  126.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50             5  
treeBuilderRecycleAdd  sample                  934.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90             5  
treeBuilderRecycleAdd  sample                 2384.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95             5  
treeBuilderRecycleAdd  sample                 3120.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99             5  
treeBuilderRecycleAdd  sample                 5808.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999            5  
treeBuilderRecycleAdd  sample                17280.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999           5  
treeBuilderRecycleAdd  sample               312320.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00             5  
treeBuilderRecycleAdd  sample            117702656.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                                 5     
    treeBuilderAdd  sample   9992328       1044.948 ?   131.496  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00             5     
    treeBuilderAdd  sample                   90.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50             5     
    treeBuilderAdd  sample                  260.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90             5     
    treeBuilderAdd  sample                  607.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95             5     
    treeBuilderAdd  sample                 2156.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99             5     
    treeBuilderAdd  sample                 8384.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999            5     
    treeBuilderAdd  sample                17856.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999           5     
    treeBuilderAdd  sample                98048.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00             5     
    treeBuilderAdd  sample            132382720.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                                10  
treeBuilderRecycleAdd  sample   8757941       1896.943 ?   116.106  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00            10  
treeBuilderRecycleAdd  sample                  144.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50            10  
treeBuilderRecycleAdd  sample                  917.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90            10  
treeBuilderRecycleAdd  sample                 2152.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95            10  
treeBuilderRecycleAdd  sample                 2980.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99            10  
treeBuilderRecycleAdd  sample                 6648.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999           10  
treeBuilderRecycleAdd  sample                17056.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999          10  
treeBuilderRecycleAdd  sample               260050.739              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00            10  
treeBuilderRecycleAdd  sample             60030976.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                                10     
    treeBuilderAdd  sample   9456081       1066.351 ?   109.690  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00            10     
    treeBuilderAdd  sample                  127.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50            10     
    treeBuilderAdd  sample                  333.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90            10     
    treeBuilderAdd  sample                  534.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95            10     
    treeBuilderAdd  sample                 1884.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99            10     
    treeBuilderAdd  sample                 7264.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999           10     
    treeBuilderAdd  sample                17760.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999          10     
    treeBuilderAdd  sample               123570.150              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00            10     
    treeBuilderAdd  sample             92274688.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                                20  
treeBuilderRecycleAdd  sample   8754165       2601.277 ?   131.160  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00            20  
treeBuilderRecycleAdd  sample                  246.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50            20  
treeBuilderRecycleAdd  sample                 1696.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90            20  
treeBuilderRecycleAdd  sample                 3680.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95            20  
treeBuilderRecycleAdd  sample                 4472.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99            20  
treeBuilderRecycleAdd  sample                 6864.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999           20  
treeBuilderRecycleAdd  sample                18720.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999          20  
treeBuilderRecycleAdd  sample               229269.350              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00            20  
treeBuilderRecycleAdd  sample            166461440.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                                20     
    treeBuilderAdd  sample  10323487       1596.081 ?   150.936  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00            20     
    treeBuilderAdd  sample                  230.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50            20     
    treeBuilderAdd  sample                  627.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90            20     
    treeBuilderAdd  sample                 1019.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95            20     
    treeBuilderAdd  sample                 2500.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99            20     
    treeBuilderAdd  sample                 7784.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999           20     
    treeBuilderAdd  sample                19584.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999          20     
    treeBuilderAdd  sample               176973.414              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00            20     
    treeBuilderAdd  sample            103546880.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                                40  
treeBuilderRecycleAdd  sample   8413786       6104.243 ?   195.964  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00            40  
treeBuilderRecycleAdd  sample                  674.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50            40  
treeBuilderRecycleAdd  sample                 4136.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90            40  
treeBuilderRecycleAdd  sample                 7528.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95            40  
treeBuilderRecycleAdd  sample                 8960.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99            40  
treeBuilderRecycleAdd  sample                12976.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999           40  
treeBuilderRecycleAdd  sample                31008.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999          40  
treeBuilderRecycleAdd  sample              2341360.845              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00            40  
treeBuilderRecycleAdd  sample             79560704.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                                40     
    treeBuilderAdd  sample   8477065       4486.574 ?   230.760  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00            40     
    treeBuilderAdd  sample                  699.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50            40     
    treeBuilderAdd  sample                 2156.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90            40     
    treeBuilderAdd  sample                 4312.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95            40     
    treeBuilderAdd  sample                 5376.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99            40     
    treeBuilderAdd  sample                 9520.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999           40     
    treeBuilderAdd  sample                28032.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999          40     
    treeBuilderAdd  sample              4583153.664              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00            40     
    treeBuilderAdd  sample            163840000.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                               100  
treeBuilderRecycleAdd  sample   7535010      11631.027 ?   225.736  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00           100  
treeBuilderRecycleAdd  sample                 1826.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50           100  
treeBuilderRecycleAdd  sample                 7504.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90           100  
treeBuilderRecycleAdd  sample                16960.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95           100  
treeBuilderRecycleAdd  sample                21408.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99           100  
treeBuilderRecycleAdd  sample                31808.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999          100  
treeBuilderRecycleAdd  sample                88576.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999         100  
treeBuilderRecycleAdd  sample             10534912.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00           100  
treeBuilderRecycleAdd  sample            100270080.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                               100     
    treeBuilderAdd  sample   7564546      10905.697 ?   235.164  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00           100     
    treeBuilderAdd  sample                 1636.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50           100     
    treeBuilderAdd  sample                 7280.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90           100     
    treeBuilderAdd  sample                14416.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95           100     
    treeBuilderAdd  sample                18976.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99           100     
    treeBuilderAdd  sample                32512.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999          100     
    treeBuilderAdd  sample                63040.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999         100     
    treeBuilderAdd  sample              8708838.195              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00           100     
    treeBuilderAdd  sample             60293120.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                              1000  
treeBuilderRecycleAdd  sample   3117183     101556.600 ?   688.145  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00          1000  
treeBuilderRecycleAdd  sample                15136.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50          1000  
treeBuilderRecycleAdd  sample                80256.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90          1000  
treeBuilderRecycleAdd  sample               169728.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95          1000  
treeBuilderRecycleAdd  sample               206848.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99          1000  
treeBuilderRecycleAdd  sample               295424.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999         1000  
treeBuilderRecycleAdd  sample               573440.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999        1000  
treeBuilderRecycleAdd  sample             16580608.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00          1000  
treeBuilderRecycleAdd  sample             79953920.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                              1000     
    treeBuilderAdd  sample   3635046      81844.338 ?   695.009  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00          1000     
    treeBuilderAdd  sample                17120.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50          1000     
    treeBuilderAdd  sample                55936.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90          1000     
    treeBuilderAdd  sample               133120.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95          1000     
    treeBuilderAdd  sample               168448.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99          1000     
    treeBuilderAdd  sample               227584.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999         1000     
    treeBuilderAdd  sample               956367.872              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999        1000     
    treeBuilderAdd  sample             20004709.990              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00          1000     
    treeBuilderAdd  sample             75235328.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                             10000  
treeBuilderRecycleAdd  sample    396416     806300.548 ?  5006.312  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00         10000  
treeBuilderRecycleAdd  sample               178176.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50         10000  
treeBuilderRecycleAdd  sample               578560.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90         10000  
treeBuilderRecycleAdd  sample              1439744.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95         10000  
treeBuilderRecycleAdd  sample              2101248.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99         10000  
treeBuilderRecycleAdd  sample              2367488.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999        10000  
treeBuilderRecycleAdd  sample             16711680.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999       10000  
treeBuilderRecycleAdd  sample             27893461.811              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00         10000  
treeBuilderRecycleAdd  sample             69468160.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                             10000     
    treeBuilderAdd  sample    544398     586738.267 ?  3652.668  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00         10000     
    treeBuilderAdd  sample               180992.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50         10000     
    treeBuilderAdd  sample               467456.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90         10000     
    treeBuilderAdd  sample               898048.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95         10000     
    treeBuilderAdd  sample              1052672.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99         10000     
    treeBuilderAdd  sample              1515520.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999        10000     
    treeBuilderAdd  sample             16334848.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999       10000     
    treeBuilderAdd  sample             28628756.070              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00         10000     
    treeBuilderAdd  sample             75759616.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                            100000  
treeBuilderRecycleAdd  sample     44146    7258758.048 ? 85405.244  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00        100000  
treeBuilderRecycleAdd  sample              1724416.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50        100000  
treeBuilderRecycleAdd  sample              5701632.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90        100000  
treeBuilderRecycleAdd  sample             11796480.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95        100000  
treeBuilderRecycleAdd  sample             15056896.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99        100000  
treeBuilderRecycleAdd  sample             28868608.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999       100000  
treeBuilderRecycleAdd  sample             61781180.416              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999      100000  
treeBuilderRecycleAdd  sample             72159775.949              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00        100000  
treeBuilderRecycleAdd  sample            153092096.000              ns/op
     [java] BTreeBuildBench.buildTreeTest                            100000     
    treeBuilderAdd  sample     50573    6333830.725 ? 79209.874  ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.00        100000     
    treeBuilderAdd  sample              1839104.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.50        100000     
    treeBuilderAdd  sample              4587520.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.90        100000     
    treeBuilderAdd  sample             10764288.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.95        100000     
    treeBuilderAdd  sample             16629760.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.99        100000     
    treeBuilderAdd  sample             28737536.000              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.999       100000     
    treeBuilderAdd  sample             57427755.008              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p0.9999      100000     
    treeBuilderAdd  sample             92259640.934              ns/op
     [java] BTreeBuildBench.buildTreeTest:buildTreeTest?p1.00        100000     
    treeBuilderAdd  sample            117833728.000              ns/op
{noformat}

So I would suggest removing {{Recycler}} (similar to 
[dc9ed46|https://github.com/apache/cassandra/commit/dc9ed463417aa8028e77e91718e4f3d6ea563210#diff-6aa10752b68c93ed354d642f9bdbe814L27]).

> BTree$Builder / io.netty.util.Recycler$Stack leaking memory
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-13929
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13929
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Thomas Steinmaurer
>            Assignee: Jay Zhuang
>            Priority: Major
>             Fix For: 3.11.x
>
>         Attachments: cassandra_3.11.0_min_memory_utilization.jpg, 
> cassandra_3.11.1_NORECYCLE_memory_utilization.jpg, 
> cassandra_3.11.1_mat_dominator_classes.png, 
> cassandra_3.11.1_mat_dominator_classes_FIXED.png, 
> cassandra_3.11.1_snapshot_heaputilization.png, 
> cassandra_3.11.1_vs_3.11.2recyclernullingpatch.png, 
> dtest_example_80_request.png, dtest_example_80_request_fix.png, 
> dtest_example_heap.png, memleak_heapdump_recyclerstack.png
>
>
> Different to CASSANDRA-13754, there seems to be another memory leak in 
> 3.11.0+ in BTree$Builder / io.netty.util.Recycler$Stack.
> * heap utilization increase after upgrading to 3.11.0 => 
> cassandra_3.11.0_min_memory_utilization.jpg
> * No difference after upgrading to 3.11.1 (snapshot build) => 
> cassandra_3.11.1_snapshot_heaputilization.png; thus most likely after fixing 
> CASSANDRA-13754, more visible now
> * MAT shows io.netty.util.Recycler$Stack as top contributing class => 
> cassandra_3.11.1_mat_dominator_classes.png
> * With -Xmx8G (CMS) and our load pattern, we have to do a rolling restart 
> after ~ 72 hours
> Verified the following fix, namely explicitly unreferencing the 
> _recycleHandle_ member (making it non-final). In 
> _org.apache.cassandra.utils.btree.BTree.Builder.recycle()_
> {code}
>         public void recycle()
>         {
>             if (recycleHandle != null)
>             {
>                 this.cleanup();
>                 builderRecycler.recycle(this, recycleHandle);
>                 recycleHandle = null; // ADDED
>             }
>         }
> {code}
> Patched a single node in our loadtest cluster with this change and after ~ 10 
> hours uptime, no sign of the previously offending class in MAT anymore => 
> cassandra_3.11.1_mat_dominator_classes_FIXED.png
> Can' say if this has any other side effects etc., but I doubt.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to