[ 
https://issues.apache.org/jira/browse/CASSANDRA-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361873#comment-16361873
 ] 

Jay Zhuang edited comment on CASSANDRA-13929 at 2/13/18 6:21 AM:
-----------------------------------------------------------------

I think it's just caching the largest tree ever built 
({{[BTree.java:907|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/btree/BTree.java#L907]}})
 and getting bigger and bigger.

Here is an example with a large number of {{IN}} in a CQL, which causes high 
heap usage for {{$Stack}} :
{noformat}
session.execute_async("select id from test.users where id in (%s)" % 
','.join(map(str, range(100000))))
{noformat}
Here is a dtest to reproduce that: 
{{[large_in_test.py|https://github.com/cooldoger/cassandra-dtest/blob/13929/large_in_test.py]}}
 {{$Stack}} uses 25% 10M heap: {{10M = 10 (request num) * 100K (element) * 8 
(obj size)}}
 !dtest_example_heap.png! 
 We could increase heap size and then increase request number, element number 
gradually to make {{$Stack}} to 80%+ heap usage. And I'm sure there're other 
use cases could build large BTree. Seems the {{$Stack}} is unable to release 
until the {{Thread}} is released and we don't that for {{ThreadPoolExecutor}}.

I'd like to propose the following patch which cleans large BTree builder ({{> 
1k elements}}) while recycling:
|Branch|uTest|
|[13929-3.11|https://github.com/cooldoger/cassandra/tree/13929-3.11]|[!https://circleci.com/gh/cooldoger/cassandra/tree/13929-3.11.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13929-3.11]|


was (Author: jay.zhuang):
I think it's just caching the largest tree ever built 
({{[BTree.java:907|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/btree/BTree.java#L907]}})
 and getting bigger and bigger.

Here is an example with a large number of {{IN}} in a CQL, which causes high 
heap usage for {{$Stack}} :
{noformat}
session.execute_async("select id from test.users where id in (%s)" % 
','.join(map(str, range(100000))))
{noformat}
Here is a dtest to reproduce that: 
{{[large_in_test.py|https://github.com/cooldoger/cassandra-dtest/blob/13929/large_in_test.py]}}
{{$Stack}} uses 25% 10M heap: {{10M = 10 (thread) * 100K (element) * 8 (obj 
size)}}
!dtest_example_heap.png! 
We could increase heap size and then increase thread number, element number 
gradually to make {{$Stack}} to 80%+ heap usage. And I'm sure there're other 
use cases could build large BTree. Seems the {{$Stack}} is unable to release 
until the {{Thread}} is released and we don't that for {{ThreadPoolExecutor}}.

I'd like to propose the following patch which cleans large BTree builder ({{> 
1k elements}}) while recycling:
| Branch | uTest |
| [13929-3.11|https://github.com/cooldoger/cassandra/tree/13929-3.11] | 
[!https://circleci.com/gh/cooldoger/cassandra/tree/13929-3.11.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13929-3.11]
 |


> BTree$Builder / io.netty.util.Recycler$Stack leaking memory
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-13929
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13929
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Thomas Steinmaurer
>            Priority: Major
>             Fix For: 3.11.x
>
>         Attachments: cassandra_3.11.0_min_memory_utilization.jpg, 
> cassandra_3.11.1_NORECYCLE_memory_utilization.jpg, 
> cassandra_3.11.1_mat_dominator_classes.png, 
> cassandra_3.11.1_mat_dominator_classes_FIXED.png, 
> cassandra_3.11.1_snapshot_heaputilization.png, 
> cassandra_3.11.1_vs_3.11.2recyclernullingpatch.png, dtest_example_heap.png
>
>
> Different to CASSANDRA-13754, there seems to be another memory leak in 
> 3.11.0+ in BTree$Builder / io.netty.util.Recycler$Stack.
> * heap utilization increase after upgrading to 3.11.0 => 
> cassandra_3.11.0_min_memory_utilization.jpg
> * No difference after upgrading to 3.11.1 (snapshot build) => 
> cassandra_3.11.1_snapshot_heaputilization.png; thus most likely after fixing 
> CASSANDRA-13754, more visible now
> * MAT shows io.netty.util.Recycler$Stack as top contributing class => 
> cassandra_3.11.1_mat_dominator_classes.png
> * With -Xmx8G (CMS) and our load pattern, we have to do a rolling restart 
> after ~ 72 hours
> Verified the following fix, namely explicitly unreferencing the 
> _recycleHandle_ member (making it non-final). In 
> _org.apache.cassandra.utils.btree.BTree.Builder.recycle()_
> {code}
>         public void recycle()
>         {
>             if (recycleHandle != null)
>             {
>                 this.cleanup();
>                 builderRecycler.recycle(this, recycleHandle);
>                 recycleHandle = null; // ADDED
>             }
>         }
> {code}
> Patched a single node in our loadtest cluster with this change and after ~ 10 
> hours uptime, no sign of the previously offending class in MAT anymore => 
> cassandra_3.11.1_mat_dominator_classes_FIXED.png
> Can' say if this has any other side effects etc., but I doubt.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to