[
https://issues.apache.org/jira/browse/CASSANDRA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
HUANG DUICAN updated CASSANDRA-14953:
-------------------------------------
Attachment: cassandra.yaml
> Failed to reclaim the memory and too many MemtableReclaimMemory pending task
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-14953
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14953
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Memtable
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
> Reporter: HUANG DUICAN
> Priority: Major
> Attachments: 1.PNG, 2.PNG, cassandra.yaml, cassandra_20190105.zip
>
>
> We found that Cassandra has a lot of write accumulation in the production
> environment, and our business has experienced a lot of write failures.
> Through the system.log, it was found that MemtableReclaimMemory was pending
> at the beginning, and then a large number of MutationStage stacks appeared at
> a certain moment.
> Finally, the heap memory is full, the GC time reaches tens of seconds, the
> node status is DN through nodetool, but the Cassandra process is still
> running.We killed the node and restarted the node, and the above situation
> disappeared.
>
> Also the number of Active MemtableReclaimMemory threads seems to stay at 1.
> (you can see the 1.PNG)
> a large number of MutationStage stacks appeared at a certain moment.
> (you can see the 2.PNG)
>
> long GC time:
> - MemtableReclaimMemory 1 156 24565 0 0
> - G1 Old Generation GC in 87121ms. G1 Old Gen: 51175946656 -> 50082999760;
> - MutationStage 128 11931622 1983820772 0 0
> - CounterMutationStage 0 0 0 0 0
> - MemtableReclaimMemory 1 156 24565 0 0
> - G1 Young Generation GC in {color:#FF0000}969ms{color}. G1 Eden Space:
> 1090519040 -> 0; G1 Old Gen: 50082999760 -> 51156741584;
> - MutationStage 128 11953653 1983820772 0 0
> - CounterMutationStage 0 0 0 0 0
> - MemtableReclaimMemory 1 156 24565 0 0
> - G1 Old Generation GC in {color:#FF0000}84785ms{color}. G1 Old Gen:
> 51173518800 -> 50180911432;
> - MutationStage 128 11967484 1983820772 0 0
> - CounterMutationStage 0 0 0 0 0
> - MemtableReclaimMemory 1 156 24565 0 0
> - G1 Young Generation GC in 611ms. G1 Eden Space: 989855744 -> 0; G1 Old
> Gen: 50180911432 -> 51153989960;
> - MutationStage 128 11975849 1983820772 0 0
> - CounterMutationStage 0 0 0 0 0
> - MemtableReclaimMemory 1 156 24565 0 0
> - G1 Old Generation GC in {color:#FF0000}85845ms{color}. G1 Old Gen:
> 51170767176 -> 50238295416;
> - MutationStage 128 11978192 1983820772 0 0
> - CounterMutationStage 0 0 0 0 0
> - MemtableReclaimMemory 1 156 24565 0 0
> - G1 Young Generation GC in 602ms. G1 Eden Space: 939524096 -> 0; G1 Old
> Gen: 50238295416 -> 51161042296;
> - MutationStage 128 11994295 1983820772 0 0
> - CounterMutationStage 0 0 0 0 0
> - MemtableReclaimMemory 1 156 24565 0 0
> - G1 Old Generation GC in {color:#FF0000}85307ms{color}. G1 Old Gen:
> 51177819512 -> 50288829624; Metaspace: 36544536 -> 36525696
> - MutationStage 128 12001932 1983820772 0 0
> - CounterMutationStage 0 0 0 0 0
> 66 - MutationStage 128 12004395 1983820772 0 0
> 66 - CounterMutationStage 0 0 0 0 0
> - MemtableReclaimMemory 1 156 24565 0 0
> 66 - MemtableReclaimMemory 1 156 24565 0 0
> - G1 Young Generation GC in 610ms. G1 Eden Space: 889192448 -> 0; G1 Old
> Gen: 50288829624 -> 51178022072;
> - MutationStage 128 12023677 1983820772 0 0
> Why is this happening?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
