Jai Bheemsen Rao Dhanwada created CASSANDRA-14840:
-----------------------------------------------------
Summary: Bootstrap of new node fails with OOM in a large cluster
Key: CASSANDRA-14840
URL: https://issues.apache.org/jira/browse/CASSANDRA-14840
Project: Cassandra
Issue Type: Bug
Components: Streaming and Messaging
Reporter: Jai Bheemsen Rao Dhanwada
We are seeing new node addition fails with OOM during bootstrap in a cluster of
more than 80 nodes and 3000 CF without any data in those CFs.
Steps to reproduce:
# Launch a 3 node cluster
# Create 3000 CF in the cluster
# Start adding nodes to the cluster one by one
# After adding 75-80 nodes, the new node bootstrap fails with OOM.
{code:java}
ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2018-10-24 03:26:47,870
JVMStabilityInspector.java:78 - Exiting due to error while processing commit
log during initialization.
java.lang.OutOfMemoryError: Java heap space
at java.util.regex.Pattern.matcher(Pattern.java:1093) ~[na:1.8.0_151]
at java.util.Formatter.parse(Formatter.java:2547) ~[na:1.8.0_151]
at java.util.Formatter.format(Formatter.java:2501) ~[na:1.8.0_151]
at java.util.Formatter.format(Formatter.java:2455) ~[na:1.8.0_151]
at java.lang.String.format(String.java:2940) ~[na:1.8.0_151]
at
org.apache.cassandra.db.commitlog.AbstractCommitLogService$1.run(AbstractCommitLogService.java:105)
~[apache-cassandra-2.1.16.jar:2.1.16]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]{code}
Cassandra Version: 2.1.16
OS: CentOS7
num_tokens: 256 on each node.
This behavior is blocking us from adding extra capacity when needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
