Hello All,
I'm having JVM unstable / OOM errors when attempting to auto bootstrap a
9th node to an existing 8 node cluster (256 tokens). Each machine has 24
cores 148GB RAM and 10TB (2TB used). Under normal operation the 8 nodes
have JVM memory configured with Xms35G and Xmx35G, and handle 2-4 billion
inserts per day. There are never updates, deletes, or sparsely populated
rows.
For the bootstrap node, I've tried memory values from 35GB to 135GB in 10GB
increments. I've tried using both memtable_allocation_types (heap_buffers
and offheap_buffers). I've not tried modifying the
memtable_cleanup_threshold but instead have tried memtable_flush_writers
from 2 to 8. I've tried memtable_(off)heap_space_in_mb from 20000 to
60000. I've tried both CMS and G1 garbage collection with various
settings.
Typically, after streaming about ~2TB of data, CPU load will hit a maximum,
and the "nodetool info" heap memory will, over the course of an hour,
approach the maximum. At that point, CPU load will drop to a single thread
with minimal activity until the system becomes unstable and eventually the
OOM error occurs.
Excerpt of the system log is below, and what I consistently see is the
MemtableFlushWriter and the MemtableReclaimMemory pending queues grow as
the memory becomes depleted, but the number of completed seems to stop
changing a few minutes after the CPU load spikes.
One other data point is there seems to be a huge number of mutations that
occur after most of the stream has occured. Concurrent_writes is set at
256 with the queue getting as high as 200K before dropping down.
Any suggestions for yaml changes or jvm changes? JVM.options is currently
the default with the memory set to the max, the current YAML file is below.
Thanks!
INFO [ScheduledTasks:1] 2018-08-06 17:49:26,329 StatusLogger.java:51 -
> MutationStage 1 2 191498052 0
> 0
> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,331 StatusLogger.java:51 -
> ViewMutationStage 0 0 0 0
> 0
> INFO [Service Thread] 2018-08-06 17:49:26,338 StatusLogger.java:51 -
> PerDiskMemtableFlushWriter_0 0 0 5865 0
> 0
> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,343 StatusLogger.java:51 -
> ReadStage 0 0 0 0
> 0
> INFO [Service Thread] 2018-08-06 17:49:26,347 StatusLogger.java:51 -
> ValidationExecutor 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,360 StatusLogger.java:51 -
> RequestResponseStage 0 0 8 0
> 0
> INFO [Service Thread] 2018-08-06 17:49:26,380 StatusLogger.java:51 -
> Sampler 0 0 0 0
> 0
> INFO [Service Thread] 2018-08-06 17:49:26,382 StatusLogger.java:51 -
> *MemtableFlushWriter
> 8 74293 4716 0 * 0
> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,388 StatusLogger.java:51 -
> ReadRepairStage 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,389 StatusLogger.java:51 -
> CounterMutationStage 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,404 StatusLogger.java:51 -
> MiscStage 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,407 StatusLogger.java:51 -
> CompactionExecutor 8 13 493 0
> 0
> INFO [Service Thread] 2018-08-06 17:49:26,410 StatusLogger.java:51 -
> InternalResponseStage 0 0 16 0
> 0
> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,413 StatusLogger.java:51 -
> *MemtableReclaimMemory
> 1 6066 356 0 * 0
> INFO [Service Thread] 2018-08-06 17:49:26,421 StatusLogger.java:51 -
> AntiEntropyStage 0 0 0 0
> 0
> INFO [Service Thread] 2018-08-06 17:49:26,430 StatusLogger.java:51 -
> CacheCleanupExecutor 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,431 StatusLogger.java:51 -
> PendingRangeCalculator 0 0 9 0
> 0
> INFO [Service Thread] 2018-08-06 17:49:26,436 StatusLogger.java:61 -
> CompactionManager 8 19
Current Yaml
num_tokens: 256
hinted_handoff_enabled: true
hinted_handoff_throttle_in_kb: 10240
max_hints_delivery_threads: 8
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 10240
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /data/cassandra/data
commitlog_directory: /data/cassandra/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
thrift_prepared_statements_cache_size_mb:
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /data/cassandra/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "10.1.1.11,10.1.1.12,10.1.1.13"
concurrent_reads: 128
concurrent_writes: 256
concurrent_counter_writes: 96
concurrent_materialized_view_writes: 32
disk_optimization_strategy: spinning
memtable_heap_space_in_mb: 61440
memtable_offheap_space_in_mb: 61440
memtable_allocation_type: heap_buffers
commitlog_total_space_in_mb: 81920
memtable_flush_writers: 8
