Hey all,
First off, thank you for taking the time to read this.
-------------------
SYSTEM SPEC
-------------------
We're using Cassandra Version 2.1.6 (please don't ask us to upgrade just yet
less you are aware of an existing bug for this issue)
We are running on AWS 4 core . 16 GB server
We are running a 4 core Cluster
We are writing about 1220 rows per second
We are querying about 3400 rows per second
There are 100 Key spaces each with 10 CF's
Our Replication factor is 3
All our data has a TTL of 10 seconds
Our "gc_grace_seconds" is 0
Durable writes is disabled
Read & write Consistency is ONE
Our HEAP size is 8GB (rest of the heap is default)
Our Compaction Strategy is Level Tiered
-XX:CMSInitiatingOccupancyFraction=50
-------------------
Cassandra.yaml
-------------------
Below are the values we've changed
hinted_handoff_enabled: false
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /cassandra
saved_caches_directory: /var/lib/cassandra/saved_caches
memtable_offheap_space_in_mb: 4096
memtable_cleanup_threshold: 0.99
memtable_allocation_type: offheap_objects
memtable_flush_writers: 4
thrift_framed_transport_size_in_mb: 150
concurrent_compactors: 4
compaction_throughput_mb_per_sec: 0
endpoint_snitch: GossipingPropertyFileSnitch
-------------------
Issue
-------------------
Over time all our CF's develop about 1 SSTable per 30 min.
Each SSTable contains only 36 rows with data that is all "Marked for deletion"
Over about 15 hours we could have up to 112000 SSTables over all nodes for all
CF's
This grinds our system to a halt and causes a major GC nearly every second.
So far the only way to get around this is to run a cron job every hour that
does a "nodetool compact".
Is there any reason that these SSTables are not being deleted during normal
compaction / minor GC / major GC ?
Regards
Stephen Walsh
This email (including any attachments) is proprietary to Aspect Software, Inc.
and may contain information that is confidential. If you have received this
message in error, please do not read, copy or forward this message. Please
notify the sender immediately, delete it from your system and destroy any
copies. You may not further disclose or distribute this email or its
attachments.
