Make data consistent (run repair), reduce gc_grace_seconds (try set it to 0 temporarily though careful as this can affect hinted handoff!) and set table’s compaction sub-property i.e. unchecked_tombstone_compaction to true. Compaction will take care of tombstones!
From: Jonathan Haddad [mailto:j...@jonhaddad.com] Sent: Thursday, February 01, 2018 11:29 AM To: user@cassandra.apache.org Subject: Re: Old tombstones not being cleaned up Changing the defaul TTL doesn’t change the TTL on the existing data, only new data. It’s only set if you don’t supply one yourself. On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen <bo.gunder...@gmail.com<mailto:bo.gunder...@gmail.com>> wrote: Hi, We are running a small 9 node Cassandra v2.1.17 cluster. The cluster generally runs fine, but we have one table that are causing OOMs because an enormous amount of tombstones. Looking at the data in the table (sstable2json), the first of the tombstones are almost a year old. The table was initially created with a gc_grace_period of 10 days, but I have now lowered it to 1 hour. I have run a full repair of the table across all nodes. I have forced several major compactions of the table by using "nodetool compact", and also tried to switch from LeveledCompaction to SizeTierCompaction and back. What could cause cassandra to keep these tombstones? sstable2json: {"key": "foo", "cells": [["0000082f-25ef-4324-bb8a-8cf013c823c1:_","0000082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135], ["000010f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","000010f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731], ["00001d7a-ce95-4c74-b67e-f8cdffec4f85:_","00001d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102], ["00001dd3-ae22-4f6e-944a-8cfa147cde68:_","00001dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006], ["000022cc-d69c-4596-89e5-3e976c0cb9a8:_","000022cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448], ["00002777-4b1a-4267-8efc-c43054e63170:_","00002777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691], ["000061e8-f48b-4484-96f1-f8b6a3ed8f9f:_","000061e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300], ["000063da-f165-449b-b65d-2b7869368734:_","000063da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634], ["0000656f-f8b5-472b-93ed-1a893002f027:_","0000656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716], ... {"key": "bar", "metadata": {"deletionInfo": {"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}}, "cells": [["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965], ["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186], ["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840], .... sstablemetadata: stablemetadata /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db SSTable: /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741 Partitioner: org.apache.cassandra.dht.Murmur3Partitioner Bloom Filter FP chance: 0.100000 Minimum timestamp: 1488976211688000 Maximum timestamp: 1517468644066000 SSTable max local deletion time: 2147483647 Compression ratio: 0.5121956624389545 Estimated droppable tombstones: 18.00161766553587 SSTable Level: 0 Repaired at: 0 ReplayPosition(segmentId=1517168739626, position=22690189) Estimated tombstone drop times:%n 1488976211: 1 1489906506: 4706 1490174752: 6111 1490449759: 6554 1490735410: 6559 1491016789: 6369 1491347982: 10216 1491680214: 13502 ... desc: CREATE TABLE xxx.yyy ( ti text, uuid text, json_data text, PRIMARY KEY (ti, uuid) ) WITH CLUSTERING ORDER BY (uuid ASC) AND bloom_filter_fp_chance = 0.1 AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' AND comment = '' AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 3600 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; jmx props(picture): [image.png]
