Re: TWCS not deleting expired sstables

[Thakrar, Jayesh]

Thank you so much Kurt - that helped!!

So here's the thing, I was logged into the server as my own id and had 
Cassandra binaries under my home directory with the default configuration.
Cassandra was running on the server with a service account and with its own 
user-id and configuration.

I assumed that like nodetool, this command would also use the Cassandra service 
JMX or server port.
However your suggestion made me realize that that's not the case.

So I copied the service account's Cassandra/conf to my local install and things 
worked.

I also looked up the source code and it did not indicate any "connection" via 
service or jmx port :)

So now need to see why the data is not getting purged.

Thanks again!!

From: kurt greaves <k...@instaclustr.com>
Date: Wednesday, January 31, 2018 at 11:27 PM
To: User <user@cassandra.apache.org>
Subject: Re: TWCS not deleting expired sstables

Well, that shouldn't happen. Seems like it's possibly not looking in the 
correct location for data directories. Try setting CASSANDRA_INCLUDE=http://cassandra.in.sh>> prior to running the script?
e.g: 
CASSANDRA_INCLUDE=/cassandra.in.sh<http://cassandra.in.sh>
 sstableexpiredblockers ae raw_logs_by_user

On 30 January 2018 at 15:34, Thakrar, Jayesh 
<jthak...@conversantmedia.com<mailto:jthak...@conversantmedia.com>> wrote:
Thanks Kurt and Kenneth.

Now only if they would work as expected.

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>ls
 -lt | tail
-rw-r--r--. 1 vchadoop vchadoop286889260 Sep 18 14:14 mc-1070-big-Index.db
-rw-r--r--. 1 vchadoop vchadoop12236 Sep 13 20:53 
mc-178-big-Statistics.db
-rw-r--r--. 1 vchadoop vchadoop   92 Sep 13 20:53 mc-178-big-TOC.txt
-rw-r--r--. 1 vchadoop vchadoop  9371211 Sep 13 20:53 
mc-178-big-CompressionInfo.db
-rw-r--r--. 1 vchadoop vchadoop   10 Sep 13 20:53 
mc-178-big-Digest.crc32
-rw-r--r--. 1 vchadoop vchadoop  13609890747<tel:(360)%20989-0747> Sep 13 20:53 
mc-178-big-Data.db
-rw-r--r--. 1 vchadoop vchadoop  1394968 Sep 13 20:53 mc-178-big-Summary.db
-rw-r--r--. 1 vchadoop vchadoop 11172592 Sep 13 20:53 mc-178-big-Filter.db
-rw-r--r--. 1 vchadoop vchadoop190508739 Sep 13 20:53 mc-178-big-Index.db
drwxr-xr-x. 2 vchadoop vchadoop   10 Sep 12 21:47 backups

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>sstableexpiredblockers
 ae raw_logs_by_user
Exception in thread "main" java.lang.IllegalArgumentException: Unknown 
keyspace/table ae.raw_logs_by_user
at 
org.apache.cassandra.tools.SSTableExpiredBlockers.main(SSTableExpiredBlockers.java:66)

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>sstableexpiredblockers
 system peers
No sstables for system.peers

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>ls
 -l ../../system/peers-37f71aca7dc2383ba70672528af04d4f/
total 308
drwxr-xr-x. 2 vchadoop vchadoop 10 Sep 11 22:59 backups
-rw-rw-r--. 1 vchadoop vchadoop 83 Jan 25 02:11 
mc-137-big-CompressionInfo.db
-rw-rw-r--. 1 vchadoop vchadoop 180369 Jan 25 02:11 mc-137-big-Data.db
-rw-rw-r--. 1 vchadoop vchadoop 10 Jan 25 02:11 mc-137-big-Digest.crc32
-rw-rw-r--. 1 vchadoop vchadoop 64 Jan 25 02:11 mc-137-big-Filter.db
-rw-rw-r--. 1 vchadoop vchadoop386 Jan 25 02:11 mc-137-big-Index.db
-rw-rw-r--. 1 vchadoop vchadoop   5171 Jan 25 02:11 mc-137-big-Statistics.db
-rw-rw-r--. 1 vchadoop vchadoop 56 Jan 25 02:11 mc-137-big-Summary.db
-rw-rw-r--. 1 vchadoop vchadoop 92 Jan 25 02:11 mc-137-big-TOC.txt
-rw-rw-r--. 1 vchadoop vchadoop 43 Jan 29 21:11 
mc-138-big-CompressionInfo.db
-rw-rw-r--. 1 vchadoop vchadoop   9723 Jan 29 21:11 mc-138-big-Data.db
-rw-rw-r--. 1 vchadoop vchadoop 10 Jan 29 21:11 mc-138-big-Digest.crc32
-rw-rw-r--. 1 vchadoop vchadoop 16 Jan 29 21:11 mc-138-big-Filter.db
-rw-rw-r--. 1 vchadoop vchadoop 17 Jan 29 21:11 mc-138-big-Index.db
-rw-rw-r--. 1 vchadoop vchadoop   5015 Jan 29 21:11 mc-138-big-Statistics.db
-rw-rw-r--. 1 vchadoop vchadoop 56 Jan 29 21:11 mc-138-big-Summary.db
-rw-rw-r--. 1 vchadoop vchadoop 92 Jan 29 21:11 mc-138-big-TOC.txt
-rw-rw-r--. 1 vchadoop vchadoop 43 Jan 29 21:53 
mc-139-big-CompressionInfo.db
-rw-rw-r--. 1 vchadoop vchadoop  18908 Jan 29 21:53 mc-139-big-Data.db
-rw-rw-r--. 1 vchadoop vchadoop 10 Jan 29 21:53 mc-139-big-Digest.crc32
-rw-rw-r--. 1 vchadoop vchadoop 16 Jan 29 21:53 mc-139-big-Filter.db
-rw-rw-r--. 1 vchadoop vchadoop 36 Jan 29 21:53 mc-139-big-Index.db
-rw-rw-r--. 1 vchadoop vchadoop   5055 Jan 29 21:53 mc-139-big-Statistics.db
-rw-rw-r--. 1 vchadoop vchadoop 56 Jan 29 21:53 mc-139-big-Summary.db
-rw-rw-r--. 1 vchadoop vchadoop 92 Jan 29 21:53 mc-139-big-TOC.txt



From: Kenneth Brotman <kenbrot...@yahoo.com.INVALID>
Date:

Re: TWCS not deleting expired sstables

[kurt greaves]

Well, that shouldn't happen. Seems like it's possibly not looking in the
correct location for data directories. Try setting CASSANDRA_INCLUDE= prior to running the script?
e.g: CASSANDRA_INCLUDE=/cassandra.in.sh
sstableexpiredblockers ae raw_logs_by_user

On 30 January 2018 at 15:34, Thakrar, Jayesh <jthak...@conversantmedia.com>
wrote:

> Thanks Kurt and Kenneth.
>
>
>
> Now only if they would work as expected.
>
>
>
> *node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>ls
> -lt | tail *
>
> -rw-r--r--. 1 vchadoop vchadoop286889260 Sep 18 14:14
> mc-1070-big-Index.db
>
> -rw-r--r--. 1 vchadoop vchadoop12236 Sep 13 20:53
> mc-178-big-Statistics.db
>
> -rw-r--r--. 1 vchadoop vchadoop   92 Sep 13 20:53
> mc-178-big-TOC.txt
>
> -rw-r--r--. 1 vchadoop vchadoop  9371211 Sep 13 20:53
> mc-178-big-CompressionInfo.db
>
> -rw-r--r--. 1 vchadoop vchadoop   10 Sep 13 20:53
> mc-178-big-Digest.crc32
>
> -rw-r--r--. 1 vchadoop vchadoop  13609890747 <(360)%20989-0747> Sep 13
> 20:53 mc-178-big-Data.db
>
> -rw-r--r--. 1 vchadoop vchadoop  1394968 Sep 13 20:53
> mc-178-big-Summary.db
>
> -rw-r--r--. 1 vchadoop vchadoop 11172592 Sep 13 20:53
> mc-178-big-Filter.db
>
> -rw-r--r--. 1 vchadoop vchadoop190508739 Sep 13 20:53
> mc-178-big-Index.db
>
> drwxr-xr-x. 2 vchadoop vchadoop   10 Sep 12 21:47 backups
>
>
>
> *node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>sstableexpiredblockers
> ae raw_logs_by_user*
>
> Exception in thread "main" java.lang.IllegalArgumentException: Unknown
> keyspace/table ae.raw_logs_by_user
>
> at org.apache.cassandra.tools.SSTableExpiredBlockers.main(
> SSTableExpiredBlockers.java:66)
>
>
>
> *node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>sstableexpiredblockers
> system peers*
>
> No sstables for system.peers
>
>
>
> *node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>ls
> -l ../../system/peers-37f71aca7dc2383ba70672528af04d4f/*
>
> total 308
>
> drwxr-xr-x. 2 vchadoop vchadoop 10 Sep 11 22:59 backups
>
> -rw-rw-r--. 1 vchadoop vchadoop 83 Jan 25 02:11
> mc-137-big-CompressionInfo.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 180369 Jan 25 02:11 mc-137-big-Data.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 10 Jan 25 02:11 mc-137-big-Digest.crc32
>
> -rw-rw-r--. 1 vchadoop vchadoop 64 Jan 25 02:11 mc-137-big-Filter.db
>
> -rw-rw-r--. 1 vchadoop vchadoop386 Jan 25 02:11 mc-137-big-Index.db
>
> -rw-rw-r--. 1 vchadoop vchadoop   5171 Jan 25 02:11
> mc-137-big-Statistics.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 56 Jan 25 02:11 mc-137-big-Summary.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 92 Jan 25 02:11 mc-137-big-TOC.txt
>
> -rw-rw-r--. 1 vchadoop vchadoop 43 Jan 29 21:11
> mc-138-big-CompressionInfo.db
>
> -rw-rw-r--. 1 vchadoop vchadoop   9723 Jan 29 21:11 mc-138-big-Data.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 10 Jan 29 21:11 mc-138-big-Digest.crc32
>
> -rw-rw-r--. 1 vchadoop vchadoop 16 Jan 29 21:11 mc-138-big-Filter.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 17 Jan 29 21:11 mc-138-big-Index.db
>
> -rw-rw-r--. 1 vchadoop vchadoop   5015 Jan 29 21:11
> mc-138-big-Statistics.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 56 Jan 29 21:11 mc-138-big-Summary.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 92 Jan 29 21:11 mc-138-big-TOC.txt
>
> -rw-rw-r--. 1 vchadoop vchadoop 43 Jan 29 21:53
> mc-139-big-CompressionInfo.db
>
> -rw-rw-r--. 1 vchadoop vchadoop  18908 Jan 29 21:53 mc-139-big-Data.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 10 Jan 29 21:53 mc-139-big-Digest.crc32
>
> -rw-rw-r--. 1 vchadoop vchadoop 16 Jan 29 21:53 mc-139-big-Filter.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 36 Jan 29 21:53 mc-139-big-Index.db
>
> -rw-rw-r--. 1 vchadoop vchadoop   5055 Jan 29 21:53
> mc-139-big-Statistics.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 56 Jan 29 21:53 mc-139-big-Summary.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 92 Jan 29 21:53 mc-139-big-TOC.txt
>
>
>
>
>
>
>
> *From: *Kenneth Brotman <kenbrot...@yahoo.com.INVALID>
> *Date: *Tuesday, January 30, 2018 at 7:37 AM
> *To: *<user@cassandra.apache.org>
> *Subject: *RE: TWCS not deleting expired sstables
>
>
>
> Wow!  It’s in the DataStax documentation: https://docs.datastax.com/en/
> dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/
> toolsSStabExpiredBlockers.html
>
>
>
> Other nice tools there as well: h

Re: TWCS not deleting expired sstables

[Thakrar, Jayesh]

Thanks Kurt and Kenneth.

Now only if they would work as expected.

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>ls
 -lt | tail
-rw-r--r--. 1 vchadoop vchadoop286889260 Sep 18 14:14 mc-1070-big-Index.db
-rw-r--r--. 1 vchadoop vchadoop12236 Sep 13 20:53 
mc-178-big-Statistics.db
-rw-r--r--. 1 vchadoop vchadoop   92 Sep 13 20:53 mc-178-big-TOC.txt
-rw-r--r--. 1 vchadoop vchadoop  9371211 Sep 13 20:53 
mc-178-big-CompressionInfo.db
-rw-r--r--. 1 vchadoop vchadoop   10 Sep 13 20:53 
mc-178-big-Digest.crc32
-rw-r--r--. 1 vchadoop vchadoop  13609890747 Sep 13 20:53 mc-178-big-Data.db
-rw-r--r--. 1 vchadoop vchadoop  1394968 Sep 13 20:53 mc-178-big-Summary.db
-rw-r--r--. 1 vchadoop vchadoop 11172592 Sep 13 20:53 mc-178-big-Filter.db
-rw-r--r--. 1 vchadoop vchadoop190508739 Sep 13 20:53 mc-178-big-Index.db
drwxr-xr-x. 2 vchadoop vchadoop   10 Sep 12 21:47 backups

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>sstableexpiredblockers
 ae raw_logs_by_user
Exception in thread "main" java.lang.IllegalArgumentException: Unknown 
keyspace/table ae.raw_logs_by_user
at 
org.apache.cassandra.tools.SSTableExpiredBlockers.main(SSTableExpiredBlockers.java:66)

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>sstableexpiredblockers
 system peers
No sstables for system.peers

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>ls
 -l ../../system/peers-37f71aca7dc2383ba70672528af04d4f/
total 308
drwxr-xr-x. 2 vchadoop vchadoop 10 Sep 11 22:59 backups
-rw-rw-r--. 1 vchadoop vchadoop 83 Jan 25 02:11 
mc-137-big-CompressionInfo.db
-rw-rw-r--. 1 vchadoop vchadoop 180369 Jan 25 02:11 mc-137-big-Data.db
-rw-rw-r--. 1 vchadoop vchadoop 10 Jan 25 02:11 mc-137-big-Digest.crc32
-rw-rw-r--. 1 vchadoop vchadoop 64 Jan 25 02:11 mc-137-big-Filter.db
-rw-rw-r--. 1 vchadoop vchadoop386 Jan 25 02:11 mc-137-big-Index.db
-rw-rw-r--. 1 vchadoop vchadoop   5171 Jan 25 02:11 mc-137-big-Statistics.db
-rw-rw-r--. 1 vchadoop vchadoop 56 Jan 25 02:11 mc-137-big-Summary.db
-rw-rw-r--. 1 vchadoop vchadoop 92 Jan 25 02:11 mc-137-big-TOC.txt
-rw-rw-r--. 1 vchadoop vchadoop 43 Jan 29 21:11 
mc-138-big-CompressionInfo.db
-rw-rw-r--. 1 vchadoop vchadoop   9723 Jan 29 21:11 mc-138-big-Data.db
-rw-rw-r--. 1 vchadoop vchadoop 10 Jan 29 21:11 mc-138-big-Digest.crc32
-rw-rw-r--. 1 vchadoop vchadoop 16 Jan 29 21:11 mc-138-big-Filter.db
-rw-rw-r--. 1 vchadoop vchadoop 17 Jan 29 21:11 mc-138-big-Index.db
-rw-rw-r--. 1 vchadoop vchadoop   5015 Jan 29 21:11 mc-138-big-Statistics.db
-rw-rw-r--. 1 vchadoop vchadoop 56 Jan 29 21:11 mc-138-big-Summary.db
-rw-rw-r--. 1 vchadoop vchadoop 92 Jan 29 21:11 mc-138-big-TOC.txt
-rw-rw-r--. 1 vchadoop vchadoop 43 Jan 29 21:53 
mc-139-big-CompressionInfo.db
-rw-rw-r--. 1 vchadoop vchadoop  18908 Jan 29 21:53 mc-139-big-Data.db
-rw-rw-r--. 1 vchadoop vchadoop 10 Jan 29 21:53 mc-139-big-Digest.crc32
-rw-rw-r--. 1 vchadoop vchadoop 16 Jan 29 21:53 mc-139-big-Filter.db
-rw-rw-r--. 1 vchadoop vchadoop 36 Jan 29 21:53 mc-139-big-Index.db
-rw-rw-r--. 1 vchadoop vchadoop   5055 Jan 29 21:53 mc-139-big-Statistics.db
-rw-rw-r--. 1 vchadoop vchadoop 56 Jan 29 21:53 mc-139-big-Summary.db
-rw-rw-r--. 1 vchadoop vchadoop 92 Jan 29 21:53 mc-139-big-TOC.txt



From: Kenneth Brotman <kenbrot...@yahoo.com.INVALID>
Date: Tuesday, January 30, 2018 at 7:37 AM
To: <user@cassandra.apache.org>
Subject: RE: TWCS not deleting expired sstables

Wow!  It’s in the DataStax documentation: 
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/toolsSStabExpiredBlockers.html

Other nice tools there as well: 
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/toolsSSTableUtilitiesTOC.html

Kenneth Brotman

From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Monday, January 29, 2018 8:20 PM
To: User
Subject: Re: TWCS not deleting expired sstables

Likely a read repair caused old data to be brought into a newer SSTable. Try 
running sstableexpiredblockers to find out if there's a newer SSTable blocking 
that one from being dropped.​

RE: TWCS not deleting expired sstables

[Kenneth Brotman]

Wow!  It’s in the DataStax documentation: 
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/toolsSStabExpiredBlockers.html

 

Other nice tools there as well: 
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/toolsSSTableUtilitiesTOC.html

 

Kenneth Brotman

 

From: kurt greaves [mailto:k...@instaclustr.com] 
Sent: Monday, January 29, 2018 8:20 PM
To: User
Subject: Re: TWCS not deleting expired sstables

 

Likely a read repair caused old data to be brought into a newer SSTable. Try 
running sstableexpiredblockers to find out if there's a newer SSTable blocking 
that one from being dropped.​

Re: TWCS not deleting expired sstables

[kurt greaves]

Likely a read repair caused old data to be brought into a newer SSTable.
Try running sstableexpiredblockers to find out if there's a newer SSTable
blocking that one from being dropped.​

Re: TWCS not deleting expired sstables

[Thakrar, Jayesh]

Hi Brian,

Thanks for your response.
Yes, I did look at that post.
In fact after reading that post, I set the "unchecked_tombstone_compaction" to 
true.

For the sstable in the example below (and its neighbors), all the data for 
those time windows have been compacted into a single sstable, so there is no 
dependency or delay caused from other sstables.

Thanks,
Jayesh


From: <brian.spind...@gmail.com>
Date: Sunday, January 28, 2018 at 1:02 PM
To: <user@cassandra.apache.org>
Subject: Re: TWCS not deleting expired sstables

I would start here:  http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html

Specifically the “Hints and repairs” and “Timestamp overlap” sections might be 
of use.
-B

On Jan 25, 2018, at 11:05 AM, Thakrar, Jayesh 
<jthak...@conversantmedia.com<mailto:jthak...@conversantmedia.com>> wrote:
Wondering if I can get some pointers to what's happening here and why sstables 
that I think should be expired are not being dropped.

Here's the table's compaction property - note also set 
"unchecked_tombstone_compaction" to true.

compaction = {'class': 
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
'compaction_window_size': '7', 'compaction_window_unit': 'DAYS', 
'max_threshold': '4', 'min_threshold': '4', 'unchecked_tombstone_compaction': 
'true'}

We insert data with timestamp and TTL programmatically.

Here's one set of sstable that I expect to be removed:

$ ls -lt *Data.db | tail -5
-rw-r--r--. 1 vchadoop vchadoop  31245097312 Sep 20 17:16 mc-1308-big-Data.db
-rw-r--r--. 1 vchadoop vchadoop  31524316252 Sep 19 14:27 mc-1187-big-Data.db
-rw-r--r--. 1 vchadoop vchadoop  21405216502 Sep 18 14:14 mc-1070-big-Data.db
-rw-r--r--. 1 vchadoop vchadoop  13609890747 Sep 13 20:53 mc-178-big-Data.db

$ date +%s
1516895877

$ date
Thu Jan 25 15:58:00 UTC 2018

$ sstablemetadata $PWD/mc-130-big-Data.db | head -20
SSTable: 
/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1/mc-130-big
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.01
Minimum timestamp: 14966028
Maximum timestamp: 14980788
SSTable min local deletion time: 1507924954
SSTable max local deletion time: 1509400832
Compressor: org.apache.cassandra.io.compress.LZ4Compressor
Compression ratio: 0.17430158132352797
TTL min: 2630598
TTL max: 4086188
First token: -9177441867697829836 (key=823134638755651936)
Last token: 9155171035305804798 (key=395118640769012487)
minClustringValues: [-1, da, 3, 1498082382078, -9223371818124305448, 
-9223371652504795402, -1]
maxClustringValues: [61818, tpt, 325, 149660280, -4611686088173246790, 
9223372014135560885, 1]
Estimated droppable tombstones: 1.1983492967652476
SSTable Level: 0
Repaired at: 0
Replay positions covered: {CommitLogPosition(segmentId=1505171071629, 
position=7157684)=CommitLogPosition(segmentId=1505171075152, position=6263269)}
totalColumnsSet: 111047277

Re: TWCS not deleting expired sstables

[brian . spindler]

I would start here:  http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html

Specifically the “Hints and repairs” and “Timestamp overlap” sections might be 
of use.  

-B

> On Jan 25, 2018, at 11:05 AM, Thakrar, Jayesh  
> wrote:
> 
> Wondering if I can get some pointers to what's happening here and why 
> sstables that I think should be expired are not being dropped.
>  
> Here's the table's compaction property - note also set 
> "unchecked_tombstone_compaction" to true.
>  
> compaction = {'class': 
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
> 'compaction_window_size': '7', 'compaction_window_unit': 'DAYS', 
> 'max_threshold': '4', 'min_threshold': '4', 'unchecked_tombstone_compaction': 
> 'true'}
>  
> We insert data with timestamp and TTL programmatically.
>  
> Here's one set of sstable that I expect to be removed:
>  
> $ ls -lt *Data.db | tail -5
> -rw-r--r--. 1 vchadoop vchadoop  31245097312 Sep 20 17:16 mc-1308-big-Data.db
> -rw-r--r--. 1 vchadoop vchadoop  31524316252 Sep 19 14:27 mc-1187-big-Data.db
> -rw-r--r--. 1 vchadoop vchadoop  21405216502 Sep 18 14:14 mc-1070-big-Data.db
> -rw-r--r--. 1 vchadoop vchadoop  13609890747 Sep 13 20:53 mc-178-big-Data.db
>  
> $ date +%s
> 1516895877
>  
> $ date 
> Thu Jan 25 15:58:00 UTC 2018
>  
> $ sstablemetadata $PWD/mc-130-big-Data.db | head -20
> SSTable: 
> /ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1/mc-130-big
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Bloom Filter FP chance: 0.01
> Minimum timestamp: 14966028
> Maximum timestamp: 14980788
> SSTable min local deletion time: 1507924954
> SSTable max local deletion time: 1509400832
> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
> Compression ratio: 0.17430158132352797
> TTL min: 2630598
> TTL max: 4086188
> First token: -9177441867697829836 (key=823134638755651936)
> Last token: 9155171035305804798 (key=395118640769012487)
> minClustringValues: [-1, da, 3, 1498082382078, -9223371818124305448, 
> -9223371652504795402, -1]
> maxClustringValues: [61818, tpt, 325, 149660280, -4611686088173246790, 
> 9223372014135560885, 1]
> Estimated droppable tombstones: 1.1983492967652476
> SSTable Level: 0
> Repaired at: 0
> Replay positions covered: {CommitLogPosition(segmentId=1505171071629, 
> position=7157684)=CommitLogPosition(segmentId=1505171075152, 
> position=6263269)}
> totalColumnsSet: 111047277
>