[
https://issues.apache.org/jira/browse/CASSANDRA-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616361#comment-14616361
]
Jeff Williams commented on CASSANDRA-9577:
------------------------------------------
I am the user mentioned by Rob above. I have run 3 major compactions now and
after each compaction it creates a new sstable, but leaves the old. About 5
hours after the compaction, JMX said I had 3 live sstables, though there were
14 in the data directory. After restarting cassandra it reported 8 live
sstables, so some came back from the dead.
Right now, JMX is reporting 10 sstables and there are 10 sstables in the data
directory, but with 6 of them from before the last major compaction. 57372 was
the result of the last compaction.
$ ls -l /var/lib/cassandra/data/trackcontent/track_content/*Data.db
-rw-r--r-- 2 cassandra cassandra 16642164891 Jun 29 06:55
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30852-Data.db
-rw-r--r-- 2 cassandra cassandra 17216513377 Jun 30 08:36
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30866-Data.db
-rw-r--r-- 2 cassandra cassandra 813683923 Jun 30 12:03
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-37136-Data.db
-rw-r--r-- 2 cassandra cassandra 855070477 Jun 30 13:15
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-43371-Data.db
-rw-r--r-- 2 cassandra cassandra 209921645 Jun 30 13:28
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-44676-Data.db
-rw-r--r-- 2 cassandra cassandra 213532360 Jul 2 03:16
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-51845-Data.db
-rw-r--r-- 2 cassandra cassandra 16916868412 Jul 6 13:34
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57372-Data.db
-rw-r--r-- 2 cassandra cassandra 256127620 Jul 6 20:27
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57377-Data.db
-rw-r--r-- 2 cassandra cassandra 8740917 Jul 7 02:10
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57378-Data.db
-rw-r--r-- 2 cassandra cassandra 1371689 Jul 7 03:23
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57379-Data.db
How is this bad for consistency? I have been assuming (perhaps incorrectly)
that reads will hit the new sstables first, so they should get the correct data
without hitting the old sstables.
I now have 3 copies of the dataset stored for this table on this node and am
getting close to filling the disk (70%). I am happy to just remove the node,
but I am wondering if you want me to try something first?
I do not see any errors in the logs, except for a exception due to "disk full"
when I tried a "nodetool cleanup" after adding a new node to the cluster. I am
assuming that this is related.
> Cassandra not performing GC on stale SStables after compaction
> --------------------------------------------------------------
>
> Key: CASSANDRA-9577
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9577
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 2.0.12.200 / DSE 4.6.1.
> Reporter: Jeff Ferland
> Assignee: Marcus Eriksson
>
> Space used (live), bytes: 878681716067
> Space used (total), bytes: 2227857083852
> jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ sudo lsof *-Data.db
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> java 4473 cassandra 446r REG 0,26 17582559172 39241
> trends-trends-jb-144864-Data.db
> java 4473 cassandra 448r REG 0,26 62040962 37431
> trends-trends-jb-144731-Data.db
> java 4473 cassandra 449r REG 0,26 829935047545 21150
> trends-trends-jb-143581-Data.db
> java 4473 cassandra 452r REG 0,26 8980406 39503
> trends-trends-jb-144882-Data.db
> java 4473 cassandra 454r REG 0,26 8980406 39503
> trends-trends-jb-144882-Data.db
> java 4473 cassandra 462r REG 0,26 9487703 39542
> trends-trends-jb-144883-Data.db
> java 4473 cassandra 463r REG 0,26 36158226 39629
> trends-trends-jb-144889-Data.db
> java 4473 cassandra 468r REG 0,26 105693505 39447
> trends-trends-jb-144881-Data.db
> java 4473 cassandra 530r REG 0,26 17582559172 39241
> trends-trends-jb-144864-Data.db
> java 4473 cassandra 535r REG 0,26 105693505 39447
> trends-trends-jb-144881-Data.db
> java 4473 cassandra 542r REG 0,26 9487703 39542
> trends-trends-jb-144883-Data.db
> java 4473 cassandra 553u REG 0,26 6431729821 39556
> trends-trends-tmp-jb-144884-Data.db
> jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ ls *-Data.db
> trends-trends-jb-142631-Data.db trends-trends-jb-143562-Data.db
> trends-trends-jb-143581-Data.db trends-trends-jb-144731-Data.db
> trends-trends-jb-144883-Data.db
> trends-trends-jb-142633-Data.db trends-trends-jb-143563-Data.db
> trends-trends-jb-144530-Data.db trends-trends-jb-144864-Data.db
> trends-trends-jb-144889-Data.db
> trends-trends-jb-143026-Data.db trends-trends-jb-143564-Data.db
> trends-trends-jb-144551-Data.db trends-trends-jb-144881-Data.db
> trends-trends-tmp-jb-144884-Data.db
> trends-trends-jb-143533-Data.db trends-trends-jb-143578-Data.db
> trends-trends-jb-144552-Data.db trends-trends-jb-144882-Data.db
> jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ cd -
> /mnt/cassandra/data/trends/trends
> jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ sudo lsof *
> jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ ls *-Data.db
> trends-trends-jb-124502-Data.db trends-trends-jb-141113-Data.db
> trends-trends-jb-141377-Data.db trends-trends-jb-141846-Data.db
> trends-trends-jb-144890-Data.db
> trends-trends-jb-125457-Data.db trends-trends-jb-141123-Data.db
> trends-trends-jb-141391-Data.db trends-trends-jb-141871-Data.db
> trends-trends-jb-41121-Data.db
> trends-trends-jb-130016-Data.db trends-trends-jb-141137-Data.db
> trends-trends-jb-141538-Data.db trends-trends-jb-141883-Data.db
> trends-trends.trends_date_idx-jb-2100-Data.db
> trends-trends-jb-139563-Data.db trends-trends-jb-141358-Data.db
> trends-trends-jb-141806-Data.db trends-trends-jb-142033-Data.db
> trends-trends-jb-141102-Data.db trends-trends-jb-141363-Data.db
> trends-trends-jb-141829-Data.db trends-trends-jb-144553-Data.db
> Compaction started INFO [CompactionExecutor:6661] 2015-06-05 14:02:36,515
> CompactionTask.java (line 120) Compacting
> [SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-124502-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141358-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141883-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141846-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141871-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141391-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-139563-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-125457-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141806-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141102-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141123-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-41121-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141137-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141113-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-142033-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141829-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141377-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141538-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141363-Data.db'),
>
> SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-130016-Data.db')]
> Compaction ended: INFO [CompactionExecutor:6661] 2015-06-09 06:54:27,527
> CompactionTask.java (line 296) Compacted 20 sstables to
> [/ebs/cassandra/data/trends/trends/trends-trends-jb-143581,].
> 1,260,539,306,144 bytes to 829,935,047,545 (~65% of original) in
> 319,910,989ms = 2.474087MB/s. 1,913,263,636 total partitions merged to
> 873,858,309. Partition merge counts were {1:391213541, 2:241603436,
> 3:102139024, 4:52438965, 5:47749892, 6:13962210, 7:9923980, 8:8363815,
> 9:3626260, 10:1413014, 11:615314, 12:336503, 13:416749, 14:55606, }
> date: Wed Jun 10 23:43:31 UTC 2015
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
