Hello.
I have a situation similar to
https://issues.apache.org/jira/browse/CASSANDRA-13153 except mine
cassandra is 3.11.1 (that issue should be fixed according to jira).
Cluster consist of 40 nodes which I have to shrink to 25 more powerful
nodes moving less powerful out from the cluster.
SizeTieredCompactionStrategy is used for target table with default
compaction options and default gc_grace_seconds (10 days).
Full sequential repair executed every day using
https://github.com/thelastpickle/cassandra-reaper
Before reaper was involved the default (incremental) 'nodetool repair'
was used.
There are couple of problems that I observe.
1. Compaction.
There are rows in target table which deleted long time ago
(gc_grace_seconds passed) but they are not compacting. I tried 'nodetool
compact' and 'nodetool repair -full -seq' with same outcome: sstables
recreated but that rows still there (I used sstabledump to detect this
state).
Some of that rows have tombstone and data in wrong order: the data
located at more recent sstable by index but relative tombstone is
located in previous sstable (shouldn't it be opposite?). CQL select does
not return that rows (correct behaviour) until node decommission.
sstables with non-compacted rows have very old repairedAt value (about
10 days before the first deleted row in sstable which should have been
compacted long time ago).
2. Streaming.
When moving the node out of cluster 'nodetool decommission' is used.
After streaming complete some old rows that was not compacted earlier is
back to life in shrinked cluster.
CQL select does return that rows as alive until running full sequential
repair using cassandra-reaper.
As workaround I tried to shut down a node and using 'nodetool
removenode' in case the node itself is streaming wrong data on
decommission but that did not work either (deleted data is back to life).
Is this a known issue?
PS: I have not tried 'nodetool scrub' yet nor dropping repairedAt for
affected sstables.
--
Best Regards,
Python_Max.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
