The read repair you have disabled is the probabilistic background repair - foreground repair due to mismatch still happens
Streaming should respect windows. Streaming doesn’t write to the memtable, only the write path puts data into the memtable. -- Jeff Jirsa > On Dec 18, 2018, at 1:49 AM, Roy Burstein <burstein....@gmail.com> wrote: > > read repair is disabled in this table : > > CREATE TABLE gil_test.my_test ( > id int, > creation_time timestamp, > ... > AND crc_check_chance = 1.0 > AND dclocal_read_repair_chance = 0.0 > AND default_time_to_live = 0 > AND gc_grace_seconds = 3600 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.0 > AND speculative_retry = 'NONE'; > > 1228 is the result of compaction from 873 and 1196, so it makes sense that > they would have the same max timestamp > 1196 is an sstable that was created during streaming (added that part of the > log before), since it was compacted right away when the node removal was done > i don't have its metadata. > > when data is being streamed from other nodes during node removal, do i have > data mixed from different time window in one stream? > also even if it is being separated, what would stop old data being stored in > the same memtable as data just being written from the application? > > >> On Mon, Dec 17, 2018 at 7:18 PM Jeff Jirsa <jji...@gmail.com> wrote: >> >> The min timestamps vary (likely due to read repairing old values into the >> memtable and flushing into these sstables), but the max timestamps for both >> are in the same second (same microsecond, even, so probably the same write): >> >> Maximum timestamp: 1544903882074190 >> Maximum timestamp: 1544903882074190 >> >> jjirsa:~ jjirsa$ date -r 1544903882 >> Sat Dec 15 11:58:02 PST 2018 >> >> TWCS buckets based on max timestamp per file, so they belong together: >> >> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java#L247 >> >> >> >>> On Sun, Dec 16, 2018 at 11:39 PM Roy Burstein <burstein....@gmail.com> >>> wrote: >>> hey jeff, attaching more information. >>> so this the situation before - 3 nodes in the cluster (3.11.3 in this case >>> but i saw same thing in 2.1 and 3.0), there is a script writing one row >>> every minute and another script doing nodetool flush every 10 minute. >>> window is defined as two hours, so after a few days this is how the >>> directory listing looks : >>> >>> drwxr-xr-x 2 cassandra cassandra 4096 Dec 11 10:38 backups >>> -rw-r--r-- 1 cassandra cassandra 646 Dec 12 05:25 mc-171-big-Index.db >>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-171-big-Filter.db >>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-171-big-Summary.db >>> -rw-r--r-- 1 cassandra cassandra 3561 Dec 12 05:25 mc-171-big-Data.db >>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-171-big-Digest.crc32 >>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25 >>> mc-171-big-CompressionInfo.db >>> -rw-r--r-- 1 cassandra cassandra 4893 Dec 12 05:25 mc-171-big-Statistics.db >>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-171-big-TOC.txt >>> -rw-r--r-- 1 cassandra cassandra 565 Dec 12 05:25 mc-172-big-Index.db >>> -rw-r--r-- 1 cassandra cassandra 96 Dec 12 05:25 mc-172-big-Filter.db >>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-172-big-Summary.db >>> -rw-r--r-- 1 cassandra cassandra 3475 Dec 12 05:25 mc-172-big-Data.db >>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-172-big-Digest.crc32 >>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25 >>> mc-172-big-CompressionInfo.db >>> -rw-r--r-- 1 cassandra cassandra 4865 Dec 12 05:25 mc-172-big-Statistics.db >>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-172-big-TOC.txt >>> -rw-r--r-- 1 cassandra cassandra 637 Dec 12 05:25 mc-173-big-Index.db >>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-173-big-Filter.db >>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-173-big-Summary.db >>> -rw-r--r-- 1 cassandra cassandra 3678 Dec 12 05:25 mc-173-big-Data.db >>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-173-big-Digest.crc32 >>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25 >>> mc-173-big-CompressionInfo.db >>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-173-big-TOC.txt >>> -rw-r--r-- 1 cassandra cassandra 4888 Dec 12 05:25 mc-173-big-Statistics.db >>> . >>> . >>> -rw-r--r-- 1 cassandra cassandra 340 Dec 15 20:10 mc-873-big-Index.db >>> -rw-r--r-- 1 cassandra cassandra 64 Dec 15 20:10 mc-873-big-Filter.db >>> -rw-r--r-- 1 cassandra cassandra 56 Dec 15 20:10 mc-873-big-Summary.db >>> -rw-r--r-- 1 cassandra cassandra 1910 Dec 15 20:10 mc-873-big-Data.db >>> -rw-r--r-- 1 cassandra cassandra 10 Dec 15 20:10 mc-873-big-Digest.crc32 >>> -rw-r--r-- 1 cassandra cassandra 51 Dec 15 20:10 >>> mc-873-big-CompressionInfo.db >>> -rw-r--r-- 1 cassandra cassandra 4793 Dec 15 20:10 mc-873-big-Statistics.db >>> -rw-r--r-- 1 cassandra cassandra 92 Dec 15 20:10 mc-873-big-TOC.txt >>> . >>> . >>> . >>> -rw-r--r-- 1 cassandra cassandra 24 Dec 17 06:50 mc-1150-big-Filter.db >>> -rw-r--r-- 1 cassandra cassandra 51 Dec 17 06:50 mc-1150-big-Index.db >>> -rw-r--r-- 1 cassandra cassandra 56 Dec 17 06:50 mc-1150-big-Summary.db >>> -rw-r--r-- 1 cassandra cassandra 10 Dec 17 06:50 mc-1150-big-Digest.crc32 >>> -rw-r--r-- 1 cassandra cassandra 226 Dec 17 06:50 mc-1150-big-Data.db >>> -rw-r--r-- 1 cassandra cassandra 43 Dec 17 06:50 >>> mc-1150-big-CompressionInfo.db >>> -rw-r--r-- 1 cassandra cassandra 92 Dec 17 06:50 mc-1150-big-TOC.txt >>> -rw-r--r-- 1 cassandra cassandra 4691 Dec 17 06:50 mc-1150-big-Statistics.db >>> >>> >>> >>> check out sstable 873 is from dec 15. at this point i remove node no 3. >>> and the log shows streamed data is creating sstables 1150-1200. it also >>> shows one of these sstables being compacted with sstable 873, resulting in >>> sstable number 1228, an operation which from what i understand should not >>> happen. >>> >>> DEBUG [StreamReceiveTask:1] 2018-12-17 06:59:54,199 >>> StreamReceiveTask.java:213 - [Stream #5ae370a0-01c9-11e9-ba69-d743ff231136] >>> Received 58 sstables from /10.106.12.34 >>> ([BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1151-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1152-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1153-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1154-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1155-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1156-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1157-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1158-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1159-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1160-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1161-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1162-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1163-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1164-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1165-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1166-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1167-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1168-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1169-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1170-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1171-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1172-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1173-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1174-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1175-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1176-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1177-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1178-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1179-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1180-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1181-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1182-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1183-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1184-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1185-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1186-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1187-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1188-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1189-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1190-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1191-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1192-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1193-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1194-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1195-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1197-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1198-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1199-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1200-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1201-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0 >>> >>> >>> >>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564 >>> TimeWindowCompactionStrategy.java:308 - bucket size 2 >= 2 and not in >>> current bucket, compacting what's here: >>> [BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db'), >>> >>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db')] >>> >>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564 >>> CompactionTask.java:155 - Compacting (5b702040-01c9-11e9-ba69-d743ff231136) >>> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db:level=0, >>> >>> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db:level=0, >>> ] >>> >>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,578 >>> CompactionTask.java:255 - Compacted (5b702040-01c9-11e9-ba69-d743ff231136) >>> 2 sstables to >>> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big,] >>> to level=0. 3.397KiB to 3.383KiB (~99% of original) in 13ms. Read >>> Throughput = 244.120KiB/s, Write Throughput = 243.067KiB/s, Row Throughput >>> = ~184/s. 71 total partitions merged to 71. Partition merge counts were >>> {1:71, } >>> >>> see sstablemetadata output for sstable 873 and 1228. as you can see the >>> 1228 min timstamp is old. >>> >>> 873: >>> WARN 07:11:30,845 Only 45.238GiB free across all data volumes. Consider >>> adding more capacity to your cluster or removing obsolete snapshots >>> SSTable: /root/gil_sstables_before/mc-873-big >>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner >>> Bloom Filter FP chance: 0.010000 >>> Minimum timestamp: 1544896864561707 >>> Maximum timestamp: 1544903882074190 >>> SSTable min local deletion time: 2147483647 >>> SSTable max local deletion time: 2147483647 >>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor >>> Compression ratio: 0.6954296160877513 >>> TTL min: 0 >>> TTL max: 0 >>> First token: -7160136740246525330 (key=42) >>> Last token: 8209703677368372043 (key=34) >>> minClustringValues: [2018-12-15 18:01Z] >>> maxClustringValues: [2018-12-15 19:58Z] >>> Estimated droppable tombstones: 0.0 >>> SSTable Level: 0 >>> Repaired at: 0 >>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001, >>> position=19800215)=CommitLogPosition(segmentId=1544706489001, >>> position=20586499)} >>> totalColumnsSet: 62 >>> totalRows: 62 >>> Estimated tombstone drop times: >>> Count Row Size Cell Count >>> 1 0 24 >>> 2 0 7 >>> 3 0 5 >>> 4 0 1 >>> 5 0 1 >>> 6 0 0 >>> 7 0 0 >>> 8 0 0 >>> 10 0 0 >>> 12 0 0 >>> 14 0 0 >>> 17 0 0 >>> 20 0 0 >>> 24 0 0 >>> 29 0 0 >>> 35 0 0 >>> 42 0 0 >>> 50 7 0 >>> 60 17 0 >>> 72 0 0 >>> 86 5 0 >>> 103 2 0 >>> 124 5 0 >>> 149 1 0 >>> 179 0 0 >>> 215 1 0 >>> 258 0 0 >>> 310 0 0 >>> 372 0 0 >>> 446 0 0 >>> 535 0 0 >>> 642 0 0 >>> 770 0 0 >>> 924 0 0 >>> 1109 0 0 >>> 1331 0 0 >>> 1597 0 0 >>> 1916 0 0 >>> 2299 0 0 >>> 2759 0 0 >>> 3311 0 0 >>> 3973 0 0 >>> 4768 0 0 >>> 5722 0 0 >>> 6866 0 0 >>> 8239 0 0 >>> 9887 0 0 >>> 11864 0 0 >>> 14237 0 0 >>> 17084 0 0 >>> 20501 0 0 >>> 24601 0 0 >>> 29521 0 0 >>> 35425 0 0 >>> 42510 0 0 >>> 51012 0 0 >>> 61214 0 0 >>> 73457 0 0 >>> 88148 0 0 >>> 105778 0 0 >>> 126934 0 0 >>> 152321 0 0 >>> 182785 0 0 >>> 219342 0 0 >>> 263210 0 0 >>> 315852 0 0 >>> 379022 0 0 >>> 454826 0 0 >>> 545791 0 0 >>> 654949 0 0 >>> 785939 0 0 >>> 943127 0 0 >>> 1131752 0 0 >>> 1358102 0 0 >>> 1629722 0 0 >>> 1955666 0 0 >>> 2346799 0 0 >>> 2816159 0 0 >>> 3379391 0 0 >>> 4055269 0 0 >>> 4866323 0 0 >>> 5839588 0 0 >>> 7007506 0 0 >>> 8409007 0 0 >>> 10090808 0 0 >>> 12108970 0 0 >>> 14530764 0 0 >>> 17436917 0 0 >>> 20924300 0 0 >>> 25109160 0 0 >>> 30130992 0 0 >>> 36157190 0 0 >>> 43388628 0 0 >>> 52066354 0 0 >>> 62479625 0 0 >>> 74975550 0 0 >>> 89970660 0 0 >>> 107964792 0 0 >>> 129557750 0 0 >>> 155469300 0 0 >>> 186563160 0 0 >>> 223875792 0 0 >>> 268650950 0 0 >>> 322381140 0 0 >>> 386857368 0 0 >>> 464228842 0 0 >>> 557074610 0 0 >>> 668489532 0 0 >>> 802187438 0 0 >>> 962624926 0 0 >>> 1155149911 0 0 >>> 1386179893 0 0 >>> 1663415872 0 0 >>> 1996099046 0 0 >>> 2395318855 0 0 >>> 2874382626 0 >>> 3449259151 0 >>> 4139110981 0 >>> 4966933177 0 >>> 5960319812 0 >>> 7152383774 0 >>> 8582860529 0 >>> 10299432635 0 >>> 12359319162 0 >>> 14831182994 0 >>> 17797419593 0 >>> 21356903512 0 >>> 25628284214 0 >>> 30753941057 0 >>> 36904729268 0 >>> 44285675122 0 >>> 53142810146 0 >>> 63771372175 0 >>> 76525646610 0 >>> 91830775932 0 >>> 110196931118 0 >>> 132236317342 0 >>> 158683580810 0 >>> 190420296972 0 >>> 228504356366 0 >>> 274205227639 0 >>> 329046273167 0 >>> 394855527800 0 >>> 473826633360 0 >>> 568591960032 0 >>> 682310352038 0 >>> 818772422446 0 >>> 982526906935 0 >>> 1179032288322 0 >>> 1414838745986 0 >>> Estimated cardinality: 38 >>> EncodingStats minTTL: 0 >>> EncodingStats minLocalDeletionTime: 1442880000 >>> EncodingStats minTimestamp: 1544896864561707 >>> KeyType: org.apache.cassandra.db.marshal.Int32Type >>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType] >>> StaticColumns: {} >>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type} >>> You have new mail in /var/spool/mail/root >>> >>> >>> >>> 1228 : >>> >>> WARN 07:10:51,187 Only 45.238GiB free across all data volumes. Consider >>> adding more capacity to your cluster or removing obsolete snapshots >>> SSTable: >>> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big >>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner >>> Bloom Filter FP chance: 0.010000 >>> Minimum timestamp: 1544896805142916 >>> Maximum timestamp: 1544903882074190 >>> SSTable min local deletion time: 2147483647 >>> SSTable max local deletion time: 2147483647 >>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor >>> Compression ratio: 0.6942828485456369 >>> TTL min: 0 >>> TTL max: 0 >>> First token: -9108684050423740263 (key=53) >>> Last token: 8669466869824630438 (key=77) >>> minClustringValues: [2018-12-15 18:00Z] >>> maxClustringValues: [2018-12-15 19:58Z] >>> Estimated droppable tombstones: 0.0 >>> SSTable Level: 0 >>> Repaired at: 0 >>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001, >>> position=19800215)=CommitLogPosition(segmentId=1544706489001, >>> position=20586499)} >>> totalColumnsSet: 113 >>> totalRows: 113 >>> Estimated tombstone drop times: >>> Count Row Size Cell Count >>> 1 0 41 >>> 2 0 21 >>> 3 0 7 >>> 4 0 1 >>> 5 0 1 >>> 6 0 0 >>> 7 0 0 >>> 8 0 0 >>> 10 0 0 >>> 12 0 0 >>> 14 0 0 >>> 17 0 0 >>> 20 0 0 >>> 24 0 0 >>> 29 0 0 >>> 35 0 0 >>> 42 0 0 >>> 50 16 0 >>> 60 25 0 >>> 72 0 0 >>> 86 17 0 >>> 103 4 0 >>> 124 7 0 >>> 149 1 0 >>> 179 0 0 >>> 215 1 0 >>> 258 0 0 >>> 310 0 0 >>> 372 0 0 >>> 446 0 0 >>> 535 0 0 >>> 642 0 0 >>> 770 0 0 >>> 924 0 0 >>> 1109 0 0 >>> 1331 0 0 >>> 1597 0 0 >>> 1916 0 0 >>> 2299 0 0 >>> 2759 0 0 >>> 3311 0 0 >>> 3973 0 0 >>> 4768 0 0 >>> 5722 0 0 >>> 6866 0 0 >>> 8239 0 0 >>> 9887 0 0 >>> 11864 0 0 >>> 14237 0 0 >>> 17084 0 0 >>> 20501 0 0 >>> 24601 0 0 >>> 29521 0 0 >>> 35425 0 0 >>> 42510 0 0 >>> 51012 0 0 >>> 61214 0 0 >>> 73457 0 0 >>> 88148 0 0 >>> 105778 0 0 >>> 126934 0 0 >>> 152321 0 0 >>> 182785 0 0 >>> 219342 0 0 >>> 263210 0 0 >>> 315852 0 0 >>> 379022 0 0 >>> 454826 0 0 >>> 545791 0 0 >>> 654949 0 0 >>> 785939 0 0 >>> 943127 0 0 >>> 1131752 0 0 >>> 1358102 0 0 >>> 1629722 0 0 >>> 1955666 0 0 >>> 2346799 0 0 >>> 2816159 0 0 >>> 3379391 0 0 >>> 4055269 0 0 >>> 4866323 0 0 >>> 5839588 0 0 >>> 7007506 0 0 >>> 8409007 0 0 >>> 10090808 0 0 >>> 12108970 0 0 >>> 14530764 0 0 >>> 17436917 0 0 >>> 20924300 0 0 >>> 25109160 0 0 >>> 30130992 0 0 >>> 36157190 0 0 >>> 43388628 0 0 >>> 52066354 0 0 >>> 62479625 0 0 >>> 74975550 0 0 >>> 89970660 0 0 >>> 107964792 0 0 >>> 129557750 0 0 >>> 155469300 0 0 >>> 186563160 0 0 >>> 223875792 0 0 >>> 268650950 0 0 >>> 322381140 0 0 >>> 386857368 0 0 >>> 464228842 0 0 >>> 557074610 0 0 >>> 668489532 0 0 >>> 802187438 0 0 >>> 962624926 0 0 >>> 1155149911 0 0 >>> 1386179893 0 0 >>> 1663415872 0 0 >>> 1996099046 0 0 >>> 2395318855 0 0 >>> 2874382626 0 >>> 3449259151 0 >>> 4139110981 0 >>> 4966933177 0 >>> 5960319812 0 >>> 7152383774 0 >>> 8582860529 0 >>> 10299432635 0 >>> 12359319162 0 >>> 14831182994 0 >>> 17797419593 0 >>> 21356903512 0 >>> 25628284214 0 >>> 30753941057 0 >>> 36904729268 0 >>> 44285675122 0 >>> 53142810146 0 >>> 63771372175 0 >>> 76525646610 0 >>> 91830775932 0 >>> 110196931118 0 >>> 132236317342 0 >>> 158683580810 0 >>> 190420296972 0 >>> 228504356366 0 >>> 274205227639 0 >>> 329046273167 0 >>> 394855527800 0 >>> 473826633360 0 >>> 568591960032 0 >>> 682310352038 0 >>> 818772422446 0 >>> 982526906935 0 >>> 1179032288322 0 >>> 1414838745986 0 >>> Estimated cardinality: 71 >>> EncodingStats minTTL: 0 >>> EncodingStats minLocalDeletionTime: 1442880000 >>> EncodingStats minTimestamp: 1544896805142916 >>> KeyType: org.apache.cassandra.db.marshal.Int32Type >>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType] >>> StaticColumns: {} >>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type} >>> >>> >>> >>>> On Fri, Dec 14, 2018 at 8:56 AM Jeff Jirsa <jji...@gmail.com> wrote: >>>> Remove node will stream data from all windows to remote nodes , so some >>>> compaction is expected >>>> >>>> Would need to see the sstablemetadata to understand what’s happening >>>> there. >>>> >>>> -- >>>> Jeff Jirsa >>>> >>>> >>>>> On Dec 13, 2018, at 10:26 PM, Roy Burstein <burstein....@gmail.com> wrote: >>>>> >>>>> Hi all , >>>>> My colleague opened Jira ticket for the issue but we are struggling with >>>>> this issue for a while and we have space issues : >>>>> >>>>> https://issues.apache.org/jira/browse/CASSANDRA-14929 >>>>> >>>>> After removing a node from the cluster, a table that is defined as TWCS, >>>>> has sstables from different time windows merged together, making old and >>>>> new data sit in the same sstable(the full example is in the ticket). >>>>> Any idea about this?workaround?Known issues? >>>>> Thanks! >>>>> Roy
