Hi, My Compression strategy in Production was *LZ4 Compression. *But I modified it to *Deflate *using alter command
For compression change, we have to use *nodetool Upgradesstables *to forcefully upgrade the compression strategy on all sstables But once upgradesstabloes command completed on all the 5 nodes in the cluster, My requests started to fail, both read and write Replication Factor - 3 Read Consistency - 1 Write Consistency - 1 FYI - I am also using lightweight transaction which uses PAXOS Cassandra Version 3.10 I am now facing Following Errors in my debug.log file and some of my requests have started to fail : Debug.log ERROR [ReadRepairStage:82952] 2018-04-09 19:05:20,669 >> CassandraDaemon.java:229 - Exception in thread >> Thread[ReadRepairStage:82952,5,main] > > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out >> - received only 0 responses. > > at > org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.db.partitions.UnfilteredPartitionIterat >> ors$2.close(UnfilteredPartitionIterators.java:182) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82) >> ~[apache-cassandra-3.10.jar:3.10] > > at > org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThr >> ow(AsyncRepairCallback.java:50) ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >> ~[apache-cassandra-3.10.jar:3.10] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> ~[na:1.8.0_144] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> ~[na:1.8.0_144] > > at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$ >> threadLocalDeallocator$0(NamedThreadFactory.java:79) >> ~[apache-cassandra-3.10.jar:3.10] > > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] > > DEBUG [ReadRepairStage:82953] 2018-04-09 19:05:22,932 >> ReadCallback.java:242 - Digest mismatch: > > org.apache.cassandra.service.DigestMismatchException: Mismatch for key >> DecoratedKey(-2666936192316364820, 5756f5b8e7b341afa22cef22c5d33260) >> (d29a0e2a05f81315f0945dee5a210060 vs d41d8cd98f00b204e9800998ecf8427e) > > at > org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92) >> ~[apache-cassandra-3.10.jar:3.10] > > at > org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233) >> ~[apache-cassandra-3.10.jar:3.10] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> [na:1.8.0_144] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> [na:1.8.0_144] > > at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$ >> threadLocalDeallocator$0(NamedThreadFactory.java:79) >> [apache-cassandra-3.10.jar:3.10] > > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] > > INFO [HintsDispatcher:767] 2018-04-09 19:05:24,874 >> HintsDispatchExecutor.java:283 - Finished hinted handoff of file >> 68c7c130-6cf8-4864-bde8-1819f238045c-1523315072851-1.hints to endpoint >> 68c7c130-6cf8-4864-bde8-1819f238045c, partially > > DEBUG [ReadRepairStage:82950] 2018-04-09 19:05:24,932 >> DataResolver.java:169 - Timeout while read-repairing after receiving all 1 >> data and digest responses > > ERROR [ReadRepairStage:82950] 2018-04-09 19:05:24,933 >> CassandraDaemon.java:229 - Exception in thread >> Thread[ReadRepairStage:82950,5,main] > > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out >> - received only 0 responses. > > at > org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.db.partitions.UnfilteredPartitionIterat >> ors$2.close(UnfilteredPartitionIterators.java:182) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82) >> ~[apache-cassandra-3.10.jar:3.10] > > at > org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThr >> ow(AsyncRepairCallback.java:50) ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >> ~[apache-cassandra-3.10.jar:3.10] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> ~[na:1.8.0_144] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> ~[na:1.8.0_144] > > at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$ >> threadLocalDeallocator$0(NamedThreadFactory.java:79) >> ~[apache-cassandra-3.10.jar:3.10] > > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] > > nodetool info shows Gossip active : true > > Thrift active : false > > Native Transport active: true > > Load : 280.43 GiB > > Generation No : 1514537104 > > Uptime (seconds) : 8810363 > > Heap Memory (MB) : 1252.06 / 3970.00 > > Off Heap Memory (MB) : 573.33 > > Data Center : dc1 > > Rack : rack1 > > *Exceptions : 18987* > > Key Cache : entries 351612, size 99.86 MiB, capacity 100 MiB, >> 11144584 hits, 21126425 requests, 0.528 recent hit rate, 14400 save period >> in seconds > > Hope anyone can help me with this. Thanks, Hitesh dua hiteshd...@gmail.com
