Re: Failed request after changing compression strategy from LZ4 to Deflate and using upgradesstable
Hi Hitesh, This list is for conversations regarding Cassandra development. Can you subscribe and post this to us...@cassandra.apache.org instead? You will get a much wider audience when you do. On Tue, Apr 10, 2018 at 10:45 PM, hitesh dua wrote: > Hi, > My Compression strategy in Production was *LZ4 Compression. *But I modified > it to *Deflate *using alter command > > For compression change, we have to use *nodetool Upgradesstables *to > forcefully upgrade the compression strategy on all sstables > > But once upgradesstabloes command completed on all the 5 nodes in the > cluster, My requests started to fail, both read and write > > Replication Factor - 3 > Read Consistency - 1 > Write Consistency - 1 > FYI - I am also using lightweight transaction which uses PAXOS > Cassandra Version 3.10 > > I am now facing Following Errors in my debug.log file and some of my > requests have started to fail : > > Debug.log > > ERROR [ReadRepairStage:82952] 2018-04-09 19:05:20,669 >>> CassandraDaemon.java:229 - Exception in thread >>> Thread[ReadRepairStage:82952,5,main] >> >> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out >>> - received only 0 responses. >> >> at >> org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.db.partitions.UnfilteredPartitionIterat >>> ors$2.close(UnfilteredPartitionIterators.java:182) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at >> org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThr >>> ow(AsyncRepairCallback.java:50) ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> ~[na:1.8.0_144] >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> ~[na:1.8.0_144] >> >> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$ >>> threadLocalDeallocator$0(NamedThreadFactory.java:79) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] >> >> DEBUG [ReadRepairStage:82953] 2018-04-09 19:05:22,932 >>> ReadCallback.java:242 - Digest mismatch: >> >> org.apache.cassandra.service.DigestMismatchException: Mismatch for key >>> DecoratedKey(-2666936192316364820, 5756f5b8e7b341afa22cef22c5d33260) >>> (d29a0e2a05f81315f0945dee5a210060 vs d41d8cd98f00b204e9800998ecf8427e) >> >> at >> org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at >> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> [na:1.8.0_144] >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> [na:1.8.0_144] >> >> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$ >>> threadLocalDeallocator$0(NamedThreadFactory.java:79) >>> [apache-cassandra-3.10.jar:3.10] >> >> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] >> >> INFO [HintsDispatcher:767] 2018-04-09 19:05:24,874 >>> HintsDispatchExecutor.java:283 - Finished hinted handoff of file >>> 68c7c130-6cf8-4864-bde8-1819f238045c-1523315072851-1.hints to endpoint >>> 68c7c130-6cf8-4864-bde8-1819f238045c, partially >> >> DEBUG [ReadRepairStage:82950] 2018-04-09 19:05:24,932 >>> DataResolver.java:169 - Timeout while read-repairing after receiving all 1 >>> data and digest responses >> >> ERROR [ReadRepairStage:82950] 2018-04-09 19:05:24,933 >>> CassandraDaemon.java:229 - Exception in thread >>> Thread[ReadRepairStage:82950,5,main] >> >> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out >>> - received only 0 responses. >> >> at >> org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.db.partitions.UnfilteredPartitionIterat >>> ors$2.close(UnfilteredPartitionIterators.java:182) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at >> org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThr >>> ow(AsyncRepairCallback.java:50) ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >>> ~[apache-cassa
Failed request after changing compression strategy from LZ4 to Deflate and using upgradesstable
Hi, My Compression strategy in Production was *LZ4 Compression. *But I modified it to *Deflate *using alter command For compression change, we have to use *nodetool Upgradesstables *to forcefully upgrade the compression strategy on all sstables But once upgradesstabloes command completed on all the 5 nodes in the cluster, My requests started to fail, both read and write Replication Factor - 3 Read Consistency - 1 Write Consistency - 1 FYI - I am also using lightweight transaction which uses PAXOS Cassandra Version 3.10 I am now facing Following Errors in my debug.log file and some of my requests have started to fail : Debug.log ERROR [ReadRepairStage:82952] 2018-04-09 19:05:20,669 >> CassandraDaemon.java:229 - Exception in thread >> Thread[ReadRepairStage:82952,5,main] > > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out >> - received only 0 responses. > > at > org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.db.partitions.UnfilteredPartitionIterat >> ors$2.close(UnfilteredPartitionIterators.java:182) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82) >> ~[apache-cassandra-3.10.jar:3.10] > > at > org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThr >> ow(AsyncRepairCallback.java:50) ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >> ~[apache-cassandra-3.10.jar:3.10] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> ~[na:1.8.0_144] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> ~[na:1.8.0_144] > > at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$ >> threadLocalDeallocator$0(NamedThreadFactory.java:79) >> ~[apache-cassandra-3.10.jar:3.10] > > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] > > DEBUG [ReadRepairStage:82953] 2018-04-09 19:05:22,932 >> ReadCallback.java:242 - Digest mismatch: > > org.apache.cassandra.service.DigestMismatchException: Mismatch for key >> DecoratedKey(-2666936192316364820, 5756f5b8e7b341afa22cef22c5d33260) >> (d29a0e2a05f81315f0945dee5a210060 vs d41d8cd98f00b204e9800998ecf8427e) > > at > org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92) >> ~[apache-cassandra-3.10.jar:3.10] > > at > org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233) >> ~[apache-cassandra-3.10.jar:3.10] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> [na:1.8.0_144] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> [na:1.8.0_144] > > at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$ >> threadLocalDeallocator$0(NamedThreadFactory.java:79) >> [apache-cassandra-3.10.jar:3.10] > > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] > > INFO [HintsDispatcher:767] 2018-04-09 19:05:24,874 >> HintsDispatchExecutor.java:283 - Finished hinted handoff of file >> 68c7c130-6cf8-4864-bde8-1819f238045c-1523315072851-1.hints to endpoint >> 68c7c130-6cf8-4864-bde8-1819f238045c, partially > > DEBUG [ReadRepairStage:82950] 2018-04-09 19:05:24,932 >> DataResolver.java:169 - Timeout while read-repairing after receiving all 1 >> data and digest responses > > ERROR [ReadRepairStage:82950] 2018-04-09 19:05:24,933 >> CassandraDaemon.java:229 - Exception in thread >> Thread[ReadRepairStage:82950,5,main] > > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out >> - received only 0 responses. > > at > org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.db.partitions.UnfilteredPartitionIterat >> ors$2.close(UnfilteredPartitionIterators.java:182) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82) >> ~[apache-cassandra-3.10.jar:3.10] > > at > org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89) >> ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThr >> ow(AsyncRepairCallback.java:50) ~[apache-cassandra-3.10.jar:3.10] > > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >> ~[apache-cassandra-3.10.jar:3.10] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> ~[na:1.8.0_144] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> ~[na:1.8.0_144] > > at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$ >> threadLocalDeallocator$0(NamedThreadFactory.java:79) >> ~[apache-cassandra-3.10.jar:3.1
