[jira] [Commented] (CASSANDRA-7408) System hints corruption - dataSize ... would be larger than file
[ https://issues.apache.org/jira/browse/CASSANDRA-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000950#comment-15000950 ] Jeff Griffith commented on CASSANDRA-7408: -- no problem [~iamaleksey] i seem to recall this being related to an issue i reported separately where a short integer was overflowing. pretty sure it's all good now. > System hints corruption - dataSize ... would be larger than file > > > Key: CASSANDRA-7408 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7408 > Project: Cassandra > Issue Type: Bug > Environment: RHEL 6.5 > Cassandra 1.2.16 > RF=3 > Thrift >Reporter: Jeff Griffith > > I've found several unresolved JIRA tickets related to SSTable corruption but > not sure if they apply to the case we are seeing in system/hints. We see > periodic exceptions such as: > {noformat} > dataSize of 144115248479299639 starting at 17209 would be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > {noformat} > Is there something we could possibly be doing from the application to cause > this sort of corruption? We also see it on some of our own column families > also some *negative* lengths which are presumably a similar corruption. > {noformat} > ERROR [HintedHandoff:57] 2014-06-17 17:08:04,690 CassandraDaemon.java (line > 191) Exception in thread Thread[HintedHandoff:57,1,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: > dataSize of 144115248479299639 starting at 17209 would be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > at > org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:441) > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:282) > at > org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:90) > at > org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:508) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: > dataSize of 144115248479299639 starting at 17209 would be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:437) > ... 6 more > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.IOException: dataSize of 144115248479299639 starting at 17209 would > be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:167) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:83) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:69) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:180) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:155) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:142) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38) > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.c
[jira] [Commented] (CASSANDRA-7408) System hints corruption - dataSize ... would be larger than file
[ https://issues.apache.org/jira/browse/CASSANDRA-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262824#comment-14262824 ] Maxim Ivanov commented on CASSANDRA-7408: - See same problem on 1.2.19, it seems to happening for system hints only. While scrubbing it is reporting following errors: {code} WARN [CompactionExecutor:17] 2015-01-02 10:09:41,454 OutputHandler.java (line 57) Non-fatal error reading row (stacktrace follows) java.io.IOError: java.io.IOException: Impossible row size 92971968326729728 at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:171) at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:526) at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:515) at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:70) at org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:280) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:250) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Impossible row size 92971968326729728 ... 10 more WARN [CompactionExecutor:17] 2015-01-02 10:09:41,456 OutputHandler.java (line 52) Row at 299608 is unreadable; skipping to next WARN [CompactionExecutor:17] 2015-01-02 10:09:41,457 OutputHandler.java (line 57) Non-fatal error reading row (stacktrace follows) java.io.IOError: java.io.IOException: Impossible row size 7881707469400978738 at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:171) at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:526) at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:515) at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:70) at org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:280) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:250) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Impossible row size 7881707469400978738 ... 10 more WARN [CompactionExecutor:17] 2015-01-02 10:09:41,459 OutputHandler.java (line 52) Row at 304082 is unreadable; skipping to next {code} > System hints corruption - dataSize ... would be larger than file > > > Key: CASSANDRA-7408 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7408 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: RHEL 6.5 > Cassandra 1.2.16 > RF=3 > Thrift >Reporter: Jeff Griffith > > I've found several unresolved JIRA tickets related to SSTable corruption but > not sure if they apply to the case we are seeing in system/hints. We see > periodic exceptions such as: > {noformat} > dataSize of 144115248479299639 starting at 17209 would be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > {noformat} > Is there something we could possibly be doing from the application to cause > this sort of corruption? We also see it on some of our own column families > also some *negative* lengths which are presumably a similar corruption. > {noformat} > ERROR [HintedHandoff:57] 2014-06-17 17:08:04,690 CassandraDaemon.java (line > 191) Exception in thread Thread[HintedHandoff:57,1,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: > dataSize of 144115248479299639 starting at 17209 would be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > at > org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:441) > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:282) > at > org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:90) > at > org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:508) >
[jira] [Commented] (CASSANDRA-7408) System hints corruption - dataSize ... would be larger than file
[ https://issues.apache.org/jira/browse/CASSANDRA-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200467#comment-14200467 ] Michael Shuler commented on CASSANDRA-7408: --- If you get this error on the same sstable, try a {{nodetool scrub}} to see if that helps. > System hints corruption - dataSize ... would be larger than file > > > Key: CASSANDRA-7408 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7408 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: RHEL 6.5 > Cassandra 1.2.16 > RF=3 > Thrift >Reporter: Jeff Griffith > > I've found several unresolved JIRA tickets related to SSTable corruption but > not sure if they apply to the case we are seeing in system/hints. We see > periodic exceptions such as: > {noformat} > dataSize of 144115248479299639 starting at 17209 would be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > {noformat} > Is there something we could possibly be doing from the application to cause > this sort of corruption? We also see it on some of our own column families > also some *negative* lengths which are presumably a similar corruption. > {noformat} > ERROR [HintedHandoff:57] 2014-06-17 17:08:04,690 CassandraDaemon.java (line > 191) Exception in thread Thread[HintedHandoff:57,1,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: > dataSize of 144115248479299639 starting at 17209 would be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > at > org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:441) > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:282) > at > org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:90) > at > org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:508) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: > dataSize of 144115248479299639 starting at 17209 would be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:437) > ... 6 more > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.IOException: dataSize of 144115248479299639 starting at 17209 would > be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:167) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:83) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:69) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:180) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:155) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:142) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38) > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(Compac
[jira] [Commented] (CASSANDRA-7408) System hints corruption - dataSize ... would be larger than file
[ https://issues.apache.org/jira/browse/CASSANDRA-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195566#comment-14195566 ] Gopinath Murugan commented on CASSANDRA-7408: - I have seen the similar issue with cassandra 1.2.13 as well..looks like a bug > System hints corruption - dataSize ... would be larger than file > > > Key: CASSANDRA-7408 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7408 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: RHEL 6.5 > Cassandra 1.2.16 > RF=3 > Thrift >Reporter: Jeff Griffith > > I've found several unresolved JIRA tickets related to SSTable corruption but > not sure if they apply to the case we are seeing in system/hints. We see > periodic exceptions such as: > {noformat} > dataSize of 144115248479299639 starting at 17209 would be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > {noformat} > Is there something we could possibly be doing from the application to cause > this sort of corruption? We also see it on some of our own column families > also some *negative* lengths which are presumably a similar corruption. > {noformat} > ERROR [HintedHandoff:57] 2014-06-17 17:08:04,690 CassandraDaemon.java (line > 191) Exception in thread Thread[HintedHandoff:57,1,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: > dataSize of 144115248479299639 starting at 17209 would be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > at > org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:441) > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:282) > at > org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:90) > at > org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:508) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: > dataSize of 144115248479299639 starting at 17209 would be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:437) > ... 6 more > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.IOException: dataSize of 144115248479299639 starting at 17209 would > be larger than file > /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length > 35542 > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:167) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:83) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:69) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:180) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:155) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:142) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38) > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.