[jira] [Commented] (CASSANDRA-13556) Corrupted SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900912#comment-16900912 ] Chakravarthi Manepalli commented on CASSANDRA-13556: I have observed the same issue in one of our setups. There were no removal or addition of columns in a recent period. I have observed the stack error in debug log. WARN [ReadStage-1] 2019-08-06 16:34:29,093 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-1,5,main]: {} java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /media/db_store/data/server_data/known_ap_ssid_list-d2cde2f0adf211e99625cd7f2acaf155/mc-5-big-Data.db at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2598) ~[apache-cassandra-3.11.1.jar:3.11.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_152] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) [apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.11.1.jar:3.11.1] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152] Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /media/db_store/data/server_data/known_ap_ssid_list-d2cde2f0adf211e99625cd7f2acaf155/mc-5-big-Data.db at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:391) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:258) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:116) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:499) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:359) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:136) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:167) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:160) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:156) ~[apache-cassandra-3.11.1.jar:3.11.1] at
[jira] [Commented] (CASSANDRA-13556) Corrupted SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143192#comment-16143192 ] Jeff Jirsa commented on CASSANDRA-13556: That sounds very much like corruption caused by CASSANDRA-13004 - did you recently add or remove any columns from that table? > Corrupted SSTables > -- > > Key: CASSANDRA-13556 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13556 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: CentOS Linux release 7.3.1611 (Core) > openjdk version "1.8.0_121" > OpenJDK Runtime Environment (build 1.8.0_121-b13) > OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode) > Python cassandra (DataStax) driver v3.6.0 >Reporter: Ihor Prokopov >Priority: Critical > Fix For: 3.9 > > > After 3 month of working, we noticed that number of compaction tasks were > growing (~600 pending tasks). SStables verification shows that some of them > were corrupted. Repairing didn't help (it was crashing with error). > Also some of requests (f.e. select * from fetcher where > domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next > error: > {color:red} > Traceback (most recent call last): > File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in > perform_simple_statement > result = future.result() > File > "/var/cassandra/apache-cassandra-3.9/bin/../lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py", > line 3650, in result > raise self._final_exception > error: unpack requires a string argument of length 4 > {color} > Table chema: > {quote} > CREATE TABLE fetcher ( > domain bigint, > uri bigint, > date date, > content_length int, > elapsed float, > encoding text, > fetched_time bigint, > flinks frozen, > flinks_count int, > html_fingerprint bigint, > indexed boolean, > adult boolean, > kws_count int, > lang_id int, > last_updated bigint, > redirect_url tuple , > revisit_date date, > revisit_interval int, > status_code int, > tokens_fingerprint bigint, > uris frozen url text, > PRIMARY KEY (domain, uri, date) > ) WITH CLUSTERING ORDER BY (uri ASC, date DESC) > AND bloom_filter_fp_chance = 0.1 > AND caching = \{'keys': 'ALL', 'rows_per_partition': 'NONE'} > AND comment = 'fetcher history' > AND compaction = \{'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', > 'sstable_size_in_mb': '256', > 'tombstone_threshold': '.2'} > AND compression = \{'chunk_length_in_kb': '64', >'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND crc_check_chance = 1.0 > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.5 > AND speculative_retry = '99PERCENTILE'; > {quote} > Corrupted > [SSTable|https://drive.google.com/file/d/0B4ZaUOv0G9oMcHpERTdlb3ozSVk/view?usp=sharing]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13556) Corrupted SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028086#comment-16028086 ] Ihor Prokopov commented on CASSANDRA-13556: --- Hi Jeff, thanks for your response a lot! Here is stacktrace: {code} ERROR [CompactionExecutor:2116] 2017-05-26 13:10:00,784 CassandraDaemon.java:226 - Exception in thread Thread[CompactionExecutor:2116,1,main] org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/cassandra/apache-cassandra-3.9/data/data/toweya/fetcher-8a86a1b0f4f611e6a6db373124acc314/mc-178837-big-Data.db at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:112) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:500) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:360) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:106) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:169) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:141) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:189) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82) ~[apache-cassandra-3.9.jar:3.9] {code} Also we noticed, that some rows returns without any errors, but fealds' values were shuffled (I mean that all fealds contains values from other fields). Best regards, Ihor > Corrupted SSTables > -- > > Key: CASSANDRA-13556 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13556 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: CentOS Linux release 7.3.1611 (Core) > openjdk version "1.8.0_121" > OpenJDK Runtime Environment (build 1.8.0_121-b13) > OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode) > Python cassandra (DataStax) driver v3.6.0 >Reporter: Ihor Prokopov >Priority: Critical > Fix For: 3.9 > > > After 3 month of working, we noticed that number of compaction tasks were > growing (~600 pending tasks). SStables verification shows that some of them > were corrupted. Repairing didn't help (it was crashing with error). > Also some of requests (f.e. select * from fetcher where > domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next > error: > {color:red} > Traceback (most recent call last): > File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in > perform_simple_statement >
[jira] [Commented] (CASSANDRA-13556) Corrupted SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026428#comment-16026428 ] Jeff Jirsa commented on CASSANDRA-13556: There should be a stack trace on the server when you run the {{SELECT}} in cqlsh. Please include that as well. > Corrupted SSTables > -- > > Key: CASSANDRA-13556 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13556 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: CentOS Linux release 7.3.1611 (Core) > openjdk version "1.8.0_121" > OpenJDK Runtime Environment (build 1.8.0_121-b13) > OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode) > Python cassandra (DataStax) driver v3.6.0 >Reporter: Ihor Prokopov > Fix For: 3.9 > > > After 3 month of working, we noticed that number of compaction tasks were > growing (~600 pending tasks). SStables verification shows that some of them > were corrupted. Repairing didn't help (it was crashing with error). > Also some of requests (f.e. select * from fetcher where > domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next > error: > {color:red} > Traceback (most recent call last): > File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in > perform_simple_statement > result = future.result() > File > "/var/cassandra/apache-cassandra-3.9/bin/../lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py", > line 3650, in result > raise self._final_exception > error: unpack requires a string argument of length 4 > {color} > Table chema: > {quote} > CREATE TABLE fetcher ( > domain bigint, > uri bigint, > date date, > content_length int, > elapsed float, > encoding text, > fetched_time bigint, > flinks frozen, > flinks_count int, > html_fingerprint bigint, > indexed boolean, > adult boolean, > kws_count int, > lang_id int, > last_updated bigint, > redirect_url tuple , > revisit_date date, > revisit_interval int, > status_code int, > tokens_fingerprint bigint, > uris frozen url text, > PRIMARY KEY (domain, uri, date) > ) WITH CLUSTERING ORDER BY (uri ASC, date DESC) > AND bloom_filter_fp_chance = 0.1 > AND caching = \{'keys': 'ALL', 'rows_per_partition': 'NONE'} > AND comment = 'fetcher history' > AND compaction = \{'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', > 'sstable_size_in_mb': '256', > 'tombstone_threshold': '.2'} > AND compression = \{'chunk_length_in_kb': '64', >'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND crc_check_chance = 1.0 > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.5 > AND speculative_retry = '99PERCENTILE'; > {quote} > Corrupted > [SSTable|https://drive.google.com/file/d/0B4ZaUOv0G9oMcHpERTdlb3ozSVk/view?usp=sharing]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org