[jira] [Commented] (CASSANDRA-12088) Upgrade corrupts SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951654#comment-15951654
]
Jeff Jirsa commented on CASSANDRA-12088:
[~cskr] - I'm trying to reproduce this, and I'm not having much luck. Did this
only happen with LWT? Were you able to reproduce it reliably from scratch, or
only with the clusters you already had created?
> Upgrade corrupts SSTables
> -
>
> Key: CASSANDRA-12088
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12088
> Project: Cassandra
> Issue Type: Bug
> Environment: OS: CentOS release 6.7 (Final)
> Cassandra version: 2.1, 3.7
>Reporter: Chandra Sekar S
>Priority: Critical
>
> When upgrading from 2.0 to 3.7, table was corrupted and an exception occurs
> when performing LWT from Java Driver. The server was upgraded from 2.0 to 2.1
> and then to 3.7. "nodetool upgradesstables" was run after each step of
> upgrade.
> Schema of affected table:
> {code}
> CREATE TABLE payment.tbl (
> c1 text,
> c2 timestamp,
> c3 text,
> s1 timestamp static,
> s2 int static,
> c4 text,
> PRIMARY KEY (c1, c2)
> ) WITH CLUSTERING ORDER BY (c2 ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> Insertion that fails:
> {code:java}
> insert into tbl (c1, s2) values ('value', 0) if not exists;
> {code}
> The stack trace in system.log of cassandra server,
> {code}
> INFO [HANDSHAKE-maven-repo.corp.zeta.in/10.1.5.13] 2016-06-24 22:23:14,887
> OutboundTcpConnection.java:514 - Handshaking version with
> maven-repo.corp.zeta.in/10.1.5.13
> ERROR [MessagingService-Incoming-/10.1.5.13] 2016-06-24 22:23:14,889
> CassandraDaemon.java:217 - Exception in thread
> Thread[MessagingService-Incoming-/10.1.5.13,5,main]
> java.io.IOError: java.io.IOException: Corrupt flags value for unfiltered
> partition (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:131)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:97)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:66)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> Caused by: java.io.IOException: Corrupt flags value for unfiltered partition
> (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:380)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> ... 11 common frames
[jira] [Commented] (CASSANDRA-12088) Upgrade corrupts SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516650#comment-15516650
]
Alex Petrov commented on CASSANDRA-12088:
-
With broken / corrupted sstables it will rarely be the same exact exception. It
might differ depending on the exact data written. If there's something
misplaced on the disk: flag, size byte, type etc, it may result into different
exceptions, all being around deserialising. Anyways, I was trying to narrow it
down, when I seen the other issue I thought there might be an overlap. Thanks
for your response.
> Upgrade corrupts SSTables
> -
>
> Key: CASSANDRA-12088
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12088
> Project: Cassandra
> Issue Type: Bug
> Environment: OS: CentOS release 6.7 (Final)
> Cassandra version: 2.1, 3.7
>Reporter: Chandra Sekar S
>Priority: Critical
>
> When upgrading from 2.0 to 3.7, table was corrupted and an exception occurs
> when performing LWT from Java Driver. The server was upgraded from 2.0 to 2.1
> and then to 3.7. "nodetool upgradesstables" was run after each step of
> upgrade.
> Schema of affected table:
> {code}
> CREATE TABLE payment.tbl (
> c1 text,
> c2 timestamp,
> c3 text,
> s1 timestamp static,
> s2 int static,
> c4 text,
> PRIMARY KEY (c1, c2)
> ) WITH CLUSTERING ORDER BY (c2 ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> Insertion that fails:
> {code:java}
> insert into tbl (c1, s2) values ('value', 0) if not exists;
> {code}
> The stack trace in system.log of cassandra server,
> {code}
> INFO [HANDSHAKE-maven-repo.corp.zeta.in/10.1.5.13] 2016-06-24 22:23:14,887
> OutboundTcpConnection.java:514 - Handshaking version with
> maven-repo.corp.zeta.in/10.1.5.13
> ERROR [MessagingService-Incoming-/10.1.5.13] 2016-06-24 22:23:14,889
> CassandraDaemon.java:217 - Exception in thread
> Thread[MessagingService-Incoming-/10.1.5.13,5,main]
> java.io.IOError: java.io.IOException: Corrupt flags value for unfiltered
> partition (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:131)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:97)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:66)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> Caused by: java.io.IOException: Corrupt flags value for unfiltered partition
> (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:380)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
[jira] [Commented] (CASSANDRA-12088) Upgrade corrupts SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516565#comment-15516565
]
Chandra Sekar S commented on CASSANDRA-12088:
-
No column was dropped during the upgrade process, but they may have been
dropped before upgrade began.
However, the exception I see in the logs is different from the one in the other
issue.
> Upgrade corrupts SSTables
> -
>
> Key: CASSANDRA-12088
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12088
> Project: Cassandra
> Issue Type: Bug
> Environment: OS: CentOS release 6.7 (Final)
> Cassandra version: 2.1, 3.7
>Reporter: Chandra Sekar S
>Priority: Critical
>
> When upgrading from 2.0 to 3.7, table was corrupted and an exception occurs
> when performing LWT from Java Driver. The server was upgraded from 2.0 to 2.1
> and then to 3.7. "nodetool upgradesstables" was run after each step of
> upgrade.
> Schema of affected table:
> {code}
> CREATE TABLE payment.tbl (
> c1 text,
> c2 timestamp,
> c3 text,
> s1 timestamp static,
> s2 int static,
> c4 text,
> PRIMARY KEY (c1, c2)
> ) WITH CLUSTERING ORDER BY (c2 ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> Insertion that fails:
> {code:java}
> insert into tbl (c1, s2) values ('value', 0) if not exists;
> {code}
> The stack trace in system.log of cassandra server,
> {code}
> INFO [HANDSHAKE-maven-repo.corp.zeta.in/10.1.5.13] 2016-06-24 22:23:14,887
> OutboundTcpConnection.java:514 - Handshaking version with
> maven-repo.corp.zeta.in/10.1.5.13
> ERROR [MessagingService-Incoming-/10.1.5.13] 2016-06-24 22:23:14,889
> CassandraDaemon.java:217 - Exception in thread
> Thread[MessagingService-Incoming-/10.1.5.13,5,main]
> java.io.IOError: java.io.IOException: Corrupt flags value for unfiltered
> partition (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:131)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:97)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:66)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> Caused by: java.io.IOException: Corrupt flags value for unfiltered partition
> (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:380)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> ... 11 common frames omitted
>
[jira] [Commented] (CASSANDRA-12088) Upgrade corrupts SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515603#comment-15515603
]
Alex Petrov commented on CASSANDRA-12088:
-
Did you have any dropped static columns on this table? Maybe this issue may be
also a manifestation of [CASSANDRA-12582]?
> Upgrade corrupts SSTables
> -
>
> Key: CASSANDRA-12088
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12088
> Project: Cassandra
> Issue Type: Bug
> Environment: OS: CentOS release 6.7 (Final)
> Cassandra version: 2.1, 3.7
>Reporter: Chandra Sekar S
>Priority: Critical
>
> When upgrading from 2.0 to 3.7, table was corrupted and an exception occurs
> when performing LWT from Java Driver. The server was upgraded from 2.0 to 2.1
> and then to 3.7. "nodetool upgradesstables" was run after each step of
> upgrade.
> Schema of affected table:
> {code}
> CREATE TABLE payment.tbl (
> c1 text,
> c2 timestamp,
> c3 text,
> s1 timestamp static,
> s2 int static,
> c4 text,
> PRIMARY KEY (c1, c2)
> ) WITH CLUSTERING ORDER BY (c2 ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> Insertion that fails:
> {code:java}
> insert into tbl (c1, s2) values ('value', 0) if not exists;
> {code}
> The stack trace in system.log of cassandra server,
> {code}
> INFO [HANDSHAKE-maven-repo.corp.zeta.in/10.1.5.13] 2016-06-24 22:23:14,887
> OutboundTcpConnection.java:514 - Handshaking version with
> maven-repo.corp.zeta.in/10.1.5.13
> ERROR [MessagingService-Incoming-/10.1.5.13] 2016-06-24 22:23:14,889
> CassandraDaemon.java:217 - Exception in thread
> Thread[MessagingService-Incoming-/10.1.5.13,5,main]
> java.io.IOError: java.io.IOException: Corrupt flags value for unfiltered
> partition (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:131)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:97)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:66)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> Caused by: java.io.IOException: Corrupt flags value for unfiltered partition
> (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:380)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> ... 11 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12088) Upgrade corrupts SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503871#comment-15503871
]
Chandra Sekar S commented on CASSANDRA-12088:
-
Sorry, I can't try these anymore. I performed `nodetool scrub` and lost some
data as well. This occurred both in our build and test environment, so it is
probably reproducible with any non-empty upgrade from 2.1 to 3.7. We've decided
to dump out data as CSV and import it into 3.7 for production upgrade.
No, I don't have the old sstables.
> Upgrade corrupts SSTables
> -
>
> Key: CASSANDRA-12088
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12088
> Project: Cassandra
> Issue Type: Bug
> Environment: OS: CentOS release 6.7 (Final)
> Cassandra version: 2.1, 3.7
>Reporter: Chandra Sekar S
>Priority: Critical
>
> When upgrading from 2.0 to 3.7, table was corrupted and an exception occurs
> when performing LWT from Java Driver. The server was upgraded from 2.0 to 2.1
> and then to 3.7. "nodetool upgradesstables" was run after each step of
> upgrade.
> Schema of affected table:
> {code}
> CREATE TABLE payment.tbl (
> c1 text,
> c2 timestamp,
> c3 text,
> s1 timestamp static,
> s2 int static,
> c4 text,
> PRIMARY KEY (c1, c2)
> ) WITH CLUSTERING ORDER BY (c2 ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> Insertion that fails:
> {code:java}
> insert into tbl (c1, s2) values ('value', 0) if not exists;
> {code}
> The stack trace in system.log of cassandra server,
> {code}
> INFO [HANDSHAKE-maven-repo.corp.zeta.in/10.1.5.13] 2016-06-24 22:23:14,887
> OutboundTcpConnection.java:514 - Handshaking version with
> maven-repo.corp.zeta.in/10.1.5.13
> ERROR [MessagingService-Incoming-/10.1.5.13] 2016-06-24 22:23:14,889
> CassandraDaemon.java:217 - Exception in thread
> Thread[MessagingService-Incoming-/10.1.5.13,5,main]
> java.io.IOError: java.io.IOException: Corrupt flags value for unfiltered
> partition (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:131)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:97)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:66)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> Caused by: java.io.IOException: Corrupt flags value for unfiltered partition
> (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:380)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
>
[jira] [Commented] (CASSANDRA-12088) Upgrade corrupts SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503793#comment-15503793
]
Alex Petrov commented on CASSANDRA-12088:
-
Does it fail only when doing an LWT or would the same exception occur when you
do the full table scan?..
If you have the old sstables at hand, could you {{sstable2json}} on
non-upgraded sstables?
> Upgrade corrupts SSTables
> -
>
> Key: CASSANDRA-12088
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12088
> Project: Cassandra
> Issue Type: Bug
> Environment: OS: CentOS release 6.7 (Final)
> Cassandra version: 2.1, 3.7
>Reporter: Chandra Sekar S
>Priority: Critical
>
> When upgrading from 2.0 to 3.7, table was corrupted and an exception occurs
> when performing LWT from Java Driver. The server was upgraded from 2.0 to 2.1
> and then to 3.7. "nodetool upgradesstables" was run after each step of
> upgrade.
> Schema of affected table:
> {code}
> CREATE TABLE payment.tbl (
> c1 text,
> c2 timestamp,
> c3 text,
> s1 timestamp static,
> s2 int static,
> c4 text,
> PRIMARY KEY (c1, c2)
> ) WITH CLUSTERING ORDER BY (c2 ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> {code}
> Insertion that fails:
> {code:java}
> insert into tbl (c1, s2) values ('value', 0) if not exists;
> {code}
> The stack trace in system.log of cassandra server,
> {code}
> INFO [HANDSHAKE-maven-repo.corp.zeta.in/10.1.5.13] 2016-06-24 22:23:14,887
> OutboundTcpConnection.java:514 - Handshaking version with
> maven-repo.corp.zeta.in/10.1.5.13
> ERROR [MessagingService-Incoming-/10.1.5.13] 2016-06-24 22:23:14,889
> CassandraDaemon.java:217 - Exception in thread
> Thread[MessagingService-Incoming-/10.1.5.13,5,main]
> java.io.IOError: java.io.IOException: Corrupt flags value for unfiltered
> partition (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:131)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:97)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.paxos.PrepareResponse$PrepareResponseSerializer.deserialize(PrepareResponse.java:66)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> Caused by: java.io.IOException: Corrupt flags value for unfiltered partition
> (isStatic flag set): 160
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:380)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> ... 11 common frames omitted
>
