[
https://issues.apache.org/jira/browse/CASSANDRA-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-21733:
-----------------------------------------
Bug Category: Parent values: Correctness(12982)Level 1 values: Recoverable
Corruption / Loss(12986)
Complexity: Normal
Discovered By: User Report
Fix Version/s: 4.0.x
4.1.x
5.0.x
6.x
7.x
Severity: Normal
Status: Open (was: Triage Needed)
> Reads fail after ALTER TABLE DROP ... USING TIMESTAMP older than the column's
> data
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-21733
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21733
> Project: Apache Cassandra
> Issue Type: Bug
> Components: CQL/Interpreter, Legacy/Local Write-Read Paths
> Reporter: Nadav Har'El
> Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x, 6.x, 7.x
>
>
> *Affects versions:*{color:#616161} 4.1.12, 5.0.9, 6.0-alpha2 (tested on the
> official Docker images); probably all versions since 4.0 or earlier{color}
> When a regular column is dropped with {{{}ALTER TABLE ... DROP col USING
> TIMESTAMP t{}}}, and the column still holds cells with timestamps newer than
> {{{}t{}}}, reads of that table fail. The coordinator returns a ReadFailure to
> the client, and the replica logs an IllegalStateException while serializing
> the read response.
> Reproducer (single node, cqlsh):
> {code:sql}
> CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> CREATE TABLE ks.t (a int PRIMARY KEY, c int, f int);
> INSERT INTO ks.t (a, c, f) VALUES (1, 2, 3);
> ALTER TABLE ks.t DROP f USING TIMESTAMP 20000;
> SELECT * FROM ks.t;
> {code}
> The {{SELECT}} fails:
> {noformat}
> ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
> message="Operation failed - received 0 responses and 1 failures: UNKNOWN from
> /127.0.0.1:7000"
> {noformat}
> and the node logs:
> {noformat}
> java.lang.RuntimeException: java.lang.IllegalStateException: [c, f] is not a
> subset of [c]
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:3159)
> ...
> Caused by: java.lang.IllegalStateException: [c, f] is not a subset of [c]
> at org.apache.cassandra.db.Columns$Serializer.encodeBitmap(Columns.java:618)
> at
> org.apache.cassandra.db.Columns$Serializer.serializeSubset(Columns.java:548)
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:242)
> ...
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:262)
> at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:453)
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2756)
> {noformat}
> (Stack trace from 6.0-alpha2; 4.1 and 5.0 fail the same way.)
> Observations:
> * The same sequence without {{USING TIMESTAMP}} (so the drop time is newer
> than the data) works correctly.
> * The INSERT doesn't need an explicit timestamp. Any data newer than the
> drop timestamp triggers the bug. In practice that can happen with clock skew,
> client-provided timestamps, or a schema restored with explicit {{DROP ...
> USING TIMESTAMP}} statements, like the ones DESCRIBE and snapshots'
> schema.cql emit for dropped columns.
> * The table doesn't need another regular column. With {{{}CREATE TABLE ks.t
> (a int PRIMARY KEY, f int){}}}, the error becomes {{{}[f] is not a subset of
> []{}}}.
> * Range scans ({{{}SELECT * FROM ks.t{}}}) fail for every table we tried.
> Single-partition reads ({{{}WHERE a = 1{}}}) also fail when the table has a
> clustering key, e.g. {{{}PRIMARY KEY (a, b){}}}, but succeed in the example
> above without one.
> * The error is in serializing the read response. The row still contains the
> dropped column {{{}f{}}}, apparently because its cells are newer than the
> drop time, but the serialization header's column set doesn't include it.
> *Why the existing unit tests don't catch this:*
> The unit tests that drop a column {{USING TIMESTAMP}} older than its data
> don't detect this bug:
> * {{{}AlterTest.testDropWithTimestamp{}}}, {{testDropStaticWithTimestamp}}
> and {{testDropMultipleWithTimestamp}} re-ADD the dropped column before
> reading the table, so they never read a table whose dropped column still has
> newer data.
> * {{CreateLikeTest.testCopyAfterAlterTable}} does exactly that: it runs
> {{ALTER TABLE ... DROP f USING TIMESTAMP 20000}} after writing {{f}} with
> current timestamps, and then reads the table. It passes because CQLTester's
> {{execute()}} runs queries through {{{}QueryProcessor.executeInternal(){}}},
> which reads locally without building a serialized {{{}ReadResponse{}}}, and
> that serialization is where the bug is. The same test fails when its
> statements are sent through the native protocol (e.g. with
> {{{}executeNet(){}}}). We found this while translating CreateLikeTest to
> ScyllaDB's Python-based CQL tests, which run against a real Cassandra node
> through the Python driver.
> Related: CASSANDRA-21607 reports a similar scenario, a dropped non-frozen
> collection that still holds cells newer than the drop time, which causes a
> NullPointerException in reads and compaction. This issue is the same
> situation for a simple (non-collection) column. It fails differently, in
> read-response serialization, but may have the same root cause.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
