[
https://issues.apache.org/jira/browse/CASSANDRA-20118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17905286#comment-17905286
]
Brandon Williams edited comment on CASSANDRA-20118 at 12/12/24 8:43 PM:
------------------------------------------------------------------------
The schema digest is calculated from all columns across all tables. 5.0 is
adding new columns to system tables and also table properties which are columns
that get added to every table, and both of these have to be communicated "in
code" since we don't pull system table schema. These new columns are what
causes disagreement. I have moved this to CASSANDRA_4 mode, which causes the
initial disagreement with 4.1. When all nodes are on CASSANDRA_4 they will
agree again.
5.0 also adds new tables in system_auth, and these can be pulled via schema
migration, and this should now work after relaxing the MS version restriction
for CASSANDRA_4, but I have moved their introduction to UPGRADING mode. Schema
should once again agree when these tables are pulled.
These changes are in my branch now:
https://github.com/driftx/cassandra/tree/CASSANDRA-20118-5.0
This means that rolling back from CASSANDRA_4 mode will at least leave you with
an unused column in compaction_history, but that is fairly minimal.
was (Author: brandon.williams):
The schema digest is calculated from all columns across all tables. 5.0 is
adding new columns to system tables and also table properties which are columns
that get added to every table, and both of these have to be communicated "in
code" since we don't pull system table schema. These new columns are what
causes disagreement. I have moved this to CASSANDRA_4 mode, which causes the
initial disagreement with 4.1. When all nodes are on CASSANDRA_4 they will
agree again.
5.0 also adds new tables in system_auth, and these can be pulled via schema
migration, and this should now work after relaxing the MS version restriction
for CASSANDRA_4, but I have move their introduction to UPGRADING mode. Schema
should once again agree when these tables are pulled.
These changes are in my branch now:
https://github.com/driftx/cassandra/tree/CASSANDRA-20118-5.0
This means that rolling back from CASSANDRA_4 mode will at least leave you with
an unused column in compaction_history, but that is fairly minimal.
> Hints ignored during Upgrade from C*4 to C*5
> --------------------------------------------
>
> Key: CASSANDRA-20118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20118
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Consistency/Hints
> Reporter: Paul Chandler
> Assignee: Brandon Williams
> Priority: Normal
> Fix For: 5.0.x
>
> Attachments: 4-1debug-third-node.log, 4-1debug.log,
> 4-1system-third-node.log, 4-1system.log, 5-0debug.log, 5-0system.log,
> image-2024-12-11-17-44-56-585.png, image-2024-12-12-11-39-25-970.png,
> system-20118.log
>
>
> I have discovered that some hints were not being processed after nodes come
> back up when a cluster in in a mixed mode with some cassandra 4 nodes and
> some cassdandra 5 nodes ( these with a storage compatibility mode CASSANDRA_4
> )
>
> When in this mode there is a schema mismatch after the first node has been
> upgraded, which continues until the last node has been upgraded.
> It seems that the hints are blocked from being sent if there is a schema
> mismatch between the 2 nodes, that can be seen at this line.
> [cassandra/src/java/org/apache/cassandra/hints/HintsDispatchTrigger.java at
> cassandra-5.0 ·
> apache/cassandra|https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/hints/HintsDispatchTrigger.java#L65]
> I have tested removing this line, and that then does allow the hint to be
> transferred normally. However I am not sure of the implications for doing
> that if the hint is for part of the schema where the actual mismatch occurs.
>
> This creates the problem when a node is being upgraded and is currently down,
> hint files will be created for it on the new cassandra 5 nodes and the old
> cassandra 4 nodes, but the hint files on the old cassandra 4 nodes will not
> be processed, due to the schema mismatch. Leading to potential data loss.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
