[
https://issues.apache.org/jira/browse/CASSANDRA-20118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17905618#comment-17905618
]
Brandon Williams edited comment on CASSANDRA-20118 at 12/13/24 10:54 PM:
-------------------------------------------------------------------------
bq. I can see a brief schema mismatch after the first node goes to UPGRADING,
but then I see the schema changes replicate to the other nodes and the schema
mismatch goes away. This is I think how it is now meant to work.
Yes indeed it is.
[Here|https://app.circleci.com/pipelines/github/driftx/cassandra/1786/workflows/23c70c72-69ab-494a-b14c-ede14bbcae3f]
is CI against the patch with
[this|https://github.com/driftx/cassandra-dtest/tree/CASSANDRA-20118] dtest
repo. I think the only new failures are the EmptyValuesTests, which I think
are failing due to [this
line|https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L343],
but I haven't figured out what I'm going to do there.
[Here|https://app.circleci.com/pipelines/github/driftx/cassandra?branch=CASSANDRA-20118-5.0-vhack]
is CI against this patch with its version set to 4.1, against
[this|https://github.com/driftx/cassandra-dtest/tree/CASSANDRA-20118-vhack]
dtest repo. I fixed the create/drop role auth problems and I think everything
that's failing is due to mismatched output now instead of throwing errors. I
don't think those cosmetics are worth bothering with in CASSANDRA_4 mode.
Aside from the EmptyValuesTests I think this is ready for review.
was (Author: brandon.williams):
bq. I can see a brief schema mismatch after the first node goes to UPGRADING,
but then I see the schema changes replicate to the other nodes and the schema
mismatch goes away. This is I think how it is now meant to work.
Yes indeed it is.
[Here|https://app.circleci.com/pipelines/github/driftx/cassandra/1788/workflows/45c3fca1-cc80-4e5e-abb6-12c72a5d99fd]
is CI against the patch with
[this|https://github.com/driftx/cassandra-dtest/tree/CASSANDRA-20118] dtest
repo. I think the only new failures are the EmptyValuesTests, which I think
are failing due to [this
line|https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L343],
but I haven't figured out what I'm going to do there.
[Here|https://app.circleci.com/pipelines/github/driftx/cassandra?branch=CASSANDRA-20118-5.0-vhack]
is CI against this patch with its version set to 4.1, against
[this|https://github.com/driftx/cassandra-dtest/tree/CASSANDRA-20118-vhack]
dtest repo. I fixed the create/drop role auth problems and I think everything
that's failing is due to mismatched output now instead of throwing errors. I
don't think those cosmetics are worth bothering with in CASSANDRA_4 mode.
Aside from the EmptyValuesTests I think this is ready for review.
> Hints ignored during Upgrade from C*4 to C*5
> --------------------------------------------
>
> Key: CASSANDRA-20118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20118
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Consistency/Hints
> Reporter: Paul Chandler
> Assignee: Brandon Williams
> Priority: Normal
> Fix For: 5.0.x
>
> Attachments: 4-1debug-third-node.log, 4-1debug.log,
> 4-1system-third-node.log, 4-1system.log, 5-0debug.log, 5-0system.log,
> debug20241213.log, image-2024-12-11-17-44-56-585.png,
> image-2024-12-12-11-39-25-970.png, system-20118.log
>
>
> I have discovered that some hints were not being processed after nodes come
> back up when a cluster in in a mixed mode with some cassandra 4 nodes and
> some cassdandra 5 nodes ( these with a storage compatibility mode CASSANDRA_4
> )
>
> When in this mode there is a schema mismatch after the first node has been
> upgraded, which continues until the last node has been upgraded.
> It seems that the hints are blocked from being sent if there is a schema
> mismatch between the 2 nodes, that can be seen at this line.
> [cassandra/src/java/org/apache/cassandra/hints/HintsDispatchTrigger.java at
> cassandra-5.0 ·
> apache/cassandra|https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/hints/HintsDispatchTrigger.java#L65]
> I have tested removing this line, and that then does allow the hint to be
> transferred normally. However I am not sure of the implications for doing
> that if the hint is for part of the schema where the actual mismatch occurs.
>
> This creates the problem when a node is being upgraded and is currently down,
> hint files will be created for it on the new cassandra 5 nodes and the old
> cassandra 4 nodes, but the hint files on the old cassandra 4 nodes will not
> be processed, due to the schema mismatch. Leading to potential data loss.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
