rstest created CASSANDRA-21445:
----------------------------------
Summary: Schema DDL accepted during mixed-major rolling upgrade is
neither propagated nor repaired, leaving permanent schema/data divergence after
the upgrade completes
Key: CASSANDRA-21445
URL: https://issues.apache.org/jira/browse/CASSANDRA-21445
Project: Apache Cassandra
Issue Type: Bug
Components: Cluster/Membership, Cluster/Schema
Reporter: rstest
h2. Description
During a rolling upgrade, while the cluster has nodes of two different major
versions, schema-changing CQL is accepted normally, but the resulting schema
mutations cannot cross the version boundary:
* {{MigrationCoordinator.shouldPushSchemaTo()}} only pushes schema to peers
whose raw messaging version equals {{{}MessagingService.current_version{}}}, so
the push to the not-yet-upgraded peer is silently skipped (the 3.11 side has
the mirror gate in {{MigrationManager.announce()}} / {{{}is30Compatible(){}}}).
* {{MigrationCoordinator.shouldPullFromEndpoint()}} refuses pulls when the
peer's release major differs ("Not pulling schema from {} because release
version ... is not major version ...") or its messaging version differs - per
CASSANDRA-13274 schema is deliberately not exchanged across majors.
We understand the exchange restriction itself is intentional. The bug is the
{*}combination{*}: DDL is accepted with no rejection, warning, or deferral
while the restriction is in force, and after the window closes there is no
mechanism that guarantees the cluster converges back to a single correct
schema. The result is a cluster that completed its rolling upgrade
"successfully" but permanently disagrees with what the same workload produces
on either pure version.
h2. Observed symptoms (four variants, same root cause)
In all four cases the identical statement sequence succeeds and converges on an
all-old cluster and on an all-new cluster, and diverges only in the lane that
performed the real rolling upgrade. The divergence is CQL-visible after the
upgrade is finalized.
# *Lost ADD COLUMN* (3.11.19 -> 4.1.10): {{ALTER TABLE ... ADD eh text}}
executed through the upgraded node while the peer is old. After both nodes are
upgraded and the upgrade finalized, {{SELECT eh, ...}} fails with
{{InvalidRequest}} (undefined column) - the column never reached the second
node.
# *Silent data divergence* (4.1.10 -> 5.0.6): {{CREATE TABLE}} on the upgraded
node, immediately followed by an {{INSERT}} routed through the still-old
coordinator. After finalization the rolling cluster returns a different row
payload than both baselines. No error is ever raised - this is the dangerous
variant.
# *Two-step rename converges to the intermediate name* (4.1.10 -> 5.0.6):
{{RENAME a TO b}} executed on the upgraded side, then {{RENAME b TO c}} on the
still-old side. After finalization the cluster exposes column {{{}b{}}};
queries on {{c}} fail with {{{}InvalidRequest{}}}.
# *Whole table permanently missing on one node* (3.11.19 -> 4.1.10): table
created on the old node while the other node was already upgraded; combined
with a peer restart/network interruption during the window, one node never
learns the table exists; {{SELECT}} coordinated by that node fails forever with
{{{}InvalidRequest "table ... does not exist"{}}}.
h2. Why the divergence is permanent (all cases)
The mixed-window propagation gap is deterministic; what makes it *permanent*
differs by variant, and we want to call out all of them explicitly because they
need different fixes:
# *Conflicting-history merge (no faults needed).* Renames are persisted by
{{SchemaKeyspace.addAlterTableToSchemaMutation()}} as delete+insert row diffs
against {{{}system_schema.columns{}}}, not as rename intents. When the two
sides finally exchange schema after both reach the new major, the
timestamp-based row merge converges - to the wrong schema (the intermediate
column name). Repair runs and still produces the bad state; retrying cannot
help because the intent was never representable.
# *Data written against stale schema (no faults needed).* A write accepted by
the not-yet-converged coordinator during the window executes against its stale
schema view. Even after schema later converges, the row-level damage is
durable; schema repair cannot un-write data. (Plain upgrade + DDL + DML - the
recorded plan contains no fault injection at all.)
# *Repair-window loss (needs unlucky timing).*
{{MigrationCoordinator.pullSchema()}} fails the pull immediately when the peer
is down ("Can't send schema pull request: node ... is down.") with no durable
retry, and a compensating {{SCHEMA_PUSH_REQ}} can expire in the outbound queue
while a link is down. If both escape hatches are missed once, nothing ever
re-attempts convergence: an entire table stayed missing on one node through
finalization and beyond.
# *Repair latency (at minimum).* Even in the most benign variant
(strictly-missing ADD COLUMN, no conflict, no faults), a freshly upgraded node
postpones schema pulls for its first 60 s of uptime
({{{}MigrationCoordinator{}}} checks {{{}getUptimeFn.getAsLong() <
MIGRATION_DELAY_IN_MS{}}}, {{{}cassandra.migration_delay_ms=60000{}}}), and
upgrade finalization is not gated on schema agreement, so clients observe the
divergence on a "finished" cluster. We have not confirmed whether this variant
eventually self-heals after the delay; the other three demonstrably do not.
h2. Steps to reproduce (simplest variant, no faults required)
Two-node cluster, both on the old version (e.g. 3.11.19), new version 4.1.10:
{code:sql}
-- step 1: both nodes OLD
CREATE KEYSPACE ks WITH replication =
{'class':'SimpleStrategy','replication_factor':2};
CREATE TABLE ks.t (pk text, PRIMARY KEY (pk));
{code}
# Rolling-upgrade node 1 to 4.1.10 (drain, stop, swap binaries, start).
Cluster is now mixed-major.
# Through *node 1* (the upgraded node):
{code:sql}
ALTER TABLE ks.t ADD eh text;{code}
Note node 1's debug log: schema push to node 2 is skipped; node 2 never logs
the column. Node 2's log shows "Not pulling schema from ... because release
version ... is not major version ..." if a pull is attempted.
# Rolling-upgrade node 2 to 4.1.10. Finalize/complete the upgrade.
# From node 2 (or with the driver round-robining):
{code:sql}
SELECT eh, pk FROM ks.t;{code}
Expected: the column exists cluster-wide (or step 3's DDL had been rejected
with a clear error).
Actual: {{InvalidRequest: Undefined column name eh}} - observed through
validation well after both nodes were upgraded and the upgrade finalized.
Variant recipes (same skeleton): (2) create the table through the upgraded node
and INSERT through the old coordinator before upgrading it - after finalization
the read results differ from a never-upgraded cluster with no error; (3)
{{RENAME a TO b}} via the upgraded node, then {{RENAME b TO c}} via the old
node - final schema keeps {{{}b{}}}; (4) create the table on the old node while
the other is upgraded, and restart/partition the upgraded peer around the
second node's upgrade - the table never appears on the peer.
h2. Root cause code pointers
* {{org.apache.cassandra.schema.MigrationCoordinator#shouldPushSchemaTo}}
(4.1.10: MigrationCoordinator.java:769) - push skipped unless raw messaging
version equals current.
* {{org.apache.cassandra.schema.MigrationCoordinator#shouldPullFromEndpoint}}
(4.1.10: MigrationCoordinator.java:353) - pull refused across release majors /
messaging versions (CASSANDRA-13274).
* {{org.apache.cassandra.schema.MigrationCoordinator#pullSchema}} (4.1.10:
MigrationCoordinator.java:658) - pull marked failed when peer is down; no
durable retry obligation.
* {{MIGRATION_DELAY}} gating (4.1.10: MigrationCoordinator.java:404,
CassandraRelevantProperties: {{{}cassandra.migration_delay_ms=60000{}}}) -
fresh nodes postpone pulls for 60 s.
* {{org.apache.cassandra.service.MigrationManager#announce}} (3.11.19:
MigrationManager.java:443) + {{is30Compatible}} - the old side's mirror push
gate.
* {{org.apache.cassandra.schema.SchemaKeyspace#addAlterTableToSchemaMutation}}
- renames serialized as delete+insert diffs, which makes variant 3
unreconcilable.
* Note for 4.1 -> 5.0: with {{{}storage_compatibility_mode: CASSANDRA_4{}}},
5.0 keeps {{current_version = VERSION_40}} (MessagingService.java:257), so the
push version check may pass while pulls remain major-blocked; schema payloads
can also be undecodable on the 4.1 peer (5.0-only
{{system_schema.column_masks}} causes {{UnknownTableException}} - we will file
that deserialization issue separately).
h2. Expected behavior / suggested fixes
Any one of these layers would have prevented the user-visible outcome; ideally
the first plus one of the others:
# *Fail loud:* reject or defer schema-changing CQL while any live peer
advertises an incompatible major/messaging version (clear client error such as
"schema changes unavailable during mixed-version upgrade"), instead of applying
locally and silently skipping propagation.
# *Repair reliably:* re-attempt failed schema pulls when a peer becomes alive
again (especially after it restarts upgraded); treat dropped/expired
{{SCHEMA_PUSH_REQ}} as pending repair rather than fire-and-forget; surface
persistent schema disagreement (gossip schema UUID mismatch beyond N minutes)
as an error-level health signal.
# *Gate finalization:* document/provide a check so that "rolling upgrade
complete" requires cluster-wide schema agreement when schema changed during the
window (e.g. a nodetool check that operators/orchestrators can gate on).
# *Preserve intent where merges are semantic:* multi-step renames cannot be
reconciled from row diffs; either block RENAME in mixed mode specifically or
carry rename intent in the schema exchange.
h2. Additional context
* CASSANDRA-13274 introduced the cross-major exchange restriction ("Fix code
to not exchange schema across major versions"); this report is about the
missing complementary guard rails around it.
* All four variants were detected by tri-differential execution (all-old vs
rolling vs all-new of the same plan), which is why we are confident the
workloads themselves are version-compatible: both pure-version clusters
converge on every sequence.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
