JingsongLi commented on code in PR #10077:
URL: https://github.com/apache/paimon/pull/10077#discussion_r4068922688
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaMergingUtils.java:
##########
@@ -68,6 +68,17 @@ public static TableSchema mergeSchemas(
return currentTableSchema;
}
+ // The automatic merge path must not relax the invariants enforced by
the explicit schema
+ // change path (see SchemaManagerUtils#assertNotUpdatingPartitionKeys /
+ // #assertNotUpdatingPrimaryKeys). Widening a key column changes its
value encoding, which
+ // silently rehashes it into another bucket, so the same logical key
would land in two
+ // different buckets and bypass deduplication. Reject such merges
instead of committing
+ // them.
+ assertNotUpdatingKeyColumn(
+ currentTableSchema, currentType, newRowType, "update",
"partition column");
+ assertNotUpdatingKeyColumn(
+ currentTableSchema, currentType, newRowType, "update",
"primary key");
Review Comment:
[P2] Update the Spark evolution tests for the new key-type contract
This correctly rejects a scenario that the existing Spark success tests
still require:
`paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/PaimonSinkTest.scala:242`
creates primary key `a INT`, feeds `MemoryStream[(Long, Date, Int)]` with type
widening enabled, and expects the key to become BIGINT. The Spark 3.4 copy at
`paimon-spark/paimon-spark-3.4/src/test/scala/org/apache/paimon/spark/PaimonSinkTest.scala:239`
does the same. Current [Spark3
CI](https://github.com/apache/paimon/actions/runs/35684011729/job/106606936417)
fails that test with `Cannot update primary key type from INT NOT NULL to
BIGINT NOT NULL: [a]`, with the stack pointing to this new guard. Please update
both successful evolution tests to retain the key type while evolving non-key
columns, and add a separate streaming rejection case checking that schema
metadata and existing rows remain unchanged. The guard should remain; the
integration tests need to reflect the intended invariant in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
