junmuz commented on code in PR #10702:
URL: https://github.com/apache/paimon/pull/10702#discussion_r4404486900
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -205,6 +206,24 @@ public static void validateTableSchema(TableSchema schema,
Set<String> dynamicOp
"Can not set %s on table without primary keys,
please define primary keys.",
CHANGELOG_PRODUCER.key()));
}
+ List<String> preserveFields =
options.changelogPreserveFieldOnRetract();
+ if (!preserveFields.isEmpty()) {
+ checkArgument(
+ options.mergeEngine() == MergeEngine.DEDUPLICATE,
+ "'%s' is only supported with 'merge-engine' =
'deduplicate'.",
+ CHANGELOG_PRODUCER_PRESERVE_FIELD_ON_RETRACT.key());
+ checkArgument(
+ changelogProducer == ChangelogProducer.LOOKUP,
Review Comment:
The internal rescale copy in SparkPostponeStagedCommitter now has
changelog-producer.preserve-field-on-retract removed. That copy forces
changelog-producer=none and writes no changelog, so the option has no effect
there. The CREATE/ALTER restriction is unchanged, and the table itself keeps
the option. This is the only place that forces changelog-producer=none on a
table copy, so no other internal copies need the same change.
I added a Spark test to PostponeBucketTableTest. It uses a postpone
fixed-bucket table with changelog-producer=lookup and preserve-field-on-retract
enabled. The test writes enough rows to trigger a load-factor rescale from 1 to
16 buckets. It then upserts an existing key and checks the table contents and
options. Without the fix, the test fails with the reported validation error
from the SparkPostponeStagedCommitter constructor. With the fix, the whole
suite passes on Spark 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]