linliu-code commented on code in PR #19110:
URL: https://github.com/apache/hudi/pull/19110#discussion_r3616404234
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamer.java:
##########
@@ -155,6 +157,15 @@ public HoodieStreamer(Config cfg, JavaSparkContext jssc,
FileSystem fs, Configur
public HoodieStreamer(Config cfg, JavaSparkContext jssc, FileSystem fs,
Configuration conf,
Option<TypedProperties> propsOverride,
Option<SourceProfileSupplier> sourceProfileSupplier) throws IOException {
this.properties = combineProperties(cfg, propsOverride,
jssc.hadoopConfiguration());
+ // The Hudi Streamer is the only write path that knows both the Debezium
payload and whether the
+ // transformer nests the CDC metadata, so it resolves the ordering field
here and threads it
+ // through both the write properties and the config that table creation
persists.
+ String resolvedDebeziumOrderingFields =
DebeziumConstants.resolveOrderingFields(cfg.payloadClassName,
+ ConfigUtils.getBooleanWithAltKeys(this.properties,
DebeziumTransformerConfig.ENABLE_NESTED_FIELDS));
+ if (resolvedDebeziumOrderingFields != null) {
+ cfg.sourceOrderingFields = resolvedDebeziumOrderingFields;
+ this.properties.setProperty(HoodieTableConfig.ORDERING_FIELDS.key(),
cfg.sourceOrderingFields);
+ }
Review Comment:
Non-blocking: this block also runs for existing **source**-path users
(`PostgresDebeziumSource`/`MysqlDebeziumSource`), not just the new transformer
— it keys off `payloadClassName`. Because it overwrites
`cfg.sourceOrderingFields` *before* `StreamSync`'s pre-existing `_event_seq ->
file,pos` normalization + ordering-field validation, a **pre-v9 MySQL table
with `_event_seq` persisted** (legitimate, since `handlePayloadAdhocConfigs`
only forces the canonical field on the v9 path) could, on restart, compare the
forced `_event_bin_file,_event_pos` against the persisted `_event_seq` and hit
`HoodieValidationException` — an upgrade break for existing users.
CI doesn't cover this: the added tests are transformer flattening +
`inferMergingConfigsForV9TableCreation` in isolation; none constructs a
`HoodieStreamer` on the source path or exercises the `StreamSync` validation,
and `TestPostgres/MysqlDebeziumSource` only cover `processDataset`.
Could we add a differential check that `(RecordMergeMode, ordering fields,
persisted ORDERING_FIELDS, whether StreamSync validation throws)` is unchanged
vs master for existing source-path configs? Must-include cell: **pre-v9 MySQL
table, persisted `_event_seq`, restarted**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
