rangareddy commented on issue #5685: URL: https://github.com/apache/hudi/issues/5685#issuecomment-5102636501
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: Both halves of what you asked for are supported now. **Replaying older-schema data into an evolved table.** Writer-schema deduction was reworked by #9743 ([HUDI-6872], merged 2023-11-10 -- absent in `release-0.14.0`, present from `release-0.14.1`). `AvroSchemaEvolutionUtils.reconcileSchemaRequirements` (`hudi-common/src/main/java/org/apache/hudi/common/schema/internal/utils/AvroSchemaEvolutionUtils.java:185`) reconciles the incoming and table schemas rather than rejecting the write -- relaxing a column to nullable where the two differ, and promoting types where a promotion is valid. **Filling defaults/null on mismatch.** This is `hoodie.write.set.null.for.missing.columns` (`HoodieCommonConfig.java:77`, `sinceVersion` 0.14.1, default `false`). Set it to `true` and a column present in the table but missing from the incoming batch is written as null instead of failing the schema compatibility check. So the concrete recipe for your replay case on a recent version (1.0.x / 1.1.x / 1.2.0) is: declare retired columns as nullable unions (`["null", T]` with `"default": null`) and set `hoodie.write.set.null.for.missing.columns=true`. Note that `hoodie.datasource.write.reconcile.schema`, suggested on 2022-06-10, is the older deprecated path -- prefer the config above. As a cross-check, the closely related #3558 (type promotion across a subset of partitions) was re-run on 0.14.0 and 1.2.0 and reads back correctly, including partitions that were never rewritten. Closing as fixed. If replaying older-schema data still fails on a recent version, please reopen with the two schemas and the write configs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
