ad1happy2go commented on issue #19654: URL: https://github.com/apache/hudi/issues/19654#issuecomment-5343634897
Following up: I can confirm this **is** a genuine regression from 0.14, and that the newly opened #19665 fixes it. (Correcting my earlier comment above, which concluded there was no regression — that comparison had missed `hoodie.datasource.write.new.columns.nullable`.) ### Root cause — two 1.x changes combine 1. **`hoodie.datasource.write.new.columns.nullable` (`MAKE_NEW_COLUMNS_NULLABLE`) was removed** by #10782. In 0.14 it made newly-added columns nullable during schema reconciliation; in 1.x the reconcile method dropped the `opts` argument and nothing reads the key anymore, so setting it on 1.x is a no-op. 2. **`HoodieAvroUtils.rewriteRecordWithNewSchema` now throws** (changed in `b77eff2`, #10135) for a field that is present in the new schema but absent from old records when it is non-nullable with no default; 0.14 backfilled `null` there. So on 1.2 the failing config (`reconcile.schema=true`, `schema.on.read.enable=false`, `avro.schema.validate=false`) throws `Field address.country has no default value and is non-nullable`, whereas 0.14.1 evolved because `new.columns.nullable=true` relaxed the new fields to nullable. ### Verified the fix (#19665) Built the PR branch (1.3.0-SNAPSHOT) and ran the exact `generateSampleDf` + `withPhoneAndCountry` repro on COW: - Failing config + `hoodie.write.set.null.for.missing.columns=true` → schema evolves (adds `phone` and nested `address.country`); a record written before the evolution but **absent from the evolve batch reads back with `phone=null`** (correct backfill). - The legacy `hoodie.datasource.write.new.columns.nullable=true` key is honored again as an alias. - Without the flag the write still throws — opt-in default preserved, so no behavior change for existing users. - Works across all paths: no-reconcile, legacy reconcile, and schema-on-read. LGTM. ### Interim workaround on 1.2 (until #19665 lands) Set **both** `hoodie.datasource.write.reconcile.schema=true` and `hoodie.schema.on.read.enable=true`, or make the added columns nullable (default `null`) in the DataFrame. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
