danny0405 opened a new pull request, #19665: URL: https://github.com/apache/hudi/pull/19665
### Describe the issue this Pull Request addresses Closes #19654. `hoodie.write.set.null.for.missing.columns` currently fills existing table fields that are absent from an incoming batch, but it does not relax newly added incoming fields before the legacy reconciliation branch runs. As a result, writes with `hoodie.datasource.write.reconcile.schema=true` and schema-on-read disabled can persist a required new field with no default, preventing existing records from being read with the evolved schema. This regressed the null-backfill behavior previously available through `hoodie.datasource.write.new.columns.nullable`. ### Summary and Changelog This restores opt-in null backfill for newly added fields without weakening Avro's default compatibility validation. - Make newly added top-level and nested fields nullable with a null default during schema canonicalization when null backfill is enabled. - Apply the policy before selecting the legacy or InternalSchema reconciliation path. - Resolve the null-backfill option through its canonical key and accept `hoodie.datasource.write.new.columns.nullable` as a compatibility alias. - Preserve the existing three-argument schema reconciliation API while adding the policy-aware overload. - Add unit and Spark regression coverage for strict validation, both reconciliation paths, commit metadata, old-row nulls, new-row values, and the compatibility alias. No code was copied. ### Impact This restores user-facing schema-evolution behavior only when null backfill is explicitly enabled. The default remains `false`, so newly added required fields without defaults still fail Avro compatibility validation. There is no storage-format change, breaking API change, or expected performance impact. ### Risk Level low. The change is confined to schema deduction for an opt-in policy. Focused unit and functional tests cover reconciliation with and without InternalSchema/schema-on-read, nested and top-level additions, persisted commit schema, existing-row null backfill, new-row values, the legacy option alias, and the unchanged strict default. ### Documentation Update Updated the inline configuration description for `hoodie.write.set.null.for.missing.columns`, including its newly restored new-field behavior and the legacy compatibility alias. No website update is required because this fixes an upgrade regression without adding a new configuration or changing its default. ### Contributor's checklist - [x] Read through Hudi's contributor guide - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
