rangareddy commented on issue #5683: URL: https://github.com/apache/hudi/issues/5683#issuecomment-5100895048
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: The reported failure is `java.lang.RuntimeException: Null-value for required field: userid_integer`, thrown from `AvroWriteSupport.writeRecordFields` inside `HoodieMergeHandle.write`. The writer schema in the stack trace shows `userid_integer` declared as a **required** `"int"` (no `null` union, no default), while the records being merged out of the pre-existing base file carry no value for it. Parquet's Avro write support then rejects the null. So the trigger is the target table's schema carrying a non-nullable column, not the column drop itself. The write path in that stack trace no longer exists: `AbstractMergeHelper` / `SparkMergeHelper` were removed and replaced by `HoodieMergeHelper` + `FileGroupReaderBasedMergeHandle`, and writer-schema deduction was reworked by #9743 ([HUDI-6872] Simplify Out Of Box Schema Evolution Functionality, merged 2023-11-10). That change is absent in `release-0.14.0` and present from `release-0.14.1` onward. Two pieces of it are directly relevant here: - `AvroSchemaEvolutionUtils.reconcileSchemaRequirements` (`hudi-common/src/main/java/org/apache/hudi/common/schema/internal/utils/AvroSchemaEvolutionUtils.java:185`) now relaxes a column's nullability during reconciliation instead of failing, by applying `updateColumnNullability(field, true)` for any column whose optionality differs between the incoming and table schemas. This was further hardened by #19337 (merged 2026-07-22). - `hoodie.write.set.null.for.missing.columns` (`HoodieCommonConfig:77`, `sinceVersion` 0.14.1, default `false`) fills a column that is missing from the incoming batch with nulls rather than failing the compatibility check. This is the supported knob for the "we stopped selecting the dropped column" case described above. We have not been able to confirm whether these cover your exact scenario, because the issue has no reproducible test case and no Hudi version. Requests for a repro were made on 2022-06-07, 2022-08-28 and 2022-10-30 (including a specific question about whether the Avro default you set was actually reflected in the registered schema) and went unanswered; there has been no activity since 2022-10-30. Closing as stale. If this is still a real problem on a recent Hudi version (1.0.x / 1.1.x / 1.2.0), please open a fresh issue with the Hudi and Spark versions, the old and new Avro schemas, the writer configs, and a minimal reproducible script. As a first thing to try on a recent version: declare the retired column as a nullable union (`["null","int"]` with `"default": null`) in the table schema, and set `hoodie.write.set.null.for.missing.columns=true` if you are dropping it from the incoming batch entirely. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
