vamsikarnika opened a new pull request, #19337: URL: https://github.com/apache/hudi/pull/19337
### Change Logs `AvroSchemaEvolutionUtils.reconcileSchema` — invoked from `HoodieSchemaUtils.deduceWriterSchema` when `hoodie.write.set.null.for.missing.columns=true` — rebuilds the writer schema **from the existing table schema** and only altered a shared column on a *data-type* change (`findType` ignores nullability). As a result, an existing column that the incoming/source schema relaxed from **required → nullable** stayed `REQUIRED` in the table indefinitely, so records arriving with `null` in that column failed record creation (and, where an error table is configured, were routed to the quarantine table). Note the `set.null.for.missing.columns=false` path already preserved the relaxation (it uses the canonicalized source schema directly); only the `=true` path silently dropped it. **Fix:** in `reconcileSchema`, detect existing columns whose incoming schema is nullable but the table is required (`nullabilityRelaxColumns`), exclude them from the early-return short-circuit, and relax them to nullable in the result via `updateColumnNullability(col, true)`. This **only ever widens, never tightens** (a nullable→required incoming leaves the table nullable), consistent with `reconcileSchemaRequirements`, and still null-fills genuinely missing columns. Tests: - `TestAvroSchemaEvolutionUtils` — direct `reconcileSchema` relax assertion + a no-tighten guard. - `TestHoodieSchemaUtils` — end-to-end `deduceWriterSchema`, parameterized over the flag. - Verified both new relax-cases **fail without the change** (`expected optional boolean but was required boolean`); full `TestAvroSchemaEvolutionUtils` (15) + `TestHoodieSchemaUtils` (11) pass with it. ### Impact Tables ingesting via the streamer/datasource with `set.null.for.missing.columns=true` will now correctly evolve an existing column to nullable when the source relaxes it, instead of leaving it required and failing/quarantining null-bearing records. Widening-only; no existing column is ever tightened, and missing-column null-filling is unchanged. ### Risk level: low Change is confined to `reconcileSchema` and only relaxes nullability of a shared column when the incoming schema is already nullable — a backwards-compatible schema widening. Covered by new unit tests and the existing `AvroSchemaEvolutionUtils` / `HoodieSchemaUtils` suites. ### Documentation Update None required — behavior now matches the documented intent of schema evolution (a valid required→nullable relaxation should be applied). ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact were stated clearly - [x] Adequate tests were added (unit tests, incl. a failing-without-fix check) - [x] CI passes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
