prashantwason opened a new pull request, #20356:
URL: https://github.com/apache/hudi/pull/20356
### Describe the issue this Pull Request addresses
Closes #20355
BigQuery sync fails for any table whose schema contains a field declared
with the Avro primitive type `"null"` (as opposed to a nullable union such as
`["null","string"]`), e.g. `{"name": "clientPromotion", "type": "null",
"default": null}`. `BigQuerySchemaResolver.getField` has no case for `NULL`, so
such a field hits the `default` branch and sync throws `RuntimeException:
Unexpected field type: NULL`.
This is the BigQuery counterpart of the Hive sync fix in #20354.
### Summary and Changelog
BigQuery sync no longer fails on schemas with `"null"` type fields. These
fields always hold null and the parquet writer never writes them, so they are
skipped and the synced BigQuery schema matches the columns in the data files.
- `BigQuerySchemaResolver.getFields`: skip fields whose schema is the
`"null"` type. `getFields` is also used for nested records, so this covers
top-level and nested fields. Nullable unions (`["null", T]`) are unaffected and
still map to `NULLABLE` columns.
- Tests: `TestBigQuerySchemaResolver#convertSchema_nullTypeField` and
`#convertSchema_nestedNullTypeField` cover top-level and nested `"null"` type
fields. Both fail with `Unexpected field type: NULL` without the fix.
### Impact
Fixes BigQuery sync for tables with `"null"` type fields. No public API or
config change.
### Risk Level
low. The change only affects fields whose type is exactly `"null"`, which
previously caused sync to throw.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
