vinishjail97 opened a new pull request, #19123:
URL: https://github.com/apache/hudi/pull/19123
### Describe the issue this Pull Request addresses
Closes #19122
Reading a Hudi table partitioned on a nested column (a partition field whose
name is a dotted path, e.g. `nested_record.level`) fails on the only batch read
path with `HoodieSchemaException: Illegal character in: nested_record.level`,
so the table cannot be read at all.
### Summary and Changelog
When a partition field is mandatory,
`HoodieFileGroupReaderBasedFileFormat#buildReaderWithPartitionValues` adds it
to the `StructType` it converts into a `HoodieSchema` (a top-level Avro field)
so it can be read from the file. For a nested partition column the field name
is a dotted path, which is not a valid Avro name, so
`convertStructTypeToHoodieSchema` throws before the read starts.
A nested partition column is never a flat top-level column in the data file:
its value is materialized from the partition path, and its root field
(`nested_record`) is already read via the data schema. This PR treats nested
partition fields (names containing `.`) as appended partition fields rather
than fields read from the file, so they are not converted into a top-level Avro
field:
- Add an `isNestedPartitionField(name)` helper (`name.contains(".")`).
- Keep nested partition fields in the appended ("remaining") partition
fields when splitting `remainingPartitionSchema` / `fixedPartitionIndexes`, so
their values are appended from the partition path.
- Exclude nested partition fields from the two `StructType -> HoodieSchema`
conversions (`requestedStructType`,
`dataStructTypeWithMandatoryPartitionFields`) so the dotted name is never
converted to an Avro field.
- Apply the same exclusion in `vectorTypes`.
This mirrors the existing root-level handling for nested mandatory columns
in `HoodieBaseRelation#appendMandatoryColumns`
(`HoodieAvroUtils.getRootLevelFieldName`).
### Impact
Tables partitioned on a nested column become readable. No change for tables
partitioned on top-level columns (the new predicate only matches dotted names).
### Risk Level
low. The behavioral change is scoped to partition fields whose name contains
`.`, which previously could not be read at all. Validated end-to-end via Apache
XTable's `ITConversionController#testPartitionedData` (HUDI partitioned on
`nested_record.level` -> ICEBERG) against a locally built snapshot: the
conversion now succeeds and the `nested_record.level = 'INFO'` filter
round-trips, confirming the partition value is materialized correctly (not just
that the crash is avoided).
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
