vinishjail97 opened a new issue, #958:
URL: https://github.com/apache/incubator-xtable/issues/958
### Search before asking
- [x] I had searched in the issues and found no similar issues.
### Please describe the bug 🐞
`ParquetSchemaExtractor` derives nullability from the parquet repetition,
and the root `MessageType` has the repetition `REPEATED`. A record is treated
as nullable when any of its fields is nullable and its own repetition is not
`REQUIRED`, so the root record becomes nullable as soon as one column is
optional, which is the case for almost every file Spark writes.
`AvroSchemaConverter` then turns the table schema into `["null", record]`,
and the Hudi target fails while generating column stats for the metadata table:
```
HoodieTableMetadataUtil.convertMetadataToColumnStatsRecords
Caused by: java.lang.IllegalStateException: Cannot get fields from schema
type: UNION
at
org.apache.hudi.common.schema.HoodieSchema.getFields(HoodieSchema.java:1156)
at
org.apache.hudi.metadata.HoodieTableMetadataUtil.getFirstNSupportedFields(HoodieTableMetadataUtil.java:1719)
```
`ConversionController` reports the Hudi sync as failed with `Failed to
generate column stats records for metadata table`. Iceberg and Delta targets
are not affected.
**To reproduce:** write a DataFrame with at least one nullable column to
plain parquet with Spark, then sync it with the parquet source to a Hudi
target. The existing `ITParquetConversionSource` cases do not catch this,
because their columns are declared non-nullable, so the root stays required.
**Expected:** the root message is the table's row, which is never null, so
its `InternalSchema` should not be nullable.
A fix that skips the nullable check for the root `MessageType` makes the
sync of TPC-DS 100GB `store_returns` to Hudi succeed, and every row read back
matches the source files. `xtable-core` unit tests and
`ITParquetConversionSource` pass with it.
Anyone in the community is welcome to pick this up; it would be good to land
in the upcoming 0.5.0 release.
### Are you willing to submit PR?
- [ ] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
