slachiewicz opened a new pull request, #989: URL: https://github.com/apache/incubator-xtable/pull/989
Fixes #958. ## What is the purpose of the pull request `ParquetSchemaExtractor` called a record nullable only when its group was not `REQUIRED` and at least one child was nullable. Parquet gives the root `MessageType` the repetition `REPEATED`, so one optional column made the whole row nullable, and the Hudi target then failed to build column stats on the `["null", record]` union. The same condition reported an optional group whose fields are all required as non-nullable, so targets declared a required struct over values that can be null. A group's nullability is now its repetition alone, and the root message is never nullable. ## Brief change log - `ParquetSchemaExtractor`: the root message is never nullable; any other group is nullable unless it is `REQUIRED`. - `TestParquetSchemaExtractor#testRecordNullability` covers the root message and the three group cases. - `ITParquetConversionSource#testSync` makes `name` nullable with one null value, the shape Spark writes by default. ## Verify this pull request This change added tests. On `main`, `testRecordNullability` fails its root and optional-group cases, and all four `testSync` cases fail on the Hudi target with `Cannot get fields from schema type: UNION`. Verified: `./mvnw -pl xtable-core verify -Dit.test=ITParquetConversionSource` and `-Dtest=TestParquetSchemaExtractor` under JDK 11 → 6/6 and 10/10 pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
