wombatu-kun opened a new pull request, #19467: URL: https://github.com/apache/hudi/pull/19467
### Describe the issue this Pull Request addresses Closes #19457 A predicate on a column whose type was widened after a base file was written fails the query, for as long as any base file still stores the original physical type: ``` Malformed Parquet file. Corrupted statistics for column "[c7] optional float c7" Caused by: java.lang.ClassCastException: class java.lang.Float cannot be cast to class java.lang.Double ``` `TupleDomainParquetPredicate.getDomain` picks its branch from the pushed-down domain's type and reads the parquet statistics as that type. The domain's type comes from the metastore, the statistics come from the file, and a type evolution is what makes the two disagree. Reading such a column always worked; only the statistics side was blind. This is schema-on-write, not schema-on-read: `HoodieSchemaCompatibilityChecker.getCompatibility` permits `double <- float`, `float <- int|long`, `long <- int` and `string <- any numeric`, and `hoodie.avro.schema.validate` defaults to `false`. ### Summary and Changelog The connector now leaves a domain out of the parquet predicate when the file's physical type cannot answer it. The engine still applies that predicate above the scan, so failing queries return their rows, and the only loss is row group pruning on the evolved column, for files written before the evolution. - `ParquetStatisticsDomains` (new): `hasComparableStatistics` mirrors `getDomain`'s dispatch, `dropIncomparableDomains` filters the descriptor-keyed tuple domain. Filtering on the descriptor rather than on the handle is what stops the check and the evaluation disagreeing about which column and type is meant, and covers both `hudi.parquet.use-column-names` modes in one pass. - `HudiPageSourceProvider.createPageSource`: wraps the existing `getParquetTupleDomain` call, which feeds both `buildPredicate` and `getFilteredRowGroups`. - Dropping is sound because `HudiMetadata.applyFilter` returns the whole regular predicate as the remaining filter, so pushdown here is an optimization only. It is the trade `remapPredicateColumnIndicesToPhysical` and `HudiColumnStatsIndexSupport.getDomainFromColumnStats` already make. - Two rejected pairs are silent wrong answers rather than crashes, a decimal column read as varchar and a string column read as decimal, which is why this is a type check and not a try/catch. Tests, where `hudi-trino` had no schema evolution coverage at all: `TestParquetStatisticsDomains` (30 type pairs, each checked against the real `getDomain`), `TestHudiEvolvedColumnPredicates` (reads a pre-evolution base file through `createPageSource`), and `TestHudiSchemaEvolutionPredicates` with its positional subclass at SQL level. Both column-resolution modes throughout. ### Impact Queries with a predicate on a type-evolved column succeed instead of failing with `HUDI_BAD_DATA`. Pruning is unchanged everywhere else, including `int -> long`, which the statistics can answer and the guard keeps. Pushdown is only enabled for base-file-only splits, so the merge path is untouched. No config or public API change. ### Risk Level low ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
