wombatu-kun opened a new issue, #19457:
URL: https://github.com/apache/hudi/issues/19457
### Bug Description
**What happened:**
A Hudi table whose schema evolved `float` -> `double` fails the whole query
through the Trino connector as soon as a predicate on the evolved column is
pushed down, for as long as any base file still stores that column as parquet
`FLOAT`:
```
TrinoException: Malformed Parquet file. Corrupted statistics for column
"[c7] optional float c7"
```
The mechanism sits in trino-parquet, not in Hudi.
`TupleDomainParquetPredicate` picks its branch on the type of the pushed-down
DOMAIN, which is the Trino type the metastore column handle carries, and then
casts the parquet statistics values to it:
```java
//
lib/trino-parquet/src/main/java/io/trino/parquet/predicate/TupleDomainParquetPredicate.java,
Trino 481
Domain domain = getDomain(column, effectivePredicateDomain.getType(), ...);
...
if (type.equals(DOUBLE)) {
Double min = (Double) minimums.get(i);
Double max = (Double) maximums.get(i);
```
After the evolution the metastore says `DOUBLE` while the base file still
holds `FLOAT`, so the statistics values are `Float` and the cast throws
`ClassCastException`. `corruptionException(...)` wraps it into a
`ParquetCorruptionException`, and `HudiPageSourceProvider.createPageSource`
maps that to `HUDI_BAD_DATA`.
The integer family is benign by contrast:
`TupleDomainParquetPredicate.asLong` accepts any
`Byte`/`Short`/`Integer`/`Long`, so an `int -> long` promotion reads fine. It
is specifically the float/double family that breaks.
**What you expected:**
The predicate to be evaluated against the promoted type, or the domain to be
dropped for base files that predate the evolution, so the query returns rows
instead of failing. Reading the same column without a predicate on it already
works.
**Steps to reproduce:**
1. Write a Hudi table with a `float` column, so at least one base file
stores it as parquet `FLOAT`.
2. Evolve that column to `double` and sync the metastore, leaving the
existing base files unrewritten.
3. Query the table through the Trino connector with a predicate on that
column.
Reported behaviour, measured during the review of #19456:
| configuration | result |
|---|---|
| `hudi.parquet.use-column-names=true` | fails, naming the evolved column |
| `hudi.parquet.use-column-names=false` | fails, naming the evolved column |
| before #19456, `hudi.parquet.use-column-names=false` | fails, naming an
unrelated column that happened to sit at the stale metastore ordinal |
This is not a regression from #19456, which rebuilt pushed-down predicate
handles on physical file ordinals. Positional mode failed before that change
too, only while naming the wrong column; #19456 strictly improves the
diagnostics here without addressing the underlying failure.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT)
**Query engine:** Trino (hudi-trino connector, Trino 481)
**Relevant configs:** `hudi.parquet.use-column-names` (either value)
### Logs and Stack Trace
```
io.trino.spi.TrinoException: Malformed Parquet file. Corrupted statistics
for column "[c7] optional float c7"
Caused by: java.lang.ClassCastException: class java.lang.Float cannot be
cast to class java.lang.Double
at
io.trino.parquet.predicate.TupleDomainParquetPredicate.getDomain(TupleDomainParquetPredicate.java)
```
### Coverage gap
`hudi-trino` carries no schema-evolution test at all today, in either
column-resolution mode, which is why this goes unnoticed. A fixture writing a
base file under one type while registering the promoted type in the metastore
would pin both the float/double failure and the benign int/long promotion.
Follow-up from the review of #19456. Context: #18780.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
