sundapeng opened a new pull request, #8995: URL: https://github.com/apache/paimon/pull/8995
### Purpose A Format Table takes its schema from the metastore, but its files are written by someone else, so the two can disagree: - A column widened on the metastore side (int -> bigint) is still INT32 in the existing files. - An unsigned INT32 column is imported as BIGINT, because that is the only Paimon type that holds all its values. Spark and Trino also show it as bigint. In both cases the query fails at two independent places: - Pushdown: `ParquetFilters` picks the column type from the declared type only, so filtering a BIGINT column stored as INT32 throws `FilterPredicate column: ecpm's declared type (java.lang.Long) does not match the schema found in file metadata`. The narrowing direction breaks too: an INT column stored as INT64 reads fine but cannot be filtered. - Read: `ParquetVectorUpdaterFactory` has updaters for narrowing but none for widening, so `LongUpdater` reads 8 bytes from a 4-byte page and fails with `Failed to read 8 bytes`. Dropping the predicate does not make the table readable. Changes: - `ParquetVectorUpdaterFactory`: add `LongFromIntegerUpdater`, `LongFromUnsignedIntegerUpdater`, `DoubleFromFloatUpdater`, mirroring the existing narrowing updaters. - `ParquetFilters`: new `pushdownType()` types the predicate after the physical type in the file. `acceptableTypes()` lists what physical types each Paimon type accepts: first what Paimon itself writes, then the widenings the read path supports. When the two cannot be reconciled it throws `UnsupportedOperationException`, which `convert()` already swallows, so the predicate is dropped but rows never are. - `ParquetSchemaConverter`: extract `isUnsignedInt()` for both sides to share. - `FormatReadBuilder`: the exception now carries the file path and table name. A split spans many files, and there was no way to tell which one broke. Three boundaries where the predicate is dropped rather than adjusted: - A literal outside the INT32 range is never truncated. `< 3000000000` truncated to `< -1294967296` would prune whole row groups and silently lose rows. - Unsigned columns are never filtered. Their statistics are ordered unsigned, and signed bounds would prune the wrong row groups. The read still widens. - A double bound is never narrowed to float. That would round the bound and lose rows at the edge. Columns missing from a file are still pushed down as before: parquet-mr skips validation for them and evaluates them as null. ### Tests New `ParquetTypeWideningTest` with 11 cases: signed INT32, unsigned INT32 (including 4294967295), float to double, the narrowing direction, literals out of range, INT32/INT64 files mixed in one read, row-group pruning still effective. 10 of them failed before the fix, all pass now. Regression: full paimon-format suite (473 tests) and the format-table tests in paimon-core (122 tests), 0 failures. Also verified on a Trino 422 cluster with the plugin built from this branch: the failing query above reproduces before the fix, and `SELECT *`, range predicates over the widened column, aggregates and partition predicates all return correct results after. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
