jackylee-ch opened a new pull request, #10901:
URL: https://github.com/apache/paimon/pull/10901
### Purpose
After a column type change that does not keep the order of values, MIN/MAX
and TopN pushdown answer from stale statistics. Stats evolution casts an old
file's min and max one by one, and casts such as STRING to INT ('100' sorts
before '2') or INT to TINYINT (200 becomes -56) break the order, so the cast
values no longer bound the file.
Flink 1.20:
```sql
CREATE TABLE t (p INT, v STRING) PARTITIONED BY (p) WITH ('sink.parallelism'
= '1');
INSERT INTO t VALUES (1, '100'), (1, '2'); -- one data file
ALTER TABLE t MODIFY v INT;
INSERT INTO t VALUES (2, 50);
SELECT MIN(v), MAX(v) FROM t; -- 50, 50; a full scan gives 2,
100
```
Spark 3.5, 4.0 and 4.1 return the same, and `ORDER BY v DESC LIMIT 1`
returns 50 instead of 100.
Filter pruning already skips such stats
(`SimpleStatsEvolutions#filterUnsafeFilter`). The MIN/MAX and TopN gates now
use an old file's bounds only if its column type equals the current one or
casts to it implicitly (a numeric widening, which keeps the order), so INT to
BIGINT still pushes down.
### Tests
`TableScanTest`, `PushDownAggregatesTest` and `BatchFileStoreITCase` cover
core TopN, Spark MIN/MAX and Flink MIN/MAX; each new case fails without the fix.
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
