LuciferYang opened a new issue, #10019: URL: https://github.com/apache/paimon/issues/10019
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master ### Compute Engine Spark (TopN / ORDER BY ... LIMIT pushdown) ### Minimal reproduce step On an append-only table, run a query that pushes a single-key TopN into Paimon: `SELECT * FROM t ORDER BY col DESC LIMIT 1` (Spark's DESC defaults to NULLS LAST), or an explicit `ORDER BY col ASC NULLS LAST LIMIT 1`. Arrange for at least two splits where one split's `col` is entirely NULL (its files' min/max are null while the null count is tracked, so `minmaxAvailable` still passes) and another split holds the real top value. `TopNDataSplitEvaluator` gates: append-only table, one sort key, a min/max-comparable type, limit ≤ 100. ### What doesn't meet your expectations? The query returns a NULL row instead of the true top value. `TopNDataSplitEvaluator.pickTopNSplits` orders the splits by the sort column's aggregate min/max and keeps the first `limit`. In the NULLS LAST branches the comparison falls to `ascCompare` / `descCompare`, which treat a null min/max as the smallest value. A split whose sort column is entirely NULL therefore sorts to the FRONT, takes one of the kept `limit` slots, and displaces the split that holds the real top value. With `LIMIT 1` and two or more splits, the reader keeps only the all-null split and emits a NULL row. A related defect in the same comparator: `ascCompare` / `descCompare` / `nullsFirstCompare` / `nullsLastCompare` return `-1` when the left operand is null without checking the right, so `compare(x, y)` and `compare(y, x)` can both return `-1` for two nulls. That violates the `Comparator` contract and can surface as `IllegalArgumentException: Comparison method violates its general contract` during the sort. Expected: under NULLS LAST a split whose sort column is provably all null (null count equals row count) is the worst candidate and sorts last, so it never displaces a split with real values. A split whose bound is merely unknown (null min/max but not provably all-null, e.g. `stats.mode=counts`) must still be read, so it sorts first conservatively. ### Anything else? Fix direction: add an `allNull` flag (null count equals row count) to the split wrapper and, in the NULLS LAST branches, order all-null splits last before falling back to the min/max comparison; and return `0` when both comparator operands are null so the comparator obeys its contract. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
