wangzhigang1999 opened a new pull request, #10317:
URL: https://github.com/apache/paimon/pull/10317
### Purpose
On PyArrow 19.0.1, filtering a `DECIMAL(15, 2)` column with `< Decimal('2')`
can return no rows where `1.00` and `1.01` should match. Arrow evaluates the
literal at a different scale from the column.
Pass the physical Arrow schema into predicate conversion and build exact
DECIMAL comparisons at the column's scale. This covers file pushdown and
post-read batch filtering, including range, `IN`, and `BETWEEN` predicates. The
bound is never rounded into an equality match.
Fixes #10316. The underlying Arrow behavior is tracked in
[apache/arrow#41011](https://github.com/apache/arrow/issues/41011).
### Tests
- Targeted predicate and read tests on PyArrow 19.0.1 and 22.0.0: 369 passed
and 32 subtests passed per version.
- Ruff format/check and Black `-S --check` passed for the new files and
changed code; Pyright reported no errors in the new module.
- A three-round local SF10 read benchmark on PyArrow 22.0.0 passed 48/48
correctness checks. Measured DECIMAL read times showed no material regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
