plusplusjiajia opened a new pull request, #10814:
URL: https://github.com/apache/paimon/pull/10814
### Purpose
As in Java `LeafBinaryFunction` and `LeafTernaryFunction`, a comparison with
a null literal matches no row and no file. The Python predicates compared
values, stats and Arrow columns with `None` instead, so a filter such as
`not_equal('v', None)` built with `PredicateBuilder` gave wrong results or
failed:
- `notEqual` returned every non-null row when a primary-key read filters the
merged rows. Java returns no rows.
- `equal`, `lessThan`, `lessOrEqual`, `greaterThan`, `greaterOrEqual`,
`between`, `notBetween`, `startsWith`, `endsWith` and `contains` raised
`TypeError`, in the row filter or when pruning files by value stats, for
example in an append table with `metadata.stats-mode: full`.
- `like` turned the null pattern into the string `'None'` and matched it.
- `notBetween` with a null bound returned rows where the Arrow expression is
the final filter, as in an append table read: Arrow evaluates `x < a OR x >
NULL` as true for `x < a`.
A null literal now fails the row test and the stats test, as in Java, and
converts to an Arrow expression that matches no row. `in` and `notIn` already
follow Java since #9132 and are unchanged.
### Tests
- `PredicateTest#test_null_literal_matches_nothing`: all 12 comparisons with
a null literal return no rows from four tables: an append table with full
stats, an append table without stats so that the Arrow filter decides, a
primary-key table, and a primary-key table with two commits to one bucket so
that rows are filtered after the merge.
- `PredicateTest#test_null_literal_by_value_stats_and_arrow`: the row test,
the stats test and the Arrow expression of each comparison reject a null
literal, and `like` does not match the string `'None'`.
- On master, 40 subtests of these two tests fail.
- The predicate, pushdown, filter, stats, Arrow, decimal and timestamp tests
pass locally: 1029 tests. flake8 is clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
