morningman opened a new pull request, #68960:
URL: https://github.com/apache/doris/pull/68960
### What problem does this PR solve?
Issue Number: None
Related PR: #67933, #68577, #67479, #68568
Problem Summary:
**In short.** This ports the tests for two branch-4.x external table fixes
whose behavior master already has: Paimon must not receive predicates that
contain a CAST (#67933), and Hive file listings must be cached per table
(#68577). It changes no production code. The other two fixes of the same batch
(#67479, #68568) are on master with their tests through #67687 and #68582.
**Background.**
- Paimon: Doris converts pushed conjuncts into connector expressions, and
that converter drops CASTs. #67933 found that `CAST(code AS INT) = 5` then
reached Paimon as `code = 5`, and Paimon pruned the data files holding `'05'`
or `' 5'`, so the query silently lost rows. branch-4.x rejected such predicates
inside its Paimon converter. On master the Paimon connector declares
`supportsCastPredicatePushdown() = false`, and
`PluginDrivenScanNode.buildRemainingFilter` keeps every conjunct with a CAST
anywhere in its tree in Doris: only the plain conjuncts go to the connector,
and the source LIMIT is turned off when a conjunct is held back.
- Hive: #68577 scoped the branch-4.2 file cache by table, so two tables
whose partitions point at the same directory no longer share a listing.
master's `HiveFileListingCache` keys entries by database, table, location and
partition values, and REFRESH TABLE drops only that table's entries.
**The problem, and what it cost.** None of this was pinned on master:
| Behavior | Tested on master before this PR |
|---|---|
| `buildRemainingFilter` keeps conjuncts with a CAST away from a connector
that does not accept CASTs, leaves them for Doris and turns the source LIMIT
off | no; tests pin only the value of Paimon's switch |
| A plain conjunct next to a CAST conjunct is still pushed | no |
| The Paimon converter pushes string equality, INT greater-than, IN and NOT
IN | INT equality only |
| Two Hive tables with the same partition directory and values keep separate
listings across a refresh | only separately (different locations, partitionless
keys); a cache key without the table name passes every existing
`HiveFileListingCacheTest` case |
| CAST predicates on a Paimon table, end to end, through the JNI and the
native reader | no |
**How this PR fixes it.**
- `PluginDrivenScanNodeCastPredicateTest` (new, fe-core): plans the nine
CAST shapes of the branch-4.x test (`=`, reversed `=`, `>`, `IN`, `NOT IN`, `IS
NULL`, `IS NOT NULL`, `OR`, decimal scale) and calls `buildRemainingFilter` on
a node whose connector does not accept CASTs: nothing is sent, the conjuncts
stay as they were and the source LIMIT is off. A plain conjunct next to a CAST
conjunct is still pushed alone, plain conjuncts are pushed whole, and the same
CAST conjunct is pushed once the connector accepts CASTs.
- `PaimonPredicateConverterTest`: string equality, INT greater-than, IN and
NOT IN are pushed, checked by evaluating the Paimon predicates on rows.
- `HiveFileListingCacheTest`: two tables with the same partition directory
and values list it separately, and refreshing one keeps the other's entry.
- `test_paimon_cast_predicate_pushdown` (new regression): writes one row per
data file so file pruning can happen, checks with EXPLAIN that the CAST
conjuncts show `predicatesFromPaimon: NONE` while plain conjuncts are pushed,
and reads the nine shapes, an AND with a plain conjunct and a LIMIT through the
JNI and the native reader.
**Results.** All the tests pass. Each new unit test fails when the logic it
guards is changed:
| Change to the code under test | Result |
|---|---|
| Remove the CAST check from `buildRemainingFilter` | 10 of the 14 cases
fail; the plain-conjunct case shows the filter sent to the connector as `((code
= 5) AND (id = 1))`, the stripped CAST that #67933 fixed |
| Drop IN support from the Paimon converter |
`plainStringRangeAndInListPredicatesArePushed` fails |
| Drop the table from the Hive cache key |
`refreshTableWithSharedPartitionLocationIsScopedPerTable` fails, the other
`HiveFileListingCacheTest` cases pass |
In the regression, the JNI and the native reader return the same rows, and
they match an internal Doris table holding the same data.
Original authors of the ported test cases: @Gabriel39 (#67933), @924060929
(#68577)
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `test_paimon_cast_predicate_pushdown` passes against the Paimon
catalog of the Iceberg docker environment; the `.out` was generated by the
script. `test_iceberg_historical_filter_planning` and
`iceberg_schema_change_ddl_with_branch` also pass there.
- [x] Unit Test
- `PluginDrivenScanNodeCastPredicateTest`,
`PaimonPredicateConverterTest` and `HiveFileListingCacheTest` pass, and fail
under the changes listed above.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
