Gabriel39 opened a new pull request, #65912:
URL: https://github.com/apache/doris/pull/65912
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
A query on a large Paimon `$files` system table used this shape:
```sql
SELECT *
FROM catalog.db.`table$files`
WHERE creation_time >= NOW() - INTERVAL 1 DAY
ORDER BY creation_time DESC, record_count
LIMIT 100;
```
The captured baseline took 26m14s and scanned 12,140,675 metadata rows while
returning no rows. Most cumulative scanner time was spent in the Java scan
path.
Doris converted the SQL predicate to a regular Paimon predicate, but Paimon
1.3.1's `FilesRead.withFilter` does not filter `creation_time`. Consequently,
historical file entries were materialized as `$files` rows and crossed JNI
before
Doris evaluated the predicate.
This change extracts a constant lower bound on `$files.creation_time` and
copies
the Paimon FilesTable with `scan.file-creation-time-millis` before split
planning.
Paimon then applies its file-creation-time manifest-entry filter while the
underlying FileStoreTable scans each FilesSplit. The original Doris
predicate is
retained as a residual correctness check.
The extraction is deliberately conservative:
- It supports `>=` and `>` with either operand order.
- It chooses the strongest lower bound across AND predicates.
- It does not infer a lower bound through OR.
- It applies only to the `$files` system table; system-table scan params and
time
travel remain rejected by the existing validation.
Expected effect: old manifest entries still need to be read because Paimon
manifest metadata has no creation-time min/max range, but old entries no
longer
become data splits, `$files` rows, or JNI traffic. The reduction depends on
the
fraction of files newer than the requested cutoff; a post-change wall-clock
number requires rerunning against the original production-scale table.
### Release note
Improve Paimon `$files` system table queries with constant `creation_time`
lower
bounds by pushing the cutoff into Paimon split planning.
### Check List (For Author)
- Test: Unit Test
- `./run-fe-ut.sh --run
org.apache.doris.datasource.paimon.source.PaimonScanNodeTest`
(15 tests, 0 failures, 0 errors)
- Behavior changed: Yes. Eligible `$files.creation_time` lower bounds now
prune
old file entries during Paimon scan planning; Doris still evaluates the
original
predicate.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
