CalvinKirs opened a new pull request, #70078:
URL: https://github.com/apache/doris/pull/70078
### What problem does this PR solve?
Issue Number: None
Related PR: #69133
Problem Summary:
When the Hive file-list cache misses, directory listing still collects every
file into an `ArrayList` before split production starts. Returning a
`FileIterator`, then calling `.iterator()` on the completed List, does not
remove this barrier. A single partition with many files therefore delays the
first split even though different partitions can be listed concurrently.
This PR:
- Advances a scan-owned, closeable filesystem cursor directly, producing
splits while the rest of the directory is still being listed. Recursive
traversal keeps the active directory stack and one file of lookahead; existing
split queues provide backpressure.
- Collects an optional complete-list cache candidate within estimated
per-directory and shared staging limits. Exceeding either limit releases the
candidate permanently for that cursor while enumeration continues.
Completed-list ownership transfers without copying the backing array.
- Publishes only after observed successful EOF, successful iterator closure
and a cancellation check. Early close, LIMIT, cancellation and listing/closure
failure discard incomplete candidates. Pre-listing publication fences prevent a
late completion from undoing metadata invalidation.
- Declines optional publication when an already-open load's cache owner
retires, including the registry-retirement/acquisition and staging/commit
races. A successful EOF no longer fails solely because ALTER CATALOG retired
its cache. Actual listing, closure and cancellation errors still propagate.
- Retains complete byte totals in a separate governed scalar cache when the
full-list candidate is abandoned or disabled. Ordinary file-based statistics
remain free of cold directory I/O; later row-count reloads can recover a
positive estimate without retaining every file object. Existing cached UNKNOWN
values follow the normal row-count refresh lifecycle.
- Fails a streaming query if listing fails after ranges have escaped. Inline
planning can roll back the failed partition's ranges under the existing
local-directory failure policy.
The million-file iterator test produces its first split after consuming one
file. This verifies incremental production; there is no same-condition old/new
latency benchmark or claimed end-to-end speedup ratio.
### Release note
Cold Hive scans can produce splits before directory listing completes. New
catalog properties bound estimated optional file-list staging:
- `hive.file_listing_cache_entry_max_bytes`: 8 MiB by default.
- `hive.file_listing_cache_pending_max_bytes`: 64 MiB per catalog cache
instance by default.
Zero disables list-candidate collection. Completed entries remain subject to
metadata-cache admission. These staging limits do not cover filesystem SDK
buffers, split queues or the entire FE heap.
Complete directory byte totals are retained independently under standard
`meta.cache.hive.file_summary.*` controls, with a one-day TTL and 10,000
locations by default, participating in catalog/global weight governance.
Initially cold file-based estimates return UNKNOWN; successful complete scans
can supply later estimates even when their full lists exceed the staging limit.
HMS statistics remain preferred.
### Check List (For Author)
- Test
- [x] Unit Test
- 572 Hive connector tests and 187 cache framework tests passed,
with zero failures/errors/skips, through `run-fe-ut.sh` with the
affected-module reactor selected via its `CUSTOM_MVN` hook.
- Added deterministic owner/registry retirement, staged-publication,
EOF, cancellation, failure, memory-budget, summary invalidation/governance and
public-statistics recovery tests.
- Negative controls reproduced the previous closed-cache exception
and expected-8000/actual-minus-one public estimate; repaired tests pass.
- `build.sh --fe` and Checkstyle passed; UI and test compilation
were skipped for the build. Scoped `git diff --check` passed.
- [x] Regression test
- Original implementation: `run-regression-test.sh` generated and
then compared all 84 output groups successfully using preloaded Parquet
fixtures and a release 4.1.3 BE.
- This validates scans, not INSERT/CTAS execution. Cluster
regression was not rerun after the follow-up repairs; affected unit suites were
rerun.
- [x] Manual test
- Original implementation: read-only native S3 check on an existing
10,000-file directory returned LIMIT 10 after four generated splits. The
following full scan returned 1,000,000 rows / sum(id)=49,500,000 and recorded
one complete listing publication.
- This check was not rerun after the follow-up repairs and is not a
baseline performance comparison.
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Cold Hive listing is incremental with bounded optional
collection; post-emission listing failure fails the query; complete scalar
statistics survive abandoned lists; retired-owner optional fills decline
retention.
- Does this need documentation?
- [ ] No.
- [x] Yes. Updated `fe/fe-connector/docs/scan-task-scheduling.md` in
this PR.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
