CalvinKirs opened a new pull request, #69835:
URL: https://github.com/apache/doris/pull/69835
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
External scans can continue listing files and retain split queues after a
LIMIT query finishes, a query is cancelled, or its deadline expires. This
change makes split enumeration follow the query's lifetime and adds a
configurable first-split deadline with progress in the error message.
| Area | What changes | How it works |
|---|---|---|
| Query lifetime | Stop split work on LIMIT completion, cancellation,
timeout, and failed/replaced planning attempts. | Query- and attempt-owned scan
scopes propagate cancellation; statement resources remain owned until the
associated tasks actually exit. |
| Producer scheduling | A stopped producer cannot be restarted by a late
wakeup or left in the executor queue. | Explicit closing/closed states prevent
resubmission; cancellation removes queued tasks and handles submission races.
Idle/queued sources close on a separate cleanup pool; a running worker closes
its own source before completing its termination future. |
| Hive/HMS and file listing | Stop requesting more partition metadata,
files, and listing pages after cancellation. | Check the scan context at
metadata RPC/retry, partition, file, and S3 pagination boundaries. Share
completed cache entries while isolating cancellable in-flight loads from
healthy callers. |
| Iceberg SDK planning | Stop owned manifest/data/delete planning jobs and
wait for their actual exit. | A source-owned executor view and cancellable
FileIO reuse the existing SDK pool. Queued jobs settle independently; SDK
bodies exit before iterators and statement resources close. |
| First split wait | Make the initial wait configurable and report
enumeration progress on timeout. | Add `first_split_wait_timeout_ms` (default
`30000`, minimum `1`, supports `SET_VAR`). The error includes processed/total
partitions, enumerated files, generated splits, and active producers. This is
separate from the existing subsequent split-fetch timeout. |
| Bounded work | Keep backpressure and allow an explicit producer cap per
scan. | Preserve split batch/queue bounds and schedule partition-batch
producers as capacity becomes available. Add `split_producer_parallelism`: `0`
retains pool-derived parallelism; whole-scan streams use one producer. |
Cancellation is cooperative. An already-running SDK operation that cannot be
interrupted may still need to return before its owner can finish cleanup.
### Release note
External scan split enumeration now stops when its query finishes or
terminates. Add the `first_split_wait_timeout_ms` and
`split_producer_parallelism` session variables. Connector/filesystem plugin API
versions become `21.0`/`2.0`; deploy plugins built with the matching FE. Native
streaming Iceberg planning uses request-local FileIO without the SDK content
cache; catalog FileIO and the Doris manifest cache are retained.
### Check List (For Author)
- Test:
- [x] Unit Test
- [x] Manual test
- [ ] Regression test (suites added; no claim of a complete regression
run on the latest master)
- Behavior changed:
- [ ] No.
- [x] Yes: query termination stops split scheduling/enumeration and
releases owned resources; first-split waits have an explicit deadline.
- Does this need documentation?
- [ ] No.
- [x] Yes: connector cancellation contract and acceptance workflow are
documented in `fe/fe-connector/README.md` and
`regression-test/script/hive_split_early_stop/README.md`; user-facing variable
reference follow-up is pending.
| Validation | Result |
|---|---|
| FE build and Java style | `build.sh --fe` and Maven Checkstyle passed. |
| Related FE tests | 495 tests across 49 classes passed through
`run-fe-ut.sh`. Covers producer state/submission races, source termination,
query/attempt ownership, cache isolation, and native Iceberg SDK data/delete
work. |
| HDFS query validation | All 488 scans succeeded: 240 measured queries, 240
warmups, four million-file SUM validations, and four separate profile queries.
Full scans were checked against native reader counts and expected SUM results. |
| HDFS LIMIT/cancel/timeout cleanup | 9/9 cases passed, three repetitions
per trigger. Maximum observed exit was 871ms, measured from LIMIT return, KILL
send, or the configured query deadline. Producers, queues, sources, listeners,
and callback registrations were released; a direct HDFS request observer saw no
new listing requests after 10s. Instrumentation was used for this separate
resource test, not performance timing. |
| Configurable first-split timeout | Manual waits of 1000ms and 2500ms
failed near their configured deadlines with progress in the error; invalid
nonpositive values are rejected. |
The PR remains a draft: synchronization with the current master is needed,
and the observed MinIO slowdowns still require attribution under controlled
host load. The HDFS results below do not establish S3 performance or a general
no-regression result. Build/test/benchmark results refer to `60351f1` on
baseline `d30058a`, before that synchronization.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
### HDFS performance measurements
| Setting | Value |
|---|---|
| Data | 10,000 actual HMS partitions × 100 physical Parquet files =
1,000,000 files; 481 bytes and one row (`id = 1`) per file. |
| Storage | Dedicated HDFS 3.2.1 NameNode/DataNode, with the same data and
server settings throughout. |
| FE comparison | Before: `d30058a80337e56f0f4930d7a75c0df50aa39ffc`; after:
`60351f116ece1427a0a7e04a58869a52dc22a57e`. |
| Common BE | `4.1.3-rc02-7126cf65d96` for both versions. |
| Execution | One online cluster, one query and one producer at a time.
Fresh FE/BE runtimes for each round; order: before → after → after → before. |
| Repetitions | Fixed 30 warmup cycles, then 20 measured queries per
scenario per round: 40 measured samples per version/scenario. |
| Configuration | Native query/catalog cache and timeout defaults.
Single-producer setting on both sides;
`experimental_enable_local_shuffle_planner=false` and `be_exec_version=10` on
both for BE 4.1.3 compatibility. |
| Cold definition | Fresh FE catalog; HDFS/OS caches were not forcibly
flushed. Timing used no proxy or Java agent. |
In the following tables, `t` is the million-file HDFS table and `p` is its
integer partition key. The 10,000-file aggregation uses `WHERE p < 100`; it is
not a separate 100-partition table.
These are observed shared-host measurements, not a guaranteed speedup.
Baseline median latency for the 10,000-file aggregation fell 13.5% between its
two rounds, and CPU affinity was not exclusive. All successful samples are
retained; both medians and means are shown. Negative changes mean lower elapsed
time.
| Scenario / SQL | Samples per version | Before median (ms) | After median
(ms) | Median change | Before mean (ms) | After mean (ms) | Mean change |
|---|---:|---:|---:|---:|---:|---:|---:|
| Fresh-catalog `SELECT * FROM t LIMIT 10` | 40 | 863.56 | 774.17 | -10.35%
| 905.38 | 813.43 | -10.16% |
| Warm-catalog `SELECT * FROM t LIMIT 10` | 40 | 114.20 | 109.90 | -3.77% |
124.91 | 115.60 | -7.45% |
| `SELECT COUNT(*), SUM(id) FROM t WHERE p < 100` (10,000 files) | 40 |
1167.90 | 1052.15 | -9.91% | 1404.32 | 1145.69 | -18.42% |
Per-round results expose drift. Million-file SUMs are separate
correctness/workload validations, one per runtime, rather than repeated
steady-state performance samples.
| Round | Version | Fresh-catalog LIMIT median (ms) | Warm-catalog LIMIT
median (ms) | 10,000-file aggregation median (ms) | Million-file SUM validation
(s; result = 1,000,000) |
|---|---|---:|---:|---:|---:|
| 1 | Before | 989.08 | 116.32 | 1273.19 | 282.615 |
| 2 | After | 753.10 | 99.75 | 1052.15 | 232.395 |
| 3 | After | 785.57 | 115.46 | 1061.70 | 232.503 |
| 4 | Before | 781.45 | 102.26 | 1101.28 | 238.465 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
