mrhhsg opened a new pull request, #68500:
URL: https://github.com/apache/doris/pull/68500
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Large non-streaming window partitions currently retain all input blocks
until the partition closes. A sufficiently large partition can therefore
exhaust BE memory even when query spill is enabled.
This change adds a capability-gated spill path for eligible non-streaming
window functions:
- keeps genuinely streaming windows on the existing in-memory path;
- incrementally detects partition boundaries and stores original input
blocks in a partition store;
- reduces full-partition aggregates while collecting input and spills input
blocks on memory revocation, a configurable threshold, or forced-spill mode;
- records bounded peer-group metadata for `percent_rank` and `cume_dist`,
with independent metadata spill;
- replays spilled partitions one block at a time from the source operator;
- preserves the existing path when any function or frame is unsupported;
- adds `spill_analytic_sink_mem_limit_bytes` and window-spill profile
counters.
The initial supported set is:
- full-partition `sum`/`sum0`/`count`/`avg` and fixed-length-state
`min`/`max`;
- `ntile` with `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`;
- `percent_rank` and `cume_dist` with `RANGE BETWEEN UNBOUNDED PRECEDING AND
CURRENT ROW`.
### Release note
Eligible non-streaming window functions can spill partitions to local
storage when query spill is enabled.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- `ENABLE_UNITY_BUILD=ON ./run-be-ut.sh --run
'--filter=AnalyticSinkOperatorTest.Spill*'` — 8/8 passed under ASAN.
- `ENABLE_UNITY_BUILD=ON ./run-be-ut.sh --run
'--filter=AnalyticSinkOperatorTest.*:BoundaryPoseTest.*'` — 21/21 passed at the
full focused-test checkpoint.
- `./build.sh --be` — Release build passed.
- `./build.sh --fe` — passed, including Checkstyle.
- `build-support/clang-format.sh`, `build-support/check-format.sh`,
`build-support/check-build-hygiene.sh`, and `git diff --check` — passed.
- [ ] Manual test
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Validation notes:
- The BE build and BE UT commands required a transient, uncommitted
workaround for an unrelated unity-build duplicate default argument in
`async_result_writer.cpp`; the file was restored afterward and is not part of
this PR.
- Clang-tidy remains blocked by baseline diagnostics, including
`be/src/core/types.h:576` reporting an unmatched `NOLINTEND` and pre-existing
warnings in the Analytic operator.
- A local SQL regression cluster was not run in this iteration.
- Behavior changed:
- [ ] No.
- [x] Yes. Eligible non-streaming analytic partitions can spill;
streaming and unsupported windows retain the existing behavior.
- Does this need documentation?
- [x] No. The implementation design is included in this PR; no
user-facing documentation change is required for this initial version.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
