mrhhsg opened a new pull request, #66088: URL: https://github.com/apache/doris/pull/66088
### What problem does this PR solve? Issue Number: None Related PR: None Source commit: https://github.com/apache/doris/commit/7261a1e48beb0940bc5c9a016f02bda65fbfe895 Problem Summary: When the scan output queue is empty while unfinished scanner work is still scheduled or pending, the downstream scan operator can be starved because scheduling only replenishes toward the minimum scan concurrency. This backports the source commit from `opt_perf_4.1` to `branch-4.1` and: - tracks scan starvation and temporarily schedules toward the configured maximum concurrency; - clears the starvation state when data arrives or scanning stops; - caps the minimum concurrency by the maximum concurrency; - aligns the default minimum active scan threads with the default local scan thread count; - raises the default `max_scanners_concurrency` from 4 to 8; - adds focused `ScannerContext` coverage for starvation-aware scheduling. ### Release note Improve scan throughput when downstream scan processing is starved for data. ### Validation - PASS: `./run-fe-ut.sh --run org.apache.doris.qe.SessionVariablesTest,org.apache.doris.qe.VariableMgrTest` (22 tests passed). - PASS: ASAN compilation of the affected BE production and test translation units: - `src/exec/CMakeFiles/Exec.dir/scan/scanner_context.cpp.o` - `src/exec/CMakeFiles/Exec.dir/scan/scanner_scheduler.cpp.o` - `test/CMakeFiles/doris_be_test.dir/exec/scan/scanner_context_test.cpp.o` - PASS: `git diff --check origin/branch-4.1...HEAD`. - NOT RUN: `ScannerContextTest.*`; relinking the monolithic ASAN `doris_be_test` binary required a broad backend rebuild. The affected production and test translation units compiled successfully, and CI will provide full validation. - NOT RUN: regression tests; this change does not alter SQL results or regression output. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test (FE focused tests; see Validation for the BE test execution gap) - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. Scanner scheduling can temporarily grow from the configured minimum to the configured maximum while the downstream operator is starved, and the default scanner concurrency increases from 4 to 8. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
