englefly opened a new pull request, #70795: URL: https://github.com/apache/doris/pull/70795
### Background Scan operators collect available runtime filters during initialization and check for additional filters when a scanner runs. A scan local state shares its residual conjuncts across scanners. Each scanner clones those expressions for execution and refreshes its clone when its applied runtime filter count changes. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Performance Problem Summary: While a runtime filter is pending, each scanner check can repeat the same cost sort under the shared conjunct lock. For 32 conjuncts and 1,000 checks with no new expressions, an isolated count dropped from 220,000 cost calls to zero. Sort the initial residual expressions before the first scanner clone and sort again when the helper appends new expressions. ### Details - **Why it was slow:** The late-arrival path stable-sorted the full conjunct list before checking for new expressions. Comparing two conjuncts walks their expression trees even when the order is already correct. - **Optimization:** Stable-sort the residual list after pushdown and before scanner initialization. On late arrival, snapshot the newly appended expressions for pruning and re-sort the shared list under the existing lock. A disabled filter still advances the applied count. - **Results:** One isolated run per variant on an Intel Core i7-9800X with Clang 20.1.7 and ASAN. The harness ran the respective `update_late_arrival_runtime_filter` implementation from master at `5bc6ed62f9b` and the candidate with a consumer stub configured to append nothing. Both started with the same pre-sorted conjuncts. Across 1,000 checks, cost calls changed from 6,000 to 0 for 3 conjuncts and from 220,000 to 0 for 32. A single conjunct and the disabled ordering control each made zero cost calls on both versions. These counts measure the late-check path in isolation. - **Applicability and cost:** The existing session variable `enable_adjust_conjunct_order_by_cost` defaults to true. Scans with runtime filter descriptors can pay for one initial sort when at least two residual conjuncts remain. Savings depend on the number of checks that find no new expressions. ### Testing - `build-support/check-format.sh` with clang-format 16 and `build-support/check-build-hygiene.sh` passed locally. - The ASAN BE unit build compiled `scan_operator.cpp`. The test translation unit passed a syntax check with the ASAN compilation flags. - Added scan unit cases for initial and late ordering, stable ties, disabled filters, peer scanner refresh, and the rows returned after applying a late IN filter. The local ASAN unit test run is pending. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: ASAN scan tests and clang-tidy pending. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Code, unit tests, and PR description. - Human review: not stated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
