airborne12 opened a new pull request, #70389: URL: https://github.com/apache/doris/pull/70389
### Background `HeapSorter` keeps the best `limit + offset` rows of a TopN as a heap of sorted blocks, worst row first. With a TopN runtime filter, and until the heap is full, every appended block was sorted and all its columns permuted into that order. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Performance Problem Summary: Only the best `limit + offset` rows of a block, the last ones of that order, can stay in the heap, yet every column of the whole block was permuted: 20% of BE CPU in a search-box TopN over 100M logs. The block is still sorted the same way, so tied rows keep their order, but only the rows the heap can keep are copied. ### Details - **Why it was slow:** under LIMIT 200 a 4096-row block permuted every column, long strings included, for 4096 rows; sorting the key itself is cheap. - **Optimization:** `sort_block_tail` computes the permutation `sort_block` computes and permutes only its last rows; `Sorter::partial_sort` uses it for the reversed sort, whose only caller is `HeapSorter`. - **Results:** branch-4.2 port, old and new code alternating in one BE: CPU per query 18-33% lower on search-box TopN queries; other TopN queries up to 40% lower, none slower; results identical. - **Applicability and cost:** TopN queries whose blocks hold more rows than `limit + offset`. Rows kept and their order are unchanged. No change to writing or storage. ### Testing New `HeapSorterTest` cases (two fail before the change; one also fails with a partial sort, which picks tied rows differently), sort and TopN unit tests, TopN, sort, search and inverted-index regression suites. ### Release note TopN queries (ORDER BY ... LIMIT) use less CPU when their scan returns many rows. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: local unit and regression tests; CI not run ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: code, tests, this description - Human review: not yet reviewed 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
