HappenLee opened a new pull request, #65966:
URL: https://github.com/apache/doris/pull/65966
### What problem does this PR solve?
Problem Summary:
In TopN lazy materialization phase 2, the V2 internal rowid fetch path
(`RowIdStorageReader::read_batch_doris_format_row`) reads segment batches
strictly serially within a `multiget_data_v2` RPC. When the fetched rows span
many segments, or one segment dominates the request, phase 2 becomes a latency
bottleneck.
This PR adds concurrent execution to this path, controlled by a new session
variable `rowid_fetch_parallel_batch_rows`:
- `0` (default): serial execution, existing behavior fully unchanged
- `-1`: parallel execution, tasks are split by segment groups (i.e., the
existing Phase-1 grouping result)
- `>0`: parallel execution, tasks are consecutive row ranges of N rows, with
rows grouped by segment within each range. This gives better load balance when
a single segment dominates the request
Design points:
- The position-indexed scatter map
(`row_id_block_idx[original_request_index] -> (batch_idx, position_in_batch)`)
already decouples the final result order from how rows are partitioned into
tasks, so both splitting strategies reuse the existing scatter logic and
produce byte-identical results to the serial path
- Tasks run on a dedicated `InternalRowIdFetch` thread pool
(`config::internal_rowid_fetch_threads`, default 16) instead of the remote scan
scheduler to avoid pool deadlock; per-request concurrency is capped by
`config::internal_rowid_fetch_max_concurrent` (default 16) via
`counting_semaphore`
- Each task uses private `seg_map` / `iterator_map` / `OlapReaderStatistics`
and timing counters (ColumnIterator is not thread-safe), merged after all tasks
finish; new `merge()` methods are added for `OlapReaderStatistics` and
`FileCacheStatistics` covering the fields used by this path
- `fetch_row_store` requests always fall back to the serial path
- The old `PMultiGetRequest` (V1) path and the external table path are not
touched
### Release note
Support parallel rowid fetch in TopN lazy materialization phase 2,
controlled by session variable `rowid_fetch_parallel_batch_rows`.
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- Deploy a cluster with this PR, run TopN lazy materialization
queries with `set rowid_fetch_parallel_batch_rows = 0 / -1 / 100` respectively,
verify result correctness (identical to serial) and compare phase-2 latency
- Behavior changed:
- [x] No. (default value 0 keeps the existing serial path)
- Does this need documentation?
- [x] Yes. The new session variable `rowid_fetch_parallel_batch_rows`
should be documented.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
