guptashivam04 opened a new pull request, #69810:
URL: https://github.com/apache/doris/pull/69810
### What problem does this PR solve?
Issue Number: None
Related PR: #69352 (BM25 statistics shared across the parallel scanners of a
tablet)
Problem Summary:
Since #69352, the scanners a parallel scan splits a tablet into share one
`TabletCollectionStatistics`. The first scanner to prepare collects the
tablet's BM25 statistics under a mutex, and every other scanner of the tablet
blocks a scanner thread on that mutex until the collection finishes. A query
that splits a tablet into k scanners therefore holds k − 1 scanner threads idle
for the whole collection, and the scanner pool (`max(48, 2 × cores)` threads by
default) is shared with every other query on the BE.
In the profile of a cold `score()` query over one tablet with many hidden
rows, split into 8 scanners, all 8 scanners spend 80–160 ms in prepare: one
collects, seven wait. Those seven hold scanner threads that other queries on
the BE could use: with a scanner pool of 8 threads, the p95 latency of another
query on the same BE is 2–3 times higher while the statistics are collected.
Fix: a scanner never waits for the statistics on a scanner thread.
- Before `prepare()`, the scheduler asks the scanner whether it can prepare
now (`Scanner::ready_to_prepare(wake_up)`, true by default). An `OlapScanner`
that shares tablet statistics takes a turn from
`TabletCollectionStatistics::take_turn()`:
- `COLLECTED`: prepare and copy the statistics.
- `COLLECT`: this scanner collects them.
- `WAIT`: another scanner is collecting; `wake_up` is kept and the
scanner is not prepared yet.
- A waiting scan task is parked by the `ScannerContext`
(`push_waiting_scan_task`): it gives back its thread and its in-flight slot, so
the context schedules other pending scanners instead. When the statistics are
published, each parked task is rescheduled (`wake_up_waiting_scan_task`). A
wake-up that arrives before the task is parked is remembered on the task, which
is then rescheduled at once.
- The collection runs outside the mutex. A scanner that took the turn but
stops before collecting (an error in prepare, pruning by a runtime filter,
close) gives the turn back, and a waiting scanner collects instead.
- Both scan schedulers go through `ScannerScheduler::_scanner_scan`, so this
covers the thread pool and the task executor
(`enable_task_executor_in_internal_table`).
Scores do not change: every scanner of a tablet still scores with the
statistics of the whole tablet.
Why not stop splitting tablets for `score()` queries: on the same tablet (35
M rows), the default split into 8 scanners runs a common-term `score()` query
in 3.2 s and a selective one in 48 ms, against 9.5 s and 78 ms with one scanner
per tablet.
### Release note
Fix BM25 `score()` queries on a tablet split across several scanners holding
scanner threads idle while the tablet's statistics are collected.
### Check List (For Author)
- Test
- [x] Regression test: all 27 suites that use `score()`
(`inverted_index_p0`, `search`, `ann_index_p0`, `variant_p0`,
`nereids_rules_p0`, `fault_injection_p0`, including
`test_bm25_score_parallel_scan_statistics` of #69352) pass on a 3-BE cluster,
with the thread pool and with `enable_task_executor_in_internal_table = true`.
- [x] Unit Test:
-
`TabletCollectionStatisticsTest.OneScannerCollectsAndTheOthersAreWokenUp`,
`GivingUpTheTurnWakesTheWaitersToCollect`, `CopyWithoutTurnCollectsOnce`.
-
`ScannerContextTest.scanner_waiting_for_statistics_gives_thread_back`: a pool
of 2 threads, where the statistics are published only after a scanner of
another tablet has produced a block. A waiting scanner blocks on its thread
until the statistics are published, as every scanner did before this change.
When the scheduler runs the waiters without parking them, they hold both
threads and the test fails after 10 s.
- `ScannerContextTest.wake_up_before_parking_reschedules_task`.
- With the existing scanner context, OLAP scanner, late-arrival
runtime filter and collection statistics tests: 187 tests pass in an ASAN build
on macOS arm64; the new tests also pass 50 repeated runs.
- [x] Manual test (macOS arm64, 8 cores; the tables on one BE), base =
master without this change. The two builds alternated, two runs of each per
setting.
- Tables: `bench.del`, one 4.8 GB tablet with 16 M rows in 16
rowsets, 12 M of them upserted and 2 M deleted with compaction off, so 18 M of
the 32 M rows in the index are hidden. It is queried cold: before each query,
its files are dropped from the OS page cache, and the query runs with the page,
segment, inverted index searcher, inverted index query and condition caches
off. `bench.big`, one 5.8 GB tablet with 35 M rows, is queried warm.
- Queries: selective term `SEARCH('content:w18000')`, common term
`SEARCH('content:jobs OR content:w17')`, `ORDER BY score() DESC LIMIT 100`.
Meanwhile a warm probe query (`count(*)` with `LIKE` on another table on the
same BE) runs every 50 ms.
- Scores: the top 100 `(id, score())` for both terms on both tables
are identical between the two builds, and identical to the same query with
`enable_parallel_scan = false` (one scanner per tablet).
- `PerScannerPrepareTime` of the 8 scanners of a cold query: base,
all 8 at 79–89 ms (selective) or 130–162 ms (common); this change, the
collecting scanner at 79–86 ms or 130–175 ms, and the other 7 under 0.4 ms
(once 1.6 ms).
- Latency with a scanner pool of 8 threads, median ms of the
`score()` query / p95 ms of the probe query, two runs each:
| Scenario | Base, thread pool | This change, thread pool | Base,
task executor | This change, task executor |
|---|---|---|---|---|
| cold, 1 client, common | 982, 864 / 186, 165 | 929, 856 / 66, 51 |
870, 850 / 177, 184 | 872, 864 / 53, 52 |
| cold, 1 client, selective | 156, 152 / 103, 80 | 158, 154 / 34, 27
| 154, 153 / 90, 77 | 154, 153 / 27, 27 |
| cold, 4 clients, common | 3127, 2813 / 139, 108 | 3032, 2749 /
139, 94 | 2899, 2910 / 146, 176 | 2916, 2710 / 97, 89 |
| cold, 8 clients, selective | 435, 412 / 169, 135 | 520, 409 / 141,
91 | 542, 511 / 390, 253 | 443, 412 / 104, 85 |
| warm, 1 client, common | 1982, 1862 / 150, 150 | 2195, 1908 / 131,
98 | 1942, 1811 / 120, 131 | 2014, 1885 / 92, 88 |
| warm, 8 clients, selective | 51, 52 / 53, 35 | 51, 51 / 63, 55 |
62, 66 / 48, 60 | 53, 51 / 56, 43 |
The `score()` queries take the same time on both builds. While the
base waits for the statistics on up to 7 of the 8 pool threads, the probe
query's p95 is 2–3 times higher; its median does not change. With the default
pool (48 threads here), the two builds are within noise in every scenario.
- Memory: the peak memory of each query (`PeakMemoryBytes` in the
audit log) is the same for the two builds in every scenario, for example a
median of 63 or 94 MB for the cold selective query, on both builds, and 97–105
MB for the cold common one. After every run, no scanner context, scanner or
scan task is left on the BE (`doris_be_scanner_ctx_cnt`,
`doris_be_scanner_cnt`, `doris_be_scanner_task_cnt` are 0).
- Cancellation: for 90 s per run, 8 clients run cold `score()`
queries that finish, are cancelled after 20–400 ms, time out after 1 s, or stop
at `LIMIT 1` while other scanners still wait. No unexpected error, no crash,
nothing left running, and the BE answers queries afterwards.
- `StatisticsCollectTime` now counts only the collection. Before, it
also summed the time the other scanners of the tablet waited for it (for
example 1.2 s against 154 ms for one common-term query).
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
