airborne12 opened a new pull request, #70007:
URL: https://github.com/apache/doris/pull/70007
### What problem does this PR solve?
Issue Number: None
Related PR: #66052, #67657
Problem Summary: Stream load into a SNII table stops scaling once flush
threads compete. On a log table with ten inverted indexes and mostly unique
terms, SNII loaded at 314 MB/s against 854 MB/s for V2, with 3.0 times the CPU
and 9.8 times the kernel CPU. The adaptive flush pool is on by default and
allows up to 4 threads per core, so this is the normal case on a busy BE. With
the pool fixed at 6 threads SNII was 23% slower than V2, and at 30 threads 60%
slower.
Two process-wide hot spots cause it, and both are reached per term:
1. `SpimiTermBuffer::report_arena_delta()` calls
`GlobalMemoryLimiter::report()` whenever the resident total changes, which is
once per new term. `report()` takes a process-wide mutex, looks the writer up
and reads the memory signals. The limiter keeps only the arena size, which
grows one 32 KiB block at a time, so nearly every call repeats the same number.
At 30 threads the flush threads blocked about 476k times per second against 21k
for V2, and every futex wait sampled was on one address. That was 13.6 cores of
kernel CPU against 3.5.
2. The Reservations that #67657 added to the term encoders change several
times per term, and each change updates two process-wide atomics through
`snii_build_consume_release()`. That lambda took 54% of the user CPU of the
flush threads.
Both scale with new terms and with threads, which is why duplicate data and
a small pool look mild.
- The limiter is now told only when the arena size changes. Nothing it
stores is lost, and pressure is still re-decided whenever the arena grows,
which is all a buffer of repeated terms ever got.
- A reporter now holds its changes back from the two process-wide counters
until they add up to 1 MiB, and publishes the remainder when it is destroyed.
One term swings tens of KiB plus a ZSTD context, so per-term reservations no
longer reach the shared cache lines.
**What this costs.** The reporter itself stays exact, so per-writer budgets
and spill thresholds do not change. The process-wide counters can lag each live
reporter by less than 1 MiB:
- `SniiIndexBuild` and `snii_index_build_mem_consumption` no longer show a
writer that stays below 1 MiB. They still return to exactly 0.
- The forced-spill decision reads the sum, so it can start late by that lag
added up: at most 1 GiB with 100 flush threads on a 10-index table, against a
share of at least 4 * `inverted_index_ram_buffer_size` (2 GiB by default). A
writer below 1 MiB could not be a victim anyway, since forced spills only pick
arenas above `snii_forced_spill_min_arena_bytes` (64 MiB by default), and the
process-level backstops read real memory.
The index bytes do not change.
| Throughput, MB/s, default adaptive flush pool | V2 | SNII | SNII vs V2 |
SNII CPU / V2 CPU |
|---|---|---|---|---|
| Log data, before | 854 | 314 | -62.9% | 3.01 |
| Log data, after | 852 | **770** | -7.7% | 1.11 |
| Short logs, before | 997 | 389 | -61.0% | 2.84 |
| Short logs, after | 1003 | **865** | -13.7% | 1.20 |
Both binaries were built from master and alternated in one window, 4 rounds
each with the order flipped, and the table shows medians. One BE pinned to 32
cores, local disk, stream load of JSON lines with 16 concurrent 100 MB batches.
Log data is expanded from a real sample of service logs, where 1% of the rows
carry base64 blobs and make up 68% of the bytes. Short logs leave those rows
out. V2 is unchanged, as it should be, and kernel CPU falls from 516 s to 43 s
on the log data.
An earlier prototype with the same two changes also held up with other
shapes and pools: fixed 30 threads -59.6% to -8.7%, fixed 6 threads -23.1% to
-14.8%, 1 GB batches -43.8% to -13.7%, and data with few unique terms -20.2% to
-11.7%.
Forced spills were checked on a real load: with
`snii_index_build_max_memory_limit_percent = 1`,
`snii_forced_spill_min_arena_bytes = 1 MiB`, and 16 concurrent 1 GB batches,
the spill directory held up to 224 files at once and the limiter logged its
shortfall warning. It stayed empty with the default share. Eight MATCH, phrase
and equality queries returned the same counts as V2 on the 11M rows, and the
tracker went back to 0.
### Release note
Improve load throughput of the SNII inverted index format on columns with
many unique terms.
### Check List (For Author)
- Test: Unit Test (`NewTermsThatKeepTheArenaStayOffTheLimiter`: new terms
inside one arena block never reach the limiter while the resident total moves;
`PressureIsDecidedWhenTheArenaNextGrows`: pressure that arrives inside a block
still ends in a forced spill; `PerTermSwingsStayOffTheSharedCounters`,
`HeldBackBytesArePublishedTogetherAtTheStep`,
`DestroyingAReporterPublishesTheLastPartialStep`,
`ConcurrentChangesOnOneReporterLoseNothing`: the coalescing, its remainder and
concurrent use; existing tracker tests that asserted exact visibility of a few
KB now use changes of at least one step; five mutants of the change each fail
tests; 2246 SNII, BKD and inverted index tests pass under ASAN).
Manual test: the A/B and the forced-spill load above.
- Behavior changed: Yes. The `SniiIndexBuild` tracker and
`snii_index_build_mem_consumption` lag each writer by less than 1 MiB, and the
forced-spill decision can start late by that lag added up.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
