airborne12 opened a new pull request, #69796:
URL: https://github.com/apache/doris/pull/69796
### What problem does this PR solve?
Issue Number: None
Related PR: #69769, #69649 (backport #69795), #69770 (backport #69788)
Problem Summary: Backport of #69769 to branch-4.2, stacked on #69795 (the
backport of #69649). #69649 stores a .prx window as PFOR instead of zstd when
zstd saves less than `snii_prx_zstd_min_saving_percent` (30) of the PFOR frame,
because every read of a zstd window pays a decompression. What that costs and
buys depends on how much zstd saves on the data. Replaying the 1.6M windows of
10M AgentLogs rows, 30 spends 8% more bytes on the windows zstd compresses and
cuts their decode time by 42%; on the windows of TextBench service logs it
spends 5% and cuts 6%; on data whose windows zstd shrinks by just under 30% it
stores every window as PFOR. No single percentage fits all data.
Each index of a segment now spends a size budget instead: `PrxZstdBudget`,
owned by its `LogicalIndexWriter` and reached through `PrxWindowLimits`, so the
resident and the spilled encoder, and the ordinary and the compaction build,
all use it. A window whose smallest frame is zstd is stored in its faster frame
(PFOR, or raw when smaller) when
- its saving is below the share whose trades, over the windows the index has
seen, fit 8% of their smallest frames, which puts the budget on the windows
zstd shrinks least, and
- the bytes the index has actually traded, this window included, still fit
8% of the smallest frames seen. This makes 8% a bound for any order of windows:
fed the TextBench windows from the largest saving to the smallest, the share
alone spends 36%.
Until it has seen 64 windows the share is 30, the previous default. The
windows counted are those that get a zstd frame (a plain payload of at least
512 bytes), the only ones whose codec a share decides; singleton and short
windows are written as before and counted neither in the allowance nor in the
warmup. The choice uses the frames the writer builds anyway: nothing is
compressed twice or scanned ahead, and the same input gives the same bytes.
`snii_prx_zstd_min_saving_percent` now defaults to -1 (the budget); 0-100 keeps
a fixed share, 0 the smallest frame.
The budget is 8% rather than less: with it the AgentLogs phrase queries
below read nearly the same frames as with 30, while in the replay a 5% budget
cuts the decode time by 31% instead of 39%.
Same binary, one stream load per table, size against the smallest frames (0):
| data | index, 30 | index, budget | table, 30 | table, budget |
|---|---|---|---|---|
| AgentLogs 99M rows | +4.13% | +3.65% | +1.92% | +1.69% |
| AgentLogs 10M rows | +4.59% | +3.37% | +2.13% | +1.57% |
| TextBench 12M rows | +0.89% | +0.79% | +0.51% | +0.45% |
- The phrase queries read the same frames with 30 and with the budget: at
99M rows all PFOR for DS1, DS6, DS7, A6 and A9 (for example 107,307 frames for
DS7), and about 11,170 zstd frames of about 259,000 for A10 with both; on
TextBench most windows stay zstd either way (582 against 583 zstd frames of 599
for 'connection reset').
- Hot queries in one process, budget data against 30 data with the same read
path (paired cycles: 6 for the 25 AgentLogs queries at 10M rows, 10 for the 15
TextBench queries): 0.96-1.06 and 0.97-1.05; every query returns the same
result.
- Against the data written before #69649 (smallest frames), the 11 AgentLogs
phrase queries at 99M rows take 0.83x in geometric mean with the budget data
(DS7 0.63x, A6 0.67x, A9 0.70x, DS4 0.74x, A10 0.77x), as with 30 (0.83x). This
run also had the read-path change of #69770, which does not touch the PFOR
windows these queries read.
- Data bytes are identical, and the index written with 0 and with 30 has
exactly the size the previous binary writes.
- Write cost: the choice adds about 48 ns per window, 79 ms over the 1.6M
AgentLogs windows.
### Branch adaptations
None. `git cherry-pick 0571eb6a58` applied cleanly on top of the backport of
#69649, which it needs: it changes the setting #69649 adds, and the automatic
pick to branch-4.2 failed because branch-4.2 does not have #69649. This PR is
stacked on that backport, #69795, so it shows both commits until #69795 merges;
merge #69795 first.
### Validation
- Built and run with both backports (this one and the stacked #69769
backport) applied to branch-4.2 at 711db43b00, the commit before #69625: its
lance-c update is not in the local third-party build, and the 16 commits after
it touch no SNII file. `PrxPod*`, `Snii*`, `DorisSniiFileReader*`,
`*InvertedIndex*`, `*IndexFileWriter*` and `*IndexFileReader*`: 1710 tests
passed, including the new budget, codec choice, writer and configuration tests.
- On master both changes passed CI.
- Not built locally on the branch-4.2 tip; this PR's BE UT run covers that.
### Release note
SNII now chooses per index how many position windows to store in the faster
but larger PFOR form, spending at most 8% of the size of the windows zstd
compresses; `snii_prx_zstd_min_saving_percent` defaults to -1 for this, and
0-100 still sets a fixed percentage.
### Check List (For Author)
- Test: Unit Test
- SniiPrxZstdBudget: the prior share, the windows zstd shrinks least are
traded first, the bound under a descending order, windows with nothing to trade.
- SniiPrxPodTest: the faster frame (PFOR or raw) and when the budget is
asked.
- SniiStreamedWriterSessionTest: windows that save 1-29% through the
ordinary and the streamed (compaction) build; 30 trades all of them, the budget
at most 8% of the smallest frames.
- SniiWriterPrxZstdSaving: the default and the config mapping.
- Manual test: index size of 10M and 99M AgentLogs and 12M TextBench loads
with 0, 30 and the budget; frames read per codec from query profiles;
interleaved hot A/B in one process.
- Behavior changed: Yes (the default .prx codec choice of newly written
segments; existing segments are read as before)
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
