airborne12 opened a new pull request, #67657: URL: https://github.com/apache/doris/pull/67657
### What problem does this PR solve? Related PR: #67537 (prerequisite) **Draft: depends on #67537, with the error-cleanup and query-performance items below still open.** This PR contains two spill-specific commits after the prerequisite. Until #67537 merges, GitHub's master comparison also includes its commits. Review the [spill-only diff](https://github.com/airborne12/apache-doris/compare/c9a1b48e1ec034731dbd1f02a14f76722986c533...4022412c1a66b74b24fe6a67c79b55187e7a52b2) for the 46 files changed by this work. Rebase onto master after the prerequisite merges. Problem Summary: SNII expands compact postings into complete document, frequency and position arrays during spill, run merging and final encoding. A frequent term or a very long document can therefore create a large allocation while the original posting arena is still live. Limiting the first spill alone moves the peak into the merge or encoder. Repeatedly merging an increasing run prefix also amplifies temporary I/O. This change reserves one hard posting workspace before allocating, shared across the writer's spill, merge and final encoding: - Append compact fragments to one spool with independently sealed, checksummed run ranges. The run directory itself can spill. - Merge contiguous groups using fan-in constrained by workspace and available file descriptors. Intermediate passes copy encoded fragments; final consumption combines boundary documents in their original position order. - Replay large position payloads and stage DD, window metadata and inline DD/PRX within the same budget. Ordinary admitted windows retain the existing resident encoder. - Keep input arena, dictionary keys/ranks, norms, NULL metadata, allocator retention and OS page cache outside this posting-workspace limit. It is an algorithmic workspace bound, not a whole-BE RSS limit. - Preserve sequential read-ahead across small-term boundaries. The added regression originally read 578,549,868 bytes from a 271,632-byte run because every term discarded the same 64 KiB cache. The persistent SNII layout and window policy established by #67537 are retained. Persistent readers are unchanged. Large streaming ZSTD frames can have different compressed bytes, so reader compatibility alone is not treated as evidence of equal query performance. ### Release note Add `snii_postings_workspace_bytes`, a positive mutable setting with a 32 MiB default, captured by newly created SNII ingestion writers and native compaction reporters. Active reporters retain their original budget. A compression context that cannot fit returns a memory-limit error; high ZSTD levels can require a larger explicit budget. The historical `snii_spill_max_run_files_per_buffer` setting now additionally limits merge fan-in. Temporary runs use a private encoded and sealed format. Large posting lists may trade additional write CPU and temporary I/O for lower peak posting memory. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test - [ ] No need to test or manual test Existing local validation applies to the exact code trees in this PR: only commit ancestry changed when rebasing onto the English-message version of #67537. No build or benchmark was rerun for that ancestry-only change. - ASAN SNII validation covered 1,337 distinct tests: 1,336 passed and one opt-in corpus test was skipped. The existing seven disabled tests were not enabled. All 16 bounded-codec tests passed again after the test-helper refactor. - Coverage includes high document frequency, long documents, 10,000 runs, multiple merge passes, boundary documents, accumulated inline DD/PRX larger than the budget, CRC/truncation/length failures, I/O errors, reservation rollback and compaction. - Complete worktree BE ASAN and Release builds passed. Release was used for performance measurements. clang-format 16, diff checks and clang-tidy checks across the 44 changed C++ files passed for introduced diagnostics; the unchanged `core/types.h` orphan-NOLINT diagnostic was recorded separately. - The final worktree BE passed `test_storage_format_snii` in an isolated cloud-mode deployment. - Runtime budget updates rejected zero and negative values. A level-19 import failed without publishing rows at 32 MiB, then succeeded at 128 MiB with phrase-query results verified. - The Wikipedia read-amplification reproduction completed in 13.563 seconds with 560,596,809 logical process-read bytes after the cache fix, versus approximately 158 GiB before that fix. This single reproduction is separate from the repeated comparisons below. **Measured tradeoffs:** 215 native writer runs and 50 real-BE imports completed, using five repeated old/new trials per tested configuration. Baseline and candidate both include #67537, isolating the spill change. Native memory below is the single-writer RSS peak increment, not entire BE memory. | Native case, 32 MiB workspace | Old / new peak RSS increment | Paired write-time change | | --- | --- | --- | | 200M documents, one token each | 1927.04 / 591.90 MiB | -5.42% (95% interval: -16.40% to +7.00%) | | 50M documents, 16 tokens each | 698.31 / 511.19 MiB | -2.73% (-10.18% to +5.33%) | | 100K documents, 2048 repeated tokens each | 207.32 / 62.79 MiB | +61.41% (+56.81% to +66.14%) | | 10M documents, four tokens, 157 runs | 99.21 / 33.01 MiB | -2.58% (-4.86% to -0.25%) | All 140 candidate native runs stayed within their configured posting workspace. The 200M-document case reduced RSS by about 69.3%; ordinary workloads did not show a universal memory benefit. Real-BE Wikipedia imports took +9.48% (+6.80% to +12.23%) and repeated-token long-document imports took +40.61% (+24.84% to +58.37%). Whole-BE RSS did not decrease consistently. All recorded posting and query result checks passed. Native diagnostic controls did not confirm a stable query regression. The real-BE query matrix stopped at 490/640 records, covering 49,000 timed requests; two adverse signals below remain unresolved. These results do not establish universal query-performance equivalence. **Open items before merge:** - [ ] Merge #67537 and rebase this PR onto master so the main PR diff contains only the spill changes. - [ ] Fix intermediate-run cleanup when a manifest write fails with `ENOSPC`. `ReducedRuns` cleanup reads its manifest through `PostingByteBuffer::read_at()`, which retries `flush()`; persistent write failure aborts cleanup and leaves intermediate run files behind. Static review confirmed this path; dedicated fault-injection coverage is still missing. - [ ] Resolve the real-BE warm Wikipedia query signals: phrase p50 +1.11% (95% interval +0.72% to +1.50%) and phrase-prefix p95 +2.47% (+0.27% to +4.73%). Long-document BE queries, completion of the BM25 matrix and independent-build confirmation were not completed. The query non-regression acceptance condition is not yet established. - Behavior changed: - [ ] No. - [x] Yes. Enforce a shared hard posting sub-budget, stream large payloads, and bound run reduction without silently changing compression level or window policy. See the release note for configuration and temporary-I/O tradeoffs. - Does this need documentation? - [ ] No. - [x] Yes. `be/src/storage/index/snii/writer/README.md` documents budget scope, configuration, temporary format, lifecycle and compatibility. User-facing documentation for the new BE configuration remains to be published. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
