airborne12 opened a new pull request, #68680:
URL: https://github.com/apache/doris/pull/68680
### What problem does this PR solve?
Issue Number: None
Related PR: #67657 (master), #67537 (SNII format alignment), #67538 (SNII
gram index).
Problem Summary:
The SNII writer could expand a large posting list into complete document,
frequency, and position arrays during spill, merge, or final encoding. This
backport shares one bounded posting workspace across those stages, writes
sealed temporary runs, and merges with a workspace-limited fan-in. The default
workspace is 32 MiB for newly created writers. It bounds posting workspace, not
whole-BE RSS.
The persistent SNII reader and proto declarations are unchanged. The 4.2
`snii.proto` noncomment declarations and field numbers match current master
exactly. Ordinary admitted posting windows retain their resident encoder. Large
streaming ZSTD frames may have different compressed bytes while retaining the
same persistent format.
### Release note
Add mutable `snii_postings_workspace_bytes` (default 32 MiB), captured when
a SNII ingestion writer or native compaction reporter is created. A compression
context that cannot fit returns a memory-limit error; high ZSTD levels can
require a larger explicit budget. `snii_spill_max_run_files_per_buffer` also
limits merge fan-in. Temporary runs use a private sealed format and are not
persistent index files.
### Check List (For Author)
- Test:
- [x] Regression test
- [x] Unit Test
- [ ] Manual test
- Behavior changed:
- [x] Yes. Bound spill and encoding workspace while preserving the
persistent reader and format.
- Does this need documentation?
- [x] Yes. `be/src/storage/index/snii/writer/README.md` documents budget
scope, temporary runs, and compatibility. User-facing configuration
documentation is a follow-up.
### Backport audit
Source merge commit: `0a3595e61870ed302eb8b732f968e48823bc0cc4`; backport
commit: `49c7bc8359a3bea9832e6fa9eec5b42a03282dcd`.
The 155 audit rows below cover every source diff hunk, with byte-identical
files collapsed as permitted: 33 byte-identical files, 109 ported hunks, 12
adapted hunks, and one inapplicable build hunk. The source runtime logic and
test assertions are retained. The master-only `STORAGE_UNITY_SKIP` hunk does
not apply because branch-4.2 has no Storage Unity Build; its recursive source
glob compiles the new `.cpp` helpers. Adaptations shorten comments and extract
a 32 KiB reservation check into a helper for the branch's clang-tidy complexity
limit. `git range-diff` shows these code/comment/build differences plus
commit-message cleanup; there are no other behavior changes.
| Source file | Source hunk | Disposition and reason |
| --- | --- | --- |
| `be/src/common/config.cpp` | `@@ -1466,3 +1466,4 @@
DEFINE_mInt64(snii_forced_spill_min_arena_bytes, "67108864");` | Ported: Source
code hunk is present on branch-4.2. |
| `be/src/common/config.cpp` | `@@ -1469,0 +1471,6 @@
DEFINE_mInt32(snii_spill_max_run_files_per_buffer, "64");` | Ported: Source
code hunk is present on branch-4.2. |
| `be/src/common/config.h` | `@@ -1501 +1501 @@
DECLARE_mInt32(snii_index_build_max_memory_limit_percent);` | Adapted: Comments
shortened to describe the 4.2 runtime behavior; declarations are preserved. |
| `be/src/common/config.h` | `@@ -1508,7 +1508,3 @@
DECLARE_mInt64(snii_forced_spill_min_arena_bytes);` | Adapted: Comments
shortened to describe the 4.2 runtime behavior; declarations are preserved. |
| `be/src/common/config.h` | `@@ -1515,0 +1512,3 @@
DECLARE_mInt32(snii_spill_max_run_files_per_buffer);` | Adapted: Comments
shortened to describe the 4.2 runtime behavior; declarations are preserved. |
| `be/src/storage/CMakeLists.txt` | `@@ -58,0 +59,8 @@
set(STORAGE_UNITY_SKIP` | N-A: branch-4.2 has no Storage Unity Build; recursive
source glob includes the new helpers. |
| `be/src/storage/compaction/compaction.cpp` | `@@ -1126 +1126,2 @@ Status
Compaction::do_inverted_index_compaction() {` | Ported: Source code hunk is
present on branch-4.2. |
| `be/src/storage/index/snii/compaction/posting_run_merger.cpp` | `all
source hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/format/dict_block.cpp` | `all source hunks` |
Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/format/dict_block.h` | `all source hunks` |
Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/format/dict_entry.cpp` | `all source hunks` |
Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/format/dict_entry.h` | `all source hunks` |
Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/format/frq_prelude.cpp` | `all source hunks` |
Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/format/frq_prelude.h` | `@@ -137,0 +138,5 @@
Status build_frq_prelude(const FrqPreludeColumns& cols, ByteSink* out);` |
Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/format/prx_pod.cpp` | `all source hunks` |
Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/format/prx_pod.h` | `all source hunks` |
Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/snii_index_writer.cpp` | `@@ -162 +162,2 @@
Status SniiIndexColumnWriter::init() {` | Ported: Source code hunk is present
on branch-4.2. |
| `be/src/storage/index/snii/snii_index_writer.cpp` | `@@ -177,4 +178,3 @@
Status SniiIndexColumnWriter::init() {` | Adapted: Process-limiter comment
shortened; runtime code is preserved. |
| `be/src/storage/index/snii/writer/README.md` | `all source hunks` |
Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/bounded_run_merge.cpp` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/compact_posting_pool.h` | `@@ -24,0 +25
@@` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/compact_posting_pool.h` | `@@ -165,0
+167,4 @@ public:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/compact_posting_pool.h` | `@@ -312,0
+318,16 @@ inline uint8_t CompactPostingPool::Cursor::next() {` | Ported:
Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/encoded_spill_run.cpp` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/encoded_spill_run.h` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/logical_index_writer.cpp` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/logical_index_writer.h` | `@@ -366,0
+367,2 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/memory_reporter.h` | `all source hunks`
| Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/posting_byte_buffer.cpp` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/posting_byte_buffer.h` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/posting_external_sort.cpp` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/posting_external_sort.h` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/posting_prx_encoder.cpp` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/posting_prx_encoder.h` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -20,0
+21 @@` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -28,0
+30,2 @@` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -34
+37,3 @@` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -40
+44,0 @@ namespace {` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -63,12
+66,0 @@ bool conservatively_fits_prx_window(uint64_t doc_count, uint64_t
position_count,` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -90
+82,19 @@ public:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -118
+128 @@ public:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -178,2
+188,6 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -183
+197,2 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -216,0
+232,5 @@ private:` | Adapted: The same 32 KiB reservation is made in a helper
to avoid a new 4.2 clang-tidy complexity warning. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -275,8
+295,17 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@
-288,17 +316,0 @@ private:` | Ported: Source code hunk is present on
branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -316,7
+328,9 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -327,2
+341,2 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -329,0
+344 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -334,2
+349,4 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -337,9
+354,3 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -346,0
+358,3 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -354,2
+368,7 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -359,2
+378,2 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -363,3
+382,21 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -376,0
+414,47 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.cpp` | `@@ -383,3
+467,9 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/posting_window_emitter.h` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/spill_run_codec.cpp` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -32 +32,3 @@
namespace doris::snii::writer {` | Adapted: Private raw-run codec comment
shortened; reader behavior is preserved. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -34,33 +36,9 @@
namespace doris::snii::writer {` | Adapted: Private raw-run codec comment
shortened; reader behavior is preserved. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -74 +52,2 @@
public:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -80,2 +59,2 @@
public:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -91,0 +71 @@
private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -99,0 +80,2 @@
private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -118 +100 @@
private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -210 +192,2 @@
private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -214,3 +197,14
@@ Status merge_run_sources(const std::vector<std::string>& run_paths,` |
Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -221,2 +215,3
@@ Status merge_run_sources(const std::vector<std::string>& run_paths,` |
Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -225,3 +220,2
@@ Status merge_run_sources(const std::vector<std::string>& run_paths,` |
Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spill_run_codec.h` | `@@ -230 +224,2 @@
Status compact_runs(const std::vector<std::string>& run_paths,` | Ported:
Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spillable_byte_buffer.h` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -24,0 +25
@@` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -26,0 +28
@@` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -35,0 +38
@@` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -37,0 +41
@@` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -38,0 +43
@@` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -105,4
+109,0 @@ std::atomic<uint64_t> g_global_forced_spills {0};` | Ported: Source
code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -185,6
+185,0 @@ void reset_global_forced_spills() {` | Ported: Source code hunk is
present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -286 +281,4
@@ SpimiTermBuffer::SpimiTermBuffer(const std::vector<std::string>* vocab, bool
has` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -311 +309,4
@@ SpimiTermBuffer::SpimiTermBuffer(bool has_positions, size_t
spill_threshold_byte` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -481 +482
@@ void SpimiTermBuffer::accumulate_without_spill_gate(uint32_t term_id,
uint32_t d` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -730,33
+731,7 @@ namespace {` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -765,43
+740,8 @@ Status reserve_tracked_vector(std::vector<T>* values, size_t target,`
| Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -808,0 +749
@@ Status sort_by_docid(std::vector<uint32_t>* docids, std::vector<uint32_t>*
freqs` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -810,18
+751,15 @@ Status sort_by_docid(std::vector<uint32_t>* docids,
std::vector<uint32_t>* freqs` | Ported: Source code hunk is present on
branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -829,3
+767,3 @@ Status sort_by_docid(std::vector<uint32_t>* docids,
std::vector<uint32_t>* freqs` | Ported: Source code hunk is present on
branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -832,0
+771,2 @@ Status sort_by_docid(std::vector<uint32_t>* docids,
std::vector<uint32_t>* freqs` | Ported: Source code hunk is present on
branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -834,9
+773,0 @@ Status sort_by_docid(std::vector<uint32_t>* docids,
std::vector<uint32_t>* freqs` | Ported: Source code hunk is present on
branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -845,11
+776,20 @@ Status sort_by_docid(std::vector<uint32_t>* docids,
std::vector<uint32_t>* freqs` | Ported: Source code hunk is present on
branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -897,2
+836,0 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -900,3 +838
@@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -918,6 +854
@@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -939 +869,0
@@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -952,44
+881,0 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1081,28
+967,10 @@ Status SpimiTermBuffer::drain_sorted_streamed(const
StreamedTermConsumer& fn) {` | Ported: Source code hunk is present on
branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1127 +995
@@ Status SpimiTermBuffer::drain_sorted_streamed(const StreamedTermConsumer&
fn) {` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1129
+996,0 @@ Status SpimiTermBuffer::drain_to_writer(RunWriter* w) {` | Ported:
Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1138,5
+1005,9 @@ Status SpimiTermBuffer::drain_to_writer(RunWriter* w) {` | Ported:
Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1146 +1017
@@ Status SpimiTermBuffer::drain_to_writer(RunWriter* w) {` | Ported: Source
code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1154,28
+1024,0 @@ Status SpimiTermBuffer::drain_to_writer(RunWriter* w) {` | Ported:
Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1183,9
+1026,12 @@ Status SpimiTermBuffer::spill_to_run() {` | Ported: Source code
hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1193
+1038,0 @@ Status SpimiTermBuffer::spill_to_run() {` | Ported: Source code hunk
is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1208,8
+1053,23 @@ Status SpimiTermBuffer::spill_to_run() {` | Ported: Source code
hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1252,2
+1112,3 @@ Status SpimiTermBuffer::merge_runs_streamed(const
StreamedTermConsumer& fn) {` | Ported: Source code hunk is present on
branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1254,0
+1116,3 @@ Status SpimiTermBuffer::merge_runs_streamed(const
StreamedTermConsumer& fn) {` | Ported: Source code hunk is present on
branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1267
+1131,11 @@ Status SpimiTermBuffer::for_each_term_sorted(const
StreamedTermConsumer& fn) {` | Ported: Source code hunk is present on
branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1290,3
+1164,4 @@ std::vector<TermPostings> SpimiTermBuffer::finalize_sorted() {` |
Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1305,2
+1180,2 @@ void SpimiTermBuffer::cleanup_runs() {` | Ported: Source code hunk
is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.cpp` | `@@ -1308
+1183,4 @@ void SpimiTermBuffer::cleanup_runs() {` | Ported: Source code hunk
is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.h` | `@@ -50,6 +50,5
@@ class GlobalMemoryLimiter; // G09 process-wide build-RAM registry (see
below)` | Adapted: Spill and merge comments shortened; runtime declarations are
preserved. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.h` | `@@ -138,3 +137,5
@@ struct TermPostings {` | Adapted: Spill and merge comments shortened;
runtime declarations are preserved. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.h` | `@@ -156,2 +157,2
@@ struct TermPostings {` | Adapted: Spill and merge comments shortened;
runtime declarations are preserved. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.h` | `@@ -261,10
+262,3 @@ public:` | Adapted: Spill and merge comments shortened; runtime
declarations are preserved. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.h` | `@@ -286,6 +280,3
@@ public:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.h` | `@@ -359,20
+349,0 @@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.h` | `@@ -398 +368,0
@@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.h` | `@@ -413,6 +382,0
@@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.h` | `@@ -420 +384 @@
private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.h` | `@@ -555,6 +519,9
@@ private:` | Ported: Source code hunk is present on branch-4.2. |
| `be/src/storage/index/snii/writer/spimi_term_buffer.h` | `@@ -597,6 +564
@@ void reset_global_forced_spills();` | Ported: Source code hunk is present on
branch-4.2. |
| `be/src/storage/index/snii/writer/term_posting_source.h` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/test/storage/index/snii/compaction/snii_index_compaction_test.cpp` |
`all source hunks` | Ported(verbatim): target file matches source blob. |
| `be/test/storage/index/snii/compaction/snii_streamed_session_test.cpp` |
`all source hunks` | Ported(verbatim): target file matches source blob. |
| `be/test/storage/index/snii/spimi_spill_rank_test.cpp` | `all source
hunks` | Ported(verbatim): target file matches source blob. |
| `be/test/storage/index/snii/staged_file_probe.h` | `all source hunks` |
Ported(verbatim): target file matches source blob. |
| `be/test/storage/index/snii/writer/bounded_posting_codec_test.cpp` | `all
source hunks` | Ported(verbatim): target file matches source blob. |
| `be/test/storage/index/snii/writer/memory_reporter_wiring_test.cpp` | `all
source hunks` | Ported(verbatim): target file matches source blob. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -110,8
+110,10 @@ Status MergeRuns(const std::vector<std::string>& run_paths, const
std::vector<st` | Ported: Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -270
+272 @@ TEST(SniiSpillRunCodec,
MergeRunSourcesAccountsReadersAndReleasesOnSuccess) {` | Ported: Source code
hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -298
+300 @@ TEST(SniiSpillRunCodec,
RunReaderDocidReservationFailureReleasesAllCharges) {` | Ported: Source code
hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -434
+436 @@ TEST(SniiSpillRunCodec,
CompactRunsMergedPostingReservationHonorsHardLimitAndRel` | Ported: Source code
hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -723,4
+725,6 @@ TEST(SniiSpillRunCodec, MergeWideTermStreamsIdenticalToMaterialized)
{` | Ported: Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -812
+816 @@ TEST(SniiSpillRunCodec, NPosExceedsFileIsCorruption) {` | Ported:
Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -846
+850,2 @@ TEST(SniiSpillRunCodec,
TruncatedPositionsFailWithoutUninitializedTail) {` | Ported: Source code hunk
is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -849,4
+854,2 @@ TEST(SniiSpillRunCodec,
TruncatedPositionsFailWithoutUninitializedTail) {` | Ported: Source code hunk
is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -894
+897,2 @@ TEST(SniiSpillRunCodec, MergeRunSourcesRejectsUnconsumedSource) {` |
Ported: Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -1072,4
+1076,6 @@ TEST(SniiSpillMergeTest, MergeWideTermStreamsMatchesMaterialized) {`
| Ported: Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -1237,0
+1244 @@ TEST(SniiSpillMergeTest, SpillMergeEqualsInMemory) {` | Ported: Source
code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spill_run_codec_test.cpp` | `@@ -1245
+1252 @@ TEST(SniiSpillMergeTest, SpillMergeEqualsInMemory) {` | Ported: Source
code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -30,13
+30,4 @@` | Adapted: Test comment shortened; assertions are preserved. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -77,5
+67,0 @@ void feed_step(SpimiTermBuffer* buf, uint32_t k) {` | Ported: Source
code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -103 +89
@@ std::map<std::string, DrainedTerm> drain(SpimiTermBuffer* buf) {` | Ported:
Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -108,9
+94,4 @@ TEST(SniiSpimiRunCap, CapIsHonoredAndCompactionSeamFires) {` | Ported:
Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -118,4
+99,3 @@ TEST(SniiSpimiRunCap, CapIsHonoredAndCompactionSeamFires) {` | Ported:
Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -147,3
+127,2 @@ TEST(SniiSpimiRunCap, CompactedDrainMatchesUncappedControl) {` |
Ported: Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -153 +132
@@ TEST(SniiSpimiRunCap, CompactedDrainMatchesUncappedControl) {` | Ported:
Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -158
+137,2 @@ TEST(SniiSpimiRunCap, CompactedDrainMatchesUncappedControl) {` |
Ported: Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -161 +141
@@ TEST(SniiSpimiRunCap, CompactedDrainMatchesUncappedControl) {` | Ported:
Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -168,0
+149 @@ TEST(SniiSpimiRunCap, CompactedDrainMatchesUncappedControl) {` |
Ported: Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -182 +163
@@ TEST(SniiSpimiRunCap, CompactedDrainMatchesUncappedControl) {` | Ported:
Source code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_run_cap_test.cpp` | `@@ -191,0
+173 @@ TEST(SniiSpimiRunCap, ZeroCapDisablesCompaction) {` | Ported: Source
code hunk is present on branch-4.2. |
| `be/test/storage/index/snii/writer/spimi_spill_memory_test.cpp` | `all
source hunks` | Ported(verbatim): target file matches source blob. |
| `be/test/storage/index/snii/writer/spimi_spill_writer_test.cpp` | `all
source hunks` | Ported(verbatim): target file matches source blob. |
| `be/test/storage/index/snii/writer/term_posting_test_utils.h` | `all
source hunks` | Ported(verbatim): target file matches source blob. |
### Validation
- BE ASAN build and unit-test binary build: passed on the final commit tree.
- BE ASAN unit tests: 150/150 passed across 12 focused suites, including the
eight pre-existing index corpus images (`SniiGoldenCorpus.WriteOrVerify`) and
four persistent writer golden-byte tests. No sanitizer report.
- Live upgrade read probe: a table and SNII index written by the prior 4.2
writer remained readable after replacing the BE with this backport. MATCH_ANY,
MATCH_PHRASE, MATCH_ALL, and row-count output matched the pre-upgrade baseline
byte for byte.
- Isolated cloud regressions:
`inverted_index_p0/storage_format/test_storage_format_snii` and
`inverted_index_p0/gram/test_gram_compaction` passed, one suite and zero
failures each.
- Clang-format 16 merged-tree preflight and full PR diff English check:
passed against `origin/branch-4.2` at
`7e4909b904ff49fc799e5a4efa02f03fe4a8638d`; no language exceptions.
- Clang-tidy 16: all 17 changed BE `.cpp` files were checked. The sole new
complexity diagnostic was removed by the helper extraction and that file now
passes. Six files still report the same existing branch diagnostics as the
pre-backport baseline, and `snii_index_writer.cpp` hits the same nine baseline
`jni-util.h` static assertions. A full clang-tidy pass is not claimed.
- No backport-specific performance benchmark is claimed. The master PR
documents workload-dependent write and query tradeoffs.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
