This is an automated email from the ASF dual-hosted git repository.
airborne12 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8fd5e503cc0 [test](be) Size two SNII spill-threshold tests to cross
the cap with libc++ (#70322)
8fd5e503cc0 is described below
commit 8fd5e503cc01c9d5c843380588bb048bb688f4dc
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Nov 6 21:44:52 2026 +0800
[test](be) Size two SNII spill-threshold tests to cross the cap with libc++
(#70322)
### Background
SNII is an inverted index storage format whose writer collects postings
in memory in a `SpimiTermBuffer`, which charges its resident bytes to a
`MemoryReporter`. Under the spill-threshold policy that index ingestion
uses, the reporter's cap only triggers spills of the posting arena and
never fails an allocation, because the persistent vocabulary (term
strings, the intern set, the slot index) may exceed the cap by design.
Two cases in `snii_streamed_session_test.cpp` check that finalizing the
index and draining the buffer work, and are accounted exactly, while the
vocabulary alone is above the cap.
### What problem does this PR solve?
Issue Number: None
Related PR: #66052 (added the two cases)
PR Type: Test
Problem Summary:
`FinalizesAfterPersistentVocabularyCrossesSpillThreshold` and
`TerminalDrainReleasesLookupStateWithoutUnderAccountingIds` fail on
every run of a BE unit test build that uses libc++, such as the macOS
build, before they reach what they test: their precondition that the
vocabulary is above the 384 KiB cap does not hold. They insert a fixed
3000 terms, which is enough only with libstdc++, because the buffer
charges each owned term string by `sizeof(std::string)`: 32 bytes with
libstdc++, 24 with libc++. This PR raises the count to 4000, which keeps
the vocabulary above the cap with both libraries without letting the
in-memory case spill. The accounting is right on both libraries, so
production code is unchanged.
### Details
- **Failure:** With libc++, `ASSERT_GT(reporter.current_bytes(),
kReporterCap)` fails in the first case (`actual: 371332 vs 393216`) and
`ASSERT_GT(before_drain, kReporterCap)` in the second (`actual: 371188
vs 393216`), on every run, in the iteration that forces a spill. The BE
UT pipeline builds with libstdc++ on Linux, and `be-ut-mac.yml` builds
the BE on macOS without building or running the unit tests, so no
pipeline has seen it since #66052.
- **Cause:** The buffer charges its owned vocabulary as
`owned_vocab_.capacity() * sizeof(std::string)` plus heap payloads,
which these short terms don't have because they fit the small-string
buffer. The capacity grows by explicit doubling and is 4096 for 3000
terms with either library, so libc++ accounts 8 × 4096 = 32,768 bytes
less for the same input. After the forced spill frees the posting arena,
371,332 bytes are left with libc++, under the 393,216-byte cap;
libstdc++ accounts 404,100 for the same input. The iteration that does
not spill keeps its arena and is above the cap with both.
- **Stability evidence:** The failure is deterministic, not timing
related. With 4000 terms both cases pass with libc++, where the forced
iteration accounts 421,332 bytes; libstdc++ accounts 32,768 bytes more
for the same input. The posting arena, which decides whether the
in-memory iteration spills, does not depend on the library: it reaches
64 KiB, below the 96 KiB (cap / 4) that a spill needs, so that iteration
still never spills.
### Testing
- `run-be-ut.sh --run --filter='SniiStreamedWriterSessionTest.*'`,
locally on macOS arm64 with ASAN (libc++): all 41 cases pass with this
change, and the two cases fail without it.
- Two mutations of `spimi_term_buffer.cpp`, run locally against the new
count: raising MEM_LIMIT_EXCEEDED once the reporter is over the cap,
which treats the spill threshold as an allocation limit, fails both
cases; keeping the intern set and the string ranks through the terminal
drain fails the second case at `during_drain < before_drain` (519,296 vs
503,296), and leaves 208,000 bytes charged at the end of both.
- These ran on `42cf4e3e5d1` plus this change; the SNII sources and
tests are the same on `bb3156d30e6`, where this PR is based. The
libstdc++ run is left to the BE UT pipeline.
### Notes for Reviewers
- Why 4000: the count must stay at or below 4096, where the vector
capacities, and with them the margins above, stay as measured. 3500
terms would leave only about 3 KB above the cap with libc++ (396,332
bytes).
---
.../snii/compaction/snii_streamed_session_test.cpp | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git
a/be/test/storage/index/inverted/snii/compaction/snii_streamed_session_test.cpp
b/be/test/storage/index/inverted/snii/compaction/snii_streamed_session_test.cpp
index 939f8fda618..e9adcb874a5 100644
---
a/be/test/storage/index/inverted/snii/compaction/snii_streamed_session_test.cpp
+++
b/be/test/storage/index/inverted/snii/compaction/snii_streamed_session_test.cpp
@@ -1022,7 +1022,12 @@ TEST(SniiStreamedWriterSessionTest,
TermHashGrowthIsReservedBeforeAllocation) {
TEST(SniiStreamedWriterSessionTest,
FinalizesAfterPersistentVocabularyCrossesSpillThreshold) {
constexpr uint64_t kReporterCap = 384U << 10;
- constexpr uint32_t kTermCount = 3000;
+ // The vocabulary must stay above the cap even after the forced spill
frees the arena, with
+ // either standard library: its strings are charged by
sizeof(std::string), 32 bytes with
+ // libstdc++ and 24 with libc++. With libc++ the forced case accounts
371,332 bytes for 3000
+ // terms, under the 393,216-byte cap, and 421,332 for 4000. The in-memory
arena stays at
+ // 64 KiB, below the cap / 4 that a spill needs, so the in-memory case
still never spills.
+ constexpr uint32_t kTermCount = 4000;
for (bool force_spill : {false, true}) {
writer::MemoryReporter reporter(
/*consume_release=*/nullptr, kReporterCap,
@@ -1067,14 +1072,17 @@ TEST(SniiStreamedWriterSessionTest,
FinalizesAfterPersistentVocabularyCrossesSpi
uint64_t prx_base = 0;
assert_ok(index.lookup("term_100000", &found, &entry, &frq_base,
&prx_base));
EXPECT_TRUE(found);
- assert_ok(index.lookup("term_102999", &found, &entry, &frq_base,
&prx_base));
+ assert_ok(index.lookup("term_" + std::to_string(100000 + kTermCount -
1), &found, &entry,
+ &frq_base, &prx_base));
EXPECT_TRUE(found);
}
}
TEST(SniiStreamedWriterSessionTest,
TerminalDrainReleasesLookupStateWithoutUnderAccountingIds) {
constexpr uint64_t kReporterCap = 384U << 10;
- constexpr uint32_t kTermCount = 3000;
+ // Sized as in FinalizesAfterPersistentVocabularyCrossesSpillThreshold, so
that the
+ // vocabulary exceeds the cap with libc++ as well.
+ constexpr uint32_t kTermCount = 4000;
for (bool force_spill : {false, true}) {
writer::MemoryReporter reporter(
/*consume_release=*/nullptr, kReporterCap,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
