eldenmoon opened a new pull request, #70723:
URL: https://github.com/apache/doris/pull/70723

   ### Background
   
   Doris stores Variant subcolumn metadata outside the segment footer in the V3 
format. Compaction reads this metadata to collect path types and statistics. It 
can reuse the segment reader across attempts.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Bugfix
   
   A transient page allocation failure can prevent later compactions from 
loading the same Variant metadata even after memory pressure subsides. The 
reader now retries the metadata load on the next call after a memory limit 
error.
   
   ### Details
   
   - **Frequency:** Affected segments keep returning a stale failure while 
subsequent compaction attempts reuse their cached reader.
   - **Root cause:** PageIO converts the allocation exception into 
`MEM_LIMIT_EXCEEDED`. The call-once guard caches this status for the lifetime 
of the reader.
   - **Why existing tests missed it:** They exercise metadata loading and 
caching with healthy page reads.
   - **Fix:** Let the Variant bulk metadata loader opt out of caching 
`MEM_LIMIT_EXCEEDED`. Concurrent attempts still use the existing lock. A retry 
skips paths that have already been merged.
   - **Other paths and branches:** Variant queries and compaction share the 
bulk loader. The same failure cache exists on branch-4.1 and branch-4.2.
   
   ### Testing
   
   - Before the fix, an ASAN test injected a single allocation exception during 
a real PageIO read. The second call on the same reader returned the stale error.
   - After the fix, the second call read the metadata and recovered the 
expected subcolumn and statistics. The third call reused the result without 
issuing more reads.
   - ASAN BE unit tests on dev-x passed 159 tests. Four tests were skipped 
because this build does not enable Nested Group writes.
   - The ASAN BE/FE build, all 14 related V3 regression suites, BE build 
hygiene, clang-format, and changed-line clang-tidy passed on dev-x.
   
   ### Release note
   
   Compaction can recover from a transient memory limit error while loading 
Variant external metadata.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No
   - Pre-submission status: Ready
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex
   - Generated: Implementation, tests, and PR description
   - Human review: Pending
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to