stinger1206 opened a new pull request, #70638:
URL: https://github.com/apache/doris/pull/70638
Cherry-pick of #67720 to branch-4.2. The change is identical to the merged
master commit; the only adaptation is that the test file keeps this branch's
`#include "storage/segment/segment_writer.h"` (master's later rename to
`vertical_segment_writer.h` is not part of this branch).
### What problem does this PR solve?
Issue Number: close #67719
Related PR: #67720
Problem Summary:
`Segment::_footer_pb` (`std::weak_ptr<SegmentFooterPB>`) was a second footer
access path outside `StoragePageCache`:
- `_get_segment_footer()` reads it via `lock()` and rebinds it
(`_footer_pb = footer_pb_shared`) from query/compaction threads whenever
the
cached footer expires, while `get_metadata_size()` also accesses the same
non-atomic `std::weak_ptr`. Concurrent assignment and `lock()` on the same
weak_ptr, if these paths overlap, is a data race and therefore undefined
behavior.
- Moreover, the weak_ptr fast path could bypass the cache lookup whenever the
footer object was still alive — including while the cache entry was
present —
so the cache could not observe real footer accesses and its LRU order could
diverge from actual usage.
In production (4.1.3) this crash family manifested as intermittent BE
SIGSEGVs, with innermost frames such as `SegmentFooterPB::ByteSizeLong`
(`Segment::get_metadata_size`) and `~SegmentFooterPB` under
`LRUCache::prune_if`, healed by BE restart with segment files byte-identical.
See #67719 for a TSAN micro-repro of the unsafe `std::weak_ptr` access
pattern.
Fix (identical to #67720):
- remove `_footer_pb` entirely;
- `_get_segment_footer()` always accesses the footer through
`StoragePageCache` (single access path: lookup → parse & insert on miss),
so
eviction and LRU recency are handled consistently;
- `get_metadata_size()` no longer reads the footer: footer memory is tracked
by `StoragePageCache`'s own memory tracking, and doing a lookup there only
for accounting would artificially refresh LRU recency on every
metadata-size sweep without any real query access;
- update the stale `StoragePageCache::insert` comment to instead recommend
going through cache lookup for every access;
- mark `Segment::_parse_footer()` and its helper `_write_error_file()` const;
- tests: concurrent eviction regression (adapted from #69811), LRU recency of
repeated footer accesses, and held-footer lifetime/cache-accounting across
eviction and reload.
### Release note
Fix unsafe Segment footer caching that could cause BE crashes and bypass
StoragePageCache LRU accounting.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Unit test: the same `SegmentFooterCacheTest` suite as #67720, including
`ConcurrentGetSegmentFooterWithEviction` (8 threads call
`_get_segment_footer()` while half of them keep erasing the footer's cache
page), `RepeatedFooterAccessesRefreshLruOrder`, and
`HeldFooterStaysValidAcrossCacheEvictionAndReload`. All 8 tests passed
locally under ASAN on master (#67720); CI re-runs them on this branch.
Manual test: the cache-only implementation is deployed on a production
canary BE and the crash family has not recurred under the
previously-crashing workload.
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]