This is an automated email from the ASF dual-hosted git repository.
bobhan1 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 58e87343513 [fix](be) remove changing segment cache blocks to index
type (#65905)
58e87343513 is described below
commit 58e87343513f708f985fdee49da81fb5e344b418
Author: bobhan1 <[email protected]>
AuthorDate: Thu Jul 23 12:53:38 2026 +0800
[fix](be) remove changing segment cache blocks to index type (#65905)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
After a cloud segment is finalized, `SegmentWriter` allocates a cache
holder over the segment index range and changes every intersecting cache
block to `INDEX`. This extra holder is not aligned to S3 multipart
buffer boundaries. In the deterministic reproduction, it creates cache
block `[241, 304]`, which crosses the boundary between buffers starting
at offsets 0 and 256.
Non-blocking close allows those multipart buffers to finish out of
order. When the buffer at offset 256 finishes first, it claims `[241,
304]` and writes its bytes at the beginning of that block. The cached
data is therefore shifted by 15 bytes, while the multipart object in
remote storage remains correct. Both a direct cache read and a cached S3
read observe the corrupted bytes.
Remove the post-finalize cache-holder allocation and
`change_cache_type(INDEX)` behavior. Segment blocks keep the cache type
selected by the file writer, and this cross-buffer cache block is no
longer created by `SegmentWriter`.
The two commits intentionally preserve the proof in history:
1. `028891cdc38` adds a deterministic end-to-end BEUT and fails on the
existing behavior.
2. `0d6964d2f80` removes the behavior and removes the temporary BEUT and
synchronization hooks.
The final PR diff contains only the production-code deletion.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `./run-be-ut.sh --run
--filter=SegmentWriterFileCacheConcurrencyTest.ConcurrentLaterPartMustNotShiftCachedSegmentBytes
-j100` on the first commit: expected failure, reproducing shifted cached
bytes in `[241, 304]`
- `./run-be-ut.sh --run --filter=CloudFileCacheWriteIndexOnlyTest.*
-j100` on the final commit: 3 tests passed
- Behavior changed: Yes. Segment cache blocks are no longer changed to
`INDEX` after segment finalization.
- Does this need documentation: No
---
be/src/storage/segment/segment_writer.cpp | 12 ------------
1 file changed, 12 deletions(-)
diff --git a/be/src/storage/segment/segment_writer.cpp
b/be/src/storage/segment/segment_writer.cpp
index 6f4ecef1140..aa480f218c5 100644
--- a/be/src/storage/segment/segment_writer.cpp
+++ b/be/src/storage/segment/segment_writer.cpp
@@ -1030,18 +1030,6 @@ Status SegmentWriter::finalize(uint64_t*
segment_file_size, uint64_t* index_size
LOG(INFO) << "segment flush consumes a lot time_ns " <<
timer.elapsed_time()
<< ", segmemt_size " << *segment_file_size;
}
- // When the cache type is not ttl(expiration time == 0), the data should
be split into normal cache queue
- // and index cache queue
- if (auto* cache_builder = _file_writer->cache_builder(); cache_builder !=
nullptr &&
-
cache_builder->_expiration_time == 0 &&
-
config::is_cloud_mode()) {
- auto index_start = _index_file_cache_info.cache_start_offset();
- auto size = *index_size + *segment_file_size;
- auto holder = cache_builder->allocate_cache_holder(index_start, size,
_tablet->tablet_id());
- for (auto& segment : holder->file_blocks) {
-
static_cast<void>(segment->change_cache_type(io::FileCacheType::INDEX));
- }
- }
return Status::OK();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
