bobhan1 opened a new pull request, #65905:
URL: https://github.com/apache/doris/pull/65905
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
After a cloud segment is finalized, `SegmentWriter` allocates a cache holder
over the segment index range and changes every intersecting cache block to
`INDEX`. This extra holder is not aligned to S3 multipart buffer boundaries. In
the deterministic reproduction, it creates cache block `[241, 304]`, which
crosses the boundary between buffers starting at offsets 0 and 256.
Non-blocking close allows those multipart buffers to finish out of order.
When the buffer at offset 256 finishes first, it claims `[241, 304]` and writes
its bytes at the beginning of that block. The cached data is therefore shifted
by 15 bytes, while the multipart object in remote storage remains correct. Both
a direct cache read and a cached S3 read observe the corrupted bytes.
Remove the post-finalize cache-holder allocation and
`change_cache_type(INDEX)` behavior. Segment blocks keep the cache type
selected by the file writer, and this cross-buffer cache block is no longer
created by `SegmentWriter`.
The two commits intentionally preserve the proof in history:
1. `028891cdc38` adds a deterministic end-to-end BEUT and fails on the
existing behavior.
2. `0d6964d2f80` removes the behavior and removes the temporary BEUT and
synchronization hooks.
The final PR diff contains only the production-code deletion.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `./run-be-ut.sh --run
--filter=SegmentWriterFileCacheConcurrencyTest.ConcurrentLaterPartMustNotShiftCachedSegmentBytes
-j100` on the first commit: expected failure, reproducing shifted cached bytes
in `[241, 304]`
- `./run-be-ut.sh --run --filter=CloudFileCacheWriteIndexOnlyTest.*
-j100` on the final commit: 3 tests passed
- Behavior changed: Yes. Segment cache blocks are no longer changed to
`INDEX` after segment finalization.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
