liaoxin01 opened a new pull request, #67347:
URL: https://github.com/apache/doris/pull/67347
### What problem does this PR solve?
Problem Summary:
A small file is handed to `PackedFileManager` when the segment is flushed
(`close(true)` /
`InvertedIndexFileWriter::begin_close()`), but its slice location is read
again much later, when the
rowset is finally closed:
- `PackedFileWriter::_wait_packed_upload()` -> `wait_upload_done()`, called
from `finish_close()`
- `CloudRowsetWriter::_collect_packed_slice_location()` ->
`get_packed_slice_location()`, when the
rowset meta is built
The gap between the two is the duration of the whole load, not the lifetime
of the packed file.
`cleanup_expired_data()` recycled `_global_slice_locations` purely on the
age of the small file
(`uploaded_file_retention_seconds`, 1800s by default), so any load running
longer than that lost the
mapping and failed with:
```
[INTERNAL_ERROR]File not found in global index:
data/<tablet_id>/<segment>_0.idx
```
reported to the coordinator as:
```
add batch req success but status isn't ok, err: [INTERNAL_ERROR]PStatus: ...
File not found in global index
```
The packed file itself had been uploaded successfully — only the in-memory
mapping was gone. In a
case we hit in production, a broker load ran for 38 minutes; the idx file
was packed and uploaded at
minute 1 and the rowset was closed at minute 38, well past the 1800s
retention.
Two more points worth noting:
- `_uploaded_packed_files` is recycled `uploaded_file_retention_seconds`
after the upload finished,
using the same config. So even with the index entry retained, a waiter
could fail with
`Packed file not found for path: ...`. Both lifetimes have to be fixed
together.
- This is not specific to inverted index files. Segments smaller than
`small_file_threshold_bytes`
go through exactly the same async-close / late-wait path.
### What is changed and how it works?
Bind the lifetime to the actual reference instead of to a fixed TTL:
- `append_small_file()` pins the slice location, and `PackedFileWriter`
releases the pin in its
destructor — i.e. after the rowset meta has been collected, which is the
last read. The background
cleanup never recycles a pinned entry; the TTL still applies to unpinned
ones, so nothing is
leaked once the writers are gone.
- Mirror the terminal upload state (UPLOADED / FAILED) of a packed file into
the slice locations it
contains, so `wait_upload_done()` can answer without looking up the
`PackedFileContext`. This
leaves the retention of `_uploaded_packed_files` untouched: only the much
smaller index entries
live as long as the load, the packed file contexts (which own the closed
`FileWriter`) are still
recycled after 30 minutes.
Behavior for short loads is unchanged.
### Release note
Fix `File not found in global index` failures for loads that run longer than
`uploaded_file_retention_seconds` when packed file is enabled.
### Check List (For Author)
- Test
- [x] Unit Test
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
