lszskye opened a new pull request, #545:
URL: https://github.com/apache/paimon-cpp/pull/545
### Purpose
Linked issue: #543
Support weighted placement of data files and distribute index files and
checkpoints across multiple external directories.
- Add `data-file.external-paths.weights` and the `weighted` strategy.
Weights apply only to this strategy; empty weights fall back to round
robin.
- Allow comma-separated roots in `global-index.external-path`, with
independent round-robin selection for index files and checkpoints.
- Recover checkpoints by scanning all configured task directories and
selecting the largest checkpoint ID. Continue ID allocation from that
maximum and clean up checkpoints across all configured roots.
- Allocate full index paths once and retain them for writing, size queries,
metadata, and cleanup, without a filename-to-path cache.
- Place newly created DV files in the configured global external directories
when `index-file-in-data-file-dir=false`.
### Tests
- Add option tests for weight parsing, ignoring weights under other
strategies, and parsing multiple global index roots.
- Add checkpoint tests for recovery across roots with duplicate older IDs,
continued task cleanup after a deletion failure, and deletion error
propagation.
- Update path factory tests and index writer tests for full-path allocation
and the revised factory interfaces.
### API and Format
- Add `Options::DATA_FILE_EXTERNAL_PATHS_WEIGHTS`.
- Breaking API change: replace `GlobalIndexFileWriter::NewFileName()` with
`NewPath()` and remove its `ToPath()` method. Callers must retain the
returned full path and pass it to `NewOutputStream()` and `GetFileSize()`.
- Update built-in index writers and their callers to use full paths.
- No storage format or protocol changes. External index locations use the
existing metadata field, and checkpoint filenames retain their format.
### Generative AI tooling
Generated-by: OpenAI Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]