lszskye opened a new pull request, #545:
URL: https://github.com/apache/paimon-cpp/pull/545

   ### Purpose
   Linked issue:  #543 
   
   Support weighted placement of data files and distribute index files and
   checkpoints across multiple external directories.
   
   - Add `data-file.external-paths.weights` and the `weighted` strategy.
     Weights apply only to this strategy; empty weights fall back to round 
robin.
   - Allow comma-separated roots in `global-index.external-path`, with
     independent round-robin selection for index files and checkpoints.
   - Recover checkpoints by scanning all configured task directories and
     selecting the largest checkpoint ID. Continue ID allocation from that
     maximum and clean up checkpoints across all configured roots.
   - Allocate full index paths once and retain them for writing, size queries,
     metadata, and cleanup, without a filename-to-path cache.
   - Place newly created DV files in the configured global external directories
     when `index-file-in-data-file-dir=false`.
   
   ### Tests
   - Add option tests for weight parsing, ignoring weights under other
     strategies, and parsing multiple global index roots.
   - Add checkpoint tests for recovery across roots with duplicate older IDs,
     continued task cleanup after a deletion failure, and deletion error
     propagation.
   - Update path factory tests and index writer tests for full-path allocation
     and the revised factory interfaces.
   
   ### API and Format
   
   - Add `Options::DATA_FILE_EXTERNAL_PATHS_WEIGHTS`.
   - Breaking API change: replace `GlobalIndexFileWriter::NewFileName()` with
     `NewPath()` and remove its `ToPath()` method. Callers must retain the
     returned full path and pass it to `NewOutputStream()` and `GetFileSize()`.
   - Update built-in index writers and their callers to use full paths.
   - No storage format or protocol changes. External index locations use the
     existing metadata field, and checkpoint filenames retain their format.
   
   ### Generative AI tooling
   
   Generated-by: OpenAI Codex (GPT-6)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to