lszskye opened a new pull request, #382:
URL: https://github.com/apache/paimon-cpp/pull/382
### Purpose
Add checkpoint support for Lumina global index builds.
This change:
- introduces `GlobalIndexCheckpointFileManager` as a plugin-facing
checkpoint storage interface;
- lets `GlobalIndexFileManager` optionally manage checkpoint scanning and
I/O;
- adds task-scoped checkpoint paths in the following form:
`[index_type]-global-index-[field_name]-[range_from]-[range_to]-[task_id]-[id].index.ckpt`
- discovers the latest checkpoint using its numeric ID and continues
allocating IDs from it;
- adds `GlobalIndexer::SupportsCheckpoint()` so checkpoint storage is
created only for index implementations that support it;
- adds an optional `task_id` to `GlobalIndexWriteTask::WriteIndex`;
- adapts Paimon checkpoint storage to Lumina's checkpoint manager;
- restores Lumina builds from the latest matching checkpoint;
- deletes matching checkpoints after a successful build;
- when a build using an existing checkpoint fails, deletes the checkpoint
and retries once from scratch;
- isolates checkpoints by index type, field, row range, and task ID.
Index implementations that do not support checkpoints keep their existing
behavior.
### Tests
src/paimon/core/global_index/global_index_file_manager_test.cpp
src/paimon/core/utils/file_store_path_factory_test.cpp
src/paimon/global_index/lumina/lumina_global_index_test.cpp
test/inte/global_index_test.cpp
### API and Format
- adds `GlobalIndexer::SupportsCheckpoint()`, which returns `false` by
default;
- adds `GlobalIndexCheckpointFileManager`;
- adds an optional `task_id` argument to `GlobalIndexWriteTask::WriteIndex`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
