wangyong9999 opened a new pull request, #245: URL: https://github.com/apache/paimon-cpp/pull/245
### Purpose Linked issue: N/A (follow-up to #194) This completes the Java-compatible source-backed primary-key BTree index lifecycle for fixed-bucket primary-key tables. #194 added the read path; this PR adds the missing maintenance path: - validate the Java-equivalent table/index prerequisites and BTree options; - restore committed source files and BTree payload metadata into each bucket writer; - read physical source rows without deletion-vector filtering, sort `(value, group row id)` with the existing bounded/spill-capable buffers, and build one payload per indexed field and positive data level; - reconcile missing, stale, duplicate, replaced, and empty-level payloads during compaction, and commit matching index ADD/DELETE entries in the same snapshot as the data changes; - retain live index manifests, payloads, and data-file extra files during snapshot expiration and orphan cleanup, while deleting retired payloads through their owning lifecycle; and - document the synchronous C++ maintenance model and its current boundaries. The implementation reuses the source metadata, BTree payload format, index manifests, commit messages, sort buffers, path factories, and reader/writer abstractions already in the repository. It adds no storage protocol, table option, index family, background scheduler, retry policy, or manual action. Java's asynchronous scheduling, retry/fairness policy, and manual rebuild actions are not ported. Existing tag-aware snapshot-expiration behavior is unchanged. Realtime and postpone-bucket writers do not automatically build source-backed payloads. ### Tests - Added coverage for schema validation, deterministic payload construction, writer restore and compaction reconciliation, snapshot expiration, orphan cleanup, and Parquet/ORC lifecycle behavior. The ORC case forces dictionary encoding and lazy decoding. - Full Debug build completed with GCC 8.3.0 and the project's `-Wall -Werror` configuration. - `paimon-common-test`: 1,461/1,461 passed. - `paimon-core-test`: 1,766/1,766 passed. - `paimon-primary-key-sorted-index-inte-test`: 12/12 passed. - `pre-commit run --all-files`: all hooks passed. - Apache RAT 0.16.1: 0 files with unapproved licenses. - `git diff --check apache/main...HEAD`: passed. ### API and Format - No new public table option or query/build API. - No new storage format or protocol. Payloads continue to use the existing BTree/global-index files, `GlobalIndexMeta`, source metadata v1, index manifests, and commit-message format. - `OrphanFilesCleaner` now supports primary-key tables and retains reachable index manifests, index payloads, and data-file extra files. It still does not enumerate `global-index.external-path`; snapshot expiration deletes retired external payloads by exact committed path. ### Documentation Updated `docs/source/user_guide/primary_key_global_index.rst` with automatic build and compaction maintenance, cleanup ownership, spill requirements, and the current synchronous/realtime/postpone boundaries. ### Generative AI tooling Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
