SteNicholas opened a new pull request, #206: URL: https://github.com/apache/paimon-cpp/pull/206
### Purpose Linked issue: close #204 Ports the two Java capabilities behind the roadmap item "Extend Data Evolution to primary-key tables and support compaction across evolved field groups" (#186): 1. **Managed BLOBs in primary-key tables** ([apache/paimon#8617](https://github.com/apache/paimon/pull/8617), [apache/paimon#9201](https://github.com/apache/paimon/pull/9201)). `PrimaryKeyBlobExternalizer` copies blob payloads into rolling `.managed.blob` packs before rows enter the merge-tree write buffer (sealed by `blob.target-file-size`, copied through the new `blob.copy-buffer-size` buffer), so data files and spills only hold small descriptors. Every data file carries a `.blobref` reference sidecar — byte-compatible with Java's `ManagedBlobReferenceFile` — in its extra files, written by `ManagedBlobReferenceCollector` and removed together with the data file everywhere companion files are collected (writer abort, uncommitted cleanup, snapshot expiration); shared packs are never deleted by table maintenance. Reads resolve descriptors back to payload bytes after merging via `ManagedBlobResolvingBatchReader` (`blob-as-descriptor` returns the raw descriptors). Schema validatio n enforces the `deduplicate`/`partial-update`/`first-row` merge engines, the managed-blob key/ordering/sequence-group rules, and rejects the unported capabilities (`data-file.external-paths`, `pk-clustering-override=true`, `blob-descriptor.source-table`). Postpone-bucket writers externalize the same way. 2. **Normal-file compaction across data-evolution field groups** ([apache/paimon#6828](https://github.com/apache/paimon/pull/6828), planning rules of [apache/paimon#9177](https://github.com/apache/paimon/pull/9177)). `DataEvolutionCompactCoordinator` groups files by exact row-id range into evolved field groups and packs contiguous runs into bins by `sum(max(file_size, source.split.open-file-cost))`; a bin becomes a task once its weight strictly exceeds `target-file-size`, a heavier-than-target group is compacted alone, a row-id gap cuts the bin, and `compaction.min.file-num` gates every task. `DataEvolutionNormalCompactTask` rewrites one run into exactly one file, preserving row ids and the merged `[min, max]` sequence-number range. `AppendCompactCoordinator::Run` routes data-evolution tables to this planner (scanning every live file instead of only small ones), rejects overrides of immutable options (row tracking, data evolution, deletion vectors, bucket, blob layout), keeps reje cting deletion-vector tables, and fails the removed legacy `data-evolution.compaction.rewrite-row-ids=true` mode like Java. Known scope cuts against current Java are documented in the user guides and in #204: `ARRAY<BLOB>`/`MAP<K, BLOB>` fields, `blob-descriptor.source-table` re-materialization, deletion-vector rewrite/materialization, the projected-manifest candidate planning with ~100k-file batches and multi-round commits, dedicated blob pack compaction, and vector-store planning (tables holding vector-store files are rejected, where Java compacts their normal files). ### Tests Unit tests: - `managed_blob_reference_file_test.cpp`: round trip with sort/dedup, Java golden-bytes interop (externally computed CRC32), corrupted/truncated/oversized-count/trailing-bytes/malformed-low-surrogate rejection, descriptor-URI parsing. - `managed_blob_reference_collector_test.cpp`: reference collection, non-descriptor values ignored, empty sidecar, abort, unknown field. - `primary_key_blob_externalizer_test.cpp`: descriptor replacement, retract drops payload, pack handover on `PrepareCommit`, rolling by target size, inline fields untouched, descriptor re-materialization, explicit close-on-seal-failure assertion via a close-tracking stream, `TestIOException` fault-injection sweep. - `managed_blob_resolving_batch_reader_test.cpp`: descriptor resolution with nulls, non-descriptor rejection, pass-through. - `data_evolution_compact_coordinator_test.cpp`: bin packing (target size, row-id gaps, heavy groups, min-file-num, per-partition planning), blob-file exclusion, vector-store rejection, mismatched-range rejection, planner and `Task::Create` input validation. - `schema_validation_test.cpp`: primary-key managed BLOB rules (merge engines, blob target size, key/sequence-field/external-paths rejections incl. empty string, sequence groups and aggregate functions, `pk-clustering-override` true/false, `blob-descriptor.source-table`). - `merge_tree_writer_test.cpp` / `postpone_bucket_writer_test.cpp`: sidecar lifecycle for committed and aborted flows, managed-blob `TestManagedBlobIOException` sweep. - `core_options_test.cpp`, `blob_format_writer_test.cpp`, `blob_writer_builder_test.cpp`, `file_store_commit_impl_test.cpp`: `blob.copy-buffer-size` parsing/bounds/pass-through/tiny-buffer chunked copy, `pk-clustering-override` commit semantics. Integration tests: - `pk_blob_table_inte_test.cpp` (new): write/read with payload and descriptor modes, compaction rebuilding exact blob references, first-row and partial-update merge engines, delete, snapshot expiration removing sidecars while shared packs survive. - `data_evolution_table_test.cpp`: compaction across evolved field groups (single and multi-partition, partition filter), rejection paths (DV tables and DV override, immutable-option overrides, legacy rewrite-row-ids), compaction across real schema evolution (INT→BIGINT type change, added column with cross-schema partial fill, dropped column), the three Java concurrency scenarios (stale compact message preserving a concurrent partial update, small-file compact conflicting with a concurrent partial update, concurrent append kept for the next merge round). - `blob_table_inte_test.cpp`: data-evolution compaction leaves dedicated `.blob` files in place and payloads readable. The tests were authored against the Java behavior and existing C++ read-path guarantees; validation relies on this PR's CI run. ### API and Format - New public options in `include/paimon/defs.h`: `blob.copy-buffer-size` (default `4 kb`, Java-aligned; the previous fixed copy buffer was 1 MB), `pk-clustering-override` (only `true` is rejected now; an explicit `false` is accepted by the commit path), `blob-descriptor.source-table` (rejected for primary-key managed BLOB tables), `data-evolution.compaction.rewrite-row-ids` (rejected when `true`). - `AppendCompactCoordinator::Run` (public API) now plans data-evolution tables across evolved field groups and rejects immutable-option overrides. - Storage format: introduces the `.blobref` sidecar (byte-compatible with Java, locked by a golden-bytes test) carried in `DataFileMeta.extra_files`, and `.managed.blob` pack files in the blob file format. No existing format is changed. ### Documentation - `docs/source/user_guide/primary_key_table.rst`: new "Managed BLOB Storage" section (behavior, restrictions, lifecycle, unsupported capabilities). - `docs/source/user_guide/compaction.rst`: new "Data-Evolution Table Compaction" section (planning rules, row-id preservation, scope cuts against Java). - `docs/source/user_guide/read.rst`: deletion-vector compaction note updated. ### Generative AI tooling Generated-by: Claude Code (Claude Opus 5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
