JingsongLi opened a new pull request, #8617: URL: https://github.com/apache/paimon/pull/8617
## Summary Add table-managed BLOB storage for primary-key tables without coupling payload bytes to data-file row positions. Raw BLOB values are externalized before MergeTree buffering, while compaction preserves stable descriptors and derives exact payload references for each output data file. ## Changes - Externalize raw primary-key BLOB values into immutable `.managed.blob` packs before sorting and preserve existing descriptors without copying payloads. - Attach one versioned and checksummed `.blobref` sidecar to every managed-BLOB data file through `DataFileMeta.extraFiles`, including files with no managed references. - Rebuild exact sidecar references from final rows after deduplication and compaction, without rewriting surviving BLOB payloads. - Restrict the initial mode to descriptor fields with the `deduplicate` merge engine and `changelog-producer=none`, and reject incompatible key, view, external-path, and PK-clustering configurations. - Keep `.managed.blob` payloads out of ordinary orphan cleanup until a root-aware garbage collector is implemented, while retaining normal lifecycle handling for `.blobref` sidecars. - Document configuration, compaction behavior, metadata ownership, limitations, and the deferred GC safety boundary. ## Testing - [x] Focused core tests covering schema validation, sidecar format and corruption detection, externalization, data-file lifecycle, compaction, and orphan cleanup (52 tests). - [x] Compile `paimon-core`, `paimon-flink-common`, and `paimon-spark-common` with dependent modules. - [x] Run Spotless checks for the affected modules. - [x] Build the Docusaurus documentation site (the repository's existing `concepts/overview` link warnings remain). - [x] Run `git diff --check`. ## Notes - Garbage collection of unreachable `.managed.blob` packs is intentionally deferred. The current orphan cleaner preserves these files to avoid deleting payloads reachable from snapshots, tags, branches, or other retained roots. - This change does not modify snapshot or `IndexManifest` schemas. BLOB reachability is modeled as an exact per-data-file dependency through `.blobref` sidecars. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
