Stephen0421 opened a new pull request, #9099: URL: https://github.com/apache/paimon/pull/9099
### Purpose Add PyPaimon support for BLOB storage on **primary-key tables**, aligned with Java managed-BLOB semantics. **Write path** - Externalize `blob-field` payloads (`BLOB`, `ARRAY<BLOB>`, `MAP<K,BLOB>`) into immutable `.managed.blob` packs before they enter the MergeTree sort buffer, via `PrimaryKeyBlobExternalizer`. - Emit per-data-file `.blobref` sidecars that reference the managed packs used by that file (`ManagedBlobReferenceFile` / collector). - Keep `blob-descriptor-field` and `blob-view-field` **inline** in the normal data file (scalar only); integrate with `DedicatedFormatWriter` and descriptor/source-table input via `BlobDescriptorReaderFactory` (`blob-descriptor.*`, `blob-descriptor.source-table`). - Enforce PK BLOB schema constraints (`merge-engine=deduplicate`, `changelog-producer=none`, key usage, inline vs managed field rules) in `schema_manager`. - Harden write lifecycle: staged-commit ownership/abort, `preserve_blob_files_on_abort` when a `BlobConsumer` retains packs, and stream dynamic-bucket HASH-index refresh across multiple checkpoints (`StreamWriteBuilder` ↔ `with_dynamic_bucket_index()` callback wiring). **Read path** - Resolve managed BLOB columns through `managed_blob_convert_record_reader` on raw/merge PK scans; support `blob-as-descriptor` and projected fields. - Fix v1 `BlobDescriptor` read paths (`from_descriptor_bytes`, descriptor/view inline resolution). - Extend `BlobViewLookup` for PK `blob-view-field` resolution and resource lifecycle. **Safety** - Default managed-BLOB write treats input as **inline bytes** unless `force_descriptor_bytes` (v2 magic or explicit descriptor contract); avoids v1 heuristic false positives on the hot path. ### Test **New / extended test modules** - `primary_key_blob_e2e_test.py` — PK managed BLOB E2E: scalar/array/map round-trip, dedup, `.blobref`, source-table copy, close/abort lifecycle - `primary_key_blob_inline_compat_test.py` — inline `blob-descriptor-field` / `blob-view-field` on PK raw & merge reads, projection, `blob-as-descriptor`, v1 managed resolution - `primary_key_blob_externalizer_test.py` — externalizer schema preservation, v1 descriptor with `force_descriptor_bytes`, inline collision guard - `managed_blob_write_ownership_test.py` — staged-commit ownership, abort/close, committer abort preserving shared packs, descriptor reader factory (static / source-table / v1) - `managed_blob_reference_file_test.py`, `managed_blob_lifecycle_test.py` - `test_primary_key_blob_read_contract.py` — read routing, `_ConvertedRow.get_blob`, v1 descriptor field, predicate validation - `test_file_store_write.py` — managed vs inline field routing into writers - `dynamic_bucket_test.py` — stream writer multi-commit HASH-index refresh (write-before-commit & commit-before-write) - `blob_test.py`, `blob_table_test.py`, `schema_manager_test.py`, `data_evolution_formats_test.py` — regressions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
