JingsongLi opened a new pull request, #9459: URL: https://github.com/apache/paimon/pull/9459
## Purpose\n\nSupport append-only frame tables where many logical rows reference one physical video payload, while keeping the ordinary .blob format unchanged.\n\n## Changes\n\n- Add an independent, versioned .shared-blob format with physical payload lengths and a row-reference index.\n- Preserve contiguous descriptor groups across file rolling and rebuild sharing during compaction.\n- Validate blob-shared-field for one scalar BLOB field on append-only data-evolution tables.\n- Add PyPaimon Blob and BlobDescriptor writes, batched ingestion, descriptor-only ScanQuery.to_torch, and a worker-local VideoFrameCollator decoder cache.\n- Document the storage format, PyPaimon ingestion, and PyTorch DataLoader decoding workflow.\n\n## Compatibility\n\nOrdinary BLOB columns continue to use the existing .blob format. Shared storage is opt-in and uses separate .shared-blob files.\n\n## Verification\n\n- Java shared/ordinary format tests: 95 passed.\n- Core schema, path, and integration tests: 70 passed.\n- CoreOptions shared-BLOB test: 1 passed.\n- Related Python suite: 91 passed, 2 skipped.\n- compileall, flake8, Spotless/static checks, and git diff checks passed.\n- A real PyTorch multi-worker integration test is included; it was not run locally because torch is not installed in the local environment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
