SteNicholas opened a new issue, #389: URL: https://github.com/apache/paimon-cpp/issues/389
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation Java Paimon ([apache/paimon#8181](https://github.com/apache/paimon/pull/8181)) and PyPaimon can write top-level `ARRAY<BLOB>` columns to dedicated `.blob` files. #387 (PR #388) adds read-only `ARRAY<BLOB>` support to Paimon C++, but C++ table creation, writes, and append compaction still reject these fields. As a result, C++ users cannot write ordered collections of multimodal objects, such as the frames or images of one sample, to a Paimon table. ### Solution Add Java-compatible write support for top-level `ARRAY<BLOB>` fields declared with `blob-field`: - **Schema validation**: Match Java `SchemaValidation` and `ColumnDirectiveUtils`: - allow `ARRAY<BLOB>` only as a top-level field declared with `blob-field`; - reject it for `blob-descriptor-field`, partition keys, and tables without data evolution; - reject `BLOB` elements nested more deeply, such as `ARRAY<ARRAY<BLOB>>`. - **Blob format writer**: Extend `BlobFormatWriter` to encode each `ARRAY<BLOB>` row as the nested payload defined in the [BLOB file spec](https://github.com/apache/paimon/blob/master/docs/docs/concepts/spec/fileformat.md). The payload contains: - array magic `1094861634`; - version 1; - element count; - concatenated element data; - Delta-Varint element length index; - index length. - **Null handling**: Distinguish null arrays (outer length `-1`), empty arrays (element count `0`), null elements (element length `-1`), and empty elements (element length `0`). - **Element sources**: Support both inline raw bytes and `BlobDescriptor` elements. Descriptor elements are materialized from their referenced storage when written, as Java does. - **Write path**: Route `ARRAY<BLOB>` fields to dedicated blob files in the data-evolution write path, and remove the write-side rejection for them. - **Compaction**: Support `ARRAY<BLOB>` in append and data-evolution compaction, including placeholder (`-2`) handling across field groups. - **Tests**: - add unit tests for payload encoding, null/empty cases, and error handling; - add round-trip tests with the C++ reader from #387; - add Java/Python cross-compatibility tests, with C++ writing and Java/Python reading. ### Anything else? - Depends on #387 / #388 for the reader side. - `MAP<K, BLOB>` write support and `ARRAY<BLOB>` in primary-key managed BLOB storage (#204) are out of scope and can be tracked separately. - No public API changes are expected. The storage format follows the existing Java spec. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
