SteNicholas opened a new issue, #415: URL: https://github.com/apache/paimon-cpp/issues/415
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation Java Paimon ([apache/paimon#8635](https://github.com/apache/paimon/pull/8635), with key types extended in [apache/paimon#8963](https://github.com/apache/paimon/pull/8963) and [apache/paimon#9057](https://github.com/apache/paimon/pull/9057)) and PyPaimon can write top-level `MAP<K, BLOB>` columns to dedicated `.blob` files. #283 (PR #278) adds read-only `MAP<K, BLOB>` support to Paimon C++, and #389 (PR #392) adds write support for `ARRAY<BLOB>`, but C++ table creation (`TableSchema::Create`) and writes (`FileStoreWrite::Create`) still reject `MAP<K, BLOB>` fields through `BlobUtils::ValidateContainerBlobWriteSchema`. As a result, C++ users cannot write keyed collections of multimodal objects, such as the images of one sample keyed by camera name, to a Paimon table. ### Solution Add Java-compatible write support for top-level `MAP<K, BLOB>` fields declared with `blob-field`: - **Schema validation**: Keep the rules consistent with Java and with `ARRAY<BLOB>` (#392): - allow `MAP<K, BLOB>` only as a top-level field declared with `blob-field`; - reject it for `blob-descriptor-field`, `blob-view-field`, partition keys, and tables without data evolution; - reject unsupported key types and `BLOB` values nested more deeply, such as `MAP<K, ARRAY<BLOB>>` or `ARRAY<MAP<K, BLOB>>`. - **Blob format writer**: Extend `BlobFormatWriter` to encode each `MAP<K, BLOB>` row as the nested payload defined in the [BLOB file spec](https://github.com/apache/paimon/blob/master/docs/docs/concepts/spec/fileformat.md) and decoded by the C++ reader from #278. The payload contains: - map magic `1296188226`; - version 1; - entry count; - concatenated key data; - concatenated BLOB data; - Delta-Varint key length index; - Delta-Varint BLOB length index; - key index length and BLOB index length. - **Key encoding**: Match the Java key encodings for the key types supported by the C++ reader: - `TINYINT`, `SMALLINT`, `INT`, `BIGINT`: fixed-width little-endian signed integers; - `BOOLEAN`: one byte, `0` or `1`; - `DATE`: four-byte little-endian days since epoch; - `DECIMAL(p, s)`: an eight-byte little-endian unscaled value for `p <= 18`, and the minimal-length big-endian two's-complement unscaled value for `p > 18`; - `STRING`: UTF-8 bytes; `BINARY`: raw bytes. - **Null handling**: Distinguish null maps (outer length `-1`), empty maps (entry count `0`), null values (BLOB length `-1`), and empty values (BLOB length `0`). Reject null keys and duplicate keys, as Java does ([apache/paimon#9107](https://github.com/apache/paimon/pull/9107)). - **Value sources**: Support both inline raw bytes and `BlobDescriptor` values. Descriptor values are materialized from their referenced storage when written, as Java does and as the `ARRAY<BLOB>` writer does. - **Placeholder**: Support data-evolution partial updates of `MAP<K, BLOB>` by recognizing an internal placeholder row on the write channel, aligned with Java `BlobMapPlaceholder`, and persisting it as a `-2` entry. The fallback reader from #278 then resolves these rows from older sequence layers. - **Write path**: Route `MAP<K, BLOB>` fields to dedicated blob files in the data-evolution write path, and remove the write-side rejection in `BlobUtils::ValidateContainerBlobWriteSchema`. - **Documentation**: Update `data_types.rst` and `write.rst`, which currently state that Paimon C++ can read but not write `MAP<kt, BLOB>` columns. - **Tests**: - add unit tests for payload encoding of every supported key type, null/empty cases, null/duplicate key rejection, and error handling; - add round-trip tests with the C++ reader from #278, including placeholder fallback across sequence layers; - add Java/Python cross-compatibility tests, with C++ writing and Java/Python reading. ### Anything else? - Depends on #283 / #278 for the reader side and builds on the `ARRAY<BLOB>` writer from #389 / #392. - `TIME` keys ([apache/paimon#9009](https://github.com/apache/paimon/pull/9009)), compaction of data-evolution tables, `paimon.map.selected-keys` for `MAP<K, BLOB>` reads, and `MAP<K, BLOB>` in primary-key managed BLOB storage (#204) are out of scope and can be tracked separately. - No public API changes are expected. The storage format follows the existing Java spec. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
