SteNicholas opened a new pull request, #416: URL: https://github.com/apache/paimon-cpp/pull/416
### Purpose Linked issue: close #415 #278 taught the C++ reader to read top-level `MAP<K, BLOB>` fields and #392 taught the C++ writer to write `ARRAY<BLOB>` fields, but tables with a `MAP<K, BLOB>` column were still rejected on creation and write. This PR makes the C++ writer store `MAP<K, BLOB>` fields in dedicated blob files, aligned with Java Paimon's `MapBlobElementSerializer` (apache/paimon#8635, with key types extended in apache/paimon#8963 and apache/paimon#9057, and duplicate keys rejected since apache/paimon#9107): - **Schema**: `TableSchema::Create` and `FileStoreWrite::Create` accept `MAP<K, BLOB>` fields whose key type is BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, DATE, DECIMAL, CHAR, VARCHAR, BINARY or VARBINARY, the key types the C++ reader supports. Other key types, including TIME, are still rejected. Blob file routing, the data-evolution and partition-key requirements and the nesting rules already apply to every blob-file field. - **Schema import**: Arrow's C schema bridge drops the metadata of a map value field, so `TableSchema::Create` restores the BLOB marker on a `LARGE_BINARY` map value, which can only be a BLOB since Paimon BYTES is `BINARY`. - **Encoding**: each row is written as the nested payload of the BLOB file spec: map magic, version, entry count, key data, value data, delta-varint key and value lengths, and both index lengths. Keys are encoded as in Java: fixed-width little-endian integers and dates, one byte for a boolean, the unscaled long of a compact decimal or its minimal big-endian two's-complement unscaled bytes otherwise, and the raw bytes of a string or binary. A null map is a `-1` entry in the file index, an empty map has an entry count of zero, and a null value has length `-1`. - **Keys**: as in Java, the keys of a row are serialized and checked to be unique by their bytes before any byte of its entry is written, so an invalid key fails the write without leaving partial data. A null key is also rejected, since Arrow map keys cannot hold the one null key Java allows, and so are a decimal key exceeding its precision and a string key that is not valid UTF-8, which the C++ reader would reject. - **Values**: a value is copied like an `ARRAY<BLOB>` element, from raw bytes or a serialized `BlobDescriptor`, through the shared element copy path, so source stream reuse, `blob-write-null-on-missing-file`, `blob-write-null-on-fetch-failure` and their metrics apply per value: only the unreachable value becomes NULL, not the map. - **Placeholders**: a map of exactly two entries with equal keys and null values, the marker the reader from #278 emits for a placeholder entry, is persisted as a placeholder entry (`-2`) in every write, as Java persists `BlobMapPlaceholder`. Since every other map must have unique keys, this marker cannot collide with a user value. - **Error context**: key errors, value errors and NULL warnings name the entry, the field and its row in the blob file. Differences from Java, also documented in `write.rst`: - A placeholder is the two-entry map marker above; Java uses a dedicated placeholder object. - A key cannot be null, as Arrow map keys are not nullable, and cannot be TIME; Java allows one null key and TIME keys. - As for an `ARRAY<BLOB>` element, a missing file is detected with `FileSystem::Exists`. Java detects a missing file for a map value only from an HTTP 404, and does not write NULL for a 404 under `blob-write-null-on-fetch-failure` alone. ### Tests Blob format writer (`blob_format_writer_test.cpp`): - `BlobFormatWriterMapBlobTest.TestMapBlobGoldenBytes`: the written bytes for an empty map; a map with an empty key, an empty value, inline and descriptor values and a null value; a null map; and a placeholder. Java's `BlobFormatWriterTest#testMapBlobGoldenBytes` uses a null key, so these bytes use a non-null key in its place. - `BlobFormatWriterMapBlobTest.TestMatchesJavaWrittenFiles`: for every supported key type and for placeholder rows, the written file is byte-identical to the blob file Java wrote for the same rows in the `map_blob_java` fixture. - `BlobFormatWriterMapBlobTest.TestMapBlobRoundTrip`: null and empty maps, empty keys and values, null values, a descriptor slice and a value larger than the copy buffer round trip, read back as data and as descriptors. - `BlobFormatWriterMapBlobTest.TestMapBlobPlaceholder`: only a two-entry map with equal keys and null values becomes a placeholder; a strict reader rejects it and a placeholder-aware reader emits the marker. - `BlobFormatWriterMapBlobTest.TestInvalidKeysFailWriteWithoutPartialData`: duplicate keys, including near-miss placeholders, an invalid UTF-8 key, a null key and a decimal key exceeding its precision fail the write with the key named and leave no partial data. - `BlobFormatWriterMapBlobTest.TestWriteNullOnUnreachableValues`: a missing descriptor fails the write with the value named, or becomes a NULL value counted by the metric under `blob-write-null-on-missing-file`. - `BlobFormatWriterMapBlobTest.TestMapBlobOutputFailureFailsWrite`: write failures and short writes at each stage of an entry, and flush failures deferred to `Finish()`. - `BlobFormatWriterTest.TestCreateWithInvalidParameters`: maps of plain binaries and maps with unsupported key types are rejected. Schema, write and compaction: - `BlobUtilsTest.ValidateContainerBlobWriteSchema` and `BlobUtilsTest.IsSupportedMapBlobKeyType`: supported and unsupported key types. - `TableSchemaTest.CreatingMapBlobSchemaIsAllowed`: the field stays `MAP<..., BLOB>` through JSON, including after the C schema bridge dropped the value metadata; `TableSchemaTest.CreatingMapBlobSchemaWithUnsupportedKeyTypeIsRejected`. - `FileStoreWriteTest.TestCreateWriterForLoadedMapBlobTableWithUnsupportedKey`: a loaded table with a TIME key is rejected on write. - `AppendCompactCoordinatorTest.TestValidateFailsOnContainerBlobTable`: compaction rejects a table with `ARRAY<BLOB>` and `MAP<K, BLOB>` columns. Integration (`blob_table_inte_test.cpp`, parameterized over the data file formats): - `BlobTableInteTest.TestMapBlobWriteAndPartialUpdate`: a write read back, then two partial updates with placeholder rows, a null map and descriptor values, including a missing one written as NULL; the table stays readable after the source files are deleted, as data and as descriptors. ### API and Format No public symbol is added or changed. `include/paimon/defs.h` documents that the blob write-null options apply per `MAP<K, BLOB>` value. The storage format is unchanged: the writer produces the `MAP<K, BLOB>` payload of the existing Paimon BLOB file spec, which the C++ reader reads since #278. The map magic number and version move to `BlobDefs`, shared by the reader and the writer. ### Documentation - `docs/source/user_guide/data_types.rst` documents the `MAP<kt, BLOB>` type, its key types and table requirements, and no longer says Paimon C++ cannot write it. - `docs/source/user_guide/write.rst` covers building `MAP<kt, BLOB>` columns, the accepted value forms, the key rules, the two-entry placeholder marker with an `arrow::MapBuilder` example, and the differences from Java. ### Generative AI tooling Generated-by: Claude Code (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
