suxiaogang223 opened a new issue, #347: URL: https://github.com/apache/paimon-cpp/issues/347
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Paimon-cpp version - Apache Paimon C++ v0.3.0 (`efbfc848`). - The problem is also present on current `main` (`ccfe346670f32b82d7c9139a8f3d356c33dc5b7d`). - Java behavior used for comparison: Apache Paimon 1.4.2. ### Minimal reproduce step Create a logical schema with a top-level Variant column and configure `variant.shreddingSchema` with a valid ROW type whose fields omit every `id`, for example: ```json { "type": "ROW", "fields": [ { "name": "payload", "type": { "type": "ROW", "fields": [ {"name": "age", "type": "INT"}, {"name": "profile", "type": { "type": "ROW", "fields": [{"name": "city", "type": "STRING"}] }} ] } } ] } ``` Pass this value to `VariantShreddingWritePlan::FromConfiguredSchema` (normally through `variant.shreddingSchema` or its Parquet fallback option) and open a writer. Paimon-cpp rejects the configuration while deserializing the first `DataField`, because `DataField::FromJson` unconditionally requires the `id` key. In an embedding engine the propagated exception is equivalent to: ```text key 'id' must exist ``` The same schema is accepted by Java Paimon 1.4.2. `DataTypeJsonParser.parseDataType` starts an `AtomicInteger` at `-1` and assigns deterministic IDs in traversal order when all field IDs are omitted. ### What doesn't meet your expectations? **Expected behavior** Paimon-cpp should accept the same configured Variant shredding schema JSON as Java Paimon: 1. If every ROW field omits `id`, assign deterministic IDs in the same traversal order as Java, starting at 0. 2. If every ROW field provides `id`, preserve those IDs. 3. If IDs are only partially present, reject the schema with an explicit diagnostic such as `Partial field id is not allowed.` **Actual behavior** Paimon-cpp accepts only the second form. A valid all-ID-omitted schema accepted by Java is rejected before the Variant shredding plan can be created. This creates a cross-language compatibility gap for engines that consume table options written by Java/Spark. Such engines must currently route an otherwise supported Variant write back to the Java SDK solely because the configured schema omits optional field IDs. ### Anything else? The mismatch comes from the generic JSON parsing path: - [`VariantShreddingWritePlan::FromConfiguredSchema`](https://github.com/apache/paimon-cpp/blob/ccfe346670f32b82d7c9139a8f3d356c33dc5b7d/src/paimon/common/data/variant/variant_shredding_write_plan.cpp#L185) delegates the configured ROW JSON to `DataTypeJsonParser::ParseType`. - [`DataTypeJsonParser::ParseRowType`](https://github.com/apache/paimon-cpp/blob/ccfe346670f32b82d7c9139a8f3d356c33dc5b7d/src/paimon/common/types/data_type_json_parser.cpp#L703) deserializes every ROW child as a `DataField`. - [`DataField::FromJson`](https://github.com/apache/paimon-cpp/blob/ccfe346670f32b82d7c9139a8f3d356c33dc5b7d/src/paimon/common/types/data_field.cpp#L67) requires `id` unconditionally. A contained fix could normalize the configured schema before normal DataType parsing: recursively count ID presence across ROW fields; generate preorder IDs only when none are present; preserve the current error behavior for partial IDs. Alternatively, the generic `DataTypeJsonParser` can implement Java's all-or-none semantics if that behavior is desired for every DataType JSON consumer. Suggested regression coverage: - all IDs omitted, including nested ROW/ARRAY/MAP descendants; - all IDs explicitly present; - partially present IDs are rejected; - configured shredding write/read round trip uses the generated IDs in the physical Parquet schema. This is separate from #344: that issue concerns missing IDs in the generated physical shredded Variant subtree, whereas this issue concerns parsing IDs in the user-configured logical shredding schema. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
