suxiaogang223 opened a new issue, #344: URL: https://github.com/apache/paimon-cpp/issues/344
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Paimon-cpp version - Apache Paimon C++ v0.3.0 (`efbfc848`) - The problem is also present on current `main` (`21228d92b30cc7e987239f0dbca4371120bdfb0b`). - Reader used to reproduce the failure: Apache Paimon Java 1.4.2. ### Minimal reproduce step 1. Create a Paimon table containing a `VARIANT` column. 2. Write at least one data file with paimon-cpp and enable Variant shredding, either with a configured shredding schema or with schema inference. A minimal typed projection such as the following is sufficient: ```text payload VARIANT typed_value ROW<age INT, city STRING> ``` 3. Inspect the Parquet schema written by paimon-cpp. The outer Paimon table field has a field ID, but the generated Variant physical fields (`metadata`, `value`, `typed_value`) and descendants of `typed_value` do not have Parquet field IDs. 4. Read the file through Apache Paimon Java 1.4.2, for example from Spark. The Java reader fails while constructing the Variant read plan, before reading any row: ```text java.lang.NullPointerException: Cannot invoke "org.apache.parquet.schema.Type$ID.intValue()" because the return value of "org.apache.parquet.schema.Type.getId()" is null at org.apache.paimon.format.parquet.ParquetSchemaConverter.convertToPaimonField(ParquetSchemaConverter.java:386) at org.apache.paimon.format.parquet.ParquetSchemaConverter.convertToPaimonField(ParquetSchemaConverter.java:406) at org.apache.paimon.format.parquet.VariantUtils.variantFileType(VariantUtils.java:49) ``` The existing `VariantParquetTest.ShreddedWriteAndReadRoundTrip` can also expose the problem by opening the raw Parquet footer and asserting that every generated field in the shredded Variant subtree has a field ID. The current C++-write/C++-read round trip succeeds because it does not verify Java interoperability or these footer IDs. ### What doesn't meet your expectations? **Expected behavior** A shredded Variant file written by paimon-cpp should be readable by the corresponding Java Paimon implementation. Its generated physical schema should carry deterministic Paimon/Parquet field IDs compatible with the schema produced by the Java writer. **Actual behavior** paimon-cpp can write and read the file itself, but Java Paimon cannot open it. `ParquetSchemaConverter` recursively converts the `typed_value` subtree and requires every converted Parquet field to have an ID. It dereferences a null `Type.getId()` for the first generated field without one. The source of the mismatch appears to be: - [`VariantShreddingSchemaImpl`](https://github.com/apache/paimon-cpp/blob/21228d92b30cc7e987239f0dbca4371120bdfb0b/src/paimon/common/data/variant/variant_shredding_utils.cpp) creates `metadata`, `value`, `typed_value`, nested object fields, and list elements with plain `arrow::field(...)`, without `paimon.id` metadata. - [`VariantShreddingWritePlan`](https://github.com/apache/paimon-cpp/blob/21228d92b30cc7e987239f0dbca4371120bdfb0b/src/paimon/common/data/variant/variant_shredding_write_plan.cpp) installs this generated physical type while preserving only the outer table field metadata. - [`ParquetFieldIdConverter`](https://github.com/apache/paimon-cpp/blob/21228d92b30cc7e987239f0dbca4371120bdfb0b/src/paimon/format/parquet/parquet_field_id_converter.cpp) copies an existing `paimon.id` to `PARQUET:field_id`; it does not assign IDs when the generated Arrow field has none. Configured shredding, inferred per-file shredding, and adaptive shredding all use this physical-schema construction path, so all modes can produce files that Java Paimon cannot read when `typed_value` is present. This blocks cross-language interoperability. For example, an engine using paimon-cpp for native writes currently has to fall back to the Java writer for shredded Variant data, otherwise Java/Spark readers cannot consume the resulting files. ### Anything else? The ordinary unshredded Variant layout does not have this problem: [`UnshreddedStructType`](https://github.com/apache/paimon-cpp/blob/21228d92b30cc7e987239f0dbca4371120bdfb0b/src/paimon/common/data/variant/variant_type_utils.cpp) explicitly assigns IDs `0` and `1` to `value` and `metadata`. Although Parquet field IDs are optional in the file-format specification and Variant physical members are identified by name, the current Java Paimon reader uses its generic Parquet-to-Paimon schema converter for `typed_value`, and that converter requires IDs. Therefore the file is not interoperable with the Java implementation that paimon-cpp targets. Suggested fix and regression coverage: 1. Assign deterministic `paimon.id` metadata to all generated shredded Variant fields, including nested object fields and list/map descendants, matching Java Paimon's physical schema conventions. 2. Ensure the IDs survive configured, inferred, and adaptive shredding paths and are emitted as Parquet field IDs. 3. Extend the C++ Parquet test to assert the raw footer IDs for a nested shredded schema. 4. Add a cross-language regression test that writes a shredded Variant file with paimon-cpp and reads it with Java Paimon (and ideally the reverse direction). ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
