suxiaogang223 opened a new pull request, #66321:
URL: https://github.com/apache/doris/pull/66321
### What problem does this PR solve?
Issue Number: Part of #65086
Problem Summary:
The Paimon JNI writer already supports primitive and complex Doris columns,
but a Doris Variant V2 value could not be written losslessly because the
generic Arrow path exposed Variant as JSON text. That loses the native Variant
`value` and `metadata` representation and does not support Variant nested in
ARRAY, MAP, or STRUCT.
This PR adds a Paimon-specific Variant V2 write protocol:
- maps Paimon VARIANT targets, including nested VARIANT nodes, to Doris
compute Variant V2 during analysis;
- requires `enable_variant_v2=true` and rejects legacy Variant V1 inputs
during analysis with an actionable error;
- transports Variant V2 through Arrow as `struct<value: binary, metadata:
binary>` while preserving SQL NULL separately from a Variant null value;
- recursively applies the binary representation inside ARRAY, MAP, and
STRUCT;
- converts the Arrow pair to Paimon `GenericVariant` in the Java JNI backend
and uses Paimon SDK accessors to validate compatibility with the bundled Paimon
version;
- keeps the existing JSON Arrow representation as the default for non-Paimon
consumers.
#### Supported write scenarios
- `INSERT INTO ... VALUES` and `INSERT INTO ... SELECT`, including UNION
inputs and reordered/omitted columns.
- `INSERT OVERWRITE` and static partition writes.
- Append-only and primary-key tables, fixed and dynamic bucket modes, and
schema evolution involving VARIANT columns.
- Top-level VARIANT and VARIANT nested in ARRAY, MAP, STRUCT, and deeper
combinations.
- Scalar/object/array values, SQL NULL, Variant null, typed primitives, long
strings, and residual object fields.
- Paimon Parquet Variant shredding. Regression coverage verifies the
physical `typed_value` layout and values, Paimon unshredding, type-mismatch
fallback to residual bytes, and reading a table containing both unshredded and
shredded files.
#### Current limitations
- Variant writes are V2-only. When `enable_variant_v2=false`, or when a
legacy Variant V1 expression is supplied to a Paimon VARIANT target, analysis
fails intentionally.
- The write path uses the Java Paimon JNI backend.
- Doris-side querying of Paimon Variant V2 is outside this PR. End-to-end
logical readback is therefore verified with the Spark/Paimon reader; raw
Parquet shredding is independently verified through Doris's S3 TVF.
- The added Paimon integration coverage uses Parquet, which is also the
format used to validate Variant shredding.
### Release note
Support writing Variant V2 values, including nested Variant values, from
Doris into Apache Paimon tables through the Java JNI writer. Set
`enable_variant_v2=true` before writing a Paimon table containing VARIANT.
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [x] Manual test
- Debug FE and BE incremental builds.
- `./run-regression-test.sh --run -s
test_paimon_write_variant_shredding`
- `./run-regression-test.sh --run -s
test_paimon_write_variant_table_modes`
- Behavior changed:
- [ ] No.
- [x] Yes. Paimon VARIANT targets now accept Variant V2 writes when
enabled and reject disabled/V1 writes during analysis.
- Does this need documentation?
- [ ] No.
- [x] Yes. User-facing documentation can follow separately.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
