morningman opened a new pull request, #69194:
URL: https://github.com/apache/doris/pull/69194
### What problem does this PR solve?
Issue Number: None
Related PR: #66620, #67323
Problem Summary:
**In short.** Ports branch-4.1 #66620 (Iceberg Variant V2 writes) and #67323
(NULL defaults for VARIANT) to master. On master Doris reads Iceberg VARIANT
columns but rejects every write to a table that has one, and rejects VARIANT in
Iceberg DDL. With this PR Doris writes VARIANT values to format-version 3
Iceberg tables with Parquet data files: top-level and nested values, through
INSERT, INSERT OVERWRITE, UPDATE, MERGE and `rewrite_data_files`. It also
creates and alters top-level VARIANT columns. For every connector sink,
including Paimon, a write is rejected when its source implicitly casts a
VARIANT to a scalar. Before this PR such a write silently stored object and
array values as NULL.
Original author: @hubgeter
**Background.** Iceberg added the VARIANT type in format version 3 and
stores it in Parquet as a group of two binary leaves, `metadata` and `value`.
Master already reads such columns (#66413). For Paimon it already encodes
Variant V2 values to Arrow (#68320, #68883), and it checks the nested shapes a
connector Variant accepts in `CheckCast`. The Iceberg side of the write path
was never ported. Two other parts of #66620 were also missing:
- the plan validator that rejects lossy common-type coercion into a VARIANT
target;
- the binding of a merge sink's output by position.
**The problem, and what it cost.**
| Statement on master | Result |
|---|---|
| `INSERT` / `UPDATE` / `MERGE` (with data) / `rewrite_data_files` on an
Iceberg table with a VARIANT column, at any depth | `Iceberg VARIANT columns
are read-only and cannot be written` |
| `CREATE TABLE ... (payload VARIANT) ENGINE=ICEBERG`, `ADD COLUMN ...
VARIANT`, `MODIFY COLUMN ... VARIANT` | `Unsupported type for Iceberg: VARIANT`
|
| `INSERT INTO paimon_t SELECT IF(TRUE, payload, 1) ...`, `UNION ALL` of a
Variant and a number, the same in `CASE`, a recursive CTE or `LATERAL VIEW` |
The common type is DECIMAL, so an object or array `payload` is written as NULL
with no error |
| `UPDATE iceberg_t SET operation = ...` on a table whose data column is
named `operation` | Merge sink outputs were matched by name, so this column
skipped the cast to its target type |
Even if the FE had let these writes through, the BE could not complete them:
- The BE could not parse `variant` in the Iceberg schema JSON.
- It had no Arrow schema mapping for VARIANT.
- It would have published the Parquet statistics of the two Variant leaves
under the field id `-1`.
**How this PR fixes it.**
- **BE.**
- The Iceberg type system parses `variant`.
- The Iceberg Arrow schema maps VARIANT to the `arrow.parquet.variant`
extension over `struct<metadata: binary, value: binary>`. Master's Variant
encoder already fills it (#68883).
- The Iceberg Parquet writer counts a Variant's values and nulls on the
`metadata` leaf, under the Variant's own field id. It records no column size
and no bounds for the Variant.
- The format v2 Iceberg reader rejects a non-NULL VARIANT initial default.
- **FE, Iceberg connector.**
- A top-level VARIANT maps to Iceberg VARIANT. A VARIANT nested in STRUCT,
ARRAY or MAP DDL is rejected.
- CREATE TABLE and CTAS need an effective format version of 3 and Parquet
data files. The format version comes from the table properties and the
catalog's `table-default` / `table-override`, and so does the file format, with
the `write-format` alias winning as it does for an existing table.
- ADD COLUMN needs the same table settings. MODIFY of a VARIANT column
changes only its comment, nullability and position, so it also works on v3 ORC
tables. A change to or from VARIANT is rejected.
- Writes of data files whose target holds a VARIANT now need format
version 3, Parquet, and no `write.parquet.shred-variants=true`. This replaces
the read-only check. DELETE and delete-only MERGE are unchanged.
- Column metrics are also collected for VARIANT fields.
- VARIANT write defaults, initial defaults and DDL defaults must be NULL
(#67323).
- **FE, engine.** `VariantWritePlanValidator` traces the lineage of each
VARIANT target column through projections, set operations, recursive CTEs,
generators, CTE consumers and scalar subqueries. It rejects an implicit cast
that turns a Variant into a non-Variant value. It is wired once for every
connector:
- in `BindSink`, for INSERT, INSERT OVERWRITE, CTAS, rewrite, and
changelog row-level DML (Paimon UPDATE / MERGE);
- in `BindExpression`, for the row-level merge sink (Iceberg UPDATE /
MERGE).
The merge sink now casts its data columns by position.
- **Differences from branch-4.x.**
- The validator's message has no `Iceberg` / `Paimon` prefix, because one
generic check serves both.
- It does not inspect the condition of an IF or of a CASE branch, or
columns read only by such a condition. Branch-4.x rejects `IF(payload['kind'] =
'event', payload, NULL)`, because comparing a Variant path with a string
implicitly casts the path, even though the written value is unchanged.
- Paimon UPDATE and MERGE are validated too. Branch-4.x validates only
Paimon INSERT.
- A Parquet leaf without a field id now fails a check (`DORIS_CHECK_GE`)
instead of being skipped. Every field of the schema the file is written with
has one.
- **Not ported.** Legacy Variant handling and the `enable_variant_v2` switch
(master is V2 only), the VALUES and MERGE action special cases (master already
casts each value and each branch to the target type), and the extension and
Arrow pool helpers (master has equivalents).
**Results.** These are from a local single-BE cluster with the Iceberg REST
and Paimon docker environments. The tables were created by Spark 4.0 with
Iceberg 1.10.1, or by Doris DDL.
| Case | master | with this PR |
|---|---|---|
| `CREATE TABLE ... (payload VARIANT) ENGINE=ICEBERG` with `format-version`
3 and Parquet | `Unsupported type for Iceberg: VARIANT` | created. With format
version 2: `Iceberg VARIANT writes require table format-version 3, but found
2`. With ORC: `... require Parquet data files, but found ORC`. `STRUCT<...,
payload: VARIANT>`: rejected |
| INSERT of 13 values into that table (objects, arrays, primitives, JSON
null, SQL NULL, `"null"`, `""`, `{}`, `[]`) | read-only | written. Spark reads
each value back through `variant_get` and `to_json`, with SQL NULL and JSON
null kept apart. The data files report 13 values, 1 null, and no bounds for the
column |
| One writer inserts 100,000 rows into `STRUCT<label, payload VARIANT>` |
read-only | One data file with several row groups. Spark's `.files` shows
100,000 values and 10,000 nulls under the nested field id, with no column size
and no bounds |
| Variant nested in STRUCT, ARRAY and MAP, and four levels deep | read-only
| Written. Spark checks the container shapes, Doris reads the nested values
back |
| `UPDATE` / `MERGE` (with `WHEN NOT MATCHED INSERT`) on a Variant column
named `operation` | read-only | Written, and Spark reads the new values |
| MERGE that keeps a Variant in one branch and inserts the number `1` in
another | read-only | The object stays intact and the number is written as the
Variant `1` |
| A table with `write.parquet.shred-variants=true`, an ORC v3 table |
read-only | Rejected with no new snapshot. `MODIFY COLUMN v VARIANT COMMENT
...` on the ORC table still works |
| INSERT into Paimon of `UNION ALL` of a Variant and `1`, `IF(TRUE, payload,
1)`, the same in `CASE`, a recursive CTE, `LATERAL VIEW` | Objects and arrays
are written as NULL (master's own suite asserted this) | All 8 rejected:
`VARIANT write cannot safely convert input column 'payload': source expression
implicitly casts VARIANT to DECIMALV3(38, 9) before sink analysis`. The same 8
statements against Iceberg are rejected too |
| Paimon `UPDATE ... SET payload = IF(TRUE, payload, 1)`, a MERGE whose
source is such a UNION, and the same UPDATE / MERGE on Iceberg | — | Rejected |
| `IF(payload['kind'] = 'object', ..., payload)` in a Paimon UPDATE, and
`CASE WHEN payload['kind'] = ... THEN payload END` in an INSERT | — | Written
(branch-4.x rejects these) |
| Iceberg table that gained a VARIANT column through Spark: the column
omitted, `DEFAULT`, `DEFAULT(payload)` in UPDATE, MERGE and `INSERT ... SELECT`
| read-only | Every such row reads NULL. `ADD COLUMN ... VARIANT DEFAULT NULL`
works. A non-NULL default and `NOT NULL DEFAULT NULL` are rejected, and the
schema is left unchanged |
- **BE UT.** 10 cases are ported from branch-4.x, and
`IcebergBuildsItsOwnSchemaAndMetadata` goes back to its Variant form:
- `ArrowSchemaUtilTest.test_variant_field` / `test_nested_variant_fields`
- `SchemaParserTest.parse_variant`
- `VariantTypeTest.to_string`
- `PrimitiveTypesTest.from_primitive_string`
- `VParquetWriterTest.WritesIcebergVariantAndCollectsLogicalMetrics` /
`WritesNestedIcebergVariant`
- `IcebergV2ReaderTest.AnnotateVariantInitialDefaultMustBeNull` /
`StructInitialDefaultKeepsVariantChildNull`
All 474 cases of the related suites pass under ASAN. Two mutations each
turn only their own two cases red: dropping the Variant metrics rule, and
accepting a non-NULL initial default. clang-tidy reports nothing on the changed
files.
- **FE UT.**
- `VariantWritePlanValidatorTest`: the 13 branch-4.x cases, plus two new
ones. One shows that IF and CASE conditions are not checked. The other shows
that columns read only by a condition are not traced.
- `BindExpressionTest`: the positional merge-output cast, and the
validator on a Variant column named `operation`.
- `ExternalRowLevelMergePlanBuilderTest`: each MERGE branch is cast to the
Variant target before the branch is selected.
- Connector tests:
- `IcebergWritePlanProviderTest`: the write capability matrix; a full
`planWrite` of a v3 Parquet, ORC and shredding table; VARIANT write defaults.
- `IcebergSchemaBuilderTest`: top-level and nested VARIANT DDL, the
effective file format precedence, and DDL defaults.
- `CatalogBackedIcebergCatalogOpsColumnEvolutionTest`: ADD COLUMN on v2,
ORC and Parquet tables, metadata-only MODIFY, and rejected type changes.
- `IcebergConnectorMetadataDdlTest`: CREATE TABLE with VARIANT.
- `IcebergSchemaUtilsTest`: the initial default.
Mutations turn their own cases red:
- in the validator, inspecting conditions, and tracing condition columns;
- in the connector, collecting metrics only for primitive fields, dropping
the CREATE check, dropping the ADD COLUMN check, and dropping the VARIANT
MODIFY branch.
All 1580 fe-connector-iceberg cases pass, and so do 131 cases in 12
fe-core classes, including the existing sink-binding, changelog and merge
builders.
- **Regression.**
- `test_iceberg_variant_read` carries the branch-4.x write block, without
the legacy-Variant cases and the `enable_variant_v2` switch, which master does
not have. Its `.out` gains rows 3–5 of `variant_nested_projection` and
`variant_nested_expressions`, the same rows as branch-4.2.
- The new `test_iceberg_variant_defaults` produces the same `.out` as
branch-4.2.
- In `test_paimon_write_variant_errors`, the 8 common-type cases are back
to `test { exception }`, and Paimon UPDATE / MERGE cases are added.
- All 72 suites under `external_table_p0/paimon/write` and
`external_table_p0/iceberg/write`, and the Iceberg row-lineage DML suites, pass
on the same cluster. The one exception is `test_paimon_write_thread_lifecycle`,
which needs Linux `/proc`.
### Release note
Doris can write VARIANT columns of Iceberg format-version 3 tables with
Parquet data files: INSERT, INSERT OVERWRITE, UPDATE, MERGE and
`rewrite_data_files`. It can also create or add top-level VARIANT columns.
A write into a VARIANT column of an Iceberg or Paimon table now fails when
its source implicitly casts a VARIANT to another type. Examples are `IF(cond,
payload, 1)` and a UNION of a Variant and a number. Before, such a write stored
object and array values as NULL. Write an explicit CAST to keep such a
conversion.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `test_iceberg_variant_read`, `test_iceberg_variant_defaults`,
`test_paimon_write_variant_errors`, as above
- [x] Unit Test
- BE, fe-core and fe-connector-iceberg, as above
- [x] Manual test (add detailed scripts or steps below)
- The Paimon and Iceberg write regression directories and the
Iceberg row-lineage DML suites on a local cluster, as above
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- Iceberg VARIANT columns become writable on v3 Parquet tables, and
top-level VARIANT DDL works.
- Writes into connector VARIANT columns reject implicit
Variant-to-scalar casts in their source.
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
- Iceberg VARIANT writes and DDL.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
