raminqaf opened a new pull request, #29369:
URL: https://github.com/apache/flink/pull/29369
## What is the purpose of the change
FLINK-40825 added `CAST` from primitive types to VARIANT. This PR casts a
whole ARRAY, MAP, ROW or STRUCTURED value into one VARIANT. An ARRAY becomes a
variant array, and a MAP, ROW or STRUCTURED value a variant object. Every leaf
is stored the same way the cast stores it on its own.
| Expression | Result
|
|-------------------------------------------------------------------|--------------------------------------------------------------|
| `CAST(ARRAY[1, NULL] AS VARIANT)` | `[1,
null]`, the `NULL` becomes a variant null |
| `CAST(MAP['a', 1, 'b', 2] AS VARIANT)` | `{"a":
1, "b": 2}` |
| `CAST(r AS VARIANT)` for `r ROW<name STRING, id BIGINT>` |
`{"id": 7, "name": "ada"}`, keys sorted, `id` a BIGINT |
| `CAST(ROW(1, 'a') AS VARIANT)` |
`{"EXPR$0": 1, "EXPR$1": "a"}` |
| `CAST(CAST(ROW(1, 'a') AS ROW<id INT, name STRING>) AS VARIANT)` |
`{"id": 1, "name": "a"}` |
| `CAST(ARRAY[PARSE_JSON(s)['a'], PARSE_JSON(s)['b']] AS VARIANT)` | the
nested VARIANTs embedded as is |
| `CAST(MAP[NULLIF(s, 'x'), 1] AS VARIANT)` with a `NULL` key | fails,
and `TRY_CAST` returns `NULL` |
| `CAST(MAP[1, 'a'] AS VARIANT)` | fails
at validation, a key must be a character string |
## Brief change log
- Commit 1, `[core]`: `BinaryVariantInternalBuilder#appendVariant` read a
nested variant, such as the result of `getField` or `getElement`, at its old
position inside a copy that starts at 0. Embedding such a value failed with
`MALFORMED_VARIANT` or silently wrote a wrong value. This also affected the
public `VariantBuilder`. It now reads the shared buffer at the variant's own
position.
- Commit 2: `VariantCastUtils` gets `appendTimestamp` and
`appendTimestampLtz`, which write a timestamp into a shared builder. No
behavior change.
- Commit 3:
- `LogicalTypeCasts` allows ARRAY, MAP, ROW and STRUCTURED to VARIANT
when every leaf casts. A MAP key must be a character string and is never
converted. MULTISET stays rejected.
- New `ToVariantConverter` in flink-table-runtime. It is built once per
type and writes the whole value into one builder, so a nested value is encoded
in a single pass. It lives next to `VariantCastUtils` so that formats can reuse
it.
- New `ConstructedToVariantCastRule`. The generated code is one
`converter.convert(input)` call.
- New `CodeGeneratorCastRule.Context#declareReusableObject`, so a rule
can hand an object to the generated code. The standalone `CastExecutor` passes
it to its constructor, like the type serializers.
- Docs: a section on casting a whole value, and the VARIANT column of
the cast matrix.
Notes for reviewers:
- A ROW is keyed by the field names of its type. The SQL `ROW` constructor
names them `EXPR$0`, `EXPR$1`, and the Table API `row()` names them `f0`, `f1`.
A ROW type always carries names, so generated names cannot be told apart from
chosen ones. `JSON_STRING` uses the names of the type the same way, and a cast
back to ROW matches by name, so the round trip holds. The docs show how to name
the fields with `CAST(... AS ROW<...>)` or `as()` in the Table API.
- A variant object sorts its keys, so the field order of a ROW is not kept.
- Building each node as its own VARIANT and copying it into its parent
re-encodes every subtree once per level. In a micro benchmark that was 5 times
slower on a typical nested row, and up to 30 times on deep nesting. The
converter writes the leaves straight into one builder. As a tree of writers it
costs about 12% over fully inlined code, but it keeps the generated code to one
call for wide schemas, and formats can reuse it.
- A `MapData` can hold the same key twice at runtime. The last value wins,
like in the MAP constructor.
- `canFail` is always true for ARRAY and MAP, since their size has no
bound. For a ROW it is true when a leaf can fail, or when the declared sizes of
its fields can exceed 16 MiB.
- A SQL `NULL` in a `VARIANT` field becomes a variant null, so the cast
back to ROW returns a variant null for it, not SQL `NULL`. The docs say so.
- Not new: walking a VARIANT recurses once per nesting level, so embedding
a VARIANT nested about 2,000 levels deep overflows a 1 MiB stack, as `toJson()`
already does. `TRY_CAST` returns `NULL` in that case. I will track this in a
separate ticket.
## Verifying this change
This change added tests and can be verified as follows:
- `BinaryVariantInternalBuilderTest` embeds a field and an element of
another variant through `VariantBuilder`. It failed before the fix.
- `ToVariantConverterTest` covers the kind of every leaf, `NULL` values
and `NULL` element types, empty values, duplicate and `NULL` map keys, an
embedded VARIANT that does not start at position 0, STRUCTURED and DISTINCT
types, the size limit, the nanosecond range of timestamps, and Java
serialization of the converter.
- `LogicalTypeCastsTest` covers the allowed and rejected shapes.
- `CastRuleProviderTest` covers rule resolution and `canFail`, including
the 16 MiB boundary for a ROW.
- `CastRulesTest` covers the standalone `CastExecutor` path, with
byte-exact results.
- `CastFunctionITCase` covers SQL and Table API round trips, the field
names of a ROW constructor and how to choose them, embedded VARIANTs, a `NULL`
map key, the size limit with `TRY_CAST`, and the validation errors.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no. The `appendVariant` fix changes the behavior of
`VariantBuilder`, which is `@PublicEvolving`, but not its API.
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes. The new
cast runs per record and builds one VARIANT per value.
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs, in `data-types.md` (English
and Chinese)
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
