raminqaf opened a new pull request, #29617:
URL: https://github.com/apache/flink/pull/29617
## What is the purpose of the change
Reading one value out of a VARIANT needs navigation plus a cast, and
navigation fails as soon as a step meets the wrong kind of value. This adds
`VARIANT_GET(variant, path[, type])` and `TRY_VARIANT_GET`, with the path
syntax proposed in FLIP-618.
On `v = PARSE_JSON('{"order_id": 17, "user": {"name": "ada"}, "items":
[{"sku": "A1", "qty": 2}], "coupon": null, "price": "12.50"}')`, from
`VariantFunctionsITCase`:
| Call | Result
|
|-------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| `VARIANT_GET(v, '$.order_id', 'INT')` | `17`
|
| `VARIANT_GET(v, '$.user.name', 'VARIANT')` | `"ada"`, as a
VARIANT
|
| `VARIANT_GET(v, '$.missing', 'INT NOT NULL')` | `NULL`, the result
type is always nullable
|
| `VARIANT_GET(v, '$.coupon')` | a variant null,
`JSON_STRING` gives `null`
|
| `VARIANT_GET(v, '$.coupon', 'STRING')` | `NULL`
|
| `VARIANT_GET(v, '$.price', 'DECIMAL(4, 2)')` | fails: `Failed to
cast the VARIANT value at path '$.price' to DECIMAL(4, 2): Cannot cast a
VARIANT STRING value to DECIMAL(4, 2).` |
| `TRY_VARIANT_GET(v, '$.price', 'DECIMAL(4, 2)')` | `NULL`
|
| `VARIANT_GET(v, '$.items[-1]')` | fails at planning:
`Invalid path '$.items[-1]': expected an index or a quoted name after
'$.items['.` |
Decisions a reviewer would otherwise ask about:
- **Paths follow FLIP-618**, which is still under discussion: `$`, `.name`,
`['name']`, `["name"]` and a 0-based `[n]`, like `JSON_VALUE` and Spark, while
`variant[INT]` is 1-based. A missing field, an index past the end, or a step on
the wrong kind of value gives NULL. A malformed literal path fails at planning.
A malformed path from a column fails when the row runs, and `TRY_VARIANT_GET`
returns NULL. This differs from the ticket text, which said an invalid path
gives NULL.
- **`type` is a string literal** such as `'INT'`, as in Spark, not
`RETURNING INT`. It needs no parser change and stays optional. The Table API
takes a `DataType` and passes its logical type.
- **The cast reuses `FromVariantConverter`**, so `VARIANT_GET(v, p, 'T')`
behaves like `CAST` of the value at `p` to `T`. Only the top level is made
nullable, so `NOT NULL` inside `T` is still checked. Spark makes every level
nullable.
- **An own path parser, `SimplePath`, instead of Jayway.**
`flink-table-common` has no Jayway dependency, which planning-time validation
needs. Jayway also accepts paths such as `$.a[` and `$['a','b']`, and the
planned edit functions need the steps. A conformance test checks that every
path `SimplePath` accepts reads the same value as Jayway, so paths mean the
same as in the JSON functions.
- **A runtime class, not codegen.** The generated code for `CAST(v['n'] AS
INT)` makes the same per-row calls, and `JSON_VALUE` only needs codegen for its
`RETURNING` syntax.
## Brief change log
- `[hotfix][docs]`: `variant[INT]` and `variant[STRING]` throw on the
wrong kind of value, but the docs said they return NULL
- `SimplePath` in `flink-table-common` parses the FLIP-618 path syntax
- `VARIANT_GET` and `TRY_VARIANT_GET`: function definitions, input and
output type strategies, runtime functions, Table API and PyFlink methods
- A conformance test of `SimplePath` against Jayway JsonPath
- Docs: function entries and one "Variant paths" section that later path
functions can point to
## Verifying this change
This change added tests and can be verified as follows:
- `SimplePathTest`: one case per parse branch, error messages and
serialization
- `SimplePathJaywayConformanceTest`: 60,000 generated paths, each accepted
one compared with Jayway on the same documents
- `VariantFunctionsITCase`: SQL and Table API results, NULL handling, the
session time zone, and validation and runtime errors
- `VariantSemanticTest`: a path that changes from row to row, including a
malformed one, for both functions
- `test_expression.py`: the PyFlink expressions. I did not run the Python
tests locally.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: yes, new methods on `BaseExpressions`
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes, the new
function. A literal path is parsed once, and a path from a column is parsed
when it changes from the previous row
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs, JavaDocs and PyFlink
docstrings
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code 2.1.310 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
