twalthr commented on code in PR #29617:
URL: https://github.com/apache/flink/pull/29617#discussion_r4398846861
##########
docs/content.zh/docs/sql/functions/built-in-functions.md:
##########
@@ -233,6 +233,35 @@ A document fails to parse if it:
In all of these cases, `TRY_PARSE_XML` returns `NULL`.
+#### Variant paths
+
+`VARIANT_GET` and `TRY_VARIANT_GET` take a path that names one place in a
Variant. A path starts
+with `$`, the whole value, followed by steps: `.name`, `['name']` or
`["name"]` for a field, and
+`[n]` for an array element. For example, on
+`PARSE_JSON('{"user": {"name": "ada"}, "items": [{"qty": 2}]}')`:
+
+| Path | Points at |
+|---------------------|------------------------------------|
+| `$` | the whole value |
+| `$.user.name` | `"ada"` |
+| `$['user']["name"]` | `"ada"` |
+| `$.items[0].qty` | `2` |
+| `$.items[1]` | nothing, the index is past the end |
+| `$.user[0]` | nothing, `user` is not an array |
+
+- Array indices start at 0, as in `JSON_VALUE`, while `variant[INT]` starts at
1. So `$.items[0]`
+ and `v['items'][1]` are the same element.
Review Comment:
```suggestion
- Array indices start at 0, as in `JSON_VALUE`, while `variant[INT]` starts
at 1. So `VARIANT_GET(v, '$.items[0]')`
and `v['items'][1]` are the same element.
```
##########
docs/content.zh/docs/sql/functions/built-in-functions.md:
##########
@@ -233,6 +233,35 @@ A document fails to parse if it:
In all of these cases, `TRY_PARSE_XML` returns `NULL`.
+#### Variant paths
+
+`VARIANT_GET` and `TRY_VARIANT_GET` take a path that names one place in a
Variant. A path starts
+with `$`, the whole value, followed by steps: `.name`, `['name']` or
`["name"]` for a field, and
+`[n]` for an array element. For example, on
+`PARSE_JSON('{"user": {"name": "ada"}, "items": [{"qty": 2}]}')`:
+
+| Path | Points at |
+|---------------------|------------------------------------|
+| `$` | the whole value |
+| `$.user.name` | `"ada"` |
+| `$['user']["name"]` | `"ada"` |
+| `$.items[0].qty` | `2` |
+| `$.items[1]` | nothing, the index is past the end |
+| `$.user[0]` | nothing, `user` is not an array |
+
+- Array indices start at 0, as in `JSON_VALUE`, while `variant[INT]` starts at
1. So `$.items[0]`
+ and `v['items'][1]` are the same element.
+- Use brackets for a name with whitespace or one of the characters `. [ ] ( )
* ' " \`, for example
+ `$['first name']`. Inside the quotes, `\\`, `\'` and `\"` stand for a
backslash and a quote.
Review Comment:
Are double quotes supported?
##########
docs/content.zh/docs/sql/functions/built-in-functions.md:
##########
@@ -233,6 +233,35 @@ A document fails to parse if it:
In all of these cases, `TRY_PARSE_XML` returns `NULL`.
+#### Variant paths
+
+`VARIANT_GET` and `TRY_VARIANT_GET` take a path that names one place in a
Variant. A path starts
+with `$`, the whole value, followed by steps: `.name`, `['name']` or
`["name"]` for a field, and
+`[n]` for an array element. For example, on
+`PARSE_JSON('{"user": {"name": "ada"}, "items": [{"qty": 2}]}')`:
+
+| Path | Points at |
+|---------------------|------------------------------------|
+| `$` | the whole value |
+| `$.user.name` | `"ada"` |
+| `$['user']["name"]` | `"ada"` |
+| `$.items[0].qty` | `2` |
+| `$.items[1]` | nothing, the index is past the end |
+| `$.user[0]` | nothing, `user` is not an array |
+
+- Array indices start at 0, as in `JSON_VALUE`, while `variant[INT]` starts at
1. So `$.items[0]`
+ and `v['items'][1]` are the same element.
+- Use brackets for a name with whitespace or one of the characters `. [ ] ( )
* ' " \`, for example
+ `$['first name']`. Inside the quotes, `\\`, `\'` and `\"` stand for a
backslash and a quote.
+- Wildcards, filters, slices, negative indices and the `lax` and `strict`
modes are not supported,
+ so a path always names one place.
+- Trailing spaces are ignored, so a path read from a `CHAR(n)` column works.
Review Comment:
```suggestion
- Trailing spaces are ignored.
```
##########
docs/data/sql_functions.yml:
##########
@@ -1451,13 +1451,69 @@ variant:
- sql: variant '[' INT ']'
table: VARIANT.at(INT)
description: |
- If the VARIANT is an ARRAY value, returns a VARIANT whose value is the
element at
- the specified index. The index starts from 1, If the index is out of
range, it returns NULL.
+ If the VARIANT is an array, returns a VARIANT holding the element at the
index. The index
+ starts at 1. If the index is out of range, returns NULL. If the VARIANT
is not an array, an
+ error is thrown. To return NULL instead, use `VARIANT_GET`.
- sql: variant '[' STRING ']'
table: VARIANT.at(STRING)
description: |
- If the VARIANT is a MAP value that has an element with this key, a
VARIANT holding
- the associated value is returned. Otherwise, NULL is returned.
+ If the VARIANT is an object that has a field with this key, returns a
VARIANT holding the
+ value of the field. If the object has no such field, returns NULL. If
the VARIANT is not an
+ object, an error is thrown. To return NULL instead, use `VARIANT_GET`.
+ - sql: VARIANT_GET(variant, path[, type])
+ table: VARIANT.variantGet(path[, returnType])
+ description: |
+ Returns the value of a Variant at a path, cast to `type`. Without
`type`, the value is
+ returned as a Variant. If the value cannot be cast to `type`, an error
is thrown. To return
+ NULL instead of an error, use the `TRY_VARIANT_GET` function. See the
"Variant paths" section
+ below the function table for the path syntax.
+
+ Returns NULL if:
+
+ * `variant` or `path` is NULL,
+ * nothing is at the path, or
+ * the value at the path is a variant null and `type` is not `'VARIANT'`.
+
+ Without `type`, or with `'VARIANT'`, a variant null stays a variant
null. It is not SQL NULL,
+ so `IS NULL` is false and `JSON_STRING` returns `'null'`.
+
+ `type` is a string literal such as `'INT'` or `'ARRAY<ROW<sku STRING,
qty INT>>'`, and must
+ not be NULL. The value is cast the way `CAST(v AS type)` casts a
Variant. The result is always
+ nullable. In the Table API, `returnType` is a `DataType`, and only its
logical type is used.
Review Comment:
```suggestion
nullable.
```
##########
docs/data/sql_functions.yml:
##########
@@ -1451,13 +1451,69 @@ variant:
- sql: variant '[' INT ']'
table: VARIANT.at(INT)
description: |
- If the VARIANT is an ARRAY value, returns a VARIANT whose value is the
element at
- the specified index. The index starts from 1, If the index is out of
range, it returns NULL.
+ If the VARIANT is an array, returns a VARIANT holding the element at the
index. The index
+ starts at 1. If the index is out of range, returns NULL. If the VARIANT
is not an array, an
+ error is thrown. To return NULL instead, use `VARIANT_GET`.
- sql: variant '[' STRING ']'
table: VARIANT.at(STRING)
description: |
- If the VARIANT is a MAP value that has an element with this key, a
VARIANT holding
- the associated value is returned. Otherwise, NULL is returned.
+ If the VARIANT is an object that has a field with this key, returns a
VARIANT holding the
+ value of the field. If the object has no such field, returns NULL. If
the VARIANT is not an
+ object, an error is thrown. To return NULL instead, use `VARIANT_GET`.
+ - sql: VARIANT_GET(variant, path[, type])
+ table: VARIANT.variantGet(path[, returnType])
+ description: |
+ Returns the value of a Variant at a path, cast to `type`. Without
`type`, the value is
+ returned as a Variant. If the value cannot be cast to `type`, an error
is thrown. To return
+ NULL instead of an error, use the `TRY_VARIANT_GET` function. See the
"Variant paths" section
+ below the function table for the path syntax.
+
+ Returns NULL if:
+
+ * `variant` or `path` is NULL,
+ * nothing is at the path, or
+ * the value at the path is a variant null and `type` is not `'VARIANT'`.
+
+ Without `type`, or with `'VARIANT'`, a variant null stays a variant
null. It is not SQL NULL,
+ so `IS NULL` is false and `JSON_STRING` returns `'null'`.
+
+ `type` is a string literal such as `'INT'` or `'ARRAY<ROW<sku STRING,
qty INT>>'`, and must
+ not be NULL. The value is cast the way `CAST(v AS type)` casts a
Variant. The result is always
+ nullable. In the Table API, `returnType` is a `DataType`, and only its
logical type is used.
+
+ ```sql
+ -- 17
Review Comment:
```suggestion
-- 17 of type INT
```
##########
docs/data/sql_functions.yml:
##########
@@ -1451,13 +1451,69 @@ variant:
- sql: variant '[' INT ']'
table: VARIANT.at(INT)
description: |
- If the VARIANT is an ARRAY value, returns a VARIANT whose value is the
element at
- the specified index. The index starts from 1, If the index is out of
range, it returns NULL.
+ If the VARIANT is an array, returns a VARIANT holding the element at the
index. The index
+ starts at 1. If the index is out of range, returns NULL. If the VARIANT
is not an array, an
+ error is thrown. To return NULL instead, use `VARIANT_GET`.
- sql: variant '[' STRING ']'
table: VARIANT.at(STRING)
description: |
- If the VARIANT is a MAP value that has an element with this key, a
VARIANT holding
- the associated value is returned. Otherwise, NULL is returned.
+ If the VARIANT is an object that has a field with this key, returns a
VARIANT holding the
+ value of the field. If the object has no such field, returns NULL. If
the VARIANT is not an
+ object, an error is thrown. To return NULL instead, use `VARIANT_GET`.
+ - sql: VARIANT_GET(variant, path[, type])
Review Comment:
```suggestion
- sql: VARIANT_GET(variant, path[, returnType])
```
##########
flink-python/pyflink/table/expression.py:
##########
@@ -2456,6 +2456,40 @@ def try_parse_xml(self, force_array=None) ->
'Expression':
else:
return _binary_op("tryParseXml")(self, force_array)
+ def variant_get(self, path, return_type: DataType = None) -> 'Expression':
+ """
+ Returns the value of a VARIANT at a path, cast to return_type. Without
return_type, the
+ value is returned as a VARIANT. If the VARIANT or the path is None, or
nothing is at the
+ path, None is returned. A variant null at the path gives None, unless
return_type is
+ omitted or VARIANT, which keeps it as a variant null. If the value
cannot be cast to
+ return_type, an error is thrown. To return None instead, use
+ :func:`~Expression.try_variant_get`.
Review Comment:
```suggestion
Returns the value of a VARIANT at a path, cast to return_type.
Without return_type, the
value is returned as a VARIANT. If the VARIANT or the path is None,
or nothing is at the
path, NULL is returned. A variant null at the path gives NULL,
unless return_type is
omitted or VARIANT, which keeps it as a variant null. If the value
cannot be cast to
return_type, an error is thrown. To return None instead, use
:func:`~Expression.try_variant_get`.
```
please revisit these docs, they are mixing SQL and Python concepts
##########
flink-python/pyflink/table/expression.py:
##########
@@ -2456,6 +2456,40 @@ def try_parse_xml(self, force_array=None) ->
'Expression':
else:
return _binary_op("tryParseXml")(self, force_array)
+ def variant_get(self, path, return_type: DataType = None) -> 'Expression':
+ """
+ Returns the value of a VARIANT at a path, cast to return_type. Without
return_type, the
+ value is returned as a VARIANT. If the VARIANT or the path is None, or
nothing is at the
+ path, None is returned. A variant null at the path gives None, unless
return_type is
+ omitted or VARIANT, which keeps it as a variant null. If the value
cannot be cast to
+ return_type, an error is thrown. To return None instead, use
+ :func:`~Expression.try_variant_get`.
+
+ Array indices in a path such as '$.items[0].qty' start at 0. See the
SQL function
+ VARIANT_GET for the path syntax. Only the logical type of return_type
is used.
+
+ Example:
+ ::
+
+ >>> lit('{"a": {"b": 1}}').parse_json().variant_get('$.a.b',
DataTypes.INT()) # 1
+ >>> lit('{"a": 1}').parse_json().variant_get('$.b',
DataTypes.INT()) # None
Review Comment:
```suggestion
>>> lit('{"a": 1}').parse_json().variant_get('$.b',
DataTypes.INT()) # NULL
```
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/BaseExpressions.java:
##########
@@ -1481,6 +1483,70 @@ public OutType tryParseXml(InType forceArray) {
unresolvedCall(TRY_PARSE_XML, toExpr(),
objectToExpression(forceArray)));
}
+ /**
+ * Returns the value of a {@link DataTypes#VARIANT()} at a path as a {@link
+ * DataTypes#VARIANT()}. If the variant or the path is {@code NULL}, or
nothing is at the path,
+ * {@code NULL} is returned. A variant null at the path stays a variant
null, which is not
+ * {@code NULL}.
+ *
+ * <p>This is a shortcut for {@code variantGet(path,
DataTypes.VARIANT())}. See {@link
+ * #variantGet(Object, DataType)}.
+ */
+ public OutType variantGet(InType path) {
+ return toApiSpecificExpression(
+ unresolvedCall(VARIANT_GET, toExpr(),
objectToExpression(path)));
+ }
+
+ /**
+ * Returns the value of a {@link DataTypes#VARIANT()} at a path such as
{@code $.items[0].qty},
+ * cast to the given type. Array indices start at 0. If the variant or the
path is {@code NULL},
+ * or nothing is at the path, {@code NULL} is returned. If the value
cannot be cast to the type,
+ * an error is thrown. To return {@code NULL} instead, use {@link
#tryVariantGet(Object,
+ * DataType)}.
+ *
+ * <p>The value is cast like {@code CAST}, so a variant null gives {@code
NULL} unless the type
+ * is {@link DataTypes#VARIANT()}. See the SQL function {@code
VARIANT_GET} for the path syntax.
+ *
+ * @param returnType the type of the result, always nullable. Only its
logical type is used.
Review Comment:
```suggestion
* @param returnType the type of the result, always nullable.
```
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/VariantGetTypeStrategy.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.types.inference.strategies;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.inference.CallContext;
+import org.apache.flink.table.types.inference.TypeStrategy;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.VariantType;
+import org.apache.flink.table.types.logical.utils.LogicalTypeCasts;
+
+import java.util.Optional;
+
+/**
+ * Type strategy for {@code VARIANT_GET(variant, path [, type])} and {@code
TRY_VARIANT_GET}.
+ *
+ * <p>The result is the type named by the {@code type} literal, or {@code
VARIANT} without one. It
+ * is always nullable, because a path that points at nothing gives {@code
NULL}.
+ */
+@Internal
+public final class VariantGetTypeStrategy implements TypeStrategy {
+
+ static final int ARG_TYPE = 2;
+
+ @Override
+ public Optional<DataType> inferType(final CallContext callContext) {
+ if (callContext.getArgumentDataTypes().size() <= ARG_TYPE) {
+ return Optional.of(DataTypes.VARIANT().nullable());
+ }
+ return Optional.of(resolveTargetType(callContext));
+ }
+
+ /**
+ * Resolves the {@code type} literal to a nullable type that {@code
VARIANT} can be cast to.
+ *
+ * @throws ValidationException if the literal is not a type or names a
type that {@code VARIANT}
+ * cannot be cast to
+ */
+ static DataType resolveTargetType(final CallContext callContext) {
+ // The input strategy only accepts a non-null literal.
+ final String typeString =
+ callContext
+ .getArgumentValue(ARG_TYPE, String.class)
+ .orElseThrow(IllegalStateException::new);
+ final DataType type;
+ try {
+ type = callContext.getDataTypeFactory().createDataType(typeString);
+ } catch (RuntimeException e) {
+ throw new ValidationException(
+ String.format(
+ "Invalid type '%s' for %s: %s",
+ typeString, callContext.getName(), e.getMessage()),
+ e);
+ }
+ final LogicalType logicalType = type.getLogicalType();
Review Comment:
```suggestion
final LogicalType logicalType = type.getLogicalType().copy(true);
```
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/VariantSemanticTest.java:
##########
@@ -111,6 +111,54 @@ public class VariantSemanticTest extends SemanticTestBase {
.runSql("INSERT INTO sink_t SELECT TRY_PARSE_JSON(v) FROM
t")
.build();
+ static final String JSON_OBJECT_A_AND_B = "{\"a\": 1, \"b\": 2}";
+
+ static final TableTestProgram TRY_VARIANT_GET_PATH_PER_ROW =
+ TableTestProgram.of(
+ "try-variant-get-path-per-row",
+ "validates TRY_VARIANT_GET reads a path that
changes from row to row")
+ .setupTableSource(
+ SourceTestStep.newBuilder("t")
+ .addSchema("v STRING", "p STRING")
+ .producedValues(
+ Row.of(JSON_OBJECT_A_AND_B, "$.a"),
+ Row.of(JSON_OBJECT_A_AND_B, "$.c"),
+ Row.of(JSON_OBJECT_A_AND_B, "$.b"),
+ Row.of(JSON_OBJECT_A_AND_B,
"$.a["),
+ Row.of(JSON_OBJECT_A_AND_B, "$.b"))
Review Comment:
occurs two times
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/VariantGetFunction.java:
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.api.config.TableConfigOptions;
+import org.apache.flink.table.api.internal.TableConfigValidation;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.data.utils.FromVariantConverter;
+import org.apache.flink.table.functions.BuiltInFunctionDefinition;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction.SpecializedContext;
+import org.apache.flink.table.types.inference.CallContext;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.table.utils.SimplePath;
+import org.apache.flink.types.variant.Variant;
+
+import javax.annotation.Nullable;
+
+import java.time.ZoneId;
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * Implementation of {@link BuiltInFunctionDefinitions#VARIANT_GET}.
+ *
+ * <p>A path that points at nothing gives {@code NULL}: a missing field, an
index past the end, a
+ * field of a non-object or an element of a non-array. The value found is cast
to the result type
+ * the way {@code CAST} does, so a variant null becomes {@code NULL} unless
the result is a {@code
+ * VARIANT}.
+ */
+@Internal
+public class VariantGetFunction extends BuiltInScalarFunction {
+
+ private static final int ARG_PATH = 1;
+
+ private final boolean isTry;
+
+ private final @Nullable SimplePath literalPath;
+
+ // Null when the result is a VARIANT, which needs no cast.
+ private final @Nullable FromVariantConverter converter;
+
+ private final String targetTypeName;
+
+ private transient @Nullable String lastPathString;
+
+ private transient @Nullable SimplePath lastPath;
+
+ private transient @Nullable IllegalArgumentException lastPathError;
+
+ public VariantGetFunction(SpecializedContext context) {
+ this(BuiltInFunctionDefinitions.VARIANT_GET, context, false);
+ }
+
+ VariantGetFunction(
+ BuiltInFunctionDefinition definition, SpecializedContext context,
boolean isTry) {
+ super(definition, context);
+ this.isTry = isTry;
+ this.literalPath = parseLiteralPath(context.getCallContext());
+ final LogicalType outputType = getOutputDataType().getLogicalType();
+ this.converter =
+ outputType.is(LogicalTypeRoot.VARIANT)
+ ? null
+ : FromVariantConverter.create(
+ outputType,
getSessionTimeZone(context.getConfiguration()));
+ this.targetTypeName = outputType.asSummaryString();
+ }
+
+ public @Nullable Object eval(@Nullable Variant variant, @Nullable
StringData path) {
+ return eval(variant, path, null);
+ }
+
+ public @Nullable Object eval(
+ @Nullable Variant variant, @Nullable StringData path, @Nullable
StringData type) {
+ // The type only matters for planning, where it defines the output
type.
+ if (variant == null || path == null) {
+ return null;
+ }
+ final SimplePath simplePath = literalPath != null ? literalPath :
parsePath(path);
+ if (simplePath == null) {
+ return null;
+ }
+ final Variant value = navigate(variant, simplePath);
+ if (value == null || converter == null) {
+ return value;
+ }
+ if (value.isNull()) {
+ return null;
+ }
+ try {
+ return converter.convert(value);
+ } catch (RuntimeException e) {
+ if (isTry) {
+ return null;
+ }
+ throw new TableRuntimeException(
+ String.format(
+ "Failed to cast the VARIANT value at path '%s' to
%s: %s",
+ simplePath, targetTypeName, e.getMessage()),
+ e);
+ }
+ }
+
+ /**
+ * Returns the literal path, or {@code null} to parse the path of each row.
+ *
+ * <p>The input strategy rejects a malformed literal path. A path that
becomes a literal only
+ * during optimization, such as a constant column of a view, skips that
check, so a malformed
+ * one is left to the rules for a path that is not a literal.
+ */
+ private static @Nullable SimplePath parseLiteralPath(CallContext
callContext) {
+ if (!callContext.isArgumentLiteral(ARG_PATH)) {
+ return null;
+ }
+ final Optional<String> path = callContext.getArgumentValue(ARG_PATH,
String.class);
+ if (path.isEmpty()) {
+ return null;
+ }
+ try {
+ return SimplePath.parse(path.get());
+ } catch (IllegalArgumentException e) {
+ return null;
+ }
+ }
+
+ /** Returns the path, or {@code null} for a malformed path in {@code
TRY_VARIANT_GET}. */
+ private @Nullable SimplePath parsePath(StringData path) {
+ final String pathString = path.toString();
+ if (!pathString.equals(lastPathString)) {
+ lastPathString = pathString;
+ try {
+ lastPath = SimplePath.parse(pathString);
+ lastPathError = null;
+ } catch (IllegalArgumentException e) {
+ lastPath = null;
+ lastPathError = e;
+ }
+ }
+ if (lastPathError != null && !isTry) {
+ throw new TableRuntimeException(lastPathError.getMessage(),
lastPathError);
+ }
+ return lastPath;
+ }
+
+ /** Returns the value at the path, or {@code null} if the path points at
nothing. */
+ @VisibleForTesting
+ static @Nullable Variant navigate(Variant variant, SimplePath path) {
+ final List<SimplePath.Step> steps = path.getSteps();
+ Variant current = variant;
+ for (final SimplePath.Step step : steps) {
+ if (step.isField()) {
+ current = current.isObject() ?
current.getField(step.getFieldName()) : null;
+ } else {
+ current = current.isArray() ?
current.getElement(step.getIndex()) : null;
+ }
+ if (current == null) {
+ return null;
+ }
+ }
+ return current;
+ }
+
+ private static ZoneId getSessionTimeZone(ReadableConfig config) {
Review Comment:
use org.apache.flink.table.planner.utils.TableConfigUtils#getLocalTimeZone
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/SimplePath.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.utils;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/**
+ * A path that points at exactly one place in a value, such as {@code
$.user.email} or {@code
+ * $['items'][0]}.
+ *
+ * <p>The syntax is the part of the JSON functions' path syntax that names a
single place: no
+ * wildcards, filters, slices, negative indices, functions or {@code lax} and
{@code strict} modes.
+ * Every path that this class accepts means the same in the JSON functions. A
path starts with
+ * {@code $}, the whole value, followed by any number of steps:
+ *
+ * <ul>
+ * <li>{@code .name} is a field. The name cannot contain whitespace or one
of the characters
+ * {@code . [ ] ( ) * ' " \}.
+ * <li>{@code ['name']} and {@code ["name"]} are a field with any name.
Inside the quotes, a
+ * backslash followed by a backslash or a quote stands for that
character.
+ * <li>{@code [n]} is the array element at the 0-based index {@code n}.
+ * </ul>
+ *
+ * <p>Trailing spaces are ignored, as in the JSON functions, so a space-padded
{@code CHAR(n)} path
+ * works.
+ */
+@Internal
+public final class SimplePath implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final String SYNTAX =
+ "A path is '$' followed by the steps .name, ['name'], [\"name\"]
or [index], "
+ + "without wildcards, filters, slices, negative indices or
functions. "
+ + "Use brackets for a name that has whitespace or one of
the characters "
+ + ". [ ] ( ) * ' \" \\ in it.";
+
+ private static final String NAME_STOP_CHARACTERS = ".[]()*'\"\\";
+
+ private static final Pattern PATH_MODE_PREFIX =
+ Pattern.compile("\\s*(strict|lax)\\s.*", Pattern.CASE_INSENSITIVE
| Pattern.DOTALL);
+
+ private final String path;
+
+ private final List<Step> steps;
+
+ private SimplePath(final String path, final List<Step> steps) {
+ this.path = path;
+ this.steps = steps;
+ }
+
+ /**
+ * Parses a path.
+ *
+ * @throws IllegalArgumentException if the path is malformed or uses
syntax that can point at
+ * more than one place
+ */
+ public static SimplePath parse(final String path) {
+ Preconditions.checkNotNull(path, "Path must not be null.");
+ return new Parser(path).parse();
+ }
+
+ /** Returns the steps after {@code $}, so the root path has none. */
+ public List<Step> getSteps() {
+ return steps;
+ }
+
+ @Override
+ public String toString() {
+ return path;
+ }
+
+ //
--------------------------------------------------------------------------------------------
+
+ /** A step to a field of an object or to an element of an array. */
+ @Internal
+ public static final class Step implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private final @Nullable String fieldName;
+
+ private final int index;
+
+ private Step(final @Nullable String fieldName, final int index) {
+ this.fieldName = fieldName;
+ this.index = index;
+ }
+
+ public static Step field(final String fieldName) {
+ return new Step(Preconditions.checkNotNull(fieldName), -1);
+ }
+
+ public static Step index(final int index) {
+ Preconditions.checkArgument(index >= 0, "Index must not be
negative.");
Review Comment:
is it guaranteed that this is effectively a ValidationException or
TableRuntimeException
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/SimplePath.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.utils;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/**
+ * A path that points at exactly one place in a value, such as {@code
$.user.email} or {@code
+ * $['items'][0]}.
+ *
+ * <p>The syntax is the part of the JSON functions' path syntax that names a
single place: no
+ * wildcards, filters, slices, negative indices, functions or {@code lax} and
{@code strict} modes.
+ * Every path that this class accepts means the same in the JSON functions. A
path starts with
+ * {@code $}, the whole value, followed by any number of steps:
+ *
+ * <ul>
+ * <li>{@code .name} is a field. The name cannot contain whitespace or one
of the characters
+ * {@code . [ ] ( ) * ' " \}.
+ * <li>{@code ['name']} and {@code ["name"]} are a field with any name.
Inside the quotes, a
+ * backslash followed by a backslash or a quote stands for that
character.
+ * <li>{@code [n]} is the array element at the 0-based index {@code n}.
+ * </ul>
+ *
+ * <p>Trailing spaces are ignored, as in the JSON functions, so a space-padded
{@code CHAR(n)} path
+ * works.
+ */
+@Internal
+public final class SimplePath implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final String SYNTAX =
+ "A path is '$' followed by the steps .name, ['name'], [\"name\"]
or [index], "
+ + "without wildcards, filters, slices, negative indices or
functions. "
+ + "Use brackets for a name that has whitespace or one of
the characters "
+ + ". [ ] ( ) * ' \" \\ in it.";
+
+ private static final String NAME_STOP_CHARACTERS = ".[]()*'\"\\";
+
+ private static final Pattern PATH_MODE_PREFIX =
+ Pattern.compile("\\s*(strict|lax)\\s.*", Pattern.CASE_INSENSITIVE
| Pattern.DOTALL);
+
+ private final String path;
+
+ private final List<Step> steps;
+
+ private SimplePath(final String path, final List<Step> steps) {
+ this.path = path;
+ this.steps = steps;
+ }
+
+ /**
+ * Parses a path.
+ *
+ * @throws IllegalArgumentException if the path is malformed or uses
syntax that can point at
+ * more than one place
+ */
+ public static SimplePath parse(final String path) {
+ Preconditions.checkNotNull(path, "Path must not be null.");
+ return new Parser(path).parse();
+ }
+
+ /** Returns the steps after {@code $}, so the root path has none. */
+ public List<Step> getSteps() {
+ return steps;
+ }
+
+ @Override
+ public String toString() {
+ return path;
+ }
+
+ //
--------------------------------------------------------------------------------------------
+
+ /** A step to a field of an object or to an element of an array. */
+ @Internal
+ public static final class Step implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private final @Nullable String fieldName;
+
+ private final int index;
+
+ private Step(final @Nullable String fieldName, final int index) {
+ this.fieldName = fieldName;
+ this.index = index;
+ }
+
+ public static Step field(final String fieldName) {
+ return new Step(Preconditions.checkNotNull(fieldName), -1);
+ }
+
+ public static Step index(final int index) {
+ Preconditions.checkArgument(index >= 0, "Index must not be
negative.");
+ return new Step(null, index);
+ }
+
+ public boolean isField() {
+ return fieldName != null;
+ }
+
+ public String getFieldName() {
+ Preconditions.checkState(fieldName != null, "Step is not a
field.");
+ return fieldName;
+ }
+
+ public int getIndex() {
+ Preconditions.checkState(fieldName == null, "Step is not an
index.");
+ return index;
+ }
+
+ @Override
+ public boolean equals(final Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ final Step that = (Step) o;
+ return index == that.index && Objects.equals(fieldName,
that.fieldName);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(fieldName, index);
+ }
+
+ @Override
+ public String toString() {
+ if (fieldName == null) {
+ return "[" + index + "]";
+ }
+ return "['" + fieldName.replace("\\", "\\\\").replace("'", "\\'")
+ "']";
+ }
+ }
+
+ //
--------------------------------------------------------------------------------------------
+
+ private static final class Parser {
+
+ private final String path;
+
+ // The end of the path without trailing spaces.
+ private final int end;
+
+ private int pos;
+
+ private Parser(final String path) {
+ this.path = path;
+ int end = path.length();
+ while (end > 0 && path.charAt(end - 1) == ' ') {
+ end--;
+ }
+ this.end = end;
+ }
+
+ private SimplePath parse() {
+ if (!path.startsWith("$")) {
+ if (PATH_MODE_PREFIX.matcher(path).matches()) {
+ throw error("the 'lax' and 'strict' modes are not
supported");
+ }
+ throw error("it must start with '$'");
+ }
+ pos = 1;
+ final List<Step> steps = new ArrayList<>();
+ while (pos < end) {
+ final char c = path.charAt(pos);
+ if (c == '.') {
+ pos++;
+ steps.add(Step.field(readName()));
+ } else if (c == '[') {
+ pos++;
+ steps.add(readBracketStep());
+ } else {
+ throw error(String.format("unexpected '%s' after '%s'", c,
consumed()));
+ }
+ }
+ return new SimplePath(path, List.copyOf(steps));
+ }
+
+ private String readName() {
+ final int start = pos;
+ while (pos < end && isNameCharacter(path.charAt(pos))) {
+ pos++;
+ }
+ if (pos == start) {
+ throw error(String.format("expected a field name after '%s'",
consumed()));
+ }
+ return path.substring(start, pos);
+ }
+
+ private Step readBracketStep() {
+ final String prefix = path.substring(0, pos - 1);
+ final char c = pos < end ? path.charAt(pos) : 0;
+ final Step step;
+ if (c == '\'' || c == '"') {
+ pos++;
+ step = Step.field(readQuotedName(c, prefix));
+ } else if (isDigit(c)) {
+ step = Step.index(readIndex());
+ } else {
+ throw error(
+ String.format("expected an index or a quoted name
after '%s'", consumed()));
+ }
+ if (pos >= end || path.charAt(pos) != ']') {
+ throw error(String.format("expected ']' after '%s'",
consumed()));
+ }
+ pos++;
+ return step;
+ }
+
+ private String readQuotedName(final char quote, final String prefix) {
+ final StringBuilder name = new StringBuilder();
+ while (pos < end) {
+ final char c = path.charAt(pos);
+ if (c == quote) {
+ pos++;
+ return name.toString();
+ }
+ if (c == '\\') {
+ final char escaped = pos + 1 < end ? path.charAt(pos + 1)
: 0;
+ if (escaped != '\\' && escaped != '\'' && escaped != '"') {
+ throw error(
+ String.format(
+ "unsupported escape sequence in the
name after '%s'. "
+ + "Only a backslash or a quote
can follow a "
+ + "backslash",
+ prefix));
+ }
+ name.append(escaped);
+ pos += 2;
+ } else {
+ name.append(c);
+ pos++;
+ }
+ }
+ throw error(String.format("the name after '%s' has no closing
quote", prefix));
+ }
+
+ private int readIndex() {
+ final int start = pos;
+ while (pos < end && isDigit(path.charAt(pos))) {
+ pos++;
+ }
+ try {
+ return Integer.parseInt(path.substring(start, pos));
+ } catch (NumberFormatException e) {
+ throw error(
+ String.format(
+ "the index after '%s' is larger than %d",
+ path.substring(0, start), Integer.MAX_VALUE));
+ }
+ }
+
+ private String consumed() {
+ return path.substring(0, pos);
+ }
+
+ private IllegalArgumentException error(final String reason) {
+ return new IllegalArgumentException(
Review Comment:
```suggestion
return new ValidationException(
```
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/BaseExpressions.java:
##########
@@ -1481,6 +1483,70 @@ public OutType tryParseXml(InType forceArray) {
unresolvedCall(TRY_PARSE_XML, toExpr(),
objectToExpression(forceArray)));
}
+ /**
+ * Returns the value of a {@link DataTypes#VARIANT()} at a path as a {@link
+ * DataTypes#VARIANT()}. If the variant or the path is {@code NULL}, or
nothing is at the path,
+ * {@code NULL} is returned. A variant null at the path stays a variant
null, which is not
+ * {@code NULL}.
+ *
+ * <p>This is a shortcut for {@code variantGet(path,
DataTypes.VARIANT())}. See {@link
+ * #variantGet(Object, DataType)}.
+ */
+ public OutType variantGet(InType path) {
+ return toApiSpecificExpression(
+ unresolvedCall(VARIANT_GET, toExpr(),
objectToExpression(path)));
+ }
+
+ /**
+ * Returns the value of a {@link DataTypes#VARIANT()} at a path such as
{@code $.items[0].qty},
+ * cast to the given type. Array indices start at 0. If the variant or the
path is {@code NULL},
+ * or nothing is at the path, {@code NULL} is returned. If the value
cannot be cast to the type,
+ * an error is thrown. To return {@code NULL} instead, use {@link
#tryVariantGet(Object,
+ * DataType)}.
+ *
+ * <p>The value is cast like {@code CAST}, so a variant null gives {@code
NULL} unless the type
+ * is {@link DataTypes#VARIANT()}. See the SQL function {@code
VARIANT_GET} for the path syntax.
+ *
+ * @param returnType the type of the result, always nullable. Only its
logical type is used.
+ */
+ public OutType variantGet(InType path, DataType returnType) {
+ return toApiSpecificExpression(
+ unresolvedCall(
+ VARIANT_GET,
+ toExpr(),
+ objectToExpression(path),
+
valueLiteral(returnType.getLogicalType().asSerializableString())));
+ }
+
+ /**
+ * Returns the value of a {@link DataTypes#VARIANT()} at a path as a {@link
+ * DataTypes#VARIANT()}. If nothing is at the path, or a path that is not
a literal is
+ * malformed, {@code NULL} is returned.
+ *
+ * <p>This is a shortcut for {@code tryVariantGet(path,
DataTypes.VARIANT())}. See {@link
+ * #tryVariantGet(Object, DataType)}.
+ */
+ public OutType tryVariantGet(InType path) {
+ return toApiSpecificExpression(
+ unresolvedCall(TRY_VARIANT_GET, toExpr(),
objectToExpression(path)));
+ }
+
+ /**
+ * Returns the value of a {@link DataTypes#VARIANT()} at a path, cast to
the given type. Unlike
+ * {@link #variantGet(Object, DataType)}, {@code NULL} is returned instead
of an error if the
+ * value cannot be cast to the type or a path that is not a literal is
malformed.
+ *
+ * @param returnType the type of the result, always nullable. Only its
logical type is used.
Review Comment:
```suggestion
* @param returnType the type of the result, always nullable.
```
##########
docs/data/sql_functions.yml:
##########
@@ -1451,13 +1451,69 @@ variant:
- sql: variant '[' INT ']'
table: VARIANT.at(INT)
description: |
- If the VARIANT is an ARRAY value, returns a VARIANT whose value is the
element at
- the specified index. The index starts from 1, If the index is out of
range, it returns NULL.
+ If the VARIANT is an array, returns a VARIANT holding the element at the
index. The index
+ starts at 1. If the index is out of range, returns NULL. If the VARIANT
is not an array, an
+ error is thrown. To return NULL instead, use `VARIANT_GET`.
- sql: variant '[' STRING ']'
table: VARIANT.at(STRING)
description: |
- If the VARIANT is a MAP value that has an element with this key, a
VARIANT holding
- the associated value is returned. Otherwise, NULL is returned.
+ If the VARIANT is an object that has a field with this key, returns a
VARIANT holding the
+ value of the field. If the object has no such field, returns NULL. If
the VARIANT is not an
+ object, an error is thrown. To return NULL instead, use `VARIANT_GET`.
+ - sql: VARIANT_GET(variant, path[, type])
+ table: VARIANT.variantGet(path[, returnType])
+ description: |
+ Returns the value of a Variant at a path, cast to `type`. Without
`type`, the value is
+ returned as a Variant. If the value cannot be cast to `type`, an error
is thrown. To return
+ NULL instead of an error, use the `TRY_VARIANT_GET` function. See the
"Variant paths" section
+ below the function table for the path syntax.
+
+ Returns NULL if:
+
+ * `variant` or `path` is NULL,
+ * nothing is at the path, or
+ * the value at the path is a variant null and `type` is not `'VARIANT'`.
+
+ Without `type`, or with `'VARIANT'`, a variant null stays a variant
null. It is not SQL NULL,
+ so `IS NULL` is false and `JSON_STRING` returns `'null'`.
+
+ `type` is a string literal such as `'INT'` or `'ARRAY<ROW<sku STRING,
qty INT>>'`, and must
+ not be NULL. The value is cast the way `CAST(v AS type)` casts a
Variant. The result is always
+ nullable. In the Table API, `returnType` is a `DataType`, and only its
logical type is used.
+
+ ```sql
+ -- 17
Review Comment:
add this to all examples
##########
flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/functions/scalar/SimplePathJaywayConformanceTest.java:
##########
Review Comment:
+1 for removing it
##########
flink-python/pyflink/table/expression.py:
##########
@@ -2456,6 +2456,40 @@ def try_parse_xml(self, force_array=None) ->
'Expression':
else:
return _binary_op("tryParseXml")(self, force_array)
+ def variant_get(self, path, return_type: DataType = None) -> 'Expression':
+ """
+ Returns the value of a VARIANT at a path, cast to return_type. Without
return_type, the
+ value is returned as a VARIANT. If the VARIANT or the path is None, or
nothing is at the
+ path, None is returned. A variant null at the path gives None, unless
return_type is
+ omitted or VARIANT, which keeps it as a variant null. If the value
cannot be cast to
+ return_type, an error is thrown. To return None instead, use
+ :func:`~Expression.try_variant_get`.
Review Comment:
JSON_LENGTH has the same issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
