[
https://issues.apache.org/jira/browse/FLINK-40319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18139672#comment-18139672
]
Ramin Gharib commented on FLINK-40319:
--------------------------------------
Fixed in master:
- 7c5b2379cb5d117ed26648f14bae4c9f11649e77
Backport to release-2.4: https://github.com/apache/flink/pull/29807
The scope is narrowed to the STRING to BINARY and VARBINARY cast. The raw-bytes
accessor and the UTF-8 check on the character-string cast are left out, see
"Out of scope" in the description.
> Allow CAST from a VARIANT string value to BINARY and VARBINARY
> --------------------------------------------------------------
>
> Key: FLINK-40319
> URL: https://issues.apache.org/jira/browse/FLINK-40319
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> {{CAST(v AS BYTES)}} fails for a string variant, whatever the string holds.
> {{VariantCastUtils#toBytes}} calls {{Variant#getBytes}}, which accepts only
> the {{BYTES}} kind. For {{PARSE_JSON('"abc"')}} the getter throws a
> {{VariantTypeException}} with the message {{Expected type BYTES but got
> STRING}}.
> This is a problem for three reasons:
> * Regular SQL allows {{STRING}} to {{BINARY}}. VARIANT already allows the
> mirror direction, {{BYTES}} to a character string. Encoding a string to UTF-8
> always succeeds.
> * JSON has no binary type, so {{PARSE_JSON}} never produces {{BYTES}}. A
> variant built from JSON can never reach a binary target, such as a
> {{MAP<STRING, BYTES>}} column.
> * Every other kind fails the same way, with the getter's
> {{VariantTypeException}}. Most other readers, such as the numeric ones, throw
> a {{TableRuntimeException}} built by {{VariantCastUtils#unsupportedKind}}
> instead. Its message names the cast and the target type.
> h3. Proposal
> Change only {{VariantCastUtils#toBytes}}:
> # Accept a stored {{STRING}} value through
> {{variant.getString().getBytes(StandardCharsets.UTF_8)}}, then pad or
> truncate it like a stored {{BYTES}} value.
> # Reject every other kind with {{unsupportedKind}}.
> {{Variant}}, {{BinaryVariant}} and {{BinaryVariantUtil}} stay unchanged.
> h3. Observable changes
> ||Expression||Before||After||
> |{{CAST(PARSE_JSON('"abc"') AS
> BYTES)}}|{{VariantTypeException}}|{{x'616263'}}|
> |{{CAST(PARSE_JSON('"abc"') AS
> BINARY(2))}}|{{VariantTypeException}}|{{x'6162'}}|
> |{{TRY_CAST(PARSE_JSON('"abc"') AS BYTES)}}|{{NULL}}|{{x'616263'}}|
> |{{CAST(PARSE_JSON('true') AS
> BYTES)}}|{{VariantTypeException}}|{{TableRuntimeException}} with the
> {{unsupportedKind}} message|
> The first row flips an existing expectation in {{CastRulesTest}}, which
> asserts that {{VARIANT_BUILDER.of("foo")}} fails a {{BYTES}} cast. The commit
> message calls this out. The table of cast targets in
> {{docs/content/docs/sql/reference/data-types.md}} gets a {{STRING}} to
> {{BINARY}} and {{VARBINARY}} entry.
> h3. Limits
> A stored {{STRING}} with an invalid UTF-8 payload loses its bytes.
> {{getString}} replaces malformed input with U+FFFD, so the cast returns {{EF
> BF BD}} in its place. {{CAST(v AS STRING)}} makes the same substitution
> today. Such a value can't come from the builders, which take a Java
> {{String}}, or from {{PARSE_JSON}}, which rejects invalid UTF-8. It can only
> come from bytes wrapped without validation, such as the binary encoding of
> the raw format from FLINK-41323. Validating string payloads at ingest, as
> FLINK-40318 proposes, would close this gap. A test pins the substitution.
> h3. Out of scope
> * A raw-bytes accessor for string values, and UTF-8 validation on the
> character-string cast. Ingest validation would make both unnecessary.
> * The {{BOOLEAN}} reader and a malformed variant. FLINK-41301 covers them.
> The {{DATE}} reader already checks the kind since FLINK-41281.
> h3. Tests
> * In {{FromVariantConverterTest}}, a non-ASCII string variant reaches
> {{BYTES}}, {{VARBINARY(3)}} and {{BINARY(8)}} as its UTF-8 bytes, truncated
> or padded like a stored {{BYTES}} value. A string payload {{C3 28}} comes out
> as {{EF BF BD 28}}.
> * In {{FromVariantConverterTest}}, a boolean, an integer and an object
> variant fail a binary cast with a {{TableRuntimeException}}.
> {{CastRulesTest}} can't show the exception type, because {{CastExecutor}}
> wraps any {{Throwable}} in a {{TableRuntimeException}}.
> * In {{CastRulesTest}}, the string case of the {{BYTES}} cast passes.
> * In {{CastFunctionITCase}}, {{CAST}} of a string variant to {{BYTES}}
> returns its bytes. {{CAST}} of a boolean one fails with the
> {{unsupportedKind}} message, and {{TRY_CAST}} returns {{NULL}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
