[
https://issues.apache.org/jira/browse/FLINK-39600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116343#comment-18116343
]
Sumin Joo commented on FLINK-39600:
-----------------------------------
Hi, [~gustavodemorais]
Flink 2.4.0 is expected to enter code freeze around mid-October.
If this FLIP is planned to be included in the 2.4 release, could you please
register it on the page below?
I've gone ahead and filled in the information for now, but please feel free to
make any corrections or updates if needed.
Thank you!
[1]
[https://cwiki.apache.org/confluence/spaces/FLINK/pages/449286596/2.4+Release]
> FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities
> ---------------------------------------------------------------------
>
> Key: FLINK-39600
> URL: https://issues.apache.org/jira/browse/FLINK-39600
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
> Reporter: Gustavo de Morais
> Assignee: Gustavo de Morais
> Priority: Critical
> Fix For: 2.4.0
>
>
> {{CAST(bytes AS STRING)}} today silently replaces invalid UTF-8 with the
> Unicode replacement character {{{}U+FFFD{}}}. The substitution is
> irreversible and produces no warning - pipelines keep running while data is
> permanently corrupted downstream. This also blocks engine optimizations that
> need injective guarantees (e.g. upsert key propagation through {{{}BINARY ->
> STRING{}}}).
> [FLIP-568|https://cwiki.apache.org/confluence/display/FLINK/FLIP-568%3A+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities]
> addresses this by:
> # Making {{CAST(bytes AS STRING)}} throw on invalid UTF-8. {{TRY_CAST}}
> returns {{{}NULL{}}}. A migration flag restores the legacy behavior.
> # Adding two SQL functions:
> ** {{IS_VALID_UTF8(bytes) -> BOOLEAN}} for routing invalid records to a
> dead-letter sink
> ** {{MAKE_VALID_UTF8(bytes) -> STRING}} as the explicit, opt-in substitution
> recipe
> # Adding {{StringData.fromUtf8Bytes(byte[])}} connector API that validates
> at ingestion and throws on invalid input.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
