Hi Sergey, Thanks for the message! I took a look at our support for UTF_16 and I don't think any additional UTF_16 support is necessary for the scope of this FLIP. For UTF-16 users should use Decode instead of Cast. Flink stores all strings as UTF-8 internally, so DECODE(bytes, 'UTF-16') already handles that case - it converts to a Java String and re-encodes as UTF-8 before storage. Since the internal representation is always UTF-8, the validation problem is fundamentally a UTF-8 concern and the functions are correctly named. Adding validation functions support for all the character sets we support for DECODE (‘US-ASCII’, ‘ISO-8859-1’, ‘UTF-8’, ‘UTF-16BE’, ‘UTF-16LE’, ‘UTF-16) might be something useful for some less frequent use cases but I'd say this is out-of-scope for this FLIP.
Kind regards, Gustavo On Fri, 20 Mar 2026 at 20:24, Sergey Nuyanzin <[email protected]> wrote: > Hi Gustavo > thanks for the proposal > > I noticed that you are proposing usage of UTF8 in names (default cast > to string is also using utf8) > however I wonder if it makes sense to introduce similar utf16 similar > functions as Flink supports this as well? > > On Fri, Mar 20, 2026 at 8:09 PM Gustavo de Morais > <[email protected]> wrote: > > > > Hi Xuyang and Timo, > > > > Thanks for the positive feedback! Regarding your suggestions, Xuyang: > > > > 1. Yes, good point - we should add the fromUtf8Bytes(byte[], int, int) > > overload as well. > > 2. This is also relevant. If we want to do validation during ingestion > > time, this might have performance implications. Since these are @Internal > > APIs, they can be changed independently from the FLIP afterwards if it > > makes sense. What's your opinion? > > > > Kind regards, > > Gustavo > > > > On Fri, 20 Mar 2026 at 08:44, Xuyang <[email protected]> wrote: > > > > > Hi, Gustavo. > > > Great catch! Thanks for driving this FLIP. Overall LGTM. I just have > two > > > minor points I'd like to confirm with you. > > > 1. Should we also add the overload function `fromUtf8Bytes(byte[], int, > > > int)` in StringData? > > > 2. Callers like `ColumnarRowData#getString` and > > > `ColumnarArrayData#getString` call `StringData.fromBytes` directly. > Should > > > these call sites be migrated in a follow-up, or intentionally left > as-is? > > > > > > > > > > > > > > > > > > -- > > > > > > Best！ > > > Xuyang > > > > > > > > > > > > At 2026-03-19 22:37:28, "Timo Walther" <[email protected]> wrote: > > > >Hi Gustavo, > > > > > > > >thank you for this excellent design document. And thanks for > discovering > > > >this data loss and driving the investigation. We should definitely fix > > > >this shortcoming. Also looking at other vendors, it is definitly a > cause > > > >for false assumptions that lead to hard-to-debug inconsistencies. > > > > > > > >+1 for this proposal. > > > > > > > >Cheers, > > > >Timo > > > > > > > > > > > >On 19.03.26 15:23, Gustavo de Morais wrote: > > > >> Hi everyone, > > > >> > > > >> Currently, CAST(bytes AS STRING) silently replaces any invalid UTF-8 > > > byte > > > >> with U+FFFD (?). The substitution is irreversible and produces no > > > warning - > > > >> the pipeline keeps running while data is permanently corrupted > > > >> downstream. This also means that a CAST from BYTES → STRING → BYTES > is > > > not > > > >> idempotent, which prevents the engine from applying certain > > > optimizations. > > > >> For example, for preserving upsert keys after such CASTs. > > > >> > > > >> I'd like to start a discussion around defining and improving the > default > > > >> behavior. I've written a short FLIP [1] proposing new utility > functions > > > to > > > >> handle this explicitly - similar to what other engines like Spark > > > already > > > >> do - and changing the default behavior to throw an error instead of > > > >> silently corrupting data, while giving users clear options to deal > with > > > >> invalid bytes. > > > >> > > > >> Looking forward to your feedback and thoughts. > > > >> > > > >> Kind regards, > > > >> Gustavo > > > >> > > > >> [1] > > > >> > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-568%3A+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities > > > >> > > > > > > > -- > Best regards, > Sergey >
