[
https://issues.apache.org/jira/browse/IMPALA-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dániel Gábor Vankó updated IMPALA-15411:
----------------------------------------
Labels: utf8 (was: )
> Add an is_valid_utf8(STRING) builtin to detect non-UTF-8 values
> ----------------------------------------------------------------
>
> Key: IMPALA-15411
> URL: https://issues.apache.org/jira/browse/IMPALA-15411
> Project: IMPALA
> Issue Type: New Feature
> Reporter: Dániel Gábor Vankó
> Assignee: Dániel Gábor Vankó
> Priority: Major
> Labels: utf8
>
> With UTF-8 validation on the Parquet write path enabled by default
> (IMPALA-12675), an INSERT/CTAS into a UTF-8-annotated STRING column now fails
> with PARQUET_INVALID_UTF8_STRING if any value is not valid UTF-8. Users
> hitting this have no easy SQL-level way to locate the offending rows in their
> source data.
> Expose a scalar builtin:
> is_valid_utf8(STRING) -> BOOLEAN
> returning true for valid UTF-8 (empty string / NULL handling to be defined),
> so users can run e.g.:
> {code:java}
> SELECT * FROM t WHERE NOT is_valid_utf8(s);{code}
> Proposal:
> - Back it with the existing impala::IsValidUtf8() in
> be/src/util/utf8-util.\{h,cc} (already SIMD-accelerated via simdutf, added in
> IMPALA-12675).
> - Register the scalar function in the builtins (frontend builtins-db/function
> registry + backend function implementation).
> - Reference it from the PARQUET_INVALID_UTF8_STRING error message and the
> impala_incompatible_changes.xml migration note so users are pointed at the
> tool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
