Dániel Gábor Vankó created IMPALA-15411:
-------------------------------------------
Summary: Add an is_valid_utf8(STRING) builtin to detect non-UTF-8
values
Key: IMPALA-15411
URL: https://issues.apache.org/jira/browse/IMPALA-15411
Project: IMPALA
Issue Type: New Feature
Reporter: Dániel Gábor Vankó
Assignee: Dániel Gábor Vankó
With UTF-8 validation on the Parquet write path enabled by default
(IMPALA-12675), an INSERT/CTAS into a UTF-8-annotated STRING column now fails
with PARQUET_INVALID_UTF8_STRING if any value is not valid UTF-8. Users hitting
this have no easy SQL-level way to locate the offending rows in their source
data.
Expose a scalar builtin:
is_valid_utf8(STRING) -> BOOLEAN
returning true for valid UTF-8 (empty string / NULL handling to be defined), so
users can run e.g.:
{code:java}
SELECT * FROM t WHERE NOT is_valid_utf8(s);{code}
Proposal:
- Back it with the existing impala::IsValidUtf8() in
be/src/util/utf8-util.\{h,cc} (already SIMD-accelerated via simdutf, added in
IMPALA-12675).
- Register the scalar function in the builtins (frontend builtins-db/function
registry + backend function implementation).
- Reference it from the PARQUET_INVALID_UTF8_STRING error message and the
impala_incompatible_changes.xml migration note so users are pointed at the tool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
