pitrou commented on code in PR #51237:
URL: https://github.com/apache/arrow/pull/51237#discussion_r3977597965
##########
cpp/src/arrow/compute/kernels/scalar_string_utf8.cc:
##########
@@ -547,6 +547,16 @@ struct Utf8NormalizeBase {
if (res < 0) {
return Status::Invalid("Cannot normalize utf8 string: ",
utf8proc_errmsg(res));
}
+ if (decompose_options_ & UTF8PROC_COMPOSE) {
+ // utf8proc_decompose() only decomposes; the canonical composition step
for
+ // NFC and NFKC is done in-place by utf8proc_normalize_utf32().
Review Comment:
Is it still useful that we call `utf8proc_decompose` first? Or should we
just decode to UTF32 codepoints ourselves?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
