singhpratech commented on issue #51225: URL: https://github.com/apache/arrow/issues/51225#issuecomment-5579337380
Reading the kernel on main (cpp/src/arrow/compute/kernels/scalar_string_utf8.cc, `Utf8NormalizeBase`): `MakeDecomposeOptions` returns `UTF8PROC_COMPOSE` for NFC and NFKC, but the only utf8proc call the kernel makes is `utf8proc_decompose`, which decomposes regardless of that flag; composition happens in `utf8proc_reencode`, which is never called. So NFC and NFKC come out as NFD and NFKD. Calling `utf8proc_reencode` on the scratch buffer with the same options after the decompose (as `utf8proc_map` does internally) would give the composed forms. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
