pitrou commented on a change in pull request #11298:
URL: https://github.com/apache/arrow/pull/11298#discussion_r721349708
##########
File path: cpp/src/arrow/compute/kernels/scalar_string.cc
##########
@@ -592,6 +592,70 @@ struct Utf8ReverseTransform : public StringTransformBase {
template <typename Type>
using Utf8Reverse = StringTransformExec<Type, Utf8ReverseTransform>;
+struct Utf8NfTransformBase : public StringTransformBase {
+ int64_t MaxCodeunits(int64_t ninputs, int64_t input_ncodeunits) override {
+ // Section 5.18 of the Unicode spec claims that the number of codepoints
for case
+ // mapping can grow by a factor of 3. This means grow by a factor of 3 in
bytes
+ // However, since we don't support all casings (SpecialCasing.txt) the
growth
+ // in bytes is actually only at max 3/2 (as covered by the unittest).
+ // Note that rounding down the 3/2 is ok, since only codepoints encoded by
+ // two code units (even) can grow to 3 code units.
+ return static_cast<int64_t>(input_ncodeunits) * 3 / 2;
+ }
+
+ int64_t Transform(const uint8_t* input, int64_t input_string_ncodeunits,
+ uint8_t* output) {
+ utf8proc_uint8_t* transformed = nullptr;
+ utf8proc_option_t options = NormalizationFormOptions();
+ auto output_string_ncodeunits =
+ utf8proc_map(input, input_string_ncodeunits, &transformed, options);
+ std::memcpy(output, transformed, output_string_ncodeunits);
+ free(transformed);
Review comment:
This is definitely non-performant as it allocates a heap area for every
input string. Ideally, we should write directly into the output buffer `output`.
I'll note the utf8proc code for this:
https://github.com/JuliaStrings/utf8proc/blob/master/utf8proc.c#L729-L757
So we would need a resizable UTF32 buffer (that can be allocated only once
for the entire duration of kernel execution).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
