pitrou commented on a change in pull request #11298:
URL: https://github.com/apache/arrow/pull/11298#discussion_r721339866
##########
File path: cpp/src/arrow/compute/kernels/scalar_string.cc
##########
@@ -592,6 +592,70 @@ struct Utf8ReverseTransform : public StringTransformBase {
template <typename Type>
using Utf8Reverse = StringTransformExec<Type, Utf8ReverseTransform>;
+struct Utf8NfTransformBase : public StringTransformBase {
+ int64_t MaxCodeunits(int64_t ninputs, int64_t input_ncodeunits) override {
+ // Section 5.18 of the Unicode spec claims that the number of codepoints
for case
+ // mapping can grow by a factor of 3. This means grow by a factor of 3 in
bytes
+ // However, since we don't support all casings (SpecialCasing.txt) the
growth
+ // in bytes is actually only at max 3/2 (as covered by the unittest).
+ // Note that rounding down the 3/2 is ok, since only codepoints encoded by
+ // two code units (even) can grow to 3 code units.
+ return static_cast<int64_t>(input_ncodeunits) * 3 / 2;
+ }
+
+ int64_t Transform(const uint8_t* input, int64_t input_string_ncodeunits,
+ uint8_t* output) {
+ utf8proc_uint8_t* transformed = nullptr;
+ utf8proc_option_t options = NormalizationFormOptions();
+ auto output_string_ncodeunits =
+ utf8proc_map(input, input_string_ncodeunits, &transformed, options);
+ std::memcpy(output, transformed, output_string_ncodeunits);
+ free(transformed);
+
+ return output_string_ncodeunits;
+ }
+
+ virtual utf8proc_option_t NormalizationFormOptions() { return
UTF8PROC_STABLE; }
Review comment:
It is strange to have a virtual function for this. It could simply be a
constructor argument, or something set up in an initialization step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
