maartenbreddels commented on a change in pull request #8621:
URL: https://github.com/apache/arrow/pull/8621#discussion_r560244159
##########
File path: cpp/src/arrow/compute/kernels/scalar_string.cc
##########
@@ -186,16 +172,51 @@ struct UTF8Transform {
}
};
+#ifdef ARROW_WITH_UTF8PROC
+
+// transforms per codepoint
+template <typename Type, typename Derived>
+struct StringTransformCodepoint : StringTransform<Type, Derived> {
+ using Base = StringTransform<Type, Derived>;
+ using offset_type = typename Base::offset_type;
+
+ bool Transform(const uint8_t* input, offset_type input_string_ncodeunits,
+ uint8_t* output, offset_type* output_written) {
+ uint8_t* output_start = output;
+ if (ARROW_PREDICT_FALSE(
+ !arrow::util::UTF8Transform(input, input +
input_string_ncodeunits, &output,
+ Derived::TransformCodepoint))) {
+ return false;
+ }
+ *output_written = static_cast<offset_type>(output - output_start);
+ return true;
+ }
+ static int64_t MaxCodepoints(offset_type input_ncodeunits) {
+ // Section 5.18 of the Unicode spec claim that the number of codepoints
for case
+ // mapping can grow by a factor of 3. This means grow by a factor of 3 in
bytes
+ // However, since we don't support all casings (SpecialCasing.txt) the
growth
+ // is actually only at max 3/2 (as covered by the unittest).
+ // Note that rounding down the 3/2 is ok, since only codepoints encoded by
+ // two code units (even) can grow to 3 code units.
+
+ return static_cast<int64_t>(input_ncodeunits) * 3 / 2;
Review comment:
Good catch, that was completely wrong, also slightly modified the text,
this is all about units/bytes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
