pitrou commented on a change in pull request #11298:
URL: https://github.com/apache/arrow/pull/11298#discussion_r741981061
##########
File path: cpp/src/arrow/compute/kernels/scalar_string.cc
##########
@@ -672,11 +678,64 @@ struct Utf8TitleTransform : public
FunctionalCaseMappingTransform {
template <typename Type>
using Utf8Title = StringTransformExec<Type, Utf8TitleTransform>;
+struct Utf8NormalizeTransform : public FunctionalCaseMappingTransform {
+ using State = OptionsWrapper<Utf8NormalizeOptions>;
+
+ const Utf8NormalizeOptions* options;
+
+ explicit Utf8NormalizeTransform(const Utf8NormalizeOptions& options)
+ : options{&options} {}
+
+ int64_t MaxCodeunits(const uint8_t* input, int64_t ninputs,
+ int64_t input_ncodeunits) override {
+ const auto option = GenerateUtf8NormalizeOption(options->method);
+ const auto n_chars =
+ utf8proc_decompose_custom(input, input_ncodeunits, NULL, 0, option,
NULL, NULL);
+
+ // convert to byte length
+ return n_chars * 4;
Review comment:
It is not obvious to me that this sufficient. Some codepoints normalize
to extremely large strings, e.g.:
```python
>>> s = "\uFDFA"
>>> len(unicodedata.normalize('NFD', s))
1
>>> len(unicodedata.normalize('NFC', s))
1
>>> len(unicodedata.normalize('NFKD', s))
18
>>> len(unicodedata.normalize('NFKC', s))
18
>>> len(unicodedata.normalize('NFKC', s).encode())
33
>>> len(unicodedata.normalize('NFKC', s).encode()) / len(s.encode())
11.0 # output utf8 can be 11 times larger than the input utf8
```
So instead of creating an output data equal to at least 11 times the size of
the input data, I think this kernel should adopt another approach and simply
grow the output data as needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
