[
https://issues.apache.org/jira/browse/ARROW-13879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17423127#comment-17423127
]
Eduardo Ponce commented on ARROW-13879:
---------------------------------------
Adding "true" binary support to Arrow (ie. not encoded as ASCII/UTF-8) seems to
be more involved than expected. Most of the kernels listed in this PR are
implemented in such a way that they could in theory work with non-encoded
binary data, but the fact that *std::string* and *std::string_view* are used in
certain parts to hold the binary data prevents "true" binary support because of
they are treated as [null-terminating
strings|https://github.com/apache/arrow/blob/master/cpp/src/arrow/vendored/string_view.hpp#L421].
Therefore, binary data containing a NULL byte (_0x00_) will yield incorrect
results.
Two possible solutions are:
* to consider _BinaryTypes_ different from _StringTypes_ where the former
always requires an explicit _length_ and the latter can depend on
null-terminating strings
* to treat strings as non-null-terminating sequence of bytes and wrap them in a
string-like container that uses an explicit _size_ instead of depending on a
NULL value
For this issue's PR, I am adding partial binary support to all the kernels
since a null byte is not supported.
> [C++] Mixed support for binary types in regex functions
> -------------------------------------------------------
>
> Key: ARROW-13879
> URL: https://issues.apache.org/jira/browse/ARROW-13879
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Reporter: Weston Pace
> Assignee: Eduardo Ponce
> Priority: Major
> Labels: kernel, pull-request-available, types
> Fix For: 6.0.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> The functions count_substring, count_substring_regex, find_substring, and
> find_substring_regex all accept binary types but the function extract_regex,
> match_substring, match_substring_regex, match_like, starts_with, ends_with,
> split_pattern, and split_pattern_regex do not.
> They should all accept binary types.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
