[
https://issues.apache.org/jira/browse/ARROW-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653294#comment-16653294
]
Antoine Pitrou commented on ARROW-3536:
---------------------------------------
I'll note that in the (probably typical) case where we want to validate many
small strings, asymptotic bytes/cycle performance can be less important than
the setup overhead. In other words, we may need some kind of "fast path" at the
beginning for small strings.
Another possible approach is to have an AdaptiveStringBuilder that validates at
the end and yields either a String array or a Binary array. For the pure ASCII
case, we can validate the whole array at once.
> [C++] Fast UTF8 validation functions
> ------------------------------------
>
> Key: ARROW-3536
> URL: https://issues.apache.org/jira/browse/ARROW-3536
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Priority: Major
> Fix For: 0.13.0
>
>
> [~lemire] discusses this topic in
> https://lemire.me/blog/2018/05/16/validating-utf-8-strings-using-as-little-as-0-7-cycles-per-byte/
> In Java there is also
> https://lemire.me/blog/2018/10/16/validating-utf-8-bytes-java-edition/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
