[
https://issues.apache.org/jira/browse/ARROW-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654094#comment-16654094
]
Daniel Lemire commented on ARROW-3536:
--------------------------------------
[~pitrou] is correct that there is probably no need for explicit vectorization
via intrinsic functions for the ASCII checks but as I pointed out elsewhere,
that's because the compiler is smart enough to autovectorize. That is maybe
unsurprising in the sense that all we need to do is to check one bit per byte.
That's something that optimizing compilers can deal with.
In fact, you can check that GNU GCC and clang can autovectorize this function...
{code:java}
bool is_ascii(char* cs, size_t len) {
char c = 0;
for(intk = 0; k < len; k++)
c |= cs[k] & 0x80;
return c == 0;
}
{code}
So that's good!
But there is some non-trivial insight involved here: the difference between the
code you see there in C and what the compiler produces is enormous. So this
code is a *lot* more efficient than you might think at first.
_The UTF8 validation function is unreadable_
This hurts!!!
:)
Kidding aside, I am willing to help out.
(Programming with function intrinsics does tend to produce unreadable code.
There are many people, me included, trying to work on this problem to alleviate
it.)
_does it accept surrogates_
They are malformed in UTF-8, so they are rejected. I don't think that this is
even debatable.
(Of course, all code can be buggy, so it would be fair to test.)
> [C++] Fast UTF8 validation functions
> ------------------------------------
>
> Key: ARROW-3536
> URL: https://issues.apache.org/jira/browse/ARROW-3536
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Priority: Major
> Fix For: 0.13.0
>
>
> [~lemire] discusses this topic in
> https://lemire.me/blog/2018/05/16/validating-utf-8-strings-using-as-little-as-0-7-cycles-per-byte/
> In Java there is also
> https://lemire.me/blog/2018/10/16/validating-utf-8-bytes-java-edition/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
