[
https://issues.apache.org/jira/browse/ARROW-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667659#comment-16667659
]
Daniel Lemire commented on ARROW-3536:
--------------------------------------
I totally understand and respect your distaste for intrinsic functions and
explicit vectorization. There is work to help solve this problem, including
work being done at the C++ standard level (and also within Java).
Your fast ASCII approach, and it is indeed almost as fast as a version using
vector instructions, is something I would qualify as being SWAR (SIMD Within A
Register) so it is SIMD technically speaking, using a 64-bit word as if it were
8 distinct byte-sized values. I am sure you will agree that it is not optimized
for readability... the simple code I just posted is probably better for
readability. So I think that the readability issue you object to as to do,
specifically, with intrinsic functions. The SWAR approach is indeed more
portable, and that's an appealing advantage. I would definitively favor it over
an approaching using mysterious-looking intrinsic functions, on portability
grounds alone.
As for autovectorization... Indeed, I am pretty sure that GNU GCC does not
autovectorize with -O2, so that's a good point. But that's also a shame (I
prefer -O3 with GCC) because, in this instance, you can write naive code,
optimized for simplicity and readability and let the optimizing compiler do the
heavy lifting for you. I think that using modern-day compilers without at
least autovectorization leaves performance on the table... but even there,
there is ample room for disagreements (e.g., as far as I can tell, the Go
compiler does not autovectorize, though Java certainly does). Note that if you
do rely on autovectorization, you have to make sure that the target machine
supports the prefered instructions.
> [C++] Fast UTF8 validation functions
> ------------------------------------
>
> Key: ARROW-3536
> URL: https://issues.apache.org/jira/browse/ARROW-3536
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Priority: Major
> Fix For: 0.13.0
>
>
> [~lemire] discusses this topic in
> https://lemire.me/blog/2018/05/16/validating-utf-8-strings-using-as-little-as-0-7-cycles-per-byte/
> In Java there is also
> https://lemire.me/blog/2018/10/16/validating-utf-8-bytes-java-edition/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
