[
https://issues.apache.org/jira/browse/ARROW-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175026#comment-16175026
]
Tim Armstrong commented on ARROW-1588:
--------------------------------------
Parquet does encodes some decimals as big-endian values when encoded as
FIXED_LEN_BYTE_ARRAY but Impala byte-swaps them when reading. I always found
this curious. This was actually a major source of runtime overhead so we ended
up with a complicated SIMD byte-swap implementation
https://github.com/apache/incubator-impala/blob/2e63752858d71cc745534367a686980e060a8180/be/src/util/bit-util.cc#L210
> [C++/Format] Harden Decimal Format
> ----------------------------------
>
> Key: ARROW-1588
> URL: https://issues.apache.org/jira/browse/ARROW-1588
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, Format
> Affects Versions: 0.7.0
> Reporter: Phillip Cloud
> Assignee: Phillip Cloud
> Fix For: 0.8.0
>
>
> We should finalize and harden the decimal format. The remaining issues are
> officially writing down the choice of making every decimal value 16 bytes and
> byte order.
> For byte order we'll need to run some benchmarks to compare little endian vs
> big endian. I plan to work on this over the next week or two.
> [[email protected]] [~wesmckinn] If there are any additional items you'd
> like to see addressed here please chime in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
