[
https://issues.apache.org/jira/browse/IMPALA-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17334600#comment-17334600
]
ASF subversion and git services commented on IMPALA-10654:
----------------------------------------------------------
Commit af6adf7618f50444f50bc53e42b676f5a377f386 in impala's branch
refs/heads/master from Amogh Margoor
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=af6adf7 ]
IMPALA-10654: Fix precision loss in DecimalValue to double conversion.
Original approach to convert DecimalValue(internal representation
of decimals) to double was not accurate.
It was:
static_cast<double>(value_) / pow(10.0, scale).
However only integers from −2^53 to 2^53 can be represented
accurately by double precision without any loss.
Hence, it would not work for numbers like -0.43149576573887316.
For DecimalValue representing -0.43149576573887316, value_ would be
-43149576573887316 and scale would be 17. As value_ < -2^53,
result would not be accurate. In newer approach we are using third
party library https://github.com/lemire/fast_double_parser, which
handles above scenario in a performant manner.
Testing:
1. Added End to End Tests covering following scenarios:
a. Test to show precision limitation of 16 in the write path
b. DecimalValue's value_ between -2^53 and 2^53.
b. value_ outside above range but abs(value_) < UINT64_MAX
c. abs(value_) > UINT64_MAX -covers DecimalValue<__int128_t>
2. Ran existing backend and end-to-end tests completely
Change-Id: I56f0652cb8f81a491b87d9b108a94c00ae6c99a1
Reviewed-on: http://gerrit.cloudera.org:8080/17303
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Improve the precision of DecimalValue<T>::ToDouble
> --------------------------------------------------
>
> Key: IMPALA-10654
> URL: https://issues.apache.org/jira/browse/IMPALA-10654
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Amogh Margoor
> Assignee: Amogh Margoor
> Priority: Major
> Labels: ramp-up
>
> From discussion of IMPALA-10350, it was noted that
> [DecimalValue<T>::ToDouble|https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/be/src/runtime/decimal-value.inline.h#L725]
> is not accurate.
> Current approach is:
> {code:java}
> static_cast<double>(value_) / pow(10.0, scale).
> {code}
> Inaccuracy is due to fact that only integers from −2^53 to 2^53 can be
> represented accurately by double precision without any loss. Hence, above
> approach would not work for numbers like -0.43149576573887316. For
> DecimalValue representing -0.43149576573887316, value_ would be
> -43149576573887316 and scale would be 17. As value_ < -2^53, result would not
> be accurate.
> Hence through discussion in IMPALA-10350, we propose to use thirdparty
> library https://github.com/lemire/fast_double_parser, which handles above
> scenario in a performant manner. Library's internal representation of Decimal
> is similar to the Impala's DecimalValue and function
> [compute_float_64|https://github.com/lemire/fast_double_parser/blob/e4f6319bfa9cbc829f7f99ae88c1d2fb205c15e8/include/fast_double_parser.h#L232]
> can be used for the conversion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
