[
https://issues.apache.org/jira/browse/IMPALA-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer updated IMPALA-7472:
------------------------------------
Description:
TimestampValue::FromSubsecondUnixTime converts a double interpreted as unix
time in seconds to TimestampValue. Impala uses it in two cases:
1: double <-> timestamp casting
2: aggregate function AVG on timestamps - it converts timestamp to double,
calculates the average on doubles, and converts the average back to timestamp.
My concern about this is double's varying's precision depending on the distance
from 0 (1970-01-01) - subsec timestamps around 2018 are rounded/distorted with
several nanosecs, which can add up if a large number of timestamps are averaged.
I think that decimal (with nanosec precision) could be used for both cases:
1. Decimal could used as an intermediate stage - double < - >decimal and
decimal < - > timestamp conversions have well defined rounding rules since
decimal V2.
2. Decimal could be used as aggregation state - this would probably make the
AVG slower, but it would be much more precise.
was:
TimestampValue::FromSubsecondUnixTime converts a double interpreted as unix
time in seconds to TimestampValue. Impala uses it in two cases:
1: double <-> timestamp casting
2: aggregate function AVG on timestamps - it converts timestamp to double,
calculates the average on doubles, and converts the average back to timestamp.
My concern about this is double's varying's precision depending on the distance
from 0 (1970-01-01) - subsec timestamps around 2018 are rounded/distorted with
several nanosecs, which can add up if a large number of timestamps are averaged.
I think that decimal (with nanosec precision) could be used for both cases:
1. Decimal could used as an intermediate stage - double <->decimal and
decimal<-> timestamp conversions have well defined rounding rules since decimal
V2.
2. Decimal could be used as aggregation state - this would probably make the
AVG slower, but it would be much more precise.
> Consider removing TimestampValue::FromSubsecondUnixTime
> -------------------------------------------------------
>
> Key: IMPALA-7472
> URL: https://issues.apache.org/jira/browse/IMPALA-7472
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Csaba Ringhofer
> Priority: Minor
> Labels: timestamp
>
> TimestampValue::FromSubsecondUnixTime converts a double interpreted as unix
> time in seconds to TimestampValue. Impala uses it in two cases:
> 1: double <-> timestamp casting
> 2: aggregate function AVG on timestamps - it converts timestamp to double,
> calculates the average on doubles, and converts the average back to timestamp.
> My concern about this is double's varying's precision depending on the
> distance from 0 (1970-01-01) - subsec timestamps around 2018 are
> rounded/distorted with several nanosecs, which can add up if a large number
> of timestamps are averaged.
> I think that decimal (with nanosec precision) could be used for both cases:
> 1. Decimal could used as an intermediate stage - double < - >decimal and
> decimal < - > timestamp conversions have well defined rounding rules since
> decimal V2.
> 2. Decimal could be used as aggregation state - this would probably make the
> AVG slower, but it would be much more precise.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
