[
https://issues.apache.org/jira/browse/IMPALA-9584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17071931#comment-17071931
]
Csaba Ringhofer edited comment on IMPALA-9584 at 3/31/20, 4:19 PM:
-------------------------------------------------------------------
I do have a theory here, simply avg(timestamp_col) is not deterministic:
select avg(timestamp_col) from functional_parquet.alltypes;
on my machine it returns
2009-12-31 14:30:20.341989994 or
2009-12-31 14:30:20.341990232
I think that the underlying cause is "double trouble":
- AVG() on timestamps converts timestamps to double and stores their sum in
double (this is a bad idea in my opinion, another reason to change to a
different mechanism is mentioned in IMPALA-7472)
- addition for doubles is not associative due to precision loss
So merging aggregates for more than 2 subsets (e.g splits) can lead to
different results depending on the order of the merges.
was (Author: csringhofer):
I do have a theory here, simply avg(timestamp_col) is not deterministic:
select avg(timestamp_col) from functional_parquet.alltypes;
on my machine it returns
2009-12-31 14:30:20.341989994 or
2009-12-31 14:30:20.341990232
I think that the underlying cause is "double trouble":
- AVG() on timestamps converts timestamps to double and stores their sum in
double
- addition for doubles is not associative due to precision loss
So merging aggregates for more than 2 subsets (e.g splits) can lead to
different results depending on the order of the merges.
> test_analytic_fns is flaky (small fractional differences in AVG)
> ----------------------------------------------------------------
>
> Key: IMPALA-9584
> URL: https://issues.apache.org/jira/browse/IMPALA-9584
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Csaba Ringhofer
> Priority: Major
> Labels: flaky-test
>
> The issue occurred here:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/2065/testReport/query_test.test_queries/TestQueries/test_analytic_fns_protocol__hs2___exec_option____batch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0____table_format__parquet_none_/
> The same test failed with all protocols.
> Failing query:
> https://github.com/apache/impala/blob/ebbe52b4bed944d3012e3679dc984827ce11d5a8/testdata/workloads/functional-query/queries/QueryTest/analytic-fns.test#L2
> Errors are like (hs2):
> 2009,3,6,3.666666666666667,2009-03-01 20:12:00.475000,'0','8' !=
> 2009,3,6,3.66666666667,2009-03-01 20:12:00.474999,'0','8'
> I think that the problem is not the difference in the 4th double column, as
> that depends on the client used and we do not require complete match during
> comparison. So the problem is likely to be the timestamp in the 5th column.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
