[
https://issues.apache.org/jira/browse/IMPALA-7963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721649#comment-16721649
]
Joe McDonnell commented on IMPALA-7963:
---------------------------------------
Added a DCHECK in stopwatch.h:
{code:java}
uint64_t RunningTime() const {
uint64_t end = Now();
if (time_ceiling_ > 0) {
if (time_ceiling_ < start_) return 0;
if (time_ceiling_ < end) end = time_ceiling_;
}
DCHECK_GE(end, start_); <-------
return end - start_;
}
{code}
This DCHECK fires, indicating that time goes backwards on centos6. We are using
CLOCK_MONOTONIC, and apparently there are some bugs where it can go backwards.
So far, we have only seen this on centos6 (specifically 6.4).
> test_empty_build_joins failed with hdfs timeout
> -----------------------------------------------
>
> Key: IMPALA-7963
> URL: https://issues.apache.org/jira/browse/IMPALA-7963
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.2.0
> Reporter: Thomas Tauber-Marshall
> Assignee: Joe McDonnell
> Priority: Blocker
> Labels: broken-build, flaky
>
> Seen in an exhaustive build on centos6:
> {noformat}
> 05:39:09 TestJoinQueries.test_empty_build_joins[batch_size: 1 | protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: parquet/none]
> 05:39:09 [gw3] linux2 -- Python 2.6.6
> /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/bin/../infra/python/env/bin/python
> 05:39:09 query_test/test_join_queries.py:97: in test_empty_build_joins
> 05:39:09 self.run_test_case('QueryTest/empty-build-joins', new_vector)
> 05:39:09 common/impala_test_suite.py:467: in run_test_case
> 05:39:09 result = self.__execute_query(target_impalad_client, query,
> user=user)
> 05:39:09 common/impala_test_suite.py:688: in __execute_query
> 05:39:09 return impalad_client.execute(query, user=user)
> 05:39:09 common/impala_connection.py:170: in execute
> 05:39:09 return self.__beeswax_client.execute(sql_stmt, user=user)
> 05:39:09 beeswax/impala_beeswax.py:182: in execute
> 05:39:09 handle = self.__execute_query(query_string.strip(), user=user)
> 05:39:09 beeswax/impala_beeswax.py:359: in __execute_query
> 05:39:09 self.wait_for_finished(handle)
> 05:39:09 beeswax/impala_beeswax.py:380: in wait_for_finished
> 05:39:09 raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> 05:39:09 E ImpalaBeeswaxException: ImpalaBeeswaxException:
> 05:39:09 E Query aborted:hdfsOpenFile() for
> hdfs://localhost:20500/test-warehouse/alltypestiny/year=2009/month=2/090201.txt
> failed to finish before the 300 second timeout
> 05:39:09 ----------------------------- Captured stderr call
> -----------------------------
> 05:39:09 -- executing against localhost:21000
> 05:39:09 use functional_parquet;
> 05:39:09
> 05:39:09 -- 2018-12-11 03:11:34,797 INFO MainThread: Started query
> d747763f9d663cd7:9abd4b9900000000
> 05:39:09 SET batch_size=1;
> 05:39:09 SET num_nodes=0;
> 05:39:09 SET disable_codegen_rows_threshold=0;
> 05:39:09 SET disable_codegen=False;
> 05:39:09 SET abort_on_error=1;
> 05:39:09 SET exec_single_node_rows_threshold=0;
> 05:39:09 -- executing against localhost:21000
> 05:39:09 select straight_join atp.id
> 05:39:09 from alltypes atp
> 05:39:09 inner join functional.alltypestiny att on atp.id = att.id
> 05:39:09 where att.int_col = 999;
> 05:39:09
> 05:39:09 -- 2018-12-11 03:11:34,816 INFO MainThread: Started query
> 5045de8553c5843c:bdc6aa1c00000000
> 05:39:09 -- executing against localhost:21000
> 05:39:09 select straight_join atp.id
> 05:39:09 from alltypes atp
> 05:39:09 right join functional.alltypestiny att on atp.id = att.id
> 05:39:09 where att.int_col = 999;
> 05:39:09
> 05:39:09 -- 2018-12-11 03:11:35,519 INFO MainThread: Started query
> 124ef451a3f65d09:f2ae4a5d00000000
> {noformat}
> Presumably caused by IMPALA-7738
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
