[jira] [Created] (IMPALA-6568) DDL statements and possibly others do not contain the Query Compilation timeline
Alexander Behm created IMPALA-6568:
--
Summary: DDL statements and possibly others do not contain the
Query Compilation timeline
Key: IMPALA-6568
URL: https://issues.apache.org/jira/browse/IMPALA-6568
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 2.11.0, Impala 2.10.0, Impala 2.9.0
Reporter: Alexander Behm
Some statements do not seem to include the "Query Compilation" timeline in the
query profile.
Repro:
{code}
create table t (i int);
describe t; <-- loads the table, but no FE timeline in profile
invalidate metadata t;
alter table t set tbproperties('numRows'='10'); <-- loads the table, but no FE
timeline in profile
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-6566) TestAdmissionController::test_require_user failed due to Impalad not starting
Tim Armstrong created IMPALA-6566:
-
Summary: TestAdmissionController::test_require_user failed due to
Impalad not starting
Key: IMPALA-6566
URL: https://issues.apache.org/jira/browse/IMPALA-6566
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Reporter: Tim Armstrong
Assignee: Tim Armstrong
{noformat}
15:51:56 ERROR
custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_require_user
15:51:56 ERRORS
15:51:56 _ ERROR at setup of TestAdmissionController.test_require_user
__
15:51:56 common/custom_cluster_test_suite.py:109: in setup_method
15:51:56 self._start_impala_cluster(cluster_args)
15:51:56 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
15:51:56 check_call(cmd + options, close_fds=True)
15:51:56 /usr/lib64/python2.6/subprocess.py:505: in check_call
15:51:56 raise CalledProcessError(retcode, cmd)
15:51:56 E CalledProcessError: Command
'['/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/bin/start-impala-cluster.py',
'--cluster_size=3', '--num_coordinators=3',
'--log_dir=/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests',
'--log_level=1', '--impalad_args="-vmodule admission-controller=3
-fair_scheduler_allocation_path
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/fair-scheduler-test2.xml
-llama_site_path
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/llama-site-test2.xml
-disable_admission_control=false -require_username" ',
'--state_store_args="-statestore_heartbeat_frequency_ms=100
-statestore_priority_update_frequency_ms=100" ']' returned non-zero exit status
1
15:51:56 Captured stdout setup
-
15:51:56 Starting State Store logging to
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
15:51:56 Starting Catalog Service logging to
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
15:51:56 Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
15:51:56 Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
15:51:56 Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
15:51:56 Error starting cluster: Expected 3 impalad(s), only 2 found
15:51:56
15:51:56 Captured stderr setup
-
15:51:56 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
15:51:56 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
15:51:56 = 76 passed, 38 skipped, 1 xfailed, 1 error in 5542.39 seconds
=
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-6564) Queries randomly fail with "CANCELLED" due to a race with IssueInitialRanges()
[
https://issues.apache.org/jira/browse/IMPALA-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-6564.
---
Resolution: Not A Bug
It turns out that the issue was introduced in one of my patches:
https://gerrit.cloudera.org/#/c/8707/32/be/src/runtime/io/disk-io-mgr.cc
This check was saving us:
{noformat}
Status DiskIoMgr::AddScanRanges(RequestContext* reader,
const vector& ranges, bool schedule_immediately) {
if (ranges.empty()) return Status::OK();
{noformat}
> Queries randomly fail with "CANCELLED" due to a race with IssueInitialRanges()
> --
>
> Key: IMPALA-6564
> URL: https://issues.apache.org/jira/browse/IMPALA-6564
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: flaky
>
> I've been chasing a flaky test that I saw in test_basic_runtime_filters when
> running against https://gerrit.cloudera.org/#/c/8966/ (the scanner buffer
> pool changes).
> I think it is a latent bug that has started reproducing more frequently. What
> I've found is:
> * Different queries fail with CANCELLED. I can repro it on my branch ~3/4
> times by running: impala-py.test tests/query_test/test_runtime_filters.py -n8
> --verbose --maxfail 1 -k basic . It happens with a variety of queries and
> file formats.
> * It seems to happen when all files are pruned out by runtime filters
> * Logging reveals IssueInitialRanges() fails with a CANCELLED status, which
> propagates up to the query status:
> {code}
> if (!initial_ranges_issued_) {
> // We do this in GetNext() to maximise the amount of work we can do while
> waiting for
> // runtime filters to show up. The scanner threads have already started
> (in Open()),
> // so we need to tell them there is work to do.
> // TODO: This is probably not worth splitting the organisational cost of
> splitting
> // initialisation across two places. Move to before the scanner threads
> start.
> Status status = IssueInitialScanRanges(state);
> if (!status.ok()) LOG(INFO) << runtime_state_->fragment_instance_id() <<
> " IssueInitialRanges() failed with status: " << status.GetDetail() << " " <<
> (void*) this;
> {code}
> * It appears that the CANCELLED comes from DiskIoMgr::AddScanRanges().
> * That function returned cancelled because a scanner thread noticed that the
> scan was complete here and cancelled the RequestContext:
> {code}
> // Done with range and it completed successfully
> if (progress_.done()) {
> // All ranges are finished. Indicate we are done.
> LOG(INFO) << runtime_state_->fragment_instance_id() << " All ranges
> done " << (void*) this;
> SetDone();
> break;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-6565) Stress test with KRPC enabled shows inconsistent results for some queries
Michael Ho created IMPALA-6565:
--
Summary: Stress test with KRPC enabled shows inconsistent results
for some queries
Key: IMPALA-6565
URL: https://issues.apache.org/jira/browse/IMPALA-6565
Project: IMPALA
Issue Type: Bug
Components: Distributed Exec
Affects Versions: Impala 3.0, Impala 2.12.0
Reporter: Michael Ho
Assignee: Michael Ho
Stress test in an 8 node cluster running at commit
881e00a8bff0469ab7860bcd0d4d4794fb04a4b8 shows some queries producing wrong
results.
{noformat}
Result hash mismatch; expected 3440873535184108466, got 3440873535184106162
01:26:30 Query ID:
c747bc20685116e8:fbac7a71
01:26:30 Query: select
01:26:30 p_brand,
01:26:30 p_type,
01:26:30 p_size,
01:26:30 count(distinct ps_suppkey) as supplier_cnt
01:26:30 from
01:26:30 partsupp,
01:26:30 part
01:26:30 where
01:26:30 p_partkey = ps_partkey
01:26:30 and p_brand <> 'Brand#45'
01:26:30 and p_type not like 'MEDIUM POLISHED%'
01:26:30 and p_size in (49, 14, 23, 45, 19, 3, 36, 9)
01:26:30 and ps_suppkey not in (
01:26:30 select
01:26:30 s_suppkey
01:26:30 from
01:26:30 supplier
01:26:30 where
01:26:30 s_comment like '%Customer%Complaints%'
01:26:30 )
01:26:30 group by
01:26:30 p_brand,
01:26:30 p_type,
01:26:30 p_size
01:26:30 order by
01:26:30 supplier_cnt desc,
01:26:30 p_brand,
01:26:30 p_type,
01:26:30 p_size
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-6538) Fix read path when Parquet min(_value)/max(_value) statistics contain NaN
[ https://issues.apache.org/jira/browse/IMPALA-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Borók-Nagy resolved IMPALA-6538. --- Resolution: Fixed Fix Version/s: Impala 2.12.0 Impala 3.0 Fixed by https://github.com/apache/impala/commit/881e00a8bff0469ab7860bcd0d4d4794fb04a4b8 > Fix read path when Parquet min(_value)/max(_value) statistics contain NaN > - > > Key: IMPALA-6538 > URL: https://issues.apache.org/jira/browse/IMPALA-6538 > Project: IMPALA > Issue Type: Sub-task >Reporter: Zoltán Borók-Nagy >Assignee: Zoltán Borók-Nagy >Priority: Major > Fix For: Impala 3.0, Impala 2.12.0 > > > (I'll only write min and max, but I'll also mean min_value and max_value by > that) > When both min and max is NaN: > * Written by Impala: > ** first element in the row group is NaN, but not all of them (Impala writer > bug) > ** all element is NaN > * Written by Hive/Parquet-mr: > ** all element is NaN > Either min or max is NaN, but not both: > * Written by Impala: > ** this cannot happen currently > * Written by Hive/Parquet-mr: > ** only the max can be NaN (needs to be checked) > Therefore, if both min and max is NaN, we can't use the statistics for > filtering. > If only the max is NaN, we still have a valid lower bound. > > A workaround can be to change the NaNs to infinities, ie. max => Inf, min => > -Inf > Based on my experiments, min/max statistics are not applied to predicates > that can be true for NaN, e.g. 'NOT x < 3' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
