[
https://issues.apache.org/jira/browse/IMPALA-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234189#comment-17234189
]
Tim Armstrong commented on IMPALA-10301:
----------------------------------------
I saw this again attaching output [^failure-output.txt] and logs for the
cluster [^test_mixed_catalog_ddls_with_invalidate_metadata.tar.gz]
I don't really understand what's going on here - it looks like the fragment
execution thread just gets stuck:
{noformat}
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.227144 10515 impala-server.cc:1242]
4a4d116a69a25cec:bd01636000000000] Registered query
query_id=4a4d116a69a25cec:bd01636000000000
session_id=00422a975195e051:c16549c5d1ebff89
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.227378 10515 Frontend.java:1532] 4a4d116a69a25cec:bd01636000000000]
Analyzing query: insert overwrite table
test_mixed_catalog_ddls_with_invalidate_metadata_8d71c4db.test_16_part
partition(j=2) values (1), (2), (3), (4), (5) db: default
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.230790 10515 BaseAuthorizationChecker.java:110]
4a4d116a69a25cec:bd01636000000000] Authorization check took 3 ms
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.230827 10515 Frontend.java:1574] 4a4d116a69a25cec:bd01636000000000]
Analysis and authorization finished.
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:W1107
13:26:55.233633 11230 executor-group.cc:164]
4a4d116a69a25cec:bd01636000000000] Executor group empty group (using
coordinator only) is unhealthy: 0 out of 1 are available.
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.233772 11230 admission-controller.cc:1646]
4a4d116a69a25cec:bd01636000000000] Trying to admit
id=4a4d116a69a25cec:bd01636000000000 in pool_name=default-pool
executor_group_name=empty group (using coordinator only)
per_host_mem_estimate=10.00 MB dedicated_coord_mem_estimate=100.00 MB
max_requests=-1 max_queued=200 max_mem=-1.00 B
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.233791 11230 admission-controller.cc:1654]
4a4d116a69a25cec:bd01636000000000] Stats: agg_num_running=2, agg_num_queued=0,
agg_mem_reserved=0, local_host(local_mem_admitted=20.00 MB,
num_admitted_running=2, num_queued=0, backend_mem_reserved=0, topN_query_stats:
queries=[1449f81ec765da17:9bd819d500000000, ee44602dffb7c0df:9fa151a700000000,
ad4e776f0df45e14:679d89b000000000], total_mem_consumed=0; pool_level_stats:
num_running=3, min=0, max=0, pool_total_mem=0, average_per_query=0)
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.233819 11230 admission-controller.cc:1200]
4a4d116a69a25cec:bd01636000000000] Admitting query
id=4a4d116a69a25cec:bd01636000000000
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.233866 11230 impala-server.cc:2062]
4a4d116a69a25cec:bd01636000000000] Registering query locations
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.233911 11230 coordinator.cc:149] 4a4d116a69a25cec:bd01636000000000]
Exec() query_id=4a4d116a69a25cec:bd01636000000000 stmt=insert overwrite table
test_mixed_catalog_ddls_with_invalidate_metadata_8d71c4db.test_16_part
partition(j=2) values (1), (2), (3), (4), (5)
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.234105 11230 coordinator.cc:473] 4a4d116a69a25cec:bd01636000000000]
starting execution on 1 backends for query_id=4a4d116a69a25cec:bd01636000000000
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.234588 9840 control-service.cc:142]
4a4d116a69a25cec:bd01636000000000] ExecQueryFInstances():
query_id=4a4d116a69a25cec:bd01636000000000
coord=impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com:27000
#instances=1
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.235033 11230 coordinator.cc:532] 4a4d116a69a25cec:bd01636000000000]
started execution on 1 backends for query_id=4a4d116a69a25cec:bd01636000000000
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.235210 11233 query-state.cc:897] 4a4d116a69a25cec:bd01636000000000]
Executing instance. instance_id=4a4d116a69a25cec:bd01636000000000
fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
impalad.impala-ec2-centos74-m5-4xlarge-ondemand-00e0.vpc.cloudera.com.jenkins.log.INFO.20201107-132618.9074:I1107
13:26:55.235864 11231 coordinator.cc:821] Coordinator waiting for backends to
finish, 1 remaining. query_id=4a4d116a69a25cec:bd01636000000000
{noformat}
> Improvement to test_local_catalog_ddls_with_invalidate_metadata_sync_ddl
> ------------------------------------------------------------------------
>
> Key: IMPALA-10301
> URL: https://issues.apache.org/jira/browse/IMPALA-10301
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Qifan Chen
> Priority: Blocker
> Labels: broken-build, flaky
> Attachments: failure-output.txt,
> test_mixed_catalog_ddls_with_invalidate_metadata.tar.gz
>
>
> In impala-cdpd-master-staging-core-s3 test, the following error was seen
> (insert into a partitioned table timeout).
> Error Message
> {code:java}
> AssertionError: Query timeout(60s): insert overwrite table
> test_local_catalog_ddls_with_invalidate_metadata_sync_ddl_b87f02d6.test_2_part
> partition(j=2) values (1), (2), (3), (4), (5) assert False
> Stacktrace
> {code}
> {code:java}
> custom_cluster/test_concurrent_ddls.py:83: in
> test_local_catalog_ddls_with_invalidate_metadata_sync_ddl
> self._run_ddls_with_invalidation(unique_database, sync_ddl=True)
> custom_cluster/test_concurrent_ddls.py:146: in _run_ddls_with_invalidation
> for i in pool.imap_unordered(run_ddls, xrange(1, NUM_ITERS + 1)):
> /usr/lib64/python2.7/multiprocessing/pool.py:655: in next
> raise value
> E AssertionError: Query timeout(60s): insert overwrite table
> test_local_catalog_ddls_with_invalidate_metadata_sync_ddl_b87f02d6.test_2_part
> partition(j=2) values (1), (2), (3), (4), (5)
> E assert False
> {code}
> The URL is
> https://master-02.jenkins.cloudera.com/view/Impala/view/Evergreen-cdpd-master-staging/job/impala-cdpd-master-staging-core-s3/lastCompletedBuild/testReport/custom_cluster.test_concurrent_ddls/TestConcurrentDdls/test_local_catalog_ddls_with_invalidate_metadata_sync_ddl/.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
