nilone created IMPALA-7874:
------------------------------
Summary: The submitted SQL is stuck and cannot be quickly entered
into the execution state.
Key: IMPALA-7874
URL: https://issues.apache.org/jira/browse/IMPALA-7874
Project: IMPALA
Issue Type: Bug
Components: Backend, Catalog
Affects Versions: Impala 2.12.0, Impala 2.10.0
Reporter: nilone
Attachments: KB)~_BMDT1FM4{~U]YD7N[I.png, ZK$_]9Q8B)~JNZJXZL1CAUC.png,
Z}SUTV0%(O(I3GW$K5$LBF8.png, `[TL{)]P0R0)C]2D~D~4FQQ.png,
}`W}LD7]`IG$T4]O~MMREAE.png
Due to the recent tough problems encountered in impala, we may decide to
abandon its use in production, This is a great pity. !!! We have used impala
for several years and have tried to solve some problems, but this time we have
tried a lot of trials and it has no effect.
The system applies impala to ETL data processing. Currently, the main
problem is that the SQL task startup delay submitted by the client is in the
CREATED state on the Coordinator 25000 web, not the Running state. These tasks
cannot be quickly queried in the CM. As you can see from the list, it is
possible to wait for a while to run past, or it may stay stuck and die for a
long time. The situation may seem to be related to metadata loading, related to
the catalogd service. We have more than 300,000 tables and 8 million
partitions, and have tried to reduce some useless tables, but in the end did
not capture slow queries on the Mysql metabase. we've tried to restart Hive
,Namenode.
Later, we found the problem was similar to that described by IMPALA-5058.
We tried to upgrade the version of impala to 2.12.0 using overwritten file
mode. The service log became more detailed, but it was still not solved.
Tracking the log of catalogd found the following rule. Once "Remaining items in
queue: 0, Loads in progress: 1" appears in some worker threads, the problem
will appear. We tried to analyze the jstack information of catalogd and found
that a large number of threads are in Waiting state.
This is our cluster environment:
CDH 5.13.1 impala-2.10.0 & 2.12.0
300 datanodes, 80 impalad
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
