[
https://issues.apache.org/jira/browse/IMPALA-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522592#comment-17522592
]
Quanlong Huang commented on IMPALA-7168:
----------------------------------------
I think another solution is setting a retry limit for UpdateCatalogCache().
E.g. if it continously fails 10 times, the coordinator should shutdown itself.
> DML query may hang if CatalogUpdateCallback() encounters repeated error
> -----------------------------------------------------------------------
>
> Key: IMPALA-7168
> URL: https://issues.apache.org/jira/browse/IMPALA-7168
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 3.0,
> Impala 2.12.0
> Reporter: Pranay Singh
> Priority: Major
>
> DML queries or INSERT will encounter a hang, if
> exec_env_->frontend()->UpdateCatalogCache() in
> ImpalaServer::CatalogUpdateCallback encounters repeated error like ENOMEM.
> This happens with SYNC_DDL set to 1 when the coordinator node is waiting for
> it's catalog version to become current.
> The scenario shows up like this, lets say there are two coordinator nodes ,
> Node A, Node B
> and catalogd and statestored are running on Node C.
> a) CREATE TABLE is executed on Node A, with SYNC_DDL set to 1, the thread
> running the query is going to block in
> impala::ImpalaServer::ProcessCatalogUpdateResult(), waiting for it's catalog
> version to become current.
> b) Meanwhile statestored running on Node C would call
> ImpalaServer::CatalogUpdateCallback on Node B via thrift RPC to do a delta
> topic update, which would not happen if we encounter repeated errors, say
> front end is low on memory (low JVM heap situation).
> c) In such case Node A will wait indefinitely waiting for it's catalog
> version to become current, till Node B is shutdown voluntarily.
> Note: This is a case where Node B is reachable (hearbeat is fine, but node is
> in a bad state, non working).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
