Csaba Ringhofer created IMPALA-8853:
---------------------------------------
Summary: Try to avoid leaking ACID locks/transactions
Key: IMPALA-8853
URL: https://issues.apache.org/jira/browse/IMPALA-8853
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.3.0
Reporter: Csaba Ringhofer
Impala can leak locks/transactions in the following ways:
1. commit/abort fails (e.g. because HMS is down) - currently Impala does not
have a re-try logic for these cases
2. Impala exits (e.g. crash or SIGTERM), and doesn't close it's open
locks/transactions
Hive has a timeout logic that should drop locks/transactions if there was no
heartbeat for them for some time. I think that Impala should not rely on this
completely though, because:
- the timeout time can be too long for some workflows, especially if there are
exclusive locks involved
- timeout logic doesn't seem to work in the Impala minicluster
Leak type 1. could be solved by periodically trying to abort transactions where
commit/abort failed, similarly how heartbeating works now
Leak type 2. could be solved by aborting open locks/transactions that were
opened by the given impalad/catalogd role during start up, so the what is
leaked during exit would be cleared during restart. This would need
impalad/catalogd to use unique "user" fields when acquiring locks/transactions
instead of the current "impala".
Note that the transaction opened by INSERT seem problematic, as these are
opened and possibly aborted by the coordinator, but commit happens on catalogd.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
