Quanlong Huang created IMPALA-14614:
---------------------------------------
Summary: Processing CREATE/DROP HMS events don't need to acquire
metastoreDdlLock
Key: IMPALA-14614
URL: https://issues.apache.org/jira/browse/IMPALA-14614
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
Assignee: Quanlong Huang
When processing CREATE_TABLE, DROP_TABLE, CREATE_DATABASE, DROP_DATABASE type
of HMS notification events, the metastoreDdlLock_ is acquired, e.g.
{code}
public boolean addTableIfNotRemovedLater(long eventId,
org.apache.hadoop.hive.metastore.api.Table msTbl) throws
DatabaseNotFoundException {
getMetastoreDdlLock().lock();
...{code}
https://github.com/apache/impala/blob/780e6683a21dae3622744a82f92e155ae06e13f2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L871
https://github.com/apache/impala/blob/780e6683a21dae3622744a82f92e155ae06e13f2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L941
https://github.com/apache/impala/blob/780e6683a21dae3622744a82f92e155ae06e13f2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L993
https://github.com/apache/impala/blob/780e6683a21dae3622744a82f92e155ae06e13f2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L1019
This lock is used to protect internal catalog operation with the corresponding
external HMS RPC to make them atomic, as memtioned in the comments:
{code}
// Lock used to ensure that CREATE[DROP] TABLE[DATABASE] operations performed
in
// catalog_ and the corresponding RPC to apply the change in HMS are atomic.
private final ReentrantLock metastoreDdlLock_ = new ReentrantLock();{code}
When processing CREATE/DROP events from HMS, catalogd doesn't trigger any HMS
create/drop RPCs so actually don't need to acquire this lock. Acquiring this
lock might impact (add lag to) event processing when there are lots of
create/drop DDLs executing in catalogd which also hold this lock.
Note that before IMPALA-10502, processing CREATE/DROP events don't need to
acquire this lock:
https://github.com/apache/impala/blob/00c8e157ddcaec3f12a09ef410d8716b3dac03a4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L682
https://github.com/apache/impala/blob/00c8e157ddcaec3f12a09ef410d8716b3dac03a4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2119
After IMPALA-10502, the code changed to
https://github.com/apache/impala/blob/7f7a631/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L706
https://github.com/apache/impala/blob/7f7a631e92c69a6dafc1f25ceb407f7b79db10e9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L721
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
