[
https://issues.apache.org/jira/browse/IMPALA-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771214#comment-17771214
]
ASF subversion and git services commented on IMPALA-12461:
----------------------------------------------------------
Commit 78b9285da457c6853e513f3852730867d4dbe632 in impala's branch
refs/heads/master from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=78b9285da ]
IMPALA-12461 part1: Avoid taking db/table level locks db/table self-event check
DB / Table level locks can be held for a long time by DDL/DML
operations in the catalogd. Trying to get the lock during self-event
check would mean blocking if there is an ongoing operation for
the given db/table.
This patch tries to solve it for the easy case of db/table level
events by using self-event specific locking that should be only
taken for short times to add/remove events from/to the in-flight
event list. The InFlightEvents object itself is used as lock
and no other of catalogd's lock must be acquired after it to
avoid deadlock.
Postponing solving this for partition level events as that would
be more complex, as both the partition list and the partitions'
in-flight event lists would need to be protected from parallel
operations that add/remove partitions.
Testing:
- ran event processing related tests
Change-Id: Ife455de09ab2e262bde1e4b5bd54c8c54c75f2cd
Reviewed-on: http://gerrit.cloudera.org:8080/20516
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> Avoid write lock on the table during self-event detection
> ---------------------------------------------------------
>
> Key: IMPALA-12461
> URL: https://issues.apache.org/jira/browse/IMPALA-12461
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Csaba Ringhofer
> Assignee: Csaba Ringhofer
> Priority: Major
>
> Saw some callstacks like this:
> {code}
> at
> org.apache.impala.catalog.CatalogServiceCatalog.tryLock(CatalogServiceCatalog.java:468)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.tryWriteLock(CatalogServiceCatalog.java:436)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.evaluateSelfEvent(CatalogServiceCatalog.java:1008)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.isSelfEvent(MetastoreEvents.java:609)
> at
> org.apache.impala.catalog.events.MetastoreEvents$BatchPartitionEvent.process(MetastoreEvents.java:1942)
> {code}
> At this point it was already checked that the event comes from Impala based
> on service id and now we are checking the table's self event list. Taking the
> table lock can be problematic as other DDL may took write lock at the same
> time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
