[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082571#comment-17082571
]
Peter Vary commented on HIVE-21354:
-----------------------------------
[~belugabehr]: AFAIK checkLock checks locks hierarchically. So it prevents
aquiring table level exclusive lock if a partition level exclusive lock is
already acquired by another query.
Explicitly mentioning table level shared lock like you suggested in your
comment for drop partition is philosophical question IMHO if the handling of
the partition level lock already prevents exclusive lock on a table anyway.
With the current implementation the performance is better with fewer number of
locks.
Having extra table level lock when querying from a single partition is a bug if
the table level lock prevents dropping other partitions, which is not a desired
behaviour.
So it all comes down if the lock check does exact matches, or it checks stuff
hierarchically.
> Lock The Entire Table If Majority Of Partitions Are Locked
> ----------------------------------------------------------
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
> Affects Versions: 4.0.0, 3.2.0
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
