[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082501#comment-17082501
]
David Mollitor edited comment on HIVE-21354 at 4/13/20, 5:18 PM:
-----------------------------------------------------------------
[~pvary] I do not think that Hive has any logic that says "if a partition of a
table is locked, then the table is locked." I think it does this in a simple
way... it comes up with a list of all the required locks and the first one is
always the table lock, the rest are the required partitions. That is to say,
it takes an explicit lock on the table,... there is no logic for an implicit
table lock:
{code:none}
EXPLAIN LOCKS SELECT * FROM web_logs;
LOCK INFORMATION:
default.web_logs -> SHARED_READ
default.web_logs.date=2015-11-18 -> SHARED_READ
default.web_logs.date=2015-11-19 -> SHARED_READ
default.web_logs.date=2015-11-20 -> SHARED_READ
default.web_logs.date=2015-11-21 -> SHARED_READ
{code}
What I would expect is, for an INSERT into a specific partition, or TRUNCATE
partition statement, those queries would take a SHARED_READ lock on the
table-level and an EXCLUSIVE lock on the specific partitions.
was (Author: belugabehr):
[~pvary] I do not think that Hive has any logic that says "if a partition of a
table is locked, then the table is locked." I think it does this in a simple
way... it comes up with a list of all the required locks and the first one is
always the table lock, the rest are the required partitions. That is to say,
it takes an explicit lock on the table,... there is no logic for an implicit
table lock:
{code:none}
EXPLAIN LOCKS SELECT * FROM web_logs;
LOCK INFORMATION:
default.web_logs -> SHARED_READ
default.web_logs.date=2015-11-18 -> SHARED_READ
default.web_logs.date=2015-11-19 -> SHARED_READ
default.web_logs.date=2015-11-20 -> SHARED_READ
default.web_logs.date=2015-11-21 -> SHARED_READ
{code}
> Lock The Entire Table If Majority Of Partitions Are Locked
> ----------------------------------------------------------
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
> Affects Versions: 4.0.0, 3.2.0
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
