[
https://issues.apache.org/jira/browse/HIVE-23236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094344#comment-17094344
]
Marton Bod edited comment on HIVE-23236 at 4/28/20, 9:53 AM:
-------------------------------------------------------------
Summary:
The new S4U-based locking is much slower (3+ times slower) and therefore do
not recommend changing to this implementation. Main reasons for slowness:
* The conflict check query in checkLock() cannot be reused in its current form
with S4U, because Postgres does not support S4U on UNION statements. One
alternative is to use OR clauses (as before) instead of UNIONs, but that is
significantly slower, as also shown by Denys's earlier tests. Another
alternative is to keep the unions (e.g. select from where field in (select
union select union select)), but this seems to cause lots of deadlocks,
especially on mysql, and even when it works, it's still not faster than the
current state.
* In our S4U, we need to retrieve and lock all result rows, can't use LIMIT 1
as in the conflict-check query, so it's more expensive
* The tables that make up our inner join are also bigger because we can't
narrow it down to only rows with smaller lockID
* When there's high concurrency, the new query increases load on HIVE_LOCKS
table which seems to sometimes slow things down in other queries targeting the
same table (e.g. acquire locks, commit)
So all these add up to the overall flow being slower than currently. We make
some gains on enqueue, but it appears to be offset by checkLock slowing down
too much.
Uploaded patch containing the approach and the benchmark test.
was (Author: marton bod):
Summary:
The new S4U-based locking is much slower (3+ times slower) and therefore do not
recommend changing to this implementation. Main reasons for slowness: * The
conflict check query in checkLock() cannot be reused in its current form with
S4U, because Postgres does not support S4U on UNION statements. One alternative
is to use OR clauses (as before) instead of UNIONs, but that is significantly
slower, as also shown by Denys's earlier tests. Another alternative is to keep
the unions (e.g. select from where field in (select union select union
select)), but this seems to cause lots of deadlocks, especially on mysql, and
even when it works, it's still not faster than the current state.
* In our S4U, we need to retrieve and lock all result rows, can't use LIMIT 1
as in the conflict-check query, so it's more expensive
* The tables that make up our inner join are also bigger because we can't
narrow it down to only rows with smaller lockID
* When there's high concurrency, the new query increases load on HIVE_LOCKS
table which seems to sometimes slow things down in other queries targeting the
same table (e.g. acquire locks, commit)
So all these add up to the overall flow being slower than currently. We make
some gains on enqueue, but it appears to be offset by checkLock slowing down
too much.
Uploaded patch containing the approach and the benchmark test.
> Remove the global lock from acquireLock
> ---------------------------------------
>
> Key: HIVE-23236
> URL: https://issues.apache.org/jira/browse/HIVE-23236
> Project: Hive
> Issue Type: Improvement
> Reporter: Marton Bod
> Assignee: Marton Bod
> Priority: Major
> Attachments: HIVE-23236.1.patch, HIVE-23236.2.patch,
> HIVE-23236.2.patch, HIVE-23236.3.patch, HIVE-23236.4.patch
>
>
> Currently we have a global lock (NEXT_LOCK_ID) when running enqueueLock,
> because the algorithm in checkLock requires the locks to have a well defined
> order, and also requires that every lock component is already stored in the
> RDBMS before checking the locks.
> Proposed approach:
> * Enqueue locks without a global S4U lock, using an auto-incremented value
> instead to get the next lock ID (modify next_lock_id table)
> * Before checking for lock conflicts, issue a S4U for all lock components we
> are requesting (db/table/partition level) in order to prevent other clients
> for doing interleaving conflict check (especially late-coming clients holding
> a lower lockID)
> * Conflict check algorithm to check not just for lower lockIDs, but also
> higher lockIDs which are already in 'acquired' state
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
