[ 
https://issues.apache.org/jira/browse/SPARK-32118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144399#comment-18144399
 ] 

Miguel Angel Fernandez Diaz commented on SPARK-32118:
-----------------------------------------------------

We hit this contention in a long-running application (a Thrift-server-like 
service with many concurrent sessions). The metastore database sits behind a 
connection pooler. One thread waited for a connection of the metastore pool 
while holding the \{{HiveExternalCatalog}} lock, and every other session that 
read the catalog stayed BLOCKED behind it for as long as the call lasted.

I have opened a PR with a smaller change than the one this ticket asks for:
https://github.com/apache/spark/pull/60446

It does not make catalog operations concurrent. It replaces the monitor used by 
\{{HiveExternalCatalog.withClient}} with a \{{ReentrantLock}} and adds an 
opt-in static conf, \{{spark.sql.hive.externalCatalog.lockTimeoutMs}} (default 
0, disabled). When it is set, a caller that cannot get the lock in time fails 
with an \{{AnalysisException}} instead of waiting with no bound. The default 
behavior does not change.

The reason for this approach is that it does not depend on the Hive client 
being thread-safe, which was the open concern in the previous PR 
(https://github.com/apache/spark/pull/28938).

Two questions for the maintainers:
1. Is this ticket the right place for the change, or should I open a separate 
one, since it mitigates the problem but does not remove the lock?
2. Would a follow-up that runs metastore calls concurrently through a pool of 
Hive clients (one client per thread, so no shared client state) be acceptable? 
I can work on it if there is interest.

> Use fine-grained read write lock for each database in HiveExternalCatalog
> -------------------------------------------------------------------------
>
>                 Key: SPARK-32118
>                 URL: https://issues.apache.org/jira/browse/SPARK-32118
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.0.0, 3.1.0
>            Reporter: Lantao Jin
>            Priority: Major
>              Labels: pull-request-available
>
> In HiveExternalCatalog, all metastore operations are synchronized by a same 
> object lock. In a heavy traffic Spark thriftserver or Spark Driver, users's 
> queries may be stuck by any a long operation. For example, if a user is 
> accessing a table which contains mass partitions, the operation 
> loadDynamicPartitions() holds the object lock for a long time. All queries 
> are blocking to wait for the lock.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to