Nikita Gorbachevski created SPARK-29550:
-------------------------------------------
Summary: Enhance locking in session catalog
Key: SPARK-29550
URL: https://issues.apache.org/jira/browse/SPARK-29550
Project: Spark
Issue Type: Bug
Components: Spark Core, SQL
Affects Versions: 2.4.4
Reporter: Nikita Gorbachevski
In my streaming application``spark.streaming.concurrentJobs`` is set to 50
which is used as size for underlying thread pool. I automatically create/alter
tables/view in runtime. I order to do that i invoke ``create ... if not exists
operations`` on driver on each batch invocation. Once i noticed that most of
batch time is spent on driver but not on executors. I did a thread dump and
figured out that most of the threads are blocked on SessionCatalog waiting for
a lock.
Existing implementation of SessionCatalog uses a single lock which is used
almost by all the methods to guard ``currentDb`` and ``tempViews`` variables. I
propose to enhance locking behaviour of SessionCatalog by :
# Employing ReadWriteLock which allows to execute read operations
concurrently.
# Replace synchronized with the corresponding read or write lock.
Also it's possible to go even further and strip locks for ``currentDb`` and
``tempViews`` but i'm not sure whether it's possible from the implementation
point of view. Probably someone will help me with this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
