[
https://issues.apache.org/jira/browse/HIVE-21581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972653#comment-16972653
]
David Mollitor commented on HIVE-21581:
---------------------------------------
[~dkuzmenko] Thank you for sharing the doc.
I assume you mean 'NN' or NameNode calls.
It is not clear to me why this lock exists. Perhaps you have better
understanding of the code?
It seems to me that the lock would be superfluous since there was a
compile-time lock. Each time this {{getInputSummary}} code gets called, there
is a new thread pool that gets created (default: 15 threads) and the work is
delegated to the pool. As I understand the code, the worst-case scenario would
be 15 threads running per query. Since the compile lock is now relaxed, we may
see more threads, and therefore more connections to the NameNode, but that is
to be expected, and preferred in order to increase query compilation time. We
can doc that the number of threads-per query should be lowered as the number of
concurrent compilations is increased.
More broadly, I had previously provided an implementation that would create a
single thread pool that all queries shared. [HIVE-19847] In this way, the
number of threads required to process a query does not automatically scale
linearly with the number of queries allowed to be compiled at once, but it
never gained any interest.
> Remove Lock in GetInputSummary
> ------------------------------
>
> Key: HIVE-21581
> URL: https://issues.apache.org/jira/browse/HIVE-21581
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
> Affects Versions: 4.0.0, 3.2.0
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21581.1.patch, HIVE-21581.2.patch,
> HIVE-21581.3.patch
>
>
> Now that Hive compile lock has been relaxed in HIVE-20535, remove the
> {{getInputSummary}} lock:
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2355]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
