[
https://issues.apache.org/jira/browse/IMPALA-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17142284#comment-17142284
]
Vihang Karajgaonkar commented on IMPALA-9824:
---------------------------------------------
[~stakiar] no worries.
Yes, the constructor in Frontend.java is instantiated only for coordinators.
The second in that list is called from ImpaladCatalog which is used in the
legacy Catalog mode. It uses 0 initial connections and I am not aware of any
paths in there which would talk to HMS. Hence I don't expect that pool to grow
at all.
The upper bound on number of connections is 32 which would happen if there are
32 concurrent threads trying to do HMS operations. This max bound should not
happen in Catalog since we limit to maximum 16 concurrent metadata load
operations in CatalogService using the config {{num_metadata_loading_threads}}.
I can see that Frontend's MetastoreClientPool can in theory have 32 open
connections to HMS if there are 32 concurrent insert queries being planned on
the transactional table on that coordinator. It also depends on the kind of
operations being done in HMS since long lasting operations will reserve the
client for a longer duration and hence its is more likely that a new connection
will need to be instantiated to serve the requests. In Frontend we mostly do
open, abort or commit transaction which are not expected to take a long and
hence I think its unlikely that we will see 32 open connections from each
coordinator. It probably is a good idea to make the max pool size configurable
so that for coordinator nodes we set it a lower value.
> MetastoreClientPool should be singleton
> ---------------------------------------
>
> Key: IMPALA-9824
> URL: https://issues.apache.org/jira/browse/IMPALA-9824
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Vihang Karajgaonkar
> Assignee: Vihang Karajgaonkar
> Priority: Minor
> Fix For: Not Applicable
>
>
> Currently, the MetastoreClientPool is instantiated at multiple places in the
> code and it would be good to refactor the code to make it a singleton. Each
> MetastoreClientPool creates multiple clients to HMS and unnecessary creation
> of multiple pools could cause problems on HMS side.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
