[
https://issues.apache.org/jira/browse/HIVE-24406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahana Bhat updated HIVE-24406:
-------------------------------
Description:
The changes to HMSHandler's getMSForConf introduced in the commit
[https://github.com/apache/hive/commit/9a47cf9f92d1c8a4e72890e3dfe2d9567f12bfb5]
makes getMSForConf and newRawStoreForConf static and hence adds an additional
HiveConf object creation step for every metastore connection created.
In a client like Presto, which creates a new metastore connection for every
query, the object creation rate shoots up leading to higher heap usage, more
frequent garbage collection, higher garbage collection times and increased
latency in the service. We noticed a constant ~100ms increase in latency of all
metastore calls during migration of HMS from 1x to 2x for Presto.
PFA the latency difference of a get_table call between 1x and 2x. Min of 22ms
in 1x vs 103ms in 2x
!Screenshot 2020-11-17 at 5.17.50 PM.png|width=591,height=222!
was:
The changes to HMSHandler's getMSForConf introduced in the commit
[https://github.com/apache/hive/commit/9a47cf9f92d1c8a4e72890e3dfe2d9567f12bfb5]
makes getMSForConf and newRawStoreForConf static and hence adds an additional
HiveConf object creation step for every metastore connection created.
In a client like Presto, which creates a new metastore connection for every
query, the object creation rate shoots up leading to higher heap usage, more
frequent garbage collection, higher garbage collection times and increased
latency in the service. We noticed a constant ~100ms increase in latency of all
metastore calls during migration of HMS from 1x to 2x for Presto.
PFA the latency difference of a get_table call between 1x and 2x.
!Screenshot 2020-11-17 at 5.17.50 PM.png|width=591,height=222!
> HiveConf creation in HiveMetastore's getMS() call adds ~100ms latency
> ---------------------------------------------------------------------
>
> Key: HIVE-24406
> URL: https://issues.apache.org/jira/browse/HIVE-24406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 2.3.4
> Reporter: Sahana Bhat
> Priority: Major
> Attachments: Screenshot 2020-11-17 at 5.17.50 PM.png
>
>
> The changes to HMSHandler's getMSForConf introduced in the commit
> [https://github.com/apache/hive/commit/9a47cf9f92d1c8a4e72890e3dfe2d9567f12bfb5]
> makes getMSForConf and newRawStoreForConf static and hence adds an
> additional HiveConf object creation step for every metastore connection
> created.
> In a client like Presto, which creates a new metastore connection for every
> query, the object creation rate shoots up leading to higher heap usage, more
> frequent garbage collection, higher garbage collection times and increased
> latency in the service. We noticed a constant ~100ms increase in latency of
> all metastore calls during migration of HMS from 1x to 2x for Presto.
> PFA the latency difference of a get_table call between 1x and 2x. Min of 22ms
> in 1x vs 103ms in 2x
> !Screenshot 2020-11-17 at 5.17.50 PM.png|width=591,height=222!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
