[
https://issues.apache.org/jira/browse/HIVE-13749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304139#comment-15304139
]
Naveen Gangam commented on HIVE-13749:
--------------------------------------
[~thejas] Thanks for continued discussion on this. However, I am not entirely
sure I understand your comment about it being dangerous for scenarios with
embedded metastore. Could you please elaborate and help me understand?
I will lay out what I found so far so that we are all on the same page. Helps
answer [~aihuaxu] questions too.
1) In a HMS heap dump from a customer, there were 66000 instances of
Configuration class stored in a HashMap (FileSystem.Cache.CACHE). These
instances along with the set of its retained objects accounted for 95% of the
total heap. Thats huge.
2) FileSystem.Cache use a map that uses a composite key with (UGI +
Configuration) objects. Because of the way, these keys are compared or because
Configuration objects used in Hive are different (because of session params and
what not), or a combination of both, we have a lot of cache misses which result
in new entries being added to this cache.
3) This Cache is storing FileSystem objects (an abstraction to the underlying
DFS), which, I am told by Hadoop engineer, are a bit expensive to instantiate.
4) HMS uses APIs like, FileSystem.get() (in HiveMetaStore) and
Path.getFileSystem()(In Warehouse.java) that cause the cache to grow.
5) I have had the customer disable this cache entirely using a hadoop property,
and they have neither seen any issues functionally nor observed any severe
performance regression while the HMS heap size remained <350MB compared to 10GB
with the cache enabled.
Since disabling the cache entirely causes no functional regression, wouldn't
adding a call to delete elements from this cache would be fine too,
functionally? Thank you
> Memory leak in Hive Metastore
> -----------------------------
>
> Key: HIVE-13749
> URL: https://issues.apache.org/jira/browse/HIVE-13749
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 1.1.0
> Reporter: Naveen Gangam
> Assignee: Naveen Gangam
> Attachments: HIVE-13749.patch, Top_Consumers7.html
>
>
> Looking a heap dump of 10GB, a large number of Configuration objects(> 66k
> instances) are being retained. These objects along with its retained set is
> occupying about 95% of the heap space. This leads to HMS crashes every few
> days.
> I will attach an exported snapshot from the eclipse MAT.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
