[
https://issues.apache.org/jira/browse/HDFS-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551317#comment-16551317
]
Barnabas Maidics edited comment on HDFS-13752 at 7/20/18 9:34 PM:
------------------------------------------------------------------
Hi [[email protected]],
I used Hive 2.1.1 and hadoop 2.7.6 locally and used VisualVM to create the heap
dump and for analysis as well. I can't attach the original heap dump because of
its size, but I've uploaded it on drive:
[https://drive.google.com/open?id=167aLQRSW2LW2qrqr6P5UcwNqSPSCCQkN]
You're right that the 42% is the retained memory, but even the URI-s itself
take 8.1%. I worked with 100k partitions (with only ~20 row in each) which may
not be the most realistic scenario but I was measuring the effect of increasing
the partition numbers.
was (Author: b.maidics):
Hi [[email protected]],
I used Hive 2.1.1 and hadoop 2.7.6 locally and used VisualVM to create the heap
dump and for analysis as well. I can't attach the original heap dump because of
its size, but I've uploaded it on drive:
https://drive.google.com/open?id=167aLQRSW2LW2qrqr6P5UcwNqSPSCCQkN
You're right that the 42% is the retained memory, but even the URI-s itself
take 8.1%. I worked with 100k partitions which may not be the most realistic
scenario but I was measuring the effect of increasing the partition numbers.
> fs.Path stores file path in java.net.URI causes big memory waste
> ----------------------------------------------------------------
>
> Key: HDFS-13752
> URL: https://issues.apache.org/jira/browse/HDFS-13752
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: fs
> Reporter: Barnabas Maidics
> Priority: Major
> Attachments: Screen Shot 2018-07-20 at 11.12.38.png
>
>
> I was looking at HiveServer2 memory usage, and a big percentage of this was
> because of org.apache.hadoop.fs.Path, where you store file paths in a
> java.net.URI object. The URI implementation stores the same string in 3
> different objects (see the attached image). In Hive when there are many
> partitions this cause a big memory usage. In my particular case 42% of memory
> was used by java.net.URI so it could be reduced to 14%.
> I wonder if the community is open to replace it with a more memory efficient
> implementation and what other things should be considered here? It can be a
> huge memory improvement for Hadoop and for Hive as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
