[
https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113415#comment-14113415
]
Karthik Kambatla commented on HDFS-6914:
----------------------------------------
Do we really want to target this to 2.5.1? If not, I would suggest targetting
2.6.
> Resolve huge memory consumption Issue with OIV processing PB-based fsimages
> ---------------------------------------------------------------------------
>
> Key: HDFS-6914
> URL: https://issues.apache.org/jira/browse/HDFS-6914
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.1
> Reporter: Hao Chen
> Labels: hdfs
> Fix For: 2.5.1
>
> Attachments: HDFS-6914.patch
>
>
> For better managing and supporting a lot of large hadoop clusters in
> production, we internally need to automatically export fsimage to delimited
> text files in LSR style and then analyse with hive or pig or build system
> metrics for real-time analyzing.
> However due to the internal layout changes introduced by the protobuf-based
> fsimage, OIV processing program consumes excessive amount of memory. For
> example, in order to export the fsimage in size of 8GB, it should have taken
> about 85GB memory which is really not reasonable and impacted performance of
> other services badly in the same server.
> To resolve above problem, I submit this patch which will reduce memory
> consumption of OIV LSR processing by 50%.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
