[
https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148691#comment-14148691
]
Hao Chen commented on HDFS-6914:
--------------------------------
Sure, I am willing if it helps, but this solution is for streaming batch
exporting based on two phases :
1) in the first phase, it will hold the attribute `name` of INode as node of
the tree structure in heap
2) in the second phase, it will traverse all INodes one by one as stream.
This is the reason why it can save most of the memory consumption. But as to
the web-based offlineimageviewer, it wants to support reactive realtime
requests so it should always hold full information i. e. all INodes in memory
where the huge memory consumption issue can't be prevented. I am not sure
whether it makes sense here ?
> Resolve huge memory consumption Issue with OIV processing PB-based fsimages
> ---------------------------------------------------------------------------
>
> Key: HDFS-6914
> URL: https://issues.apache.org/jira/browse/HDFS-6914
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.1
> Reporter: Hao Chen
> Labels: hdfs
> Attachments: HDFS-6914.patch, HDFS-6914.v2.patch
>
>
> For better managing and supporting a lot of large hadoop clusters in
> production, we internally need to automatically export fsimage to delimited
> text files in LSR style and then analyse with hive or pig or build system
> metrics for real-time analyzing.
> However due to the internal layout changes introduced by the protobuf-based
> fsimage, OIV processing program consumes excessive amount of memory. For
> example, in order to export the fsimage in size of 8GB, it should have taken
> about 85GB memory which is really not reasonable and impacted performance of
> other services badly in the same server.
> To resolve above problem, I submit this patch which will reduce memory
> consumption of OIV LSR processing by 50%.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
