[
https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107095#comment-14107095
]
Kihwal Lee commented on HDFS-6914:
----------------------------------
We do this with the fsimage_oiv/oiv_legacy that was added in HDFS-6293. The
memory requirement is constantly small regardless of the size of fsimage. Is
there any reason not to use it?
> Resolve huge memory consumption Issue with OIV processing PB-based fsimages
> ---------------------------------------------------------------------------
>
> Key: HDFS-6914
> URL: https://issues.apache.org/jira/browse/HDFS-6914
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.1
> Reporter: Hao Chen
> Labels: hdfs
> Fix For: 2.5.1
>
> Attachments: HDFS-6914.patch
>
>
> For better managing and supporting a lot of large hadoop clusters in
> production, we internally need to automatically export fsimage to delimited
> text files in LSR style and then analyse with hive or pig or build system
> metrics for real-time analyzing.
> However due to the internal layout changes introduced by the protobuf-based
> fsimage, OIV processing program consumes excessive amount of memory. For
> example, in order to export the fsimage in size of 8GB, it should have taken
> about 85GB memory which is really not reasonable and impacted performance of
> other services badly in the same server.
> To resolve above problem, I submit this patch which will reduce memory
> consumption of OIV LSR processing by 50%.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
