[
https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hao Chen updated HDFS-6914:
---------------------------
Attachment: HDFS-6914.v2.patch
Clean code to keep changes minimal and make the coding style consistent
> Resolve huge memory consumption Issue with OIV processing PB-based fsimages
> ---------------------------------------------------------------------------
>
> Key: HDFS-6914
> URL: https://issues.apache.org/jira/browse/HDFS-6914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.4.1
> Reporter: Hao Chen
> Labels: hdfs
> Attachments: HDFS-6914.patch, HDFS-6914.v2.patch
>
>
> For better managing and supporting a lot of large hadoop clusters in
> production, we internally need to automatically export fsimage to delimited
> text files in LSR style and then analyse with hive or pig or build system
> metrics for real-time analyzing.
> However due to the internal layout changes introduced by the protobuf-based
> fsimage, OIV processing program consumes excessive amount of memory. For
> example, in order to export the fsimage in size of 8GB, it should have taken
> about 85GB memory which is really not reasonable and impacted performance of
> other services badly in the same server.
> To resolve above problem, I submit this patch which will reduce memory
> consumption of OIV LSR processing by 50%.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
