[
https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148963#comment-14148963
]
ASF GitHub Bot commented on HDFS-6914:
--------------------------------------
GitHub user haoch opened a pull request:
https://github.com/apache/hadoop-common/pull/25
HDFS-6914. Resolve huge memory consumption Issue with OIV processing
PB-based fsimages
For better managing and supporting a lot of large hadoop clusters in
production, we internally need to automatically export fsimage to delimited
text files in LSR style and then analyse with hive or pig or build system
metrics for real-time analyzing.
However due to the internal layout changes introduced by the
protobuf-based fsimage, OIV processing program consumes excessive amount of
memory. For example, in order to export the fsimage in size of 8GB, it should
have taken about 85GB memory which is really not reasonable and impacted
performance of other services badly in the same server.
To resolve above problem, I submit this patch which will reduce memory
consumption of OIV LSR processing by 50%.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/haoch/hadoop-common HDFS-6914-branch-2.4.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hadoop-common/pull/25.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #25
----
commit 340e333f627b3d64c9463819da04f97a3deecb6b
Author: Chen,Hao <cn.haoc...@gmail.com>
Date: 2014-09-26T09:23:24Z
HDFS-6914. Resolve huge memory consumption Issue with OIV processing
PB-based fsimages
----
> Resolve huge memory consumption Issue with OIV processing PB-based fsimages
> ---------------------------------------------------------------------------
>
> Key: HDFS-6914
> URL: https://issues.apache.org/jira/browse/HDFS-6914
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.1
> Reporter: Hao Chen
> Labels: hdfs
> Attachments: HDFS-6914.patch, HDFS-6914.v2.patch
>
>
> For better managing and supporting a lot of large hadoop clusters in
> production, we internally need to automatically export fsimage to delimited
> text files in LSR style and then analyse with hive or pig or build system
> metrics for real-time analyzing.
> However due to the internal layout changes introduced by the protobuf-based
> fsimage, OIV processing program consumes excessive amount of memory. For
> example, in order to export the fsimage in size of 8GB, it should have taken
> about 85GB memory which is really not reasonable and impacted performance of
> other services badly in the same server.
> To resolve above problem, I submit this patch which will reduce memory
> consumption of OIV LSR processing by 50%.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
