Hao Chen created HDFS-6914:
------------------------------
Summary: Resolve huge memory consumption Issue with OIV processing
PB-based fsimages
Key: HDFS-6914
URL: https://issues.apache.org/jira/browse/HDFS-6914
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Hao Chen
Fix For: 2.5.1
For better managing and supporting a lot of large hadoop clusters in
production, we internally need to automatically export fsimage to delimited
text files in LSR style and then analyse with hive or pig or build system
metrics for real-time analyzing.
However due to the internal layout changes introduced by the protobuf-based
fsimage, OIV processing program consumes excessive amount of memory. For
example, in order to export the fsimage in size of 8GB, it should have taken
about 85GB memory which is really not reasonable and impacted performance of
other services badly in the same server.
To resolve above problem, I submit this patch which will reduce memory
consumption of OIV LSR processing by 50%.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
