[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6293:
-----------------------------
Assignee: Haohui Mai
Status: Patch Available (was: Open)
To demonstrate my points, I'm attaching a patch which stores the current
PB-based fsimage into a LevelDB, and perform lsr on top of the LevelDB.
The tool that converts the fsimage into LevelDB reads the whole {{INODE_DIR}}
section into memory, then store the json representation of each inode into the
key {{IN || parent_id || localName}}. That way all children for a particular
inode are co-located thus it is efficient to run operations like lsr.
The conversion tool takes 16bytes * number of inodes to convert the fsimage.
For a fsimage that have 400M inodes, the tool takes around 6.4G of memory,
which could be run on a commodity machine. The lsr tool only requires O(1)
memory.
> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>
> Key: HDFS-6293
> URL: https://issues.apache.org/jira/browse/HDFS-6293
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Kihwal Lee
> Assignee: Haohui Mai
> Priority: Blocker
> Attachments: HDFS-6293.000.patch, Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf.
> Due to the internal layout changes introduced by the protobuf-based fsimage,
> OIV consumes excessive amount of memory. We have tested with a fsimage with
> about 140M files/directories. The peak heap usage when processing this image
> in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
> the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
> heap (max new size was 1GB). It should be possible to process any image with
> the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.
> I also noticed that the secret manager section has no tokens while there were
> unexpired tokens in the original image (pre-2.4.0). I did not check whether
> they were also missing in the new pb fsimage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
