[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989348#comment-13989348
]
Haohui Mai commented on HDFS-6293:
----------------------------------
Just to recap:
# The requirement is to build an offline tool that can process PB-based fsimage.
# The namespace is mostly a hierarchal structure with snapshots. That is the
exact reason why the PB-based fsimage have moved towards a record based storage.
# It is useful to recover the hierarchal structure in the namespace in some use
cases. Based on design choices made in (2), any in-memory processing algorithm
requires Theta(n) memory, where n is the number of inodes. It requires too much
resources.
# Various solutions to leverage the resource on SNN have been proposed.
Here are my two cents:
# Though the current sets of tools do load the whole fsimage into memory and
process it, there is no reason that any offline tool has to be implemented in
that way. For example, building an index can solve the above use cases.
# The namespace is no longer a tree with snapshots. Forcing it into a
hierarchal structure sometimes requires fitting square pegs through round pipes.
# Note that the goal of SBN / SNN are to improve the reliability of the system.
The simpler the code is, the more likely the code can be throughly reasoned
about and become more reliable. Personally I don't like any solutions that add
complexity to SBN / SNN to solve the use case of offline image viewer. It
doesn't seem right to solve an offline problem using an online machine that is
accounted for the reliability of the system.
> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>
> Key: HDFS-6293
> URL: https://issues.apache.org/jira/browse/HDFS-6293
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Kihwal Lee
> Priority: Blocker
> Attachments: HDFS-6293.000.patch, Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf.
> Due to the internal layout changes introduced by the protobuf-based fsimage,
> OIV consumes excessive amount of memory. We have tested with a fsimage with
> about 140M files/directories. The peak heap usage when processing this image
> in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
> the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
> heap (max new size was 1GB). It should be possible to process any image with
> the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.
> I also noticed that the secret manager section has no tokens while there were
> unexpired tokens in the original image (pre-2.4.0). I did not check whether
> they were also missing in the new pb fsimage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
