[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993859#comment-13993859
]
Kihwal Lee commented on HDFS-6293:
----------------------------------
Thanks for the patch, Haohui.
I tested the output generated by it. I modified the patch for easier testing.
It took 9 minutes for NN to read in a 14GB PB-based fsimage with 140M files &
directories. Saving in the old format took a little less than 5 minutes.
Adding few minutes to SBN's checkpointing time doesn't seem terrible. I was
then able to process the image using the 2.3 OIV.
I will post a patch with little bit of tweaks.
bq. Note that the v2 patch only intends to be committed to branch-2. For trunk
it might make more sense to export the records in the fsimage directly
processing tools like hive.
We now have more time to think about alternatives. Once we have a better
design, we can introduce it in branch-2 and deprecate this approach.
> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>
> Key: HDFS-6293
> URL: https://issues.apache.org/jira/browse/HDFS-6293
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Kihwal Lee
> Assignee: Haohui Mai
> Priority: Blocker
> Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
> HDFS-6293.002-save-deprecated-fsimage.patch, Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf.
> Due to the internal layout changes introduced by the protobuf-based fsimage,
> OIV consumes excessive amount of memory. We have tested with a fsimage with
> about 140M files/directories. The peak heap usage when processing this image
> in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
> the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
> heap (max new size was 1GB). It should be possible to process any image with
> the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.
> I also noticed that the secret manager section has no tokens while there were
> unexpired tokens in the original image (pre-2.4.0). I did not check whether
> they were also missing in the new pb fsimage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
