[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992501#comment-13992501
]
Haohui Mai commented on HDFS-6293:
----------------------------------
bq. In the results above, the amount of memory on the machine is far larger
than the image so everything happens in memory and seeks are free.
Reads on fsimage are mostly sequential, so it really doesn't matter whether the
whole fsimage can fit into the memory or not.
bq. Can you run an experiment with a large fsimage (25G or so) with a
representative fs hierarchy (not totally flat) and then generate DB and convert
to LSR on a smaller machine (16G or so)?
The fsimage that I've experimented with originates from a production cluster.
It was in the old format which requires a big machine to do convert it to a
PB-based fsimage. I have to strip it down to fit it into my machine. Please see
HDFS-5698 on how the image is generated. If you can send me your PB-based
fsimage then I can experiment with it.
Since the image comes from a production cluster, the fs hierarchy is definitely
not flat. I generated the DB by in a Java VM with 22G heap.
> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>
> Key: HDFS-6293
> URL: https://issues.apache.org/jira/browse/HDFS-6293
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Kihwal Lee
> Assignee: Haohui Mai
> Priority: Blocker
> Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
> HDFS-6293.002-save-deprecated-fsimage.patch, Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf.
> Due to the internal layout changes introduced by the protobuf-based fsimage,
> OIV consumes excessive amount of memory. We have tested with a fsimage with
> about 140M files/directories. The peak heap usage when processing this image
> in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
> the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
> heap (max new size was 1GB). It should be possible to process any image with
> the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.
> I also noticed that the secret manager section has no tokens while there were
> unexpired tokens in the original image (pre-2.4.0). I did not check whether
> they were also missing in the new pb fsimage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
