Kihwal Lee created HDFS-6293:
--------------------------------
Summary: Issues with OIV processing PB-based fsimages
Key: HDFS-6293
URL: https://issues.apache.org/jira/browse/HDFS-6293
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Priority: Blocker
There are issues with OIV when processing fsimages in protobuf.
Due to the internal layout changes introduced by the protobuf-based fsimage,
OIV consumes excessive amount of memory. We have tested with a fsimage with
about 140M files/directories. The peak heap usage when processing this image in
pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting the
image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of heap
(max new size was 1GB). It should be possible to process any image with the
default heap size of 1.5GB.
Another issue is the complete change of format/content in OIV's XML output. I
also noticed that the secret manager section has no tokens while there were
unexpired tokens in the original image (pre-2.4.0). I did not check whether
they were also missing in the new pb fsimage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
