[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988498#comment-13988498
]
Andrew Wang commented on HDFS-6293:
-----------------------------------
I created two subtasks for PB and an HTTP interface. I have looked at the PB
fsimage also, even with JSON we'd need to do similar things to avoid having a
giant array. This requires a bit of custom parsing with Jackson or whatever, so
it's still extra work.
bq. This report has the same state as fsimage, given it is done right after the
checkpoint.
My concern is in an HA environment, we may write out the fsimage, copy it over,
and then fail while writing out the second listing. If edit logs get cleaned up
in the meantime, we might have a gap between the listing and the start of the
edit logs.
bq. Certainly, where possible. But you have all the information in the jira and
have an opportunity to discuss it, right?
I'm not sure how to interpret this. I just feel that Marcelo or myself could
have shared this feedback more immediately over the higher-bandwidth medium of
a phone, and we clearly had an interest in this JIRA since Marcelo's been
commenting since the beginning. I'm not sure why you'd be offended that I asked
that the rest of us be included in future phone calls.
> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>
> Key: HDFS-6293
> URL: https://issues.apache.org/jira/browse/HDFS-6293
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Kihwal Lee
> Priority: Blocker
> Attachments: Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf.
> Due to the internal layout changes introduced by the protobuf-based fsimage,
> OIV consumes excessive amount of memory. We have tested with a fsimage with
> about 140M files/directories. The peak heap usage when processing this image
> in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
> the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
> heap (max new size was 1GB). It should be possible to process any image with
> the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.
> I also noticed that the secret manager section has no tokens while there were
> unexpired tokens in the original image (pre-2.4.0). I did not check whether
> they were also missing in the new pb fsimage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
