[
https://issues.apache.org/jira/browse/HDFS-6673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14282014#comment-14282014
]
Lei (Eddy) Xu commented on HDFS-6673:
-------------------------------------
[~andrew.wang] Here is the performance results
The tests were run on a 2-socket Xeon E5540 machine, 24GB RAM. The 3.2 GB
fsimage and the leveldb are stored in two separate local-attached SATA 1TB
disks, respectively.
{code}
$ HADOOP_HEAPSIZE=1024 time -p hdfs oiv -i fsimage_new_pb -o /dev/null -p
Delimited -t /data/1/lei/fse.db
15/01/18 15:14:14 INFO offlineImageViewer.FSImageHandler: Loading 68 strings
15/01/18 15:14:14 INFO offlineImageViewer.PBImageTextWriter: Loading directories
15/01/18 15:14:14 INFO offlineImageViewer.PBImageTextWriter: Loading
directories in INode section.
15/01/18 15:15:34 INFO offlineImageViewer.PBImageTextWriter: Found 4188717
INode directories.
15/01/18 15:15:34 INFO offlineImageViewer.PBImageTextWriter: Finished loading
directories: 80235ms
15/01/18 15:15:34 INFO offlineImageViewer.PBImageTextWriter: Loading INode
directory section.
15/01/18 15:17:12 INFO offlineImageViewer.PBImageTextWriter: Scanned 3731860
INode directories to build namespace.
15/01/18 15:17:12 INFO offlineImageViewer.PBImageTextWriter: Finished loading
INode directory section in 97964ms
15/01/18 15:17:12 INFO offlineImageViewer.PBImageTextWriter: Found 30600809
inodes in inode section
15/01/18 15:24:50 INFO offlineImageViewer.PBImageTextWriter: Outputted
30600809 inodes.
real 638.05
user 665.23
sys 31.04
{code}
It uses {{10:38}} to generate Delimited outputs for this 3.2 GB fsimage, using
only 1GB heap size. As compared to this OIV delimited outputs, the time to
transform the old format fsimage to this 3.2 GB protobuf-based fsimage is
{{5:39}}, with 16 GB heap size.
> Add Delimited format supports for PB OIV tool
> ---------------------------------------------
>
> Key: HDFS-6673
> URL: https://issues.apache.org/jira/browse/HDFS-6673
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: 2.4.0
> Reporter: Lei (Eddy) Xu
> Assignee: Lei (Eddy) Xu
> Priority: Minor
> Attachments: HDFS-6673.000.patch, HDFS-6673.001.patch,
> HDFS-6673.002.patch, HDFS-6673.003.patch
>
>
> The new oiv tool, which is designed for Protobuf fsimage, lacks a few
> features supported in the old {{oiv}} tool.
> This task adds supports of _Delimited_ processor to the oiv tool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
