[
https://issues.apache.org/jira/browse/HDFS-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13864779#comment-13864779
]
Haohui Mai commented on HDFS-5722:
----------------------------------
The short answer is forward compatibility -- the old code should be able to
skip unknown data in the new FSImage rather than always bailing out.
You're right that it is not possible to skip forward by an arbitrary number of
protobuf types without the size information, and the file has to be read in
strictly sequential order. The old code has to always bail out because it does
not know how to deal with the unknown data.
To address this problem, the proposed format contains the size information so
that the code can skip unknown types. The following EBNF grammars sketch thew
new format:
{code}
FSImage := FileHeader Section*
FileHeader := SectionDescriptor*
SectionDescriptor := Name Offset Length
Section := <blob>
Name := string
Offset := long
Length := long
{code}
Let's say we're adding the information of data-node caches into the FSImage.
Under the old scheme, the information is put at the end of the FSImage. There
is no forward compatibility since the old code which does not understand the
information at all has no idea how to skip it. The new scheme allows a simple
solution. You put the information in a separate section -- the old code can
skip that section and continue to function since now the sizes are available. A
real use case is that a tool that counts the statistics of the file system can
continue to work on the new FSImage without any changes.
The design requires putting offset and length in the FSImage, and having
compression inside the file makes things difficult. Therefore this jira
proposes to move compression from FSImage to the higher-level application logic.
> Implement compression in the HTTP server of SNN / SBN instead of FSImage
> ------------------------------------------------------------------------
>
> Key: HDFS-5722
> URL: https://issues.apache.org/jira/browse/HDFS-5722
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Haohui Mai
>
> The current FSImage format support compression, there is a field in the
> header which specifies the compression codec used to compress the data in the
> image. The main motivation was to reduce the number of bytes to be
> transferred between SNN / SBN / NN.
> The main disadvantage, however, is that it requires the client to access the
> FSImage in strictly sequential order. This might not fit well with the new
> design of FSImage. For example, serializing the data in protobuf allows the
> client to quickly skip data that it does not understand. The compression
> built-in the format, however, complicates the calculation of offsets and
> lengths. Recovering from a corrupted, compressed FSImage is also non-trivial
> as off-the-shelf tools like bzip2recover is inapplicable.
> This jira proposes to move the compression from the format of the FSImage to
> the transport layer, namely, the HTTP server of SNN / SBN. This design
> simplifies the format of FSImage, opens up the opportunity to quickly
> navigate through the FSImage, and eases the process of recovery. It also
> retains the benefits of reducing the number of bytes to be transferred across
> the wire since there are compression on the transport layer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
