[
https://issues.apache.org/jira/browse/HDFS-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845817#comment-17845817
]
ASF GitHub Bot commented on HDFS-17524:
---------------------------------------
XbaoWu opened a new pull request, #6820:
URL: https://github.com/apache/hadoop/pull/6820
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Hadoop 3.2 introduced optimization features for HDFS StringTable
(https://github.com/apache/hadoop/commit/b60ca37914b22550e3630fa02742d40697decb31),
It resulted in lower versions of Hadoop upgraded to 3.2 and later versions not
supporting downgrade operations. Therefore, we have added Transformed Processor
to support downgrading. In the upgrade with large version span, it avoids the
low version can not read the fs-image generated by the 3.2 and later version,
which support for the 3.2 and later version to be downgraded to the low version.
For more information about this PR, please refer to the following issue：
[HDFS-17524](https://issues.apache.org/jira/browse/HDFS-17524) OIV: add
Transformed processor which reconstructs an fsimage from another fsimage file.
### How was this patch tested?
add TestOfflineImageViewer.testTransformedWriter()
add TestOfflineImageViewer.testTransformedWriterWrongLayoutVersion()
add TestOfflineImageViewer.testTransformedWriterReasonableLayoutVersion()
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> OIV: add Transformed processor which reconstructs an fsimage from another
> fsimage file
> --------------------------------------------------------------------------------------
>
> Key: HDFS-17524
> URL: https://issues.apache.org/jira/browse/HDFS-17524
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
> Affects Versions: 3.2.0, 3.3.4
> Reporter: Xiaobao Wu
> Priority: Major
>
> *Background：*
> The Image file generated by the existing Hadoop 3.3.4 version cannot be
> forward compatible . In the high version of HDFS, the fsimage file conversion
> tool is provided to support the generation of forward compatible fsimage file
> to support the downgrade operation.
> {*}Description{*}：
> Because there are differences in the structure and loading methods of some
> Sections between high and low versions of fsimage files, especially the
> StringTable Section. This will make it impossible to downgrade to a lower
> version of HDFS ( e.g., 3.1.1 ) in higher versions ( e.g., 3.3.4 ), because
> when the lower version of HDFS loads the fsimage file generated by the higher
> version of HDFS, there will be an ArrayIndexOutOfBoundsException.
>
> The code differences are as follows：
> {code:java}
> // 3.3.4
> static SerialNumberManager.StringTable loadStringTable(InputStream in)
> throws IOException {
> ··· ···
> SerialNumberManager.StringTable stringTable =
> SerialNumberManager.newStringTable(s.getNumEntry(), s.getMaskBits());
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto
> .StringTableSection.Entry.parseDelimitedFrom(in);
> stringTable.put(e.getId(), e.getStr());
> }
> return stringTable;
> }
> // 3.1.1
> static String[] loadStringTable(InputStream in) throws IOException {
> ··· ···
> String[] stringTable = new String[s.getNumEntry() + 1];
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto
> .StringTableSection.Entry.parseDelimitedFrom(in);
> // ArrayIndexOutOfBoundsException is triggered when loading a higher
> version of the fsimage file.
> stringTable[e.getId()] = e.getStr();
> }
> return stringTable;
> }{code}
> {*}Solution{*}：
> Solution Reference from HDFS-17463
> !http://www.kdocs.cn/api/v3/office/copy/Mm0rd3BzNEx2Y29zaUdsQkczVnRUV2JwR2RvVWNVdk9aT3dRc2czUXRYdit1ekZ4UmN3UWFLN0hwOTZidnJ1L2ZxaW5PaUNHRmU1bGNyS3lRUGZRbE1vR2I4MlQvS0ppOUZxbVRnQ2o2SUNJZGFoeVNzMUFjR2tKTStsTjZpUTFwanpmcTRML0JFTDJHcXV4aGpESVFXS1RTeEkyZk5sb25LOEEyT0lHbDJydVlIZEJ2dXlyYVozM2pkZGdacEtWQnR3SUQ0MXUwV1RINTMyaDluV2FRTWNjS2p5Nm0rZngzbGNGdEd4cFpLdjFpWUtWK2UyMDZhVVFYUWVHZXlwZEQ0c25MWU93NFY0PQ==/attach/object/K3TLVNAYAAQFQ?|width=693!
> From the figure, it can be seen that the Id arrangement of StringTable in the
> fsimage file has changed from a compact arrangement to a decentralized
> arrangement, that is, USER, GROUP and XATTR are no longer mixed. The
> arrangement is divided into different storage areas and arranged separately.
> * With the sub-sections feature introduced in HDFS-14617, Protobuf can
> support compatible reading.
> * When saving fsimage files in high and low versions, the main difference is
> the arrangement of Entry（e.g., USER, GROUP, and XATTR ） in StringTable.
> * We will add a conversion tool to convert the Id arrangement of the high
> version fsimage file StringTable to a compact arrangement, so that the low
> version can be compatible with this format fsimage file.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
