[ https://issues.apache.org/jira/browse/HDFS-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDFS-17524: ---------------------------------- Labels: pull-request-available (was: ) > OIV: add Transformed processor which reconstructs an fsimage from another > fsimage file > -------------------------------------------------------------------------------------- > > Key: HDFS-17524 > URL: https://issues.apache.org/jira/browse/HDFS-17524 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools > Affects Versions: 3.2.0, 3.3.4 > Reporter: Xiaobao Wu > Priority: Major > Labels: pull-request-available > > *Background:* > The Image file generated by the existing Hadoop 3.3.4 version cannot be > forward compatible . In the high version of HDFS, the fsimage file conversion > tool is provided to support the generation of forward compatible fsimage file > to support the downgrade operation. > {*}Description{*}: > Because there are differences in the structure and loading methods of some > Sections between high and low versions of fsimage files, especially the > StringTable Section. This will make it impossible to downgrade to a lower > version of HDFS ( e.g., 3.1.1 ) in higher versions ( e.g., 3.3.4 ), because > when the lower version of HDFS loads the fsimage file generated by the higher > version of HDFS, there will be an ArrayIndexOutOfBoundsException. > > The code differences are as follows: > {code:java} > // 3.3.4 > static SerialNumberManager.StringTable loadStringTable(InputStream in) > throws IOException { > ··· ··· > SerialNumberManager.StringTable stringTable = > SerialNumberManager.newStringTable(s.getNumEntry(), s.getMaskBits()); > for (int i = 0; i < s.getNumEntry(); ++i) { > FsImageProto.StringTableSection.Entry e = FsImageProto > .StringTableSection.Entry.parseDelimitedFrom(in); > stringTable.put(e.getId(), e.getStr()); > } > return stringTable; > } > // 3.1.1 > static String[] loadStringTable(InputStream in) throws IOException { > ··· ··· > String[] stringTable = new String[s.getNumEntry() + 1]; > for (int i = 0; i < s.getNumEntry(); ++i) { > FsImageProto.StringTableSection.Entry e = FsImageProto > .StringTableSection.Entry.parseDelimitedFrom(in); > // ArrayIndexOutOfBoundsException is triggered when loading a higher > version of the fsimage file. > stringTable[e.getId()] = e.getStr(); > } > return stringTable; > }{code} > {*}Solution{*}: > Solution Reference from HDFS-17463 > !http://www.kdocs.cn/api/v3/office/copy/Mm0rd3BzNEx2Y29zaUdsQkczVnRUV2JwR2RvVWNVdk9aT3dRc2czUXRYdit1ekZ4UmN3UWFLN0hwOTZidnJ1L2ZxaW5PaUNHRmU1bGNyS3lRUGZRbE1vR2I4MlQvS0ppOUZxbVRnQ2o2SUNJZGFoeVNzMUFjR2tKTStsTjZpUTFwanpmcTRML0JFTDJHcXV4aGpESVFXS1RTeEkyZk5sb25LOEEyT0lHbDJydVlIZEJ2dXlyYVozM2pkZGdacEtWQnR3SUQ0MXUwV1RINTMyaDluV2FRTWNjS2p5Nm0rZngzbGNGdEd4cFpLdjFpWUtWK2UyMDZhVVFYUWVHZXlwZEQ0c25MWU93NFY0PQ==/attach/object/K3TLVNAYAAQFQ?|width=693! > From the figure, it can be seen that the Id arrangement of StringTable in the > fsimage file has changed from a compact arrangement to a decentralized > arrangement, that is, USER, GROUP and XATTR are no longer mixed. The > arrangement is divided into different storage areas and arranged separately. > * With the sub-sections feature introduced in HDFS-14617, Protobuf can > support compatible reading. > * When saving fsimage files in high and low versions, the main difference is > the arrangement of Entry(e.g., USER, GROUP, and XATTR ) in StringTable. > * We will add a conversion tool to convert the Id arrangement of the high > version fsimage file StringTable to a compact arrangement, so that the low > version can be compatible with this format fsimage file. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org