[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[ https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845793#comment-17845793 ] ASF GitHub Bot commented on HDFS-17463: --- hiwangzhihui commented on PR #6736: URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2106760458 @Hexiaoqiao I agree with your propose , We will proceed with work for it. Thank you for your guide. > Support the switch StringTable Split ID feature > --- > > Key: HDFS-17463 > URL: https://issues.apache.org/jira/browse/HDFS-17463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > Attachments: Image_struct.png, error.png > > > desc: > * > Hadoop 3.2 introduced optimization features for HDFS StringTable > (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of > Hadoop upgraded to 3.2 and later versions not supporting downgrade > operations. > !error.png! > * This issue has also been discussed in HDFS-14831, and it is recommended to > revert the feature, but it cannot fundamentally solve the problem。 > * > Therefore, we have added an optimization to support downgrading > > Solution: > * First, we will add the "dfs. image. save. splitId. stringTable" conf > switch "StringTable optimization feature" is enabled > * When the conf value is false, an Image file compatible with lower versions > of HDFS is generated to support downgrading. > * > The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 > is shown in the following figure. > * With the sub-sections feature introduced in HDFS-14617, Protobuf can > support compatible reading. > * > The data structure causing incompatible differences is mainly StringTable. > !Image_struct.png|width=396,height=163! > * In "dfs.image.save.splitId.stringTable = false " the Id generation order > of StringTable starts from 0 to Integer.Max. When true, the Id value range > follows the latest rules. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[ https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841984#comment-17841984 ] ASF GitHub Bot commented on HDFS-17463: --- Hexiaoqiao commented on PR #6736: URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2082660933 @XbaoWu @hiwangzhihui Thanks for your works. I prefer to push this feature into tool module. a. I totally agree that fsimage is not compatible with all versions (include some segment format, parellel loading etc) especially with some old verions, but some end users could need it. b. However after traverse all active branches, it shows all of them are keeping the same format now. c. Even this feature is ready online now, if you meet the compatible issues about fsimage, you need to restart NameNode to generate another new one, so I think fsimage tools could be more convenient. What do you think about? > Support the switch StringTable Split ID feature > --- > > Key: HDFS-17463 > URL: https://issues.apache.org/jira/browse/HDFS-17463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > Attachments: Image_struct.png, error.png > > > desc: > * > Hadoop 3.2 introduced optimization features for HDFS StringTable > (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of > Hadoop upgraded to 3.2 and later versions not supporting downgrade > operations. > !error.png! > * This issue has also been discussed in HDFS-14831, and it is recommended to > revert the feature, but it cannot fundamentally solve the problem。 > * > Therefore, we have added an optimization to support downgrading > > Solution: > * First, we will add the "dfs. image. save. splitId. stringTable" conf > switch "StringTable optimization feature" is enabled > * When the conf value is false, an Image file compatible with lower versions > of HDFS is generated to support downgrading. > * > The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 > is shown in the following figure. > * With the sub-sections feature introduced in HDFS-14617, Protobuf can > support compatible reading. > * > The data structure causing incompatible differences is mainly StringTable. > !Image_struct.png|width=396,height=163! > * In "dfs.image.save.splitId.stringTable = false " the Id generation order > of StringTable starts from 0 to Integer.Max. When true, the Id value range > follows the latest rules. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841982#comment-17841982
]
ASF GitHub Bot commented on HDFS-17463:
---
hadoop-yetus commented on PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2082632595
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 01s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 86m 09s | | trunk passed |
| +1 :green_heart: | compile | 5m 49s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 141m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 20s | | the patch passed |
| +1 :green_heart: | compile | 3m 19s | | the patch passed |
| +1 :green_heart: | javac | 3m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 02s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 154m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 02s | | The patch does not
generate ASF License warnings. |
| | | 408m 47s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6736 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | MINGW64_NT-10.0-17763 6b971c86bc4b 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / d4197ab0fab16d60037594b06089181b8db46467 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6736/4/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6736/4/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Support the switch StringTable Split ID feature
> ---
>
> Key: HDFS-17463
> URL: https://issues.apache.org/jira/browse/HDFS-17463
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4
>Reporter: wangzhihui
>Priority: Major
> Labels: pull-request-available
> Attachments: Image_struct.png, error.png
>
>
> desc:
> *
> Hadoop 3.2 introduced optimization features for HDFS StringTable
> (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of
> Hadoop upgraded to 3.2 and later versions not supporting downgrade
> operations.
> !error.png!
> * This issue has also been discussed in HDFS-14831, and it is recommended to
> revert the feature, but it cannot fundamentally solve the problem。
> *
> Therefore, we have added an optimization to support downgrading
>
> Solution:
> * First, we will add the "dfs. image. save. splitId. stringTable" conf
> switch "StringTable optimization feature" is enabled
> * When the conf value is false, an Image file compatible with lower versions
> of HDFS is generated to support downgrading.
> *
> The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2
> is shown in the following figure.
>
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[ https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841571#comment-17841571 ] wangzhihui commented on HDFS-17463: --- hi, [~hexiaoqiao] Please help assess it. Thanks. > Support the switch StringTable Split ID feature > --- > > Key: HDFS-17463 > URL: https://issues.apache.org/jira/browse/HDFS-17463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > Attachments: Image_struct.png, error.png > > > desc: > * > Hadoop 3.2 introduced optimization features for HDFS StringTable > (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of > Hadoop upgraded to 3.2 and later versions not supporting downgrade > operations. > !error.png! > * This issue has also been discussed in HDFS-14831, and it is recommended to > revert the feature, but it cannot fundamentally solve the problem。 > * > Therefore, we have added an optimization to support downgrading > > Solution: > * First, we will add the "dfs. image. save. splitId. stringTable" conf > switch "StringTable optimization feature" is enabled > * When the conf value is false, an Image file compatible with lower versions > of HDFS is generated to support downgrading. > * > The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 > is shown in the following figure. > * With the sub-sections feature introduced in HDFS-14617, Protobuf can > support compatible reading. > * > The data structure causing incompatible differences is mainly StringTable. > !Image_struct.png|width=396,height=163! > * In "dfs.image.save.splitId.stringTable = false " the Id generation order > of StringTable starts from 0 to Integer.Max. When true, the Id value range > follows the latest rules. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[ https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841557#comment-17841557 ] wangzhihui commented on HDFS-17463: --- [~ste...@apache.org] I get it. Thanks for your remind. > Support the switch StringTable Split ID feature > --- > > Key: HDFS-17463 > URL: https://issues.apache.org/jira/browse/HDFS-17463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > Attachments: Image_struct.png, error.png > > > desc: > * > Hadoop 3.2 introduced optimization features for HDFS StringTable > (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of > Hadoop upgraded to 3.2 and later versions not supporting downgrade > operations. > !error.png! > * This issue has also been discussed in HDFS-14831, and it is recommended to > revert the feature, but it cannot fundamentally solve the problem。 > * > Therefore, we have added an optimization to support downgrading > > Solution: > * First, we will add the "dfs. image. save. splitId. stringTable" conf > switch "StringTable optimization feature" is enabled > * When the conf value is false, an Image file compatible with lower versions > of HDFS is generated to support downgrading. > * > The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 > is shown in the following figure. > * With the sub-sections feature introduced in HDFS-14617, Protobuf can > support compatible reading. > * > The data structure causing incompatible differences is mainly StringTable. > !Image_struct.png|width=396,height=163! > * In "dfs.image.save.splitId.stringTable = false " the Id generation order > of StringTable starts from 0 to Integer.Max. When true, the Id value range > follows the latest rules. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[ https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841321#comment-17841321 ] Steve Loughran commented on HDFS-17463: --- Sorry, HDFS is not my world. Ask on the hdfs developer list for reviewers. > Support the switch StringTable Split ID feature > --- > > Key: HDFS-17463 > URL: https://issues.apache.org/jira/browse/HDFS-17463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > Attachments: Image_struct.png, error.png > > > desc: > * > Hadoop 3.2 introduced optimization features for HDFS StringTable > (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of > Hadoop upgraded to 3.2 and later versions not supporting downgrade > operations. > !error.png! > * This issue has also been discussed in HDFS-14831, and it is recommended to > revert the feature, but it cannot fundamentally solve the problem。 > * > Therefore, we have added an optimization to support downgrading > > Solution: > * First, we will add the "dfs. image. save. splitId. stringTable" conf > switch "StringTable optimization feature" is enabled > * When the conf value is false, an Image file compatible with lower versions > of HDFS is generated to support downgrading. > * > The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 > is shown in the following figure. > * With the sub-sections feature introduced in HDFS-14617, Protobuf can > support compatible reading. > * > The data structure causing incompatible differences is mainly StringTable. > !Image_struct.png|width=396,height=163! > * In "dfs.image.save.splitId.stringTable = false " the Id generation order > of StringTable starts from 0 to Integer.Max. When true, the Id value range > follows the latest rules. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840762#comment-17840762
]
ASF GitHub Bot commented on HDFS-17463:
---
hadoop-yetus commented on PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2076952024
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 53s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 274m 42s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 432m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6736 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 38640423e977 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d4197ab0fab16d60037594b06089181b8db46467 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/4/testReport/ |
| Max. process+thread count | 2639 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[ https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840658#comment-17840658 ] wangzhihui commented on HDFS-17463: --- hi , [~ste...@apache.org] We have added a feature to support downgrade capability. Please help review whether the community needs to merge this feature. > Support the switch StringTable Split ID feature > --- > > Key: HDFS-17463 > URL: https://issues.apache.org/jira/browse/HDFS-17463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > Attachments: Image_struct.png, error.png > > > desc: > * > Hadoop 3.2 introduced optimization features for HDFS StringTable > (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of > Hadoop upgraded to 3.2 and later versions not supporting downgrade > operations. > !error.png! > * This issue has also been discussed in HDFS-14831, and it is recommended to > revert the feature, but it cannot fundamentally solve the problem。 > * > Therefore, we have added an optimization to support downgrading > > Solution: > * First, we will add the "dfs. image. save. splitId. stringTable" conf > switch "StringTable optimization feature" is enabled > * When the conf value is false, an Image file compatible with lower versions > of HDFS is generated to support downgrading. > * > The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 > is shown in the following figure. > * With the sub-sections feature introduced in HDFS-14617, Protobuf can > support compatible reading. > * > The data structure causing incompatible differences is mainly StringTable. > !Image_struct.png|width=396,height=163! > * In "dfs.image.save.splitId.stringTable = false " the Id generation order > of StringTable starts from 0 to Integer.Max. When true, the Id value range > follows the latest rules. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840183#comment-17840183
]
ASF GitHub Bot commented on HDFS-17463:
---
hadoop-yetus commented on PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2073100512
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 31s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 210m 20s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 307m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6736 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux d39cd7a88ad5 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8399f99ae05faafbd8e272ca638038b8806ec94f |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/3/testReport/ |
| Max. process+thread count | 4366 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840147#comment-17840147
]
ASF GitHub Bot commented on HDFS-17463:
---
hadoop-yetus commented on PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2072822967
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 00s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 87m 03s | | trunk passed |
| +1 :green_heart: | compile | 6m 15s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 150m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 38s | | the patch passed |
| +1 :green_heart: | compile | 3m 33s | | the patch passed |
| +1 :green_heart: | javac | 3m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 164m 04s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 55s | | The patch does not
generate ASF License warnings. |
| | | 429m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6736 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | MINGW64_NT-10.0-17763 263e1c7dd5ce 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / f78c740a23e1a9d140a56098297642541a6f8343 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6736/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6736/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Support the switch StringTable Split ID feature
> ---
>
> Key: HDFS-17463
> URL: https://issues.apache.org/jira/browse/HDFS-17463
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4
>Reporter: wangzhihui
>Priority: Major
> Labels: pull-request-available
> Attachments: Image_struct.png, error.png
>
>
> desc:
> *
> Hadoop 3.2 introduced optimization features for HDFS StringTable
> (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of
> Hadoop upgraded to 3.2 and later versions not supporting downgrade
> operations.
> !error.png!
> * This issue has also been discussed in HDFS-14831, and it is recommended to
> revert the feature, but it cannot fundamentally solve the problem。
> *
> Therefore, we have added an optimization to support downgrading
>
> Solution:
> * First, we will add the "dfs. image. save. splitId. stringTable" conf
> switch "StringTable optimization feature" is enabled
> * When the conf value is false, an Image file compatible with lower versions
> of HDFS is generated to support downgrading.
> *
> The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2
> is shown in the following figure.
>
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839023#comment-17839023
]
ASF GitHub Bot commented on HDFS-17463:
---
hadoop-yetus commented on PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2066734147
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 30s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 252 unchanged
- 0 fixed = 259 total (was 252) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 209m 23s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 300m 52s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestFileContextAcl |
| | hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot |
| | hadoop.hdfs.server.namenode.TestAclConfigFlag |
| | hadoop.hdfs.server.namenode.TestNameNodeAcl |
| | hadoop.hdfs.server.namenode.TestFSImageWithAcl |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl |
| | hadoop.hdfs.web.TestWebHDFSAcl |
| | hadoop.hdfs.server.namenode.TestSerialFSImage |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6736 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837695#comment-17837695
]
ASF GitHub Bot commented on HDFS-17463:
---
hadoop-yetus commented on PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2058962419
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 12s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 29s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 250 unchanged
- 2 fixed = 259 total (was 252) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 203m 20s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 302m 51s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestFileContextAcl |
| | hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot |
| | hadoop.hdfs.server.namenode.TestAclConfigFlag |
| | hadoop.hdfs.server.namenode.TestNameNodeAcl |
| | hadoop.hdfs.server.namenode.TestFSImageWithAcl |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl |
| | hadoop.hdfs.web.TestWebHDFSAcl |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6736 |
| Optional Tests | dupname asflicense compile javac javadoc
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837689#comment-17837689
]
ASF GitHub Bot commented on HDFS-17463:
---
XbaoWu commented on code in PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#discussion_r1567230524
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java:
##
@@ -115,6 +119,36 @@ public static StringTable getStringTable() {
return map;
}
+ // returns serial snapshot of current values for a save.
+ public static StringTable getSerialStringTable() {
+// approximate size for capacity.
+int size = 0;
+for (final SerialNumberManager snm : values) {
+ size += snm.size();
+}
+
+int tableMaskBits = getMaskBits();
+StringTable map = new StringTable(size, 0);
+Set entrySet = new HashSet();
+AtomicInteger index = new AtomicInteger();
+
+Map entryMap = new TreeMap();
+for (final SerialNumberManager snm : values) {
+ final int mask = snm.getMask(tableMaskBits);
+ for (Entry entry : snm.entrySet()) {
+entryMap.put(entry.getKey() | mask, entry.getValue());
+ }
+}
+
+entryMap.forEach((key, value) -> {
+ if(entrySet.add(value)){
+map.put(index.incrementAndGet(), value);
Review Comment:
incrementAndGet method is executed when the fsimage file is persisted to
ensure that the index is continuous.
> Support the switch StringTable Split ID feature
> ---
>
> Key: HDFS-17463
> URL: https://issues.apache.org/jira/browse/HDFS-17463
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4
>Reporter: wangzhihui
>Priority: Major
> Labels: pull-request-available
> Attachments: Image_struct.png, error.png
>
>
> desc:
> *
> Hadoop 3.2 introduced optimization features for HDFS StringTable
> (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of
> Hadoop upgraded to 3.2 and later versions not supporting downgrade
> operations.
> !error.png!
> * This issue has also been discussed in HDFS-14831, and it is recommended to
> revert the feature, but it cannot fundamentally solve the problem。
> *
> Therefore, we have added an optimization to support downgrading
>
> Solution:
> * First, we will add the "dfs. image. save. splitId. stringTable" conf
> switch "StringTable optimization feature" is enabled
> * When the conf value is false, an Image file compatible with lower versions
> of HDFS is generated to support downgrading.
> *
> The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2
> is shown in the following figure.
> * With the sub-sections feature introduced in HDFS-14617, Protobuf can
> support compatible reading.
> *
> The data structure causing incompatible differences is mainly StringTable.
> !Image_struct.png|width=396,height=163!
> * In "dfs.image.save.splitId.stringTable = false " the Id generation order
> of StringTable starts from 0 to Integer.Max. When true, the Id value range
> follows the latest rules.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837683#comment-17837683
]
ASF GitHub Bot commented on HDFS-17463:
---
XbaoWu commented on code in PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#discussion_r1567205509
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java:
##
@@ -115,6 +119,36 @@ public static StringTable getStringTable() {
return map;
}
+ // returns serial snapshot of current values for a save.
+ public static StringTable getSerialStringTable() {
Review Comment:
Okay, I 'll add this test case as soon as possible.
> Support the switch StringTable Split ID feature
> ---
>
> Key: HDFS-17463
> URL: https://issues.apache.org/jira/browse/HDFS-17463
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4
>Reporter: wangzhihui
>Priority: Major
> Labels: pull-request-available
> Attachments: Image_struct.png, error.png
>
>
> desc:
> *
> Hadoop 3.2 introduced optimization features for HDFS StringTable
> (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of
> Hadoop upgraded to 3.2 and later versions not supporting downgrade
> operations.
> !error.png!
> * This issue has also been discussed in HDFS-14831, and it is recommended to
> revert the feature, but it cannot fundamentally solve the problem。
> *
> Therefore, we have added an optimization to support downgrading
>
> Solution:
> * First, we will add the "dfs. image. save. splitId. stringTable" conf
> switch "StringTable optimization feature" is enabled
> * When the conf value is false, an Image file compatible with lower versions
> of HDFS is generated to support downgrading.
> *
> The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2
> is shown in the following figure.
> * With the sub-sections feature introduced in HDFS-14617, Protobuf can
> support compatible reading.
> *
> The data structure causing incompatible differences is mainly StringTable.
> !Image_struct.png|width=396,height=163!
> * In "dfs.image.save.splitId.stringTable = false " the Id generation order
> of StringTable starts from 0 to Integer.Max. When true, the Id value range
> follows the latest rules.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837681#comment-17837681
]
ASF GitHub Bot commented on HDFS-17463:
---
XbaoWu commented on code in PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#discussion_r1567203473
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java:
##
@@ -628,25 +638,59 @@ private void loadRootINode(INodeSection.INode p) {
public final static class Saver {
private long numImageErrors;
-private static long buildPermissionStatus(INodeAttributes n) {
- return n.getPermissionLong();
+private static long buildPermissionStatus(INodeAttributes n,
+boolean enableSaveSplitIdStringTable) {
+ if(enableSaveSplitIdStringTable){
+return n.getPermissionLong();
+ } else {
+StringTable serialStringTable =
SerialNumberManager.getSerialStringTable();
+Map stringMap = StreamSupport
+.stream(serialStringTable.spliterator(), false)
+.collect(Collectors.toMap(Map.Entry::getValue, Map.Entry::getKey));
+
+long permission = n.getPermissionLong();
+permission = USER.BITS.combine(stringMap.get(n.getUserName()),
permission);
+permission = GROUP.BITS.combine(stringMap.get(n.getGroupName()),
permission);
+return permission;
+ }
}
-private static AclFeatureProto.Builder buildAclEntries(AclFeature f) {
+private static AclFeatureProto.Builder buildAclEntries(AclFeature f,
+boolean enableSaveSplitIdStringTable) {
AclFeatureProto.Builder b = AclFeatureProto.newBuilder();
+ Map stringMap = null;
+ if(!enableSaveSplitIdStringTable){
+StringTable serialStringTable =
SerialNumberManager.getSerialStringTable();
+stringMap = StreamSupport.stream(serialStringTable.spliterator(),
false)
+.collect(Collectors.toMap(Map.Entry::getValue,
Map.Entry::getKey));
+ }
+
for (int pos = 0, e; pos < f.getEntriesSize(); pos++) {
e = f.getEntryAt(pos);
+if(!enableSaveSplitIdStringTable){
+ e = (int)
NAME.BITS.combine(stringMap.get(AclEntryStatusFormat.getName(e)), e);
+}
b.addEntries(e);
}
return b;
}
-private static XAttrFeatureProto.Builder buildXAttrs(XAttrFeature f) {
+private static XAttrFeatureProto.Builder buildXAttrs(XAttrFeature f,
boolean enableSaveSplitIdStringTable) {
XAttrFeatureProto.Builder b = XAttrFeatureProto.newBuilder();
+ Map stringMap = null;
+ if(!enableSaveSplitIdStringTable){
+StringTable serialStringTable =
SerialNumberManager.getSerialStringTable();
Review Comment:
Yes, when enableSaveSplitIdStringTable is false, getSerialStringTable is
called ; conversely, getStringTable is called. The mapping relations stored in
these two StringTable are different.
> Support the switch StringTable Split ID feature
> ---
>
> Key: HDFS-17463
> URL: https://issues.apache.org/jira/browse/HDFS-17463
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4
>Reporter: wangzhihui
>Priority: Major
> Labels: pull-request-available
> Attachments: Image_struct.png, error.png
>
>
> desc:
> *
> Hadoop 3.2 introduced optimization features for HDFS StringTable
> (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of
> Hadoop upgraded to 3.2 and later versions not supporting downgrade
> operations.
> !error.png!
> * This issue has also been discussed in HDFS-14831, and it is recommended to
> revert the feature, but it cannot fundamentally solve the problem。
> *
> Therefore, we have added an optimization to support downgrading
>
> Solution:
> * First, we will add the "dfs. image. save. splitId. stringTable" conf
> switch "StringTable optimization feature" is enabled
> * When the conf value is false, an Image file compatible with lower versions
> of HDFS is generated to support downgrading.
> *
> The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2
> is shown in the following figure.
> * With the sub-sections feature introduced in HDFS-14617, Protobuf can
> support compatible reading.
> *
> The data structure causing incompatible differences is mainly StringTable.
> !Image_struct.png|width=396,height=163!
> * In "dfs.image.save.splitId.stringTable = false " the Id generation order
> of StringTable starts from 0 to Integer.Max. When true, the Id value range
> follows the latest rules.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe,
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837678#comment-17837678
]
ASF GitHub Bot commented on HDFS-17463:
---
XbaoWu commented on code in PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#discussion_r1567198629
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclEntryStatusFormat.java:
##
@@ -47,7 +47,7 @@ public enum AclEntryStatusFormat implements
LongBitFormat.Enum {
private static final AclEntryType[] ACL_ENTRY_TYPE_VALUES =
AclEntryType.values();
- private final LongBitFormat BITS;
Review Comment:
Because the “private” BITS cannot be import by FSImageFormatPBINode. This
change may not be appropriate, I will write a getter for BITS is a better
solution.
> Support the switch StringTable Split ID feature
> ---
>
> Key: HDFS-17463
> URL: https://issues.apache.org/jira/browse/HDFS-17463
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4
>Reporter: wangzhihui
>Priority: Major
> Labels: pull-request-available
> Attachments: Image_struct.png, error.png
>
>
> desc:
> *
> Hadoop 3.2 introduced optimization features for HDFS StringTable
> (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of
> Hadoop upgraded to 3.2 and later versions not supporting downgrade
> operations.
> !error.png!
> * This issue has also been discussed in HDFS-14831, and it is recommended to
> revert the feature, but it cannot fundamentally solve the problem。
> *
> Therefore, we have added an optimization to support downgrading
>
> Solution:
> * First, we will add the "dfs. image. save. splitId. stringTable" conf
> switch "StringTable optimization feature" is enabled
> * When the conf value is false, an Image file compatible with lower versions
> of HDFS is generated to support downgrading.
> *
> The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2
> is shown in the following figure.
> * With the sub-sections feature introduced in HDFS-14617, Protobuf can
> support compatible reading.
> *
> The data structure causing incompatible differences is mainly StringTable.
> !Image_struct.png|width=396,height=163!
> * In "dfs.image.save.splitId.stringTable = false " the Id generation order
> of StringTable starts from 0 to Integer.Max. When true, the Id value range
> follows the latest rules.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837566#comment-17837566
]
ASF GitHub Bot commented on HDFS-17463:
---
hiwangzhihui commented on code in PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#discussion_r1566889645
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java:
##
@@ -115,6 +119,36 @@ public static StringTable getStringTable() {
return map;
}
+ // returns serial snapshot of current values for a save.
+ public static StringTable getSerialStringTable() {
+// approximate size for capacity.
+int size = 0;
+for (final SerialNumberManager snm : values) {
+ size += snm.size();
+}
+
+int tableMaskBits = getMaskBits();
+StringTable map = new StringTable(size, 0);
+Set entrySet = new HashSet();
+AtomicInteger index = new AtomicInteger();
+
+Map entryMap = new TreeMap();
+for (final SerialNumberManager snm : values) {
+ final int mask = snm.getMask(tableMaskBits);
+ for (Entry entry : snm.entrySet()) {
+entryMap.put(entry.getKey() | mask, entry.getValue());
+ }
+}
+
+entryMap.forEach((key, value) -> {
+ if(entrySet.add(value)){
+map.put(index.incrementAndGet(), value);
Review Comment:
Why not execute the incrementAndGet method when persistently collecting
information?
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java:
##
@@ -628,25 +638,59 @@ private void loadRootINode(INodeSection.INode p) {
public final static class Saver {
private long numImageErrors;
-private static long buildPermissionStatus(INodeAttributes n) {
- return n.getPermissionLong();
+private static long buildPermissionStatus(INodeAttributes n,
+boolean enableSaveSplitIdStringTable) {
+ if(enableSaveSplitIdStringTable){
+return n.getPermissionLong();
+ } else {
+StringTable serialStringTable =
SerialNumberManager.getSerialStringTable();
+Map stringMap = StreamSupport
+.stream(serialStringTable.spliterator(), false)
+.collect(Collectors.toMap(Map.Entry::getValue, Map.Entry::getKey));
+
+long permission = n.getPermissionLong();
+permission = USER.BITS.combine(stringMap.get(n.getUserName()),
permission);
+permission = GROUP.BITS.combine(stringMap.get(n.getGroupName()),
permission);
+return permission;
+ }
}
-private static AclFeatureProto.Builder buildAclEntries(AclFeature f) {
+private static AclFeatureProto.Builder buildAclEntries(AclFeature f,
+boolean enableSaveSplitIdStringTable) {
AclFeatureProto.Builder b = AclFeatureProto.newBuilder();
+ Map stringMap = null;
+ if(!enableSaveSplitIdStringTable){
+StringTable serialStringTable =
SerialNumberManager.getSerialStringTable();
+stringMap = StreamSupport.stream(serialStringTable.spliterator(),
false)
+.collect(Collectors.toMap(Map.Entry::getValue,
Map.Entry::getKey));
+ }
+
for (int pos = 0, e; pos < f.getEntriesSize(); pos++) {
e = f.getEntryAt(pos);
+if(!enableSaveSplitIdStringTable){
+ e = (int)
NAME.BITS.combine(stringMap.get(AclEntryStatusFormat.getName(e)), e);
+}
b.addEntries(e);
}
return b;
}
-private static XAttrFeatureProto.Builder buildXAttrs(XAttrFeature f) {
+private static XAttrFeatureProto.Builder buildXAttrs(XAttrFeature f,
boolean enableSaveSplitIdStringTable) {
XAttrFeatureProto.Builder b = XAttrFeatureProto.newBuilder();
+ Map stringMap = null;
+ if(!enableSaveSplitIdStringTable){
+StringTable serialStringTable =
SerialNumberManager.getSerialStringTable();
Review Comment:
When enableSaveSplitIdStringTable=false, different StringMaps should be
obtained to store data
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclEntryStatusFormat.java:
##
@@ -47,7 +47,7 @@ public enum AclEntryStatusFormat implements
LongBitFormat.Enum {
private static final AclEntryType[] ACL_ENTRY_TYPE_VALUES =
AclEntryType.values();
- private final LongBitFormat BITS;
Review Comment:
Why remove the “private”
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java:
##
@@ -115,6 +119,36 @@ public static StringTable getStringTable() {
return map;
}
+ // returns serial snapshot of current values for a save.
+ public static StringTable getSerialStringTable() {
Review Comment:
Please add test cases.
> Support the switch StringTable Split ID feature
> ---
>
> Key:
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[ https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837542#comment-17837542 ] ASF GitHub Bot commented on HDFS-17463: --- XbaoWu opened a new pull request, #6736: URL: https://github.com/apache/hadoop/pull/6736 ### Description of PR Hadoop 3.2 introduced optimization features for HDFS StringTable (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of Hadoop upgraded to 3.2 and later versions not supporting downgrade operations. Therefore, we have added an optimization to support downgrading. In the upgrade with large version span, it avoids the low version can not read the fs-image generated by the 3.2 and later version, which support for the 3.2 and later version to be downgraded to the low version. For more information about this PR, please refer to the following issue： [HDFS-17463](https://issues.apache.org/jira/browse/HDFS-17463) Support the switch StringTable Split ID feature ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Support the switch StringTable Split ID feature > --- > > Key: HDFS-17463 > URL: https://issues.apache.org/jira/browse/HDFS-17463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4 >Reporter: wangzhihui >Priority: Major > Attachments: Image_struct.png, error.png > > > desc: > * > Hadoop 3.2 introduced optimization features for HDFS StringTable > (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of > Hadoop upgraded to 3.2 and later versions not supporting downgrade > operations. > !error.png! > * This issue has also been discussed in HDFS-14831, and it is recommended to > revert the feature, but it cannot fundamentally solve the problem。 > * > Therefore, we have added an optimization to support downgrading > > Solution: > * First, we will add the "dfs. image. save. splitId. stringTable" conf > switch "StringTable optimization feature" is enabled > * When the conf value is false, an Image file compatible with lower versions > of HDFS is generated to support downgrading. > * > The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 > is shown in the following figure. > * With the sub-sections feature introduced in HDFS-14617, Protobuf can > support compatible reading. > * > The data structure causing incompatible differences is mainly StringTable. > !Image_struct.png|width=396,height=163! > * In "dfs.image.save.splitId.stringTable = false " the Id generation order > of StringTable starts from 0 to Integer.Max. When true, the Id value range > follows the latest rules. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[ https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836514#comment-17836514 ] Xiaobao Wu commented on HDFS-17463: --- [~hiwangzhihui] Okay, I 'll support this feature as soon as possible. > Support the switch StringTable Split ID feature > --- > > Key: HDFS-17463 > URL: https://issues.apache.org/jira/browse/HDFS-17463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4 >Reporter: wangzhihui >Priority: Major > Attachments: Image_struct.png, error.png > > > desc: > * > Hadoop 3.2 introduced optimization features for HDFS StringTable > (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of > Hadoop upgraded to 3.2 and later versions not supporting downgrade > operations. > !error.png! > * This issue has also been discussed in HDFS-14831, and it is recommended to > revert the feature, but it cannot fundamentally solve the problem。 > * > Therefore, we have added an optimization to support downgrading > > Solution: > * First, we will add the "dfs. image. save. splitId. stringTable" conf > switch "StringTable optimization feature" is enabled > * When the conf value is false, an Image file compatible with lower versions > of HDFS is generated to support downgrading. > * > The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 > is shown in the following figure. > * With the sub-sections feature introduced in HDFS-14617, Protobuf can > support compatible reading. > * > The data structure causing incompatible differences is mainly StringTable. > !Image_struct.png|width=396,height=163! > * In "dfs.image.save.splitId.stringTable = false " the Id generation order > of StringTable starts from 0 to Integer.Max. When true, the Id value range > follows the latest rules. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
