[jira] [Commented] (HDFS-17321) RBF: Add RouterAutoMsyncService for auto msync in Router
[
https://issues.apache.org/jira/browse/HDFS-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841327#comment-17841327
]
ASF GitHub Bot commented on HDFS-17321:
---
hadoop-yetus commented on PR #6404:
URL: https://github.com/apache/hadoop/pull/6404#issuecomment-2079830301
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 01s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 89m 58s | | trunk passed |
| +1 :green_heart: | compile | 4m 55s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 00s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 146m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 53s | | the patch passed |
| +1 :green_heart: | compile | 2m 24s | | the patch passed |
| +1 :green_heart: | javac | 2m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 02s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 07s | | the patch passed |
| +1 :green_heart: | shadedclient | 176m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 8m 22s | | The patch does not
generate ASF License warnings. |
| | | 439m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6404 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | MINGW64_NT-10.0-17763 daf58be70672 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / efcc74feef4208036e29f21129682ad6155ccd89 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6404/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6404/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: Add RouterAutoMsyncService for auto msync in Router
>
>
> Key: HDFS-17321
> URL: https://issues.apache.org/jira/browse/HDFS-17321
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
>
> Router should have the ability to to auto msync to a nameservice. And it can
> ensure router periodically refreshes its record of a namespace's state.
> Different from HDFS-17027, this is controled by router itself without
> configuring with AbstractNNFailoverProxyProvider.
> And HDFS-16890 maybe lead to many read requests into active NN at the same
> time.
> This PR provides a new way to implememts auto msync in Router.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17367) Add PercentUsed for Different StorageTypes in JMX
[ https://issues.apache.org/jira/browse/HDFS-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841276#comment-17841276 ] ASF GitHub Bot commented on HDFS-17367: --- zhtttylz commented on PR #6735: URL: https://github.com/apache/hadoop/pull/6735#issuecomment-2079628784 Thanks @haiyang1987 for your feedback. I've updated the pull request. Please review it at your convenience, thanks! > Add PercentUsed for Different StorageTypes in JMX > - > > Key: HDFS-17367 > URL: https://issues.apache.org/jira/browse/HDFS-17367 > Project: Hadoop HDFS > Issue Type: Improvement > Components: metrics, namenode >Affects Versions: 3.5.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > > Currently, the NameNode only displays PercentUsed for the entire cluster. We > plan to add corresponding PercentUsed metrics for different StorageTypes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17367) Add PercentUsed for Different StorageTypes in JMX
[
https://issues.apache.org/jira/browse/HDFS-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841272#comment-17841272
]
ASF GitHub Bot commented on HDFS-17367:
---
hadoop-yetus commented on PR #6735:
URL: https://github.com/apache/hadoop/pull/6735#issuecomment-2079609229
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 31s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 49s | | trunk passed |
| +1 :green_heart: | compile | 17m 44s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 16m 13s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 34s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 57s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 6m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 11s | | the patch passed |
| +1 :green_heart: | compile | 16m 49s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 16m 49s | | the patch passed |
| +1 :green_heart: | compile | 16m 15s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 16m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 28s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 52s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 6m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 38s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 233m 43s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6735/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 14s | | The patch does not
generate ASF License warnings. |
| | | 500m 59s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6735/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6735 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets
markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs
checkstyle |
| uname | Linux 9472a5f3af00 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 83bbd6730bdebaa97ab1c501b82228405adc308b |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
[jira] [Commented] (HDFS-17329) DiskBalancerCluster#nodesToProcess should better use ArrayList
[
https://issues.apache.org/jira/browse/HDFS-17329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841248#comment-17841248
]
ASF GitHub Bot commented on HDFS-17329:
---
hadoop-yetus commented on PR #6423:
URL: https://github.com/apache/hadoop/pull/6423#issuecomment-2079431542
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 110m 43s | | trunk passed |
| +1 :green_heart: | compile | 7m 23s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 02s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 03s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 178m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 56s | | the patch passed |
| +1 :green_heart: | compile | 4m 12s | | the patch passed |
| +1 :green_heart: | javac | 4m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 48s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 05s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 197m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 7m 25s | | The patch does not
generate ASF License warnings. |
| | | 519m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6423 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 adc8ab3d366b 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 326682188fe751c5f81708384ab4493c94b9e05e |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6423/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6423/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> DiskBalancerCluster#nodesToProcess should better use ArrayList
> --
>
> Key: HDFS-17329
> URL: https://issues.apache.org/jira/browse/HDFS-17329
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: diskbalancer
>Affects Versions: 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Trivial
> Labels: pull-request-available
>
> Currently, nodesToProcess uses LinkedList to store elements. But in
> computePlan method, we use code like below:
> {code:java}
> nodesToProcess.get(x); {code}
> So, we should better change LinkedList to ArrayList here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17327) Fix ReconfigurableBase class's reconfigureProperty method judgment logic.
[
https://issues.apache.org/jira/browse/HDFS-17327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841224#comment-17841224
]
ASF GitHub Bot commented on HDFS-17327:
---
hadoop-yetus commented on PR #6421:
URL: https://github.com/apache/hadoop/pull/6421#issuecomment-2079381100
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 85m 24s | | trunk passed |
| +1 :green_heart: | compile | 37m 18s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 36s | | trunk passed |
| -1 :x: | mvnsite | 4m 11s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6421/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 10m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 153m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 11s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 8m 55s | | the patch passed |
| +1 :green_heart: | compile | 34m 54s | | the patch passed |
| +1 :green_heart: | javac | 34m 54s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 5m 49s | | the patch passed |
| -1 :x: | mvnsite | 4m 12s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6421/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 10m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 166m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 31s | | The patch does not
generate ASF License warnings. |
| | | 503m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6421 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 33900c775089 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 2e2769ddf733bbaf154535f92d3d9dd6fa300f5c |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6421/1/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6421/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Fix ReconfigurableBase class's reconfigureProperty method judgment logic.
> -
>
> Key: HDFS-17327
> URL: https://issues.apache.org/jira/browse/HDFS-17327
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhaobo Huang
>Assignee: Zhaobo Huang
>Priority: Major
> Labels: pull-request-available
>
> 1. If it is a null value, after reconfig, It may be that the current system
> value is in effect, not necessarily the default value.
> 2. {{ReconfigurableBase#reconfigureProperty}} judges the condition
> incorrectly, When the effectValue is not null, the system value needs to be
> changed.
> 3. In other words, certain variables actually have effected, but the current
> configuration of the system is refreshed to
[jira] [Commented] (HDFS-17335) Add metrics for syncWaitQ in FSEditLogAsync
[
https://issues.apache.org/jira/browse/HDFS-17335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841133#comment-17841133
]
ASF GitHub Bot commented on HDFS-17335:
---
hadoop-yetus commented on PR #6431:
URL: https://github.com/apache/hadoop/pull/6431#issuecomment-2079069487
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 01s | | markdownlint was not available.
|
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 88m 52s | | trunk passed |
| +1 :green_heart: | compile | 38m 51s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 48s | | trunk passed |
| -1 :x: | mvnsite | 4m 18s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6431/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 10m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 161m 03s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 9m 24s | | the patch passed |
| +1 :green_heart: | compile | 35m 56s | | the patch passed |
| +1 :green_heart: | javac | 35m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 5m 46s | | the patch passed |
| -1 :x: | mvnsite | 4m 20s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6431/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 11m 08s | | the patch passed |
| +1 :green_heart: | shadedclient | 169m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 40s | | The patch does not
generate ASF License warnings. |
| | | 521m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6431 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets
markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs
checkstyle |
| uname | MINGW64_NT-10.0-17763 25eab3e968f1 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 165a5568f6e1051d7f7dd88c49bad9e63c2cf0ee |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6431/1/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6431/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Add metrics for syncWaitQ in FSEditLogAsync
> ---
>
> Key: HDFS-17335
> URL: https://issues.apache.org/jira/browse/HDFS-17335
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Minor
> Labels: pull-request-available
>
> To monitor syncWaitQ in FSEditLogAsync, we add a metric syncPendingCount.
> The reason we add this metrics is that when dequeueEdit() return null, the
> boolean variable doSync is set to
[jira] [Commented] (HDFS-17338) Improve the doc related to permissions.
[
https://issues.apache.org/jira/browse/HDFS-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841116#comment-17841116
]
ASF GitHub Bot commented on HDFS-17338:
---
hadoop-yetus commented on PR #6440:
URL: https://github.com/apache/hadoop/pull/6440#issuecomment-2078989021
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 88m 56s | | trunk passed |
| +1 :green_heart: | compile | 39m 39s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 20s | | trunk passed |
| -1 :x: | mvnsite | 4m 08s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6440/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 4m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 143m 09s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 00s | | the patch passed |
| +1 :green_heart: | compile | 37m 45s | | the patch passed |
| +1 :green_heart: | javac | 37m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 37s | | the patch passed |
| -1 :x: | mvnsite | 4m 46s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6440/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 5m 01s | | the patch passed |
| +1 :green_heart: | shadedclient | 160m 06s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 30s | | The patch does not
generate ASF License warnings. |
| | | 487m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6440 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 6660d75d3dc9 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 7fb0de8832244b28a4aff191c299d5fd5222b647 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6440/1/testReport/
|
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6440/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Improve the doc related to permissions.
> ---
>
> Key: HDFS-17338
> URL: https://issues.apache.org/jira/browse/HDFS-17338
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Zhaobo Huang
>Priority: Minor
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17438) RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
[
https://issues.apache.org/jira/browse/HDFS-17438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841081#comment-17841081
]
ASF GitHub Bot commented on HDFS-17438:
---
hadoop-yetus commented on PR #6655:
URL: https://github.com/apache/hadoop/pull/6655#issuecomment-2078798013
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 42s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 20s | | trunk passed |
| +1 :green_heart: | compile | 17m 21s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 16m 8s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 10s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 4m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 16m 37s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 16m 37s | | the patch passed |
| +1 :green_heart: | compile | 16m 13s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 16m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 6s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 42s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 4m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 27s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 29m 42s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6655/11/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | | The patch does not
generate ASF License warnings. |
| | | 283m 39s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterClientRejectOverload |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6655/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6655 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 73f9aa6bb17b 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5
20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c561fdf655791a1fc74b67514974a38c0eec04be |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
[jira] [Commented] (HDFS-17438) RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
[
https://issues.apache.org/jira/browse/HDFS-17438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841058#comment-17841058
]
ASF GitHub Bot commented on HDFS-17438:
---
hadoop-yetus commented on PR #6655:
URL: https://github.com/apache/hadoop/pull/6655#issuecomment-2078682775
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 7s | | trunk passed |
| +1 :green_heart: | compile | 8m 59s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 8m 11s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 8m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 8m 34s | | the patch passed |
| +1 :green_heart: | compile | 8m 13s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 8m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 27s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 29m 39s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 187m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6655/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6655 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2be76a1925e8 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c561fdf655791a1fc74b67514974a38c0eec04be |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6655/12/testReport/ |
| Max. process+thread count | 2970 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output |
[jira] [Commented] (HDFS-17344) Last packet will be splited into two parts when write block
[
https://issues.apache.org/jira/browse/HDFS-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841008#comment-17841008
]
ASF GitHub Bot commented on HDFS-17344:
---
hadoop-yetus commented on PR #6478:
URL: https://github.com/apache/hadoop/pull/6478#issuecomment-2078402028
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 87m 48s | | trunk passed |
| +1 :green_heart: | compile | 5m 03s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 142m 06s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 06s | | the patch passed |
| +1 :green_heart: | compile | 2m 28s | | the patch passed |
| +1 :green_heart: | javac | 2m 28s | | the patch passed |
| -1 :x: | blanks | 0m 01s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6478/1/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 2m 00s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 147m 03s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 09s | | The patch does not
generate ASF License warnings. |
| | | 401m 05s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6478 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 aaf6227edc21 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / af5ddbc3a650b44fc77d5ff9ff5ffdd1ce15c551 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6478/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6478/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Last packet will be splited into two parts when write block
> ---
>
> Key: HDFS-17344
> URL: https://issues.apache.org/jira/browse/HDFS-17344
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.5.0
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> As mentioned in
> [https://github.com/apache/hadoop/pull/6368#issuecomment-1899635293]
> This Jira try to solve that problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17357) NioInetPeer.close() should close socket connection.
[
https://issues.apache.org/jira/browse/HDFS-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840974#comment-17840974
]
ASF GitHub Bot commented on HDFS-17357:
---
hadoop-yetus commented on PR #6502:
URL: https://github.com/apache/hadoop/pull/6502#issuecomment-2078169223
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 85m 14s | | trunk passed |
| +1 :green_heart: | compile | 5m 07s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 139m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 01s | | the patch passed |
| +1 :green_heart: | compile | 2m 35s | | the patch passed |
| +1 :green_heart: | javac | 2m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 05s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 147m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 07s | | The patch does not
generate ASF License warnings. |
| | | 396m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6502 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 0a31c1737811 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / a1c48f705b7f6726982b19caf2737a38ed936c68 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6502/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6502/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> NioInetPeer.close() should close socket connection.
> ---
>
> Key: HDFS-17357
> URL: https://issues.apache.org/jira/browse/HDFS-17357
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Assignee: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> NioInetPeer.close() now do not close socket connection.
> And I found 3w+ connections leakage in datanode . And I found many warn
> message as blew.
> 2024-01-22 15:27:57,500 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> hostname:50010:DataXceiverServer
>
> When any Exception is found in DataXceiverServer, it will execute clostStream.
> IOUtils.closeStream(peer) -> Peer.close() -> NioInetPeer.close()
> But NioInetPeer.close() is not invoked with close socket connection. And
> this will lead to connection leakage.
> Other subClass of Peer's close() is implemented with socket.close(). See
> EncryptedPeer, DomainPeer, BasicInetPeer
>
>
> This solution can be reporduced as following:
> (1) Client write data to HDFS
> (2) datanode Xceiver count max to DFS_DATANODE_MAX_RECEIVER_THREADS_KEY ， the
> new Xceiver will fail and throw IOException . And the socket will not release.
> (3)
[jira] [Commented] (HDFS-17364) Use WeakReferencedElasticByteBufferPool in DFSStripedInputStream
[
https://issues.apache.org/jira/browse/HDFS-17364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840969#comment-17840969
]
ASF GitHub Bot commented on HDFS-17364:
---
hadoop-yetus commented on PR #6514:
URL: https://github.com/apache/hadoop/pull/6514#issuecomment-2078145607
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 107m 55s | | trunk passed |
| +1 :green_heart: | compile | 6m 03s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 172m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 44s | | the patch passed |
| +1 :green_heart: | compile | 3m 00s | | the patch passed |
| +1 :green_heart: | javac | 3m 00s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 185m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 11s | | The patch does not
generate ASF License warnings. |
| | | 494m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6514 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 6320a162bf26 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 36954d34bac56942b8827587a13fed7893b51efa |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6514/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6514/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Use WeakReferencedElasticByteBufferPool in DFSStripedInputStream
>
>
> Key: HDFS-17364
> URL: https://issues.apache.org/jira/browse/HDFS-17364
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Priority: Major
> Labels: pull-request-available
>
> DFSStripedInputStream uses ElasticByteBufferPool to allocate byte buffers for
> the "curStripeBuf". This is used for non-positional (stateful) reads and is
> allocated with a size of numDataBlocks * cellSize. For RS-6-3-1024k, that
> means each DFSStripedInputStream could allocate a 6mb buffer. When the IS is
> finished, the buffer is put back in the pool. Over time and with spikes of
> concurrent reads, the pool grows and most of the buffers sit there unused.
>
> WeakReferencedElasticByteBufferPool was introduced HADOOP-18105 and mitigates
> this issue because the excess buffers can be GC'd once they are no longer
> needed. We should use this same pool in DFSStripedInputStream
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional
[jira] [Commented] (HDFS-17360) Record the number of times a block is read during a certain time period.
[
https://issues.apache.org/jira/browse/HDFS-17360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840960#comment-17840960
]
ASF GitHub Bot commented on HDFS-17360:
---
hadoop-yetus commented on PR #6505:
URL: https://github.com/apache/hadoop/pull/6505#issuecomment-2078110928
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 00s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 90m 25s | | trunk passed |
| +1 :green_heart: | compile | 5m 59s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 03s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 04s | | trunk passed |
| +1 :green_heart: | shadedclient | 150m 08s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 24s | | the patch passed |
| +1 :green_heart: | compile | 3m 27s | | the patch passed |
| +1 :green_heart: | javac | 3m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 04s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 157m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 18s | | The patch does not
generate ASF License warnings. |
| | | 425m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6505 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | MINGW64_NT-10.0-17763 f4145dde94d1 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / c430cc09a1662a5eb33cbfae7e6ecb608e93d55f |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6505/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6505/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Record the number of times a block is read during a certain time period.
>
>
> Key: HDFS-17360
> URL: https://issues.apache.org/jira/browse/HDFS-17360
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Zhaobo Huang
>Assignee: Zhaobo Huang
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17367) Add PercentUsed for Different StorageTypes in JMX
[
https://issues.apache.org/jira/browse/HDFS-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840925#comment-17840925
]
ASF GitHub Bot commented on HDFS-17367:
---
hadoop-yetus commented on PR #6735:
URL: https://github.com/apache/hadoop/pull/6735#issuecomment-2078012457
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 40s | | trunk passed |
| +1 :green_heart: | compile | 19m 20s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 17m 29s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 29s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 46s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 5m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 7s | | the patch passed |
| +1 :green_heart: | compile | 18m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 18m 30s | | the patch passed |
| +1 :green_heart: | compile | 17m 27s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 17m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 40s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 25s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 43s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 6m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 43m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 20m 22s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 260m 32s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6735/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 9s | | The patch does not
generate ASF License warnings. |
| | | 536m 0s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6735/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6735 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets
markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs
checkstyle |
| uname | Linux 4523d666c18b 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b2e6af9266c4cd71d6106a33a8380bd8d842c8ba |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
[jira] [Commented] (HDFS-17372) CommandProcessingThread#queue should use LinkedBlockingDeque to prevent high priority command blocked by low priority command
[
https://issues.apache.org/jira/browse/HDFS-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840896#comment-17840896
]
ASF GitHub Bot commented on HDFS-17372:
---
hadoop-yetus commented on PR #6530:
URL: https://github.com/apache/hadoop/pull/6530#issuecomment-2077843466
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 37s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 214m 18s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 304m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6530/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6530 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e1206fda9ab8 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 53cb0560e3fa5aae7cc975dba68a773e37440d1a |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6530/14/testReport/ |
| Max. process+thread count | 4504 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6530/14/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[jira] [Commented] (HDFS-17374) EC: StripedBlockReader#newConnectedPeer should set SO_TIMEOUT and SO_KEEPALIVE
[
https://issues.apache.org/jira/browse/HDFS-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840839#comment-17840839
]
ASF GitHub Bot commented on HDFS-17374:
---
hadoop-yetus commented on PR #6536:
URL: https://github.com/apache/hadoop/pull/6536#issuecomment-2077450806
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 93m 48s | | trunk passed |
| +1 :green_heart: | compile | 6m 30s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 09s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 06s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 154m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 59s | | the patch passed |
| +1 :green_heart: | compile | 3m 42s | | the patch passed |
| +1 :green_heart: | javac | 3m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 34s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 167m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 52s | | The patch does not
generate ASF License warnings. |
| | | 444m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6536 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 1b55e27aca49 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / f775e5a301f6777a42056cbc8fd42dc34ef178e4 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6536/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6536/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> EC: StripedBlockReader#newConnectedPeer should set SO_TIMEOUT and SO_KEEPALIVE
> --
>
> Key: HDFS-17374
> URL: https://issues.apache.org/jira/browse/HDFS-17374
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17372) CommandProcessingThread#queue should use LinkedBlockingDeque to prevent high priority command blocked by low priority command
[ https://issues.apache.org/jira/browse/HDFS-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840782#comment-17840782 ] ASF GitHub Bot commented on HDFS-17372: --- hfutatzhanghb commented on PR #6530: URL: https://github.com/apache/hadoop/pull/6530#issuecomment-2077114489 @Hexiaoqiao @zhangshuyan0 @ZanderXu Sir, this PR has run rightly for about two months. Could you please help me push this PR forward ? thanks a lot. > CommandProcessingThread#queue should use LinkedBlockingDeque to prevent high > priority command blocked by low priority command > - > > Key: HDFS-17372 > URL: https://issues.apache.org/jira/browse/HDFS-17372 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840777#comment-17840777
]
ASF GitHub Bot commented on HDFS-17381:
---
hadoop-yetus commented on PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#issuecomment-2077058346
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 107m 58s | | trunk passed |
| +1 :green_heart: | compile | 49m 43s | | trunk passed |
| +1 :green_heart: | checkstyle | 7m 02s | | trunk passed |
| -1 :x: | mvnsite | 5m 07s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6551/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 17m 03s | | trunk passed |
| +1 :green_heart: | shadedclient | 205m 05s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 40s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 12m 31s | | the patch passed |
| +1 :green_heart: | compile | 44m 26s | | the patch passed |
| +1 :green_heart: | javac | 44m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 6m 55s | | the patch passed |
| -1 :x: | mvnsite | 5m 09s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6551/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 16m 05s | | the patch passed |
| +1 :green_heart: | shadedclient | 210m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 46s | | The patch does not
generate ASF License warnings. |
| | | 649m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6551 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 9c4b94b77753 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / b0f92fb05bd02bf03620e8f0ce6a0f5892465ca0 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6551/1/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-client hadoop-tools/hadoop-distcp U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6551/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Distcp of EC files should not be limited to DFS.
>
>
> Key: HDFS-17381
> URL: https://issues.apache.org/jira/browse/HDFS-17381
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Sadanand Shenoy
>Assignee: Sadanand Shenoy
>Priority: Major
> Labels: pull-request-available
>
> Currently EC file support in distcp is limited to DFS and the code checks if
> the given FS instance is DistributedFileSystem, In Ozone, EC is supported
> now, and this limitation can be removed and a general contract for any
> filesystem that supports EC files should be
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840762#comment-17840762
]
ASF GitHub Bot commented on HDFS-17463:
---
hadoop-yetus commented on PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2076952024
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 53s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 274m 42s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 432m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6736 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 38640423e977 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d4197ab0fab16d60037594b06089181b8db46467 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/4/testReport/ |
| Max. process+thread count | 2639 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was
[jira] [Commented] (HDFS-17382) Add Apache Log4j Extras Library to Hadoop 3.3 for Enhanced Log Rolling Capabilities
[
https://issues.apache.org/jira/browse/HDFS-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840758#comment-17840758
]
ASF GitHub Bot commented on HDFS-17382:
---
hadoop-yetus commented on PR #6584:
URL: https://github.com/apache/hadoop/pull/6584#issuecomment-2076934071
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 01s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.4 Compile Tests _ |
| +0 :ok: | mvndep | 1m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 87m 15s | | branch-3.4 passed |
| +1 :green_heart: | compile | 38m 06s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 14m 54s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 13m 49s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 284m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 16s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 22m 44s | | the patch passed |
| +1 :green_heart: | compile | 35m 51s | | the patch passed |
| +1 :green_heart: | javac | 35m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 14m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 13m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 163m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 4m 34s | | hadoop-project in the patch
passed. |
| -1 :x: | unit | 289m 01s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6584/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 6m 00s | | hadoop-client-minicluster in
the patch passed. |
| -1 :x: | asflicense | 8m 37s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6584/1/artifact/out/results-asflicense.txt)
| The patch generated 2 ASF License warnings. |
| | | 835m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.cli.TestCacheAdminCLI |
| | hadoop.cli.TestXAttrCLI |
| | hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot |
| | hadoop.fs.viewfs.TestViewFileSystemHdfs |
| | hadoop.fs.viewfs.TestViewFileSystemLinkFallback |
| | hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash |
| | hadoop.fs.viewfs.TestViewFileSystemLinkRegex |
| |
hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract |
| | hadoop.fs.viewfs.TestViewFsAtHdfsRoot |
| | hadoop.fs.viewfs.TestViewFsHdfs |
| | hadoop.fs.viewfs.TestViewFsLinkFallback |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.qjournal.server.TestJournalNode |
| | hadoop.hdfs.qjournal.server.TestJournalNodeMXBean |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.aliasmap.TestSecureAliasMap |
| | hadoop.hdfs.server.blockmanagement.TestBlockReportLease |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithShortCircuitRead |
| |
hadoop.hdfs.server.common.blockaliasmap.impl.TestInMemoryLevelDBAliasMapClient |
| |
hadoop.hdfs.server.common.blockaliasmap.impl.TestLevelDBFileRegionAliasMap |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetCache |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl |
| | hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
| |
[jira] [Commented] (HDFS-17372) CommandProcessingThread#queue should use LinkedBlockingDeque to prevent high priority command blocked by low priority command
[ https://issues.apache.org/jira/browse/HDFS-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840755#comment-17840755 ] ASF GitHub Bot commented on HDFS-17372: --- hadoop-yetus commented on PR #6530: URL: https://github.com/apache/hadoop/pull/6530#issuecomment-2076916456 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | -1 :x: | patch | 0m 41s | | https://github.com/apache/hadoop/pull/6530 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6530 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6530/1/console | | versions | git=2.44.0.windows.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > CommandProcessingThread#queue should use LinkedBlockingDeque to prevent high > priority command blocked by low priority command > - > > Key: HDFS-17372 > URL: https://issues.apache.org/jira/browse/HDFS-17372 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17446) The DataNode adds a log to report its progress during addToReplicasMap execution.
[
https://issues.apache.org/jira/browse/HDFS-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840738#comment-17840738
]
ASF GitHub Bot commented on HDFS-17446:
---
hadoop-yetus commented on PR #6685:
URL: https://github.com/apache/hadoop/pull/6685#issuecomment-2076830219
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 43s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 280m 41s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6685/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 422m 29s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6685/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6685 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e72ed1fbf8e6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bc12b7d74f0e10de7e0b09b771b76eca3d9ff0be |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6685/10/testReport/ |
| Max. process+thread count | 3488 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-17378) Missing operationType for some operations in authorizer
[
https://issues.apache.org/jira/browse/HDFS-17378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840689#comment-17840689
]
ASF GitHub Bot commented on HDFS-17378:
---
hadoop-yetus commented on PR #6553:
URL: https://github.com/apache/hadoop/pull/6553#issuecomment-2076556974
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 86m 20s | | trunk passed |
| +1 :green_heart: | compile | 5m 53s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 140m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 23s | | the patch passed |
| +1 :green_heart: | compile | 3m 26s | | the patch passed |
| +1 :green_heart: | javac | 3m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 150m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 17s | | The patch does not
generate ASF License warnings. |
| | | 404m 06s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6553 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 1f78799f76e5 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / d68e060e85d2f356da6885c2e33508abe8cbf11c |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6553/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6553/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Missing operationType for some operations in authorizer
> ---
>
> Key: HDFS-17378
> URL: https://issues.apache.org/jira/browse/HDFS-17378
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.3.6
>Reporter: Sebastian Bernauer
>Priority: Minor
> Labels: pull-request-available
>
> In HDFS-14743 the operationType was implemented as a thread-local, which
> requires the caller to set the thread-local before calling the
> {{AccessControlEnforcer}} interface.
> Most operations will set the operationType to a String, but some operations
> set it to {{{}null{}}}.
> This causes the FSPermissionChecker to always call the old (deprecated)
> {{AccessControlEnforcer.checkPermission}} API (see
> [https://github.com/apache/hadoop/blob/50d256ef3c2531563bc6ba96dec6b78e154b4697/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java#L431])
> Instead of setting the operationType to {{null}} (e.g.
> [here|https://github.com/apache/hadoop/blob/50d256ef3c2531563bc6ba96dec6b78e154b4697/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L2767])
> we should set the correct
[jira] [Commented] (HDFS-17352) Add configuration to control whether DN delete this replica from disk when client requests a missing block
[
https://issues.apache.org/jira/browse/HDFS-17352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840685#comment-17840685
]
ASF GitHub Bot commented on HDFS-17352:
---
hadoop-yetus commented on PR #6559:
URL: https://github.com/apache/hadoop/pull/6559#issuecomment-2076543325
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 01s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 97m 05s | | trunk passed |
| +1 :green_heart: | compile | 6m 28s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 07s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 08s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 153m 07s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 48s | | the patch passed |
| +1 :green_heart: | compile | 3m 42s | | the patch passed |
| +1 :green_heart: | javac | 3m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 36s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 167m 08s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 41s | | The patch does not
generate ASF License warnings. |
| | | 445m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6559 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | MINGW64_NT-10.0-17763 d777404a85ec 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / cada170bb5603c7921ae6eae957dff26deb76c5d |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6559/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6559/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Add configuration to control whether DN delete this replica from disk when
> client requests a missing block
> ---
>
> Key: HDFS-17352
> URL: https://issues.apache.org/jira/browse/HDFS-17352
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> As discussed at
> https://github.com/apache/hadoop/pull/6464#issuecomment-1902959898
> we should add configuration to control whether DN delete this replica from
> disk when client requests a missing block.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17401) EC: Excess internal block may not be able to be deleted correctly when it's stored in fallback storage
[
https://issues.apache.org/jira/browse/HDFS-17401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840653#comment-17840653
]
ASF GitHub Bot commented on HDFS-17401:
---
hadoop-yetus commented on PR #6597:
URL: https://github.com/apache/hadoop/pull/6597#issuecomment-2076378724
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 109m 52s | | trunk passed |
| +1 :green_heart: | compile | 7m 27s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 08s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 179m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 14s | | the patch passed |
| +1 :green_heart: | compile | 4m 33s | | the patch passed |
| +1 :green_heart: | javac | 4m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 198m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 8m 19s | | The patch does not
generate ASF License warnings. |
| | | 523m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 fe745cdfa6fa 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 1fdac019fd28f997cd662f23fcf90c84be9e4613 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6597/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6597/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> EC: Excess internal block may not be able to be deleted correctly when it's
> stored in fallback storage
> --
>
> Key: HDFS-17401
> URL: https://issues.apache.org/jira/browse/HDFS-17401
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.6
>Reporter: Ruinan Gu
>Assignee: Ruinan Gu
>Priority: Major
> Labels: pull-request-available
>
> Excess internal block can't be deleted correctly when it's stored in fallback
> storage.
> Simple case:
> EC-RS-6-3-1024k file is stored using ALL_SSD storage policy(SSD is default
> storage type and DISK is fallback storage type), if the block group is as
> follows
> [0(SSD), 0(SSD), 1(SSD), 2(SSD), 3(SSD), 4(SSD), 5(SSD), 6(SSD), 7(SSD),
> 8(DISK)]
> The are two index 0 internal block and one of them should be chosen to
> delete.But the current implement chooses the index 0 internal blocks as
> candidates but DISK as exess storage type.As a result, the exess storage
> type(DISK) can not correspond to the exess internal blocks' storage type(SSD)
> correctly, and the exess internal block can not be deleted correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail:
[jira] [Commented] (HDFS-17397) Choose another DN as soon as possible, when encountering network issues
[
https://issues.apache.org/jira/browse/HDFS-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840636#comment-17840636
]
ASF GitHub Bot commented on HDFS-17397:
---
hadoop-yetus commented on PR #6591:
URL: https://github.com/apache/hadoop/pull/6591#issuecomment-2076278535
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 92m 28s | | trunk passed |
| +1 :green_heart: | compile | 5m 11s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 148m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 28s | | the patch passed |
| +1 :green_heart: | compile | 2m 48s | | the patch passed |
| +1 :green_heart: | javac | 2m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 06s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 01s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 161m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 27s | | The patch does not
generate ASF License warnings. |
| | | 426m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6591 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 12bdaf86eb0b 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 73d6c12734975d4adbd52f39e810478322b55a9f |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6591/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6591/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Choose another DN as soon as possible, when encountering network issues
> ---
>
> Key: HDFS-17397
> URL: https://issues.apache.org/jira/browse/HDFS-17397
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: xleoken
>Priority: Minor
> Labels: pull-request-available
> Attachments: hadoop.png
>
>
> Choose another DN as soon as possible, when encountering network issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[
https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840628#comment-17840628
]
ASF GitHub Bot commented on HDFS-17384:
---
hfutatzhanghb commented on code in PR #6762:
URL: https://github.com/apache/hadoop/pull/6762#discussion_r1578763344
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -6043,7 +6079,7 @@ void updatePipeline(
updatePipelineInternal(clientName, oldBlock, newBlock, newNodes,
newStorageIDs, logRetryCache);
} finally {
- writeUnlock("updatePipeline");
+ writeUnlock(FSNamesystemLockMode.GLOBAL, "updatePipeline");
Review Comment:
@ZanderXu Hi, sir. Have a question here. In documentaion, it is said that
updateBlockForPipeline and updatePipeline are only involved by blocks
operations. Why we use FSNamesystemLockMode.GLOBAL here?
> [FGL] Replace the global lock with global FS Lock and global BM lock
>
>
> Key: HDFS-17384
> URL: https://issues.apache.org/jira/browse/HDFS-17384
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: FGL, pull-request-available
>
> First, we can replace the current global lock with two locks, global FS lock
> and global BM lock.
> The global FS lock is used to make directory tree-related operations
> thread-safe.
> The global BM lock is used to make block-related operations and DN-related
> operations thread-safe.
>
> For some operations involving both directory tree and block or DN, the global
> FS lock and the global BM lock are acquired.
>
> The lock order should be:
> * The global FS lock
> * The global BM lock
>
> There are some special requirements for this ticket.
> * End-user can choose to use global lock or fine-grained lock through
> configuration.
> * Try not to modify the current implementation logic as much as possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17499) removeQueuedBlock in PendingDataNodeMessages has memory leak
[
https://issues.apache.org/jira/browse/HDFS-17499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840617#comment-17840617
]
ASF GitHub Bot commented on HDFS-17499:
---
hadoop-yetus commented on PR #6772:
URL: https://github.com/apache/hadoop/pull/6772#issuecomment-2076151769
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 20s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 43m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 269m 43s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 432m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6772/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6772 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3232c21f6f76 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 72e8113586364b18a8b536aa4062f2129b324236 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6772/1/testReport/ |
| Max. process+thread count | 2487 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6772/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[jira] [Commented] (HDFS-17402) StartupSafeMode should not exit when resources are from low to available
[
https://issues.apache.org/jira/browse/HDFS-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840599#comment-17840599
]
ASF GitHub Bot commented on HDFS-17402:
---
hadoop-yetus commented on PR #6618:
URL: https://github.com/apache/hadoop/pull/6618#issuecomment-2076062911
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 96m 45s | | trunk passed |
| +1 :green_heart: | compile | 6m 36s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 03s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 155m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 46s | | the patch passed |
| +1 :green_heart: | compile | 3m 55s | | the patch passed |
| +1 :green_heart: | javac | 3m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 163m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 39s | | The patch does not
generate ASF License warnings. |
| | | 444m 09s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6618 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 e13068a72d28 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 21fb4a3fb7d6153f6c54f8486f600e8f6ac97532 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6618/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6618/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> StartupSafeMode should not exit when resources are from low to available
>
>
> Key: HDFS-17402
> URL: https://issues.apache.org/jira/browse/HDFS-17402
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zilong Zhu
>Priority: Major
> Labels: pull-request-available
>
> After HDFS-17231, NameNode can exit safemode automatically when resources are
> from low to available. It used
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem#leaveSafeMode, this
> function will change BMSafeModeStatus. However, NameNode entering resource
> low safe mode doesn't change BMSafeModeStatus in
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem#enterSafeMode. This is
> not equal
> Now:
> a. NN enter StartupSafeMode
> b. NN enter ResourceLowSafeMode
> c. NN resources from low to available
> d. NN safemode off
>
> Expectations：
> a. NN enter StartupSafeMode
> b. NN enter ResourceLowSafeMode
> c. NN resources from low to available
> d. NN exit ResourceLowSafeMode but in StartupSafeMode
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17438) RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
[
https://issues.apache.org/jira/browse/HDFS-17438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840558#comment-17840558
]
ASF GitHub Bot commented on HDFS-17438:
---
hadoop-yetus commented on PR #6655:
URL: https://github.com/apache/hadoop/pull/6655#issuecomment-2075751591
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 90m 37s | | trunk passed |
| +1 :green_heart: | compile | 40m 32s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 09s | | trunk passed |
| -1 :x: | mvnsite | 4m 41s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 9m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 170m 01s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 172m 26s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 8m 11s | | the patch passed |
| +1 :green_heart: | compile | 39m 02s | | the patch passed |
| +1 :green_heart: | javac | 39m 02s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 6m 12s | | the patch passed |
| -1 :x: | mvnsite | 4m 39s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 10m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 178m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 07s | | The patch does not
generate ASF License warnings. |
| | | 544m 32s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6655 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 d2057a3233aa 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 5c14fd29eabcd45938699729b360bd373af87d2c |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/1/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
> -
>
> Key: HDFS-17438
> URL: https://issues.apache.org/jira/browse/HDFS-17438
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-17438.001.patch
>
>
> At present, when the status of all namenodes in an ns in the router is the
> same, the namenode which is the newest reported will be placed at the top of
> the cache. when the
[jira] [Updated] (HDFS-17499) removeQueuedBlock in PendingDataNodeMessages has memory leak
[ https://issues.apache.org/jira/browse/HDFS-17499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17499: -- Labels: pull-request-available (was: ) > removeQueuedBlock in PendingDataNodeMessages has memory leak > > > Key: HDFS-17499 > URL: https://issues.apache.org/jira/browse/HDFS-17499 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Danny Becker >Priority: Major > Labels: pull-request-available > > Introduced by HDFS-17477. The PendingDataNodeMessages#removeQueuedBlock() > will create an empty list stored in queueByBlockId for every incremental > block report processed by the BlockManager. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17499) removeQueuedBlock in PendingDataNodeMessages has memory leak
[ https://issues.apache.org/jira/browse/HDFS-17499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840536#comment-17840536 ] ASF GitHub Bot commented on HDFS-17499: --- dannytbecker opened a new pull request, #6772: URL: https://github.com/apache/hadoop/pull/6772 ### Description of PR Introduced by [HDFS-17477](https://issues.apache.org/jira/browse/HDFS-17477). The PendingDataNodeMessages#removeQueuedBlock() will create an empty list stored in queueByBlockId for every incremental block report processed by the BlockManager. ### How was this patch tested? Tested with TestLargeBlockReport#testBlockReportSucceedsWithLargerLengthLimit() ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > removeQueuedBlock in PendingDataNodeMessages has memory leak > > > Key: HDFS-17499 > URL: https://issues.apache.org/jira/browse/HDFS-17499 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Danny Becker >Priority: Major > > Introduced by HDFS-17477. The PendingDataNodeMessages#removeQueuedBlock() > will create an empty list stored in queueByBlockId for every incremental > block report processed by the BlockManager. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17439) Improve NNThroughputBenchmark to allow non super user to use the tool
[
https://issues.apache.org/jira/browse/HDFS-17439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840522#comment-17840522
]
ASF GitHub Bot commented on HDFS-17439:
---
hadoop-yetus commented on PR #6677:
URL: https://github.com/apache/hadoop/pull/6677#issuecomment-2075445476
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 01s | | markdownlint was not available.
|
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 105m 33s | | trunk passed |
| +1 :green_heart: | compile | 46m 51s | | trunk passed |
| +1 :green_heart: | checkstyle | 7m 01s | | trunk passed |
| -1 :x: | mvnsite | 5m 22s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6677/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 12m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 192m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 39s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 11m 28s | | the patch passed |
| +1 :green_heart: | compile | 44m 13s | | the patch passed |
| +1 :green_heart: | javac | 44m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 6m 53s | | the patch passed |
| -1 :x: | mvnsite | 5m 10s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6677/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 12m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 204m 00s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 31s | | The patch does not
generate ASF License warnings. |
| | | 623m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6677 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets
markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs
checkstyle |
| uname | MINGW64_NT-10.0-17763 6776337c4aef 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / ba26e23d24761de4ff88b31836f6b1f731756656 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6677/1/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6677/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Improve NNThroughputBenchmark to allow non super user to use the tool
> -
>
> Key: HDFS-17439
> URL: https://issues.apache.org/jira/browse/HDFS-17439
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: benchmarks, namenode
>Reporter: Fateh Singh
>Priority: Major
> Labels: pull-request-available
>
> The NNThroughputBenchmark can only be used with hdfs user or any user with
> super user privileges since entering/exiting safemode is a privileged
> operation. However, when using super user, ACL checks are skipped. Hence it
> renders the tool to be useless when testing namenode performance along with
>
[jira] [Commented] (HDFS-17446) The DataNode adds a log to report its progress during addToReplicasMap execution.
[
https://issues.apache.org/jira/browse/HDFS-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840478#comment-17840478
]
ASF GitHub Bot commented on HDFS-17446:
---
hadoop-yetus commented on PR #6685:
URL: https://github.com/apache/hadoop/pull/6685#issuecomment-2075182513
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 114m 45s | | trunk passed |
| +1 :green_heart: | compile | 8m 34s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 9m 03s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 191m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 38s | | the patch passed |
| +1 :green_heart: | compile | 5m 05s | | the patch passed |
| +1 :green_heart: | javac | 5m 05s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 46s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 217m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 8m 26s | | The patch does not
generate ASF License warnings. |
| | | 563m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6685 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 da27b6e6c2af 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 150f9834c98b60a1d0013945927a007d22280259 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6685/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6685/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> The DataNode adds a log to report its progress during addToReplicasMap
> execution.
> -
>
> Key: HDFS-17446
> URL: https://issues.apache.org/jira/browse/HDFS-17446
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: xiaojunxiang
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-03-28-23-53-29-057.png,
> image-2024-04-01-17-26-05-433.png
>
>
> Datanodes do not print logs when addToReplicasMap is just started, so we will
> not be aware of its progress, or even wait up to 4 hours for a large cluster.
> This is very confusing.
> A cluster is as follows:
> !image-2024-03-28-23-53-29-057.png!
>
> The optimized result is as follows:
> (Because Scan thread is ForkJoinPool, so the printing may be repeated, but it
> will affect the overall progress of the presentation
> )
> !image-2024-04-01-17-26-05-433.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[
https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840413#comment-17840413
]
ASF GitHub Bot commented on HDFS-17444:
---
hadoop-yetus commented on PR #6694:
URL: https://github.com/apache/hadoop/pull/6694#issuecomment-2074815022
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 89m 11s | | trunk passed |
| +1 :green_heart: | compile | 6m 24s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 142m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 28s | | the patch passed |
| +1 :green_heart: | compile | 3m 23s | | the patch passed |
| +1 :green_heart: | javac | 3m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 04s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 155m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 20s | | The patch does not
generate ASF License warnings. |
| | | 414m 41s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6694 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 5b7668c3c5d3 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 745dd1fd0c1d69030ec3ab0dd7365b731d382603 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6694/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6694/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Add getJournalSyncerStatus jmx metrics, to Indicates whether the
> JournalSyncer thread for the every namespace has enter working state.
> --
>
> Key: HDFS-17444
> URL: https://issues.apache.org/jira/browse/HDFS-17444
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4, 3.3.6
>Reporter: xiaojunxiang
>Assignee: xiaojunxiang
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-04-01-12-16-26-932.png
>
>
> The JornalNode JVM process is not immediately in a normal state until the
> JournalSyncer thread is started.
> For some management platforms such as Ambari rolling restart JournalNode, we
> need a jmx metric to determine whether the JournalSyncer thread is started
> and enter working state for current namespace before restarting the next
> JournalNode. Otherwise, restart too quickly and more than half of
> JournalNodes will be out of order, causing the NameNode to die.
> When i add it , the effect is as follows:
> !image-2024-04-01-12-16-26-932.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail:
[jira] [Commented] (HDFS-17454) Fix namenode fsck swallows the exception stacktrace, this can help us to troubleshooting log.
[ https://issues.apache.org/jira/browse/HDFS-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840399#comment-17840399 ] ASF GitHub Bot commented on HDFS-17454: --- xiaojunxiang2023 commented on PR #6709: URL: https://github.com/apache/hadoop/pull/6709#issuecomment-2074706653 > Can you add a test for this, considering now we are propagating the real error back to the client. > > I think if the actual error is being thrown in that case on the client side we can live with just the message, rather than having the entire trace. @ayushtkn @hiwangzhihui Hi，I think the stacktrace can better help analyze problem. And I have aready add the unit tests and successly pass the yetus. Please help me review it again. > Fix namenode fsck swallows the exception stacktrace, this can help us to > troubleshooting log. > - > > Key: HDFS-17454 > URL: https://issues.apache.org/jira/browse/HDFS-17454 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: xiaojunxiang >Priority: Minor > Labels: pull-request-available > Attachments: image-2024-04-05-15-40-37-147.png, > image-2024-04-05-15-41-38-420.png, image-2024-04-07-13-22-22-493.png, > image-2024-04-07-13-22-46-684.png > > > When I used `hdfs fsck /xxx.txt -move`, missing error, but I can't kown the > reason, because the exception stacktrace doesn't append to LOG, original code: > !image-2024-04-05-15-40-37-147.png! > > When I fix it, look, we can see the exception stacktrace: > !image-2024-04-07-13-22-22-493.png! > !image-2024-04-07-13-22-46-684.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17454) Fix namenode fsck swallows the exception stacktrace, this can help us to troubleshooting log.
[
https://issues.apache.org/jira/browse/HDFS-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840389#comment-17840389
]
ASF GitHub Bot commented on HDFS-17454:
---
hadoop-yetus commented on PR #6709:
URL: https://github.com/apache/hadoop/pull/6709#issuecomment-2074675857
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 91m 59s | | trunk passed |
| +1 :green_heart: | compile | 6m 20s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 03s | | trunk passed |
| +1 :green_heart: | shadedclient | 150m 01s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 38s | | the patch passed |
| +1 :green_heart: | compile | 3m 36s | | the patch passed |
| +1 :green_heart: | javac | 3m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 161m 09s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 28s | | The patch does not
generate ASF License warnings. |
| | | 430m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6709 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 c868f865a709 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / d1b50f01a9ec635128adf2f54c66f882f39eb0b3 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6709/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6709/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Fix namenode fsck swallows the exception stacktrace, this can help us to
> troubleshooting log.
> -
>
> Key: HDFS-17454
> URL: https://issues.apache.org/jira/browse/HDFS-17454
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.6
>Reporter: xiaojunxiang
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2024-04-05-15-40-37-147.png,
> image-2024-04-05-15-41-38-420.png, image-2024-04-07-13-22-22-493.png,
> image-2024-04-07-13-22-46-684.png
>
>
> When I used `hdfs fsck /xxx.txt -move`, missing error, but I can't kown the
> reason, because the exception stacktrace doesn't append to LOG, original code:
> !image-2024-04-05-15-40-37-147.png!
>
> When I fix it, look, we can see the exception stacktrace:
> !image-2024-04-07-13-22-22-493.png!
> !image-2024-04-07-13-22-46-684.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17497) Logic for committed blocks is mixed when computing file size
[
https://issues.apache.org/jira/browse/HDFS-17497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840351#comment-17840351
]
ASF GitHub Bot commented on HDFS-17497:
---
hadoop-yetus commented on PR #6765:
URL: https://github.com/apache/hadoop/pull/6765#issuecomment-2074475186
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 31s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 3s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 290 unchanged - 1
fixed = 290 total (was 291) |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 269m 3s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 427m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6765/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6765 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3ed817d3780c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0aa96155ae7aed9c69d8c0ede601fffd4bc8c17f |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6765/2/testReport/ |
| Max. process+thread count | 2789 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6765/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840350#comment-17840350
]
ASF GitHub Bot commented on HDFS-17488:
---
hadoop-yetus commented on PR #6759:
URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2074474125
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 26s | | trunk passed |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 1s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 232m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 374m 52s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6759 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 1f3c0fcbc898 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1f166985eedcfea6aedcc62046908f86ed9827fa |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/4/testReport/ |
| Max. process+thread count | 4094 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-17458) Remove unnecessary BP lock in ReplicaMap
[
https://issues.apache.org/jira/browse/HDFS-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840301#comment-17840301
]
ASF GitHub Bot commented on HDFS-17458:
---
hadoop-yetus commented on PR #6717:
URL: https://github.com/apache/hadoop/pull/6717#issuecomment-2074182504
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 110m 52s | | trunk passed |
| +1 :green_heart: | compile | 7m 26s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 179m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 11s | | the patch passed |
| +1 :green_heart: | compile | 4m 15s | | the patch passed |
| +1 :green_heart: | javac | 4m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 06s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 191m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 18s | | The patch does not
generate ASF License warnings. |
| | | 515m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6717 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 8b1fb94b8141 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 4acc5b4369d4e0528645386df1720ee3bb8cced3 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6717/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6717/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Remove unnecessary BP lock in ReplicaMap
>
>
> Key: HDFS-17458
> URL: https://issues.apache.org/jira/browse/HDFS-17458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.4.0
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> In HDFS-16429 we make LightWeightResizableGSet to be thread safe, and in
> HDFS-16511 we change some methods in ReplicaMap to acquire read lock instead
> of acquiring write lock.
> This PR try to remove unnecessary Block_Pool read lock further.
> Recently, I performed stress tests on datanodes to measure their read/write
> operations/second.
> Before we removing some lock, it can only achieve ~2K write ops. After
> optimizing, it can achieve more than 5K write ops.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17464) Improve some logs output in class FsDatasetImpl
[ https://issues.apache.org/jira/browse/HDFS-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840293#comment-17840293 ] ASF GitHub Bot commented on HDFS-17464: --- hfutatzhanghb commented on PR #6724: URL: https://github.com/apache/hadoop/pull/6724#issuecomment-2074091225 @zhangshuyan0 @ZanderXu @haiyang1987 Hi, sir. please help me reivew this simple modification when you have free time, thanks~ > Improve some logs output in class FsDatasetImpl > --- > > Key: HDFS-17464 > URL: https://issues.apache.org/jira/browse/HDFS-17464 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17456) Fix the dfsused statistics of datanode are incorrect when appending a file.
[
https://issues.apache.org/jira/browse/HDFS-17456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840292#comment-17840292
]
ASF GitHub Bot commented on HDFS-17456:
---
hadoop-yetus commented on PR #6713:
URL: https://github.com/apache/hadoop/pull/6713#issuecomment-2074088648
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 89m 22s | | trunk passed |
| +1 :green_heart: | compile | 6m 03s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 02s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 02s | | trunk passed |
| +1 :green_heart: | shadedclient | 148m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 58s | | the patch passed |
| +1 :green_heart: | compile | 3m 31s | | the patch passed |
| +1 :green_heart: | javac | 3m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 159m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 21s | | The patch does not
generate ASF License warnings. |
| | | 423m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6713 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 ebc41b0c2c32 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / ecf2c6ae3012d4fe3ac718774e203062dc64b85f |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6713/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6713/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Fix the dfsused statistics of datanode are incorrect when appending a file.
> ---
>
> Key: HDFS-17456
> URL: https://issues.apache.org/jira/browse/HDFS-17456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.3.3
>Reporter: fuchaohong
>Priority: Major
> Labels: pull-request-available
>
> In our production env, the namenode page showed that the datanode space had
> been used up, but the actual datanode machine still had a lot of free space.
> After troubleshooting, the dfsused statistics of datanode are incorrect when
> appending a file. The following is the dfsused after each append of 100.
> |*Error*|*Expect*|
> |0|0|
> |100|100|
> |300|200|
> |600|300|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17464) Improve some logs output in class FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840290#comment-17840290
]
ASF GitHub Bot commented on HDFS-17464:
---
hadoop-yetus commented on PR #6724:
URL: https://github.com/apache/hadoop/pull/6724#issuecomment-2074072638
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 112m 04s | | trunk passed |
| +1 :green_heart: | compile | 8m 02s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 188m 07s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 41s | | the patch passed |
| +1 :green_heart: | compile | 4m 59s | | the patch passed |
| +1 :green_heart: | javac | 4m 59s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 31s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 04s | | the patch passed |
| +1 :green_heart: | shadedclient | 210m 01s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 8m 21s | | The patch does not
generate ASF License warnings. |
| | | 548m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6724 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 0d90d6683388 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 7693441ac77911b7fe3ce75935d02b9545eedb51 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6724/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6724/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Improve some logs output in class FsDatasetImpl
> ---
>
> Key: HDFS-17464
> URL: https://issues.apache.org/jira/browse/HDFS-17464
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.4.0
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Minor
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17462) NPE in Router concat when trg is an empty file.
[
https://issues.apache.org/jira/browse/HDFS-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840281#comment-17840281
]
ASF GitHub Bot commented on HDFS-17462:
---
hadoop-yetus commented on PR #6722:
URL: https://github.com/apache/hadoop/pull/6722#issuecomment-2074005099
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 87m 10s | | trunk passed |
| +1 :green_heart: | compile | 4m 55s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 137m 05s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 03s | | the patch passed |
| +1 :green_heart: | compile | 2m 24s | | the patch passed |
| +1 :green_heart: | javac | 2m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 08s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 151m 08s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 26s | | The patch does not
generate ASF License warnings. |
| | | 398m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6722 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 81e6d3c33564 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / cff78d9cb5d00eb38a44ab14958473155a7c31e1 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6722/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6722/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> NPE in Router concat when trg is an empty file.
> ---
>
> Key: HDFS-17462
> URL: https://issues.apache.org/jira/browse/HDFS-17462
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.10.2, 3.3.6
>Reporter: NaihaoFan
>Priority: Minor
> Labels: pull-request-available
>
> When trg of Router concat is an empty file, it will trigger NPE in Router,
> and the concat will fail, example:
> This is because when trg is an empty file, NameNode will return
> lastLocatedBlock as null in the response of getBlockLocations. And Router
> will not check null of lastLocatedBlock returned, instead Router will use it
> to get block pool id directly.
> Trg of concat is an empty file should be allowed in router since this case is
> supported by concat of NameNode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17497) Logic for committed blocks is mixed when computing file size
[
https://issues.apache.org/jira/browse/HDFS-17497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840264#comment-17840264
]
ASF GitHub Bot commented on HDFS-17497:
---
ZanderXu commented on PR #6765:
URL: https://github.com/apache/hadoop/pull/6765#issuecomment-2073863664
@xkrogen master, I very much hope you can review this PR when you are
available, since you are familiar with ~~HDFS-10843~~
> Logic for committed blocks is mixed when computing file size
>
>
> Key: HDFS-17497
> URL: https://issues.apache.org/jira/browse/HDFS-17497
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> One in-writing HDFS file may contains multiple committed blocks, as follows
> (assume one file contains three blocks):
> || ||Block 1||Block 2||Block 3||
> |Case 1|Complete|Commit|UnderConstruction|
> |Case 2|Complete|Commit|Commit|
> |Case 3|Commit|Commit|Commit|
>
> But the logic for committed blocks is mixed when computing file size, it
> ignores the bytes of the last committed block and contains the bytes of other
> committed blocks.
> {code:java}
> public final long computeFileSize(boolean includesLastUcBlock,
> boolean usePreferredBlockSize4LastUcBlock) {
> if (blocks.length == 0) {
> return 0;
> }
> final int last = blocks.length - 1;
> //check if the last block is BlockInfoUnderConstruction
> BlockInfo lastBlk = blocks[last];
> long size = lastBlk.getNumBytes();
> // the last committed block is not complete, so it's bytes may be ignored.
> if (!lastBlk.isComplete()) {
> if (!includesLastUcBlock) {
>size = 0;
> } else if (usePreferredBlockSize4LastUcBlock) {
>size = isStriped()?
>getPreferredBlockSize() *
>((BlockInfoStriped)lastBlk).getDataBlockNum() :
>getPreferredBlockSize();
> }
> }
> // The bytes of other committed blocks are calculated into the file length.
> for (int i = 0; i < last; i++) {
> size += blocks[i].getNumBytes();
> }
> return size;
> } {code}
> The bytes of one committed block will not be changed, so the bytes of the
> last committed block should be calculated into the file length too.
>
> And the logic for committed blocks is mixed too when computing file length in
> DFSInputStream. Normally DFSInputStream does not need to get visible length
> for committed block regardless of whether the committed block is the last
> block or not.
>
> -HDFS-10843- encountered one bug which actually caused by the committed
> block, but -HDFS-10843- fixed that bug by updating quota usage when
> completing block. The num of bytes of the committed block will no longer
> change, so we should update the quota usage when the block is committed,
> which can reduce the delta quota usage in time.
>
> So there are somethings we need to do:
> * Unify the calculation logic for all committed blocks in
> {{computeFileSize}} of {{INodeFile}}
> * Unify the calculation logic for all committed blocks in {{getFileLength}}
> of {{DFSInputStream}}
> * Update quota usage when committing block
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17467) IncrementalBlockReportManager#getPerStorageIBR may throw NPE when remove volumes
[ https://issues.apache.org/jira/browse/HDFS-17467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840263#comment-17840263 ] ASF GitHub Bot commented on HDFS-17467: --- hfutatzhanghb commented on PR #6730: URL: https://github.com/apache/hadoop/pull/6730#issuecomment-2073859407 > @hfutatzhanghb thanks for your report. It looks like [HDFS-17488](https://issues.apache.org/jira/browse/HDFS-17488). Let's review [HDFS-17488](https://issues.apache.org/jira/browse/HDFS-17488) together to fix this bug if you have time. Thanks @ZanderXu OK, Sir. I will review HDFS-17488 soonly. Thanks for reminding. > IncrementalBlockReportManager#getPerStorageIBR may throw NPE when remove > volumes > > > Key: HDFS-17467 > URL: https://issues.apache.org/jira/browse/HDFS-17467 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > When we remove volumes, it may cause > IncrementalBlockReportManager#getPerStorageIBR throws NPE. > Consider below situation: > 1、we have down createRbw、finalizeBlock. But have not done > datanode.closeBlock in method `BlockReceiver.PacketResponder#finalizeBlock`. > 2、we remove volume which replica was written to and it executes such code: > `storageMap.remove(storageUuid);` > 3、 we begin to execute datanode.closeBlock which try to send IBR to NameNode. > but when getting DatanodeStorage from storageMap using > storageUuid, we will get null because we have remove this storageUuid key > from storageMap. > 4、Throw NPE in getPerStorageIBR method, because ConcurrentHashMap don't allow > null key. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17467) IncrementalBlockReportManager#getPerStorageIBR may throw NPE when remove volumes
[ https://issues.apache.org/jira/browse/HDFS-17467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840262#comment-17840262 ] ASF GitHub Bot commented on HDFS-17467: --- ZanderXu commented on PR #6730: URL: https://github.com/apache/hadoop/pull/6730#issuecomment-2073833703 @hfutatzhanghb thanks for your report. It looks like HDFS-17488. Let's review HDFS-17488 together to fix this bug if you have time. Thanks > IncrementalBlockReportManager#getPerStorageIBR may throw NPE when remove > volumes > > > Key: HDFS-17467 > URL: https://issues.apache.org/jira/browse/HDFS-17467 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > > When we remove volumes, it may cause > IncrementalBlockReportManager#getPerStorageIBR throws NPE. > Consider below situation: > 1、we have down createRbw、finalizeBlock. But have not done > datanode.closeBlock in method `BlockReceiver.PacketResponder#finalizeBlock`. > 2、we remove volume which replica was written to and it executes such code: > `storageMap.remove(storageUuid);` > 3、 we begin to execute datanode.closeBlock which try to send IBR to NameNode. > but when getting DatanodeStorage from storageMap using > storageUuid, we will get null because we have remove this storageUuid key > from storageMap. > 4、Throw NPE in getPerStorageIBR method, because ConcurrentHashMap don't allow > null key. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17469) Audit log for reportBadBlocks RPC
[ https://issues.apache.org/jira/browse/HDFS-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840261#comment-17840261 ] ASF GitHub Bot commented on HDFS-17469: --- ZanderXu commented on PR #6731: URL: https://github.com/apache/hadoop/pull/6731#issuecomment-2073830839 Merged. Thanks @cxzl25 for your contribution. > Audit log for reportBadBlocks RPC > - > > Key: HDFS-17469 > URL: https://issues.apache.org/jira/browse/HDFS-17469 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: dzcxzl >Priority: Minor > Labels: pull-request-available > > After [HDFS-10347|https://issues.apache.org/jira/browse/HDFS-10347], we can > know the DN corresponding to the reported bad block, but we do not know the > reported Client IP. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17469) Audit log for reportBadBlocks RPC
[ https://issues.apache.org/jira/browse/HDFS-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840260#comment-17840260 ] ASF GitHub Bot commented on HDFS-17469: --- ZanderXu merged PR #6731: URL: https://github.com/apache/hadoop/pull/6731 > Audit log for reportBadBlocks RPC > - > > Key: HDFS-17469 > URL: https://issues.apache.org/jira/browse/HDFS-17469 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: dzcxzl >Priority: Minor > Labels: pull-request-available > > After [HDFS-10347|https://issues.apache.org/jira/browse/HDFS-10347], we can > know the DN corresponding to the reported bad block, but we do not know the > reported Client IP. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17466) Move FsVolumeList#getVolumes() invocation out of DataSetLock
[
https://issues.apache.org/jira/browse/HDFS-17466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840252#comment-17840252
]
ASF GitHub Bot commented on HDFS-17466:
---
hadoop-yetus commented on PR #6728:
URL: https://github.com/apache/hadoop/pull/6728#issuecomment-2073685786
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 90m 04s | | trunk passed |
| +1 :green_heart: | compile | 6m 10s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 00s | | trunk passed |
| +1 :green_heart: | shadedclient | 148m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 55s | | the patch passed |
| +1 :green_heart: | compile | 3m 38s | | the patch passed |
| +1 :green_heart: | javac | 3m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 159m 09s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 21s | | The patch does not
generate ASF License warnings. |
| | | 425m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6728 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 17e48f8a2a3c 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 814377ae9d9dde8bbe8786e0da491220cf156fbe |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6728/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6728/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Move FsVolumeList#getVolumes() invocation out of DataSetLock
>
>
> Key: HDFS-17466
> URL: https://issues.apache.org/jira/browse/HDFS-17466
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.4.0
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17467) IncrementalBlockReportManager#getPerStorageIBR may throw NPE when remove volumes
[
https://issues.apache.org/jira/browse/HDFS-17467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840240#comment-17840240
]
ASF GitHub Bot commented on HDFS-17467:
---
hadoop-yetus commented on PR #6730:
URL: https://github.com/apache/hadoop/pull/6730#issuecomment-2073563142
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 88m 00s | | trunk passed |
| +1 :green_heart: | compile | 5m 58s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 148m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 29s | | the patch passed |
| +1 :green_heart: | compile | 3m 24s | | the patch passed |
| +1 :green_heart: | javac | 3m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 56s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 155m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 08s | | The patch does not
generate ASF License warnings. |
| | | 418m 05s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6730 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 2106f698c8b4 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / a23b3b2f0304c55902d615953973ba72ab42ce89 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6730/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6730/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> IncrementalBlockReportManager#getPerStorageIBR may throw NPE when remove
> volumes
>
>
> Key: HDFS-17467
> URL: https://issues.apache.org/jira/browse/HDFS-17467
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.4.0
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> When we remove volumes, it may cause
> IncrementalBlockReportManager#getPerStorageIBR throws NPE.
> Consider below situation:
> 1、we have down createRbw、finalizeBlock. But have not done
> datanode.closeBlock in method `BlockReceiver.PacketResponder#finalizeBlock`.
> 2、we remove volume which replica was written to and it executes such code:
> `storageMap.remove(storageUuid);`
> 3、 we begin to execute datanode.closeBlock which try to send IBR to NameNode.
> but when getting DatanodeStorage from storageMap using
> storageUuid, we will get null because we have remove this storageUuid key
> from storageMap.
> 4、Throw NPE in getPerStorageIBR method, because ConcurrentHashMap don't allow
> null key.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HDFS-17469) Audit log for reportBadBlocks RPC
[
https://issues.apache.org/jira/browse/HDFS-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840238#comment-17840238
]
ASF GitHub Bot commented on HDFS-17469:
---
hadoop-yetus commented on PR #6731:
URL: https://github.com/apache/hadoop/pull/6731#issuecomment-2073517962
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 85m 37s | | trunk passed |
| +1 :green_heart: | compile | 5m 45s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 143m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 20s | | the patch passed |
| +1 :green_heart: | compile | 3m 17s | | the patch passed |
| +1 :green_heart: | javac | 3m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 152m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 07s | | The patch does not
generate ASF License warnings. |
| | | 407m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6731 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 f997501a3e3b 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 17b822eb25e449b13a908eebf6c7a15628356b8c |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6731/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6731/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Audit log for reportBadBlocks RPC
> -
>
> Key: HDFS-17469
> URL: https://issues.apache.org/jira/browse/HDFS-17469
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: dzcxzl
>Priority: Minor
> Labels: pull-request-available
>
> After [HDFS-10347|https://issues.apache.org/jira/browse/HDFS-10347], we can
> know the DN corresponding to the reported bad block, but we do not know the
> reported Client IP.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17470) FsVolumeList#getNextVolume can be moved out of DataSetLock
[
https://issues.apache.org/jira/browse/HDFS-17470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840223#comment-17840223
]
ASF GitHub Bot commented on HDFS-17470:
---
hadoop-yetus commented on PR #6733:
URL: https://github.com/apache/hadoop/pull/6733#issuecomment-2073412719
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 108m 51s | | trunk passed |
| +1 :green_heart: | compile | 7m 38s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 05s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 175m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 36s | | the patch passed |
| +1 :green_heart: | compile | 4m 16s | | the patch passed |
| +1 :green_heart: | javac | 4m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 49s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 05s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 195m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 41s | | The patch does not
generate ASF License warnings. |
| | | 513m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6733 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 2785c41ce87a 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 382dfcc3c3e8c4fdd85102c8b1cd602d4fc35298 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6733/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6733/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> FsVolumeList#getNextVolume can be moved out of DataSetLock
> --
>
> Key: HDFS-17470
> URL: https://issues.apache.org/jira/browse/HDFS-17470
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> FsVolumeList#getNextVolume can be out of BLOCK_POOl read lock.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17497) Logic for committed blocks is mixed when computing file size
[
https://issues.apache.org/jira/browse/HDFS-17497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840218#comment-17840218
]
ASF GitHub Bot commented on HDFS-17497:
---
hadoop-yetus commented on PR #6765:
URL: https://github.com/apache/hadoop/pull/6765#issuecomment-2073379567
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 48s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 2s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6765/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 290 unchanged
- 1 fixed = 294 total (was 291) |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 268m 19s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6765/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 426m 24s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6765/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6765 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux be83e02b2463 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0c54e4a633f06fcaceb4781e63f06bb8a76b0fc8 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840183#comment-17840183
]
ASF GitHub Bot commented on HDFS-17463:
---
hadoop-yetus commented on PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2073100512
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 31s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 210m 20s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 307m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6736 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux d39cd7a88ad5 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8399f99ae05faafbd8e272ca638038b8806ec94f |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/3/testReport/ |
| Max. process+thread count | 4366 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6736/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was
[jira] [Commented] (HDFS-17367) Add PercentUsed for Different StorageTypes in JMX
[
https://issues.apache.org/jira/browse/HDFS-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840182#comment-17840182
]
ASF GitHub Bot commented on HDFS-17367:
---
hadoop-yetus commented on PR #6735:
URL: https://github.com/apache/hadoop/pull/6735#issuecomment-2073091922
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 01s | | markdownlint was not available.
|
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 01s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 91m 20s | | trunk passed |
| +1 :green_heart: | compile | 41m 20s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 10s | | trunk passed |
| -1 :x: | mvnsite | 4m 39s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6735/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 11m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 172m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 39s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 9m 54s | | the patch passed |
| +1 :green_heart: | compile | 38m 19s | | the patch passed |
| +1 :green_heart: | javac | 38m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 6m 13s | | the patch passed |
| -1 :x: | mvnsite | 4m 46s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6735/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 11m 05s | | the patch passed |
| +1 :green_heart: | shadedclient | 179m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 22s | | The patch does not
generate ASF License warnings. |
| | | 550m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6735 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets
markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs
checkstyle |
| uname | MINGW64_NT-10.0-17763 611b5ed007dc 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 71c5c8543faf6f7acac12e23f9d9b1d86c8d44aa |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6735/1/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6735/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Add PercentUsed for Different StorageTypes in JMX
> -
>
> Key: HDFS-17367
> URL: https://issues.apache.org/jira/browse/HDFS-17367
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: metrics, namenode
>Affects Versions: 3.5.0
>Reporter: Hualong Zhang
>Assignee: Hualong Zhang
>Priority: Major
> Labels: pull-request-available
>
> Currently, the NameNode only displays PercentUsed for the entire cluster. We
> plan to add corresponding PercentUsed metrics for different StorageTypes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[
https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840147#comment-17840147
]
ASF GitHub Bot commented on HDFS-17463:
---
hadoop-yetus commented on PR #6736:
URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2072822967
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 00s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 87m 03s | | trunk passed |
| +1 :green_heart: | compile | 6m 15s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 150m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 38s | | the patch passed |
| +1 :green_heart: | compile | 3m 33s | | the patch passed |
| +1 :green_heart: | javac | 3m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 164m 04s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 55s | | The patch does not
generate ASF License warnings. |
| | | 429m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6736 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | MINGW64_NT-10.0-17763 263e1c7dd5ce 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / f78c740a23e1a9d140a56098297642541a6f8343 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6736/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6736/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Support the switch StringTable Split ID feature
> ---
>
> Key: HDFS-17463
> URL: https://issues.apache.org/jira/browse/HDFS-17463
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4
>Reporter: wangzhihui
>Priority: Major
> Labels: pull-request-available
> Attachments: Image_struct.png, error.png
>
>
> desc:
> *
> Hadoop 3.2 introduced optimization features for HDFS StringTable
> (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of
> Hadoop upgraded to 3.2 and later versions not supporting downgrade
> operations.
> !error.png!
> * This issue has also been discussed in HDFS-14831, and it is recommended to
> revert the feature, but it cannot fundamentally solve the problem。
> *
> Therefore, we have added an optimization to support downgrading
>
> Solution:
> * First, we will add the "dfs. image. save. splitId. stringTable" conf
> switch "StringTable optimization feature" is enabled
> * When the conf value is false, an Image file compatible with lower versions
> of HDFS is generated to support downgrading.
> *
> The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2
> is shown in the following figure.
>
[jira] [Commented] (HDFS-17496) DataNode supports more fine-grained dataset lock based on blockid
[
https://issues.apache.org/jira/browse/HDFS-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840139#comment-17840139
]
ASF GitHub Bot commented on HDFS-17496:
---
hadoop-yetus commented on PR #6764:
URL: https://github.com/apache/hadoop/pull/6764#issuecomment-2072709220
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6764/2/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 58s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6764/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 54 unchanged -
0 fixed = 58 total (was 54) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 247m 46s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6764/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 402m 14s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.namenode.ha.TestDNFencing |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6764/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6764 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4ac18b1b0858 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC
[jira] [Commented] (HDFS-17484) Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually
[
https://issues.apache.org/jira/browse/HDFS-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840134#comment-17840134
]
ASF GitHub Bot commented on HDFS-17484:
---
hadoop-yetus commented on PR #6758:
URL: https://github.com/apache/hadoop/pull/6758#issuecomment-2072683616
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 0s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 251 unchanged
- 0 fixed = 252 total (was 251) |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 228m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 373m 43s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6758 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 0983654d82e7 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5
20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0b75e7bd36f06728bcf31f08686c476d580270eb |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
[jira] [Commented] (HDFS-17496) DataNode supports more fine-grained dataset lock based on blockid
[
https://issues.apache.org/jira/browse/HDFS-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840133#comment-17840133
]
ASF GitHub Bot commented on HDFS-17496:
---
hadoop-yetus commented on PR #6764:
URL: https://github.com/apache/hadoop/pull/6764#issuecomment-2072671948
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 18s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6764/1/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 9s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6764/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 54 unchanged -
0 fixed = 58 total (was 54) |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 241m 17s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6764/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 397m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.server.namenode.ha.TestDNFencing |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6764/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6764 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux fe96732bd6b6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk /
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840102#comment-17840102
]
ASF GitHub Bot commented on HDFS-17488:
---
hadoop-yetus commented on PR #6759:
URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2072325438
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 39s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 30s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 204m 49s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 293m 24s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBPOfferService |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6759 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 258959f150db 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 07f35e4bb9fa7af0e940cb91c970bd4eaf5c5ae0 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/3/testReport/ |
| Max. process+thread count | 4188 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-17497) Logic for committed blocks is mixed when computing file size
[
https://issues.apache.org/jira/browse/HDFS-17497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840099#comment-17840099
]
ASF GitHub Bot commented on HDFS-17497:
---
ZanderXu opened a new pull request, #6765:
URL: https://github.com/apache/hadoop/pull/6765
One in-writing HDFS file may contains multiple committed blocks, as
follows (assume one file contains three blocks):
| Block 1 | Block 2 | Block 3
> Logic for committed blocks is mixed when computing file size
>
>
> Key: HDFS-17497
> URL: https://issues.apache.org/jira/browse/HDFS-17497
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: ZanderXu
>Priority: Major
>
> One in-writing HDFS file may contains multiple committed blocks, as follows
> (assume one file contains three blocks):
> || ||Block 1||Block 2||Block 3||
> |Case 1|Complete|Commit|UnderConstruction|
> |Case 2|Complete|Commit|Commit|
> |Case 3|Commit|Commit|Commit|
>
> But the logic for committed blocks is mixed when computing file size, it
> ignores the bytes of the last committed block and contains the bytes of other
> committed blocks.
> {code:java}
> public final long computeFileSize(boolean includesLastUcBlock,
> boolean usePreferredBlockSize4LastUcBlock) {
> if (blocks.length == 0) {
> return 0;
> }
> final int last = blocks.length - 1;
> //check if the last block is BlockInfoUnderConstruction
> BlockInfo lastBlk = blocks[last];
> long size = lastBlk.getNumBytes();
> // the last committed block is not complete, so it's bytes may be ignored.
> if (!lastBlk.isComplete()) {
> if (!includesLastUcBlock) {
>size = 0;
> } else if (usePreferredBlockSize4LastUcBlock) {
>size = isStriped()?
>getPreferredBlockSize() *
>((BlockInfoStriped)lastBlk).getDataBlockNum() :
>getPreferredBlockSize();
> }
> }
> // The bytes of other committed blocks are calculated into the file length.
> for (int i = 0; i < last; i++) {
> size += blocks[i].getNumBytes();
> }
> return size;
> } {code}
> The bytes of one committed block will not be changed, so the bytes of the
> last committed block should be calculated into the file length too.
>
> And the logic for committed blocks is mixed too when computing file length in
> DFSInputStream. Normally DFSInputStream doesn't get visible length for
> committed block regardless of whether the committed block is the last block
> or not.
>
> HDFS-10843 noticed one bug which actually caused by the committed block, but
> HDFS-10843 fixed that bug in another way.
> The num of bytes of the committed block will no longer change, so we should
> update the quota usage when the block is committed, which can reduce the
> delta quota usage in time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17497) Logic for committed blocks is mixed when computing file size
[
https://issues.apache.org/jira/browse/HDFS-17497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-17497:
--
Labels: pull-request-available (was: )
> Logic for committed blocks is mixed when computing file size
>
>
> Key: HDFS-17497
> URL: https://issues.apache.org/jira/browse/HDFS-17497
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> One in-writing HDFS file may contains multiple committed blocks, as follows
> (assume one file contains three blocks):
> || ||Block 1||Block 2||Block 3||
> |Case 1|Complete|Commit|UnderConstruction|
> |Case 2|Complete|Commit|Commit|
> |Case 3|Commit|Commit|Commit|
>
> But the logic for committed blocks is mixed when computing file size, it
> ignores the bytes of the last committed block and contains the bytes of other
> committed blocks.
> {code:java}
> public final long computeFileSize(boolean includesLastUcBlock,
> boolean usePreferredBlockSize4LastUcBlock) {
> if (blocks.length == 0) {
> return 0;
> }
> final int last = blocks.length - 1;
> //check if the last block is BlockInfoUnderConstruction
> BlockInfo lastBlk = blocks[last];
> long size = lastBlk.getNumBytes();
> // the last committed block is not complete, so it's bytes may be ignored.
> if (!lastBlk.isComplete()) {
> if (!includesLastUcBlock) {
>size = 0;
> } else if (usePreferredBlockSize4LastUcBlock) {
>size = isStriped()?
>getPreferredBlockSize() *
>((BlockInfoStriped)lastBlk).getDataBlockNum() :
>getPreferredBlockSize();
> }
> }
> // The bytes of other committed blocks are calculated into the file length.
> for (int i = 0; i < last; i++) {
> size += blocks[i].getNumBytes();
> }
> return size;
> } {code}
> The bytes of one committed block will not be changed, so the bytes of the
> last committed block should be calculated into the file length too.
>
> And the logic for committed blocks is mixed too when computing file length in
> DFSInputStream. Normally DFSInputStream doesn't get visible length for
> committed block regardless of whether the committed block is the last block
> or not.
>
> HDFS-10843 noticed one bug which actually caused by the committed block, but
> HDFS-10843 fixed that bug in another way.
> The num of bytes of the committed block will no longer change, so we should
> update the quota usage when the block is committed, which can reduce the
> delta quota usage in time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17367) Add PercentUsed for Different StorageTypes in JMX
[ https://issues.apache.org/jira/browse/HDFS-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840093#comment-17840093 ] ASF GitHub Bot commented on HDFS-17367: --- zhtttylz commented on code in PR #6735: URL: https://github.com/apache/hadoop/pull/6735#discussion_r1576202576 ## hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md: ## @@ -326,6 +326,15 @@ Each metrics record contains tags such as HAState and Hostname as additional inf | `FSN(Read/Write)LockOverallNanosAvgTime` | Average time of holding the lock by all operations in nanoseconds | | `PendingSPSPaths` | The number of paths to be processed by storage policy satisfier | +BlockManager +- + +BlockManager metrics indicate each StorageType's usage in the cluster with detailed entries. Review Comment: Thank you very much for your suggestions; I will modify the code as soon as possible. > Add PercentUsed for Different StorageTypes in JMX > - > > Key: HDFS-17367 > URL: https://issues.apache.org/jira/browse/HDFS-17367 > Project: Hadoop HDFS > Issue Type: Improvement > Components: metrics, namenode >Affects Versions: 3.5.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > > Currently, the NameNode only displays PercentUsed for the entire cluster. We > plan to add corresponding PercentUsed metrics for different StorageTypes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840091#comment-17840091
]
ASF GitHub Bot commented on HDFS-17488:
---
hadoop-yetus commented on PR #6759:
URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2072209318
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 13s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 32s | | trunk passed |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 227m 24s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 385m 19s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6759 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 770ac906a345 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4b773c6a8b2014e2e21151e2d9163e871994912a |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/2/testReport/ |
| Max. process+thread count | 3559 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-17367) Add PercentUsed for Different StorageTypes in JMX
[ https://issues.apache.org/jira/browse/HDFS-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840083#comment-17840083 ] ASF GitHub Bot commented on HDFS-17367: --- haiyang1987 commented on code in PR #6735: URL: https://github.com/apache/hadoop/pull/6735#discussion_r1576147893 ## hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md: ## @@ -326,6 +326,15 @@ Each metrics record contains tags such as HAState and Hostname as additional inf | `FSN(Read/Write)LockOverallNanosAvgTime` | Average time of holding the lock by all operations in nanoseconds | | `PendingSPSPaths` | The number of paths to be processed by storage policy satisfier | +BlockManager +- + +BlockManager metrics indicate each StorageType's usage in the cluster with detailed entries. Review Comment: Maybe BlockManager will add other metrics in the future and we can update it to a more generic description? > Add PercentUsed for Different StorageTypes in JMX > - > > Key: HDFS-17367 > URL: https://issues.apache.org/jira/browse/HDFS-17367 > Project: Hadoop HDFS > Issue Type: Improvement > Components: metrics, namenode >Affects Versions: 3.5.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > > Currently, the NameNode only displays PercentUsed for the entire cluster. We > plan to add corresponding PercentUsed metrics for different StorageTypes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17475) Add a command to check if files are readable
[
https://issues.apache.org/jira/browse/HDFS-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840077#comment-17840077
]
ASF GitHub Bot commented on HDFS-17475:
---
hadoop-yetus commented on PR #6745:
URL: https://github.com/apache/hadoop/pull/6745#issuecomment-2072123314
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 01s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 110m 13s | | trunk passed |
| +1 :green_heart: | compile | 7m 56s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 189m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 52s | | the patch passed |
| +1 :green_heart: | compile | 4m 18s | | the patch passed |
| +1 :green_heart: | javac | 4m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 195m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 32s | | The patch does not
generate ASF License warnings. |
| | | 528m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6745 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | MINGW64_NT-10.0-17763 a4bf9fc4d5fd 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 23a2e2b6a31fec60a164742dc6df4344b2fcd5e4 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6745/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6745/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Add a command to check if files are readable
>
>
> Key: HDFS-17475
> URL: https://issues.apache.org/jira/browse/HDFS-17475
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Felix N
>Assignee: Felix N
>Priority: Minor
> Labels: pull-request-available
>
> Sometimes a job can fail due to one unreadable file down the line due to
> missing replicas or dead DNs or other reason. This command should allow users
> to check whether files are readable by checking for metadata on DNs without
> executing full read pipelines of the files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17451) RBF: fix spotbugs for redundant nullcheck of dns.
[ https://issues.apache.org/jira/browse/HDFS-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840069#comment-17840069 ] ASF GitHub Bot commented on HDFS-17451: --- haiyang1987 commented on PR #6697: URL: https://github.com/apache/hadoop/pull/6697#issuecomment-2072033364 Committed to trunk. Thanks @KeeProMise for your contributions! and @ZanderXu @simbadzina @ayushtkn @slfan1989 review! > RBF: fix spotbugs for redundant nullcheck of dns. > - > > Key: HDFS-17451 > URL: https://issues.apache.org/jira/browse/HDFS-17451 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > > h2. Dodgy code Warnings > ||Code||Warning|| > |RCN|Redundant nullcheck of dns, which is known to be non-null in > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType)| > | |[Bug type RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (click for > details)|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6655/8/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html#RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE] > In class org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer > In method > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) > Value loaded from dns > Return value of > org.apache.hadoop.thirdparty.com.google.common.cache.LoadingCache.get(Object) > of type Object > Redundant null check at RouterRpcServer.java:[line 1093]| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17451) RBF: fix spotbugs for redundant nullcheck of dns.
[ https://issues.apache.org/jira/browse/HDFS-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840068#comment-17840068 ] ASF GitHub Bot commented on HDFS-17451: --- haiyang1987 merged PR #6697: URL: https://github.com/apache/hadoop/pull/6697 > RBF: fix spotbugs for redundant nullcheck of dns. > - > > Key: HDFS-17451 > URL: https://issues.apache.org/jira/browse/HDFS-17451 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > > h2. Dodgy code Warnings > ||Code||Warning|| > |RCN|Redundant nullcheck of dns, which is known to be non-null in > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType)| > | |[Bug type RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (click for > details)|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6655/8/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html#RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE] > In class org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer > In method > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) > Value loaded from dns > Return value of > org.apache.hadoop.thirdparty.com.google.common.cache.LoadingCache.get(Object) > of type Object > Redundant null check at RouterRpcServer.java:[line 1093]| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17451) RBF: fix spotbugs for redundant nullcheck of dns.
[ https://issues.apache.org/jira/browse/HDFS-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840066#comment-17840066 ] ASF GitHub Bot commented on HDFS-17451: --- KeeProMise commented on PR #6697: URL: https://github.com/apache/hadoop/pull/6697#issuecomment-2072025405 @ZanderXu @haiyang1987 thanks for you review. > RBF: fix spotbugs for redundant nullcheck of dns. > - > > Key: HDFS-17451 > URL: https://issues.apache.org/jira/browse/HDFS-17451 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > > h2. Dodgy code Warnings > ||Code||Warning|| > |RCN|Redundant nullcheck of dns, which is known to be non-null in > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType)| > | |[Bug type RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (click for > details)|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6655/8/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html#RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE] > In class org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer > In method > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) > Value loaded from dns > Return value of > org.apache.hadoop.thirdparty.com.google.common.cache.LoadingCache.get(Object) > of type Object > Redundant null check at RouterRpcServer.java:[line 1093]| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[ https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840057#comment-17840057 ] ASF GitHub Bot commented on HDFS-17384: --- ZanderXu commented on PR #6762: URL: https://github.com/apache/hadoop/pull/6762#issuecomment-2071967284 The spotbugs will be fixed by HDFS-17451. And the failed UT is not imported by this PR. > [FGL] Replace the global lock with global FS Lock and global BM lock > > > Key: HDFS-17384 > URL: https://issues.apache.org/jira/browse/HDFS-17384 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: FGL, pull-request-available > > First, we can replace the current global lock with two locks, global FS lock > and global BM lock. > The global FS lock is used to make directory tree-related operations > thread-safe. > The global BM lock is used to make block-related operations and DN-related > operations thread-safe. > > For some operations involving both directory tree and block or DN, the global > FS lock and the global BM lock are acquired. > > The lock order should be: > * The global FS lock > * The global BM lock > > There are some special requirements for this ticket. > * End-user can choose to use global lock or fine-grained lock through > configuration. > * Try not to modify the current implementation logic as much as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[
https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840054#comment-17840054
]
ASF GitHub Bot commented on HDFS-17384:
---
hadoop-yetus commented on PR #6762:
URL: https://github.com/apache/hadoop/pull/6762#issuecomment-2071959089
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 47 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 14s | | trunk passed |
| +1 :green_heart: | compile | 17m 33s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 16m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 0s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| -1 :x: | spotbugs | 1m 34s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/1/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-rbf in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 34m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 3s | | the patch passed |
| +1 :green_heart: | compile | 16m 51s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 16m 51s | | the patch passed |
| +1 :green_heart: | compile | 16m 21s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 16m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 36s | | root: The patch generated
0 new + 1232 unchanged - 16 fixed = 1232 total (was 1248) |
| +1 :green_heart: | mvnsite | 3m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 3m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 6m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 237m 5s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 29m 40s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | unit | 0m 50s | | hadoop-fs2img in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 9s | | The patch does not
generate ASF License warnings. |
| | | 506m 7s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6762 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell
[jira] [Commented] (HDFS-17476) fix: False positive "Observer Node is too far behind" due to long overflow.
[
https://issues.apache.org/jira/browse/HDFS-17476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840026#comment-17840026
]
ASF GitHub Bot commented on HDFS-17476:
---
hadoop-yetus commented on PR #6747:
URL: https://github.com/apache/hadoop/pull/6747#issuecomment-2071793284
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 91m 28s | | trunk passed |
| +1 :green_heart: | compile | 6m 17s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 00s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 152m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 48s | | the patch passed |
| +1 :green_heart: | compile | 3m 43s | | the patch passed |
| +1 :green_heart: | javac | 3m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 162m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 46s | | The patch does not
generate ASF License warnings. |
| | | 434m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6747 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 cf76230213b6 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 4208a984b11713148b6b6eba9f898d3d16c41fdd |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6747/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6747/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> fix: False positive "Observer Node is too far behind" due to long overflow.
> ---
>
> Key: HDFS-17476
> URL: https://issues.apache.org/jira/browse/HDFS-17476
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Critical
> Labels: pull-request-available
> Attachments: HDFS-17476.patch, image-2024-04-18-10-57-10-481.png
>
>
> In the code GlobalStateIdContext#receiveRequestState(), if clientStateId is a
> small negative number, clientStateId-serverStateId may be greater than
> （ESTIMATED_TRANSACTIONS_PER_SECOND due to overflow
> * TimeUnit.MILLISECONDS.toSeconds(clientWaitTime)
> * ESTIMATED_SERVER_TIME_MULTIPLIER）,
> resulting in false positives that Observer Node is too far behind.
> !image-2024-04-18-10-57-10-481.png|width=742,height=110!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17484) Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually
[
https://issues.apache.org/jira/browse/HDFS-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840018#comment-17840018
]
ASF GitHub Bot commented on HDFS-17484:
---
hadoop-yetus commented on PR #6758:
URL: https://github.com/apache/hadoop/pull/6758#issuecomment-2071762108
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 0s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 1s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/4/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 251 unchanged
- 0 fixed = 253 total (was 251) |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 227m 39s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 369m 17s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6758 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 03c58b1c9a42 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5
20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9cceb198d62d185f49bfd65ce5641c39035f5188 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
[jira] [Commented] (HDFS-17496) DataNode supports more fine-grained dataset lock based on blockid
[
https://issues.apache.org/jira/browse/HDFS-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840014#comment-17840014
]
ASF GitHub Bot commented on HDFS-17496:
---
hfutatzhanghb opened a new pull request, #6764:
URL: https://github.com/apache/hadoop/pull/6764
### Description of PR
### How was this patch tested?
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> DataNode supports more fine-grained dataset lock based on blockid
> -
>
> Key: HDFS-17496
> URL: https://issues.apache.org/jira/browse/HDFS-17496
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Attachments: image-2024-04-23-16-17-07-057.png
>
>
> Recently, we used NvmeSSD as volumes in datanodes and performed some stress
> tests.
> We found that NvmeSSD and HDD disks achieve similar performance when create
> lots of small files, such as 10KB.
> This phenomenon is counterintuitive. After analyzing the metric monitoring ,
> we found that fsdataset lock became the bottleneck in high concurrency
> scenario.
>
> Currently, we have two level locks which are BLOCK_POOL and VOLUME. We can
> further split the volume lock to DIR lock.
> DIR lock is defined as below： given a blockid, we can determine which subdir
> this block will be placed in finalized dir. We just use
> subdir[0-31]/subdir[0-31] as the
> name of DIR lock.
> More details, please refer to method DatanodeUtil#idToBlockDir：
> {code:java}
> public static File idToBlockDir(File root, long blockId) {
> int d1 = (int) ((blockId >> 16) & 0x1F);
> int d2 = (int) ((blockId >> 8) & 0x1F);
> String path = DataStorage.BLOCK_SUBDIR_PREFIX + d1 + SEP +
> DataStorage.BLOCK_SUBDIR_PREFIX + d2;
> return new File(root, path);
> } {code}
> The performance comparison is as below:
> experimental setup:
> 3 DataNodes with single disk.
> 10 Cients concurrent write and delete files after writing.
> 550 threads per Client.
> !image-2024-04-23-16-17-07-057.png!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17496) DataNode supports more fine-grained dataset lock based on blockid
[
https://issues.apache.org/jira/browse/HDFS-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-17496:
--
Labels: pull-request-available (was: )
> DataNode supports more fine-grained dataset lock based on blockid
> -
>
> Key: HDFS-17496
> URL: https://issues.apache.org/jira/browse/HDFS-17496
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-04-23-16-17-07-057.png
>
>
> Recently, we used NvmeSSD as volumes in datanodes and performed some stress
> tests.
> We found that NvmeSSD and HDD disks achieve similar performance when create
> lots of small files, such as 10KB.
> This phenomenon is counterintuitive. After analyzing the metric monitoring ,
> we found that fsdataset lock became the bottleneck in high concurrency
> scenario.
>
> Currently, we have two level locks which are BLOCK_POOL and VOLUME. We can
> further split the volume lock to DIR lock.
> DIR lock is defined as below： given a blockid, we can determine which subdir
> this block will be placed in finalized dir. We just use
> subdir[0-31]/subdir[0-31] as the
> name of DIR lock.
> More details, please refer to method DatanodeUtil#idToBlockDir：
> {code:java}
> public static File idToBlockDir(File root, long blockId) {
> int d1 = (int) ((blockId >> 16) & 0x1F);
> int d2 = (int) ((blockId >> 8) & 0x1F);
> String path = DataStorage.BLOCK_SUBDIR_PREFIX + d1 + SEP +
> DataStorage.BLOCK_SUBDIR_PREFIX + d2;
> return new File(root, path);
> } {code}
> The performance comparison is as below:
> experimental setup:
> 3 DataNodes with single disk.
> 10 Cients concurrent write and delete files after writing.
> 550 threads per Client.
> !image-2024-04-23-16-17-07-057.png!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840010#comment-17840010
]
ASF GitHub Bot commented on HDFS-17488:
---
kokonguyen191 commented on code in PR #6759:
URL: https://github.com/apache/hadoop/pull/6759#discussion_r1575868360
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/IncrementalBlockReportManager.java:
##
@@ -304,4 +310,4 @@ void clearIBRs() {
int getPendingIBRSize() {
return pendingIBRs.size();
}
-}
\ No newline at end of file
+}
Review Comment:
Done. Actually took me quite a while to find the setting to disable it in IDE
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:
##
@@ -2745,14 +2745,18 @@ public void checkAndUpdate(String bpid, ScanInfo
scanInfo)
curDirScannerNotifyCount = 0;
lastDirScannerNotifyTime = startTimeMs;
}
-try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.VOLUME, bpid,
-vol.getStorageID())) {
+String storageUuid = vol.getStorageID();
+try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.VOLUME,
bpid, storageUuid)) {
memBlockInfo = volumeMap.get(bpid, blockId);
if (memBlockInfo != null &&
memBlockInfo.getState() != ReplicaState.FINALIZED) {
// Block is not finalized - ignore the difference
return;
}
+ if (!storageMap.containsKey(storageUuid)) {
Review Comment:
Moved the block up
> DN can fail IBRs with NPE when a volume is removed
> --
>
> Key: HDFS-17488
> URL: https://issues.apache.org/jira/browse/HDFS-17488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Felix N
>Assignee: Felix N
>Priority: Major
> Labels: pull-request-available
>
>
> Error logs
> {code:java}
> 2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830
> heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode
> (BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool
> BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid
> 1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246)
> at
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920)
> at java.lang.Thread.run(Thread.java:748) {code}
> The root cause is in BPOfferService#notifyNamenodeBlock, happens when it's
> called on a block belonging to a volume already removed prior. Because the
> volume was already removed
>
> {code:java}
> private void notifyNamenodeBlock(ExtendedBlock block, BlockStatus status,
> String delHint, String storageUuid, boolean isOnTransientStorage) {
> checkBlock(block);
> final ReceivedDeletedBlockInfo info = new ReceivedDeletedBlockInfo(
> block.getLocalBlock(), status, delHint);
> final DatanodeStorage storage = dn.getFSDataset().getStorage(storageUuid);
>
> // storage == null here because it's already removed earlier.
> for (BPServiceActor actor : bpServices) {
> actor.getIbrManager().notifyNamenodeBlock(info, storage,
> isOnTransientStorage);
> }
> } {code}
> so IBRs with a null storage are now pending.
> The reason why notifyNamenodeBlock can trigger on such blocks is up in
> DirectoryScanner#reconcile
> {code:java}
> public void reconcile() throws IOException {
> LOG.debug("reconcile start DirectoryScanning");
> scan();
> // If a volume is removed here after scan() already finished running,
> // diffs is stale and checkAndUpdate will run on a removed volume
> // HDFS-14476: run checkAndUpdate with batch to avoid holding the lock too
> // long
> int loopCount = 0;
> synchronized (diffs) {
> for (final Map.Entry entry : diffs.getEntries()) {
> dataset.checkAndUpdate(entry.getKey(), entry.getValue());
> ...
> } {code}
> Inside checkAndUpdate, memBlockInfo is null because all the block meta in
> memory is removed during the volume removal, but diskFile still exists. Then
> DataNode#notifyNamenodeDeletedBlock (and further down the line,
> notifyNamenodeBlock) is called on this block.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840008#comment-17840008
]
ASF GitHub Bot commented on HDFS-17488:
---
kokonguyen191 commented on code in PR #6759:
URL: https://github.com/apache/hadoop/pull/6759#discussion_r1575867553
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/IncrementalBlockReportManager.java:
##
@@ -169,11 +169,17 @@ private synchronized StorageReceivedDeletedBlocks[]
generateIBRs() {
: pendingIBRs.entrySet()) {
final PerStorageIBR perStorage = entry.getValue();
-// Send newly-received and deleted blockids to namenode
+ // Send newly-received and deleted blockids to namenode
final ReceivedDeletedBlockInfo[] rdbi = perStorage.removeAll();
- if (rdbi != null) {
-reports.add(new StorageReceivedDeletedBlocks(entry.getKey(), rdbi));
+ if (rdbi == null) {
Review Comment:
I intentionally put it like this to reduce nesting and make the logic a bit
easier to read
> DN can fail IBRs with NPE when a volume is removed
> --
>
> Key: HDFS-17488
> URL: https://issues.apache.org/jira/browse/HDFS-17488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Felix N
>Assignee: Felix N
>Priority: Major
> Labels: pull-request-available
>
>
> Error logs
> {code:java}
> 2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830
> heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode
> (BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool
> BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid
> 1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246)
> at
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920)
> at java.lang.Thread.run(Thread.java:748) {code}
> The root cause is in BPOfferService#notifyNamenodeBlock, happens when it's
> called on a block belonging to a volume already removed prior. Because the
> volume was already removed
>
> {code:java}
> private void notifyNamenodeBlock(ExtendedBlock block, BlockStatus status,
> String delHint, String storageUuid, boolean isOnTransientStorage) {
> checkBlock(block);
> final ReceivedDeletedBlockInfo info = new ReceivedDeletedBlockInfo(
> block.getLocalBlock(), status, delHint);
> final DatanodeStorage storage = dn.getFSDataset().getStorage(storageUuid);
>
> // storage == null here because it's already removed earlier.
> for (BPServiceActor actor : bpServices) {
> actor.getIbrManager().notifyNamenodeBlock(info, storage,
> isOnTransientStorage);
> }
> } {code}
> so IBRs with a null storage are now pending.
> The reason why notifyNamenodeBlock can trigger on such blocks is up in
> DirectoryScanner#reconcile
> {code:java}
> public void reconcile() throws IOException {
> LOG.debug("reconcile start DirectoryScanning");
> scan();
> // If a volume is removed here after scan() already finished running,
> // diffs is stale and checkAndUpdate will run on a removed volume
> // HDFS-14476: run checkAndUpdate with batch to avoid holding the lock too
> // long
> int loopCount = 0;
> synchronized (diffs) {
> for (final Map.Entry entry : diffs.getEntries()) {
> dataset.checkAndUpdate(entry.getKey(), entry.getValue());
> ...
> } {code}
> Inside checkAndUpdate, memBlockInfo is null because all the block meta in
> memory is removed during the volume removal, but diskFile still exists. Then
> DataNode#notifyNamenodeDeletedBlock (and further down the line,
> notifyNamenodeBlock) is called on this block.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839996#comment-17839996
]
ASF GitHub Bot commented on HDFS-17488:
---
ZanderXu commented on code in PR #6759:
URL: https://github.com/apache/hadoop/pull/6759#discussion_r1575829376
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/IncrementalBlockReportManager.java:
##
@@ -169,11 +169,17 @@ private synchronized StorageReceivedDeletedBlocks[]
generateIBRs() {
: pendingIBRs.entrySet()) {
final PerStorageIBR perStorage = entry.getValue();
-// Send newly-received and deleted blockids to namenode
+ // Send newly-received and deleted blockids to namenode
final ReceivedDeletedBlockInfo[] rdbi = perStorage.removeAll();
- if (rdbi != null) {
-reports.add(new StorageReceivedDeletedBlocks(entry.getKey(), rdbi));
+ if (rdbi == null) {
Review Comment:
```
if (rdbi != null) {
// Null storage, should not happen
if (entry.getKey() == null) {
dnMetrics.incrNullStorageBlockReports();
continue;
}
reports.add(new StorageReceivedDeletedBlocks(entry.getKey(), rdbi));
}
```
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:
##
@@ -2745,14 +2745,18 @@ public void checkAndUpdate(String bpid, ScanInfo
scanInfo)
curDirScannerNotifyCount = 0;
lastDirScannerNotifyTime = startTimeMs;
}
-try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.VOLUME, bpid,
-vol.getStorageID())) {
+String storageUuid = vol.getStorageID();
+try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.VOLUME,
bpid, storageUuid)) {
memBlockInfo = volumeMap.get(bpid, blockId);
if (memBlockInfo != null &&
memBlockInfo.getState() != ReplicaState.FINALIZED) {
// Block is not finalized - ignore the difference
return;
}
+ if (!storageMap.containsKey(storageUuid)) {
Review Comment:
How about moving this logic to line 2750？
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/IncrementalBlockReportManager.java:
##
@@ -304,4 +310,4 @@ void clearIBRs() {
int getPendingIBRSize() {
return pendingIBRs.size();
}
-}
\ No newline at end of file
+}
Review Comment:
can rollback this change
> DN can fail IBRs with NPE when a volume is removed
> --
>
> Key: HDFS-17488
> URL: https://issues.apache.org/jira/browse/HDFS-17488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Felix N
>Assignee: Felix N
>Priority: Major
> Labels: pull-request-available
>
>
> Error logs
> {code:java}
> 2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830
> heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode
> (BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool
> BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid
> 1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246)
> at
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920)
> at java.lang.Thread.run(Thread.java:748) {code}
> The root cause is in BPOfferService#notifyNamenodeBlock, happens when it's
> called on a block belonging to a volume already removed prior. Because the
> volume was already removed
>
> {code:java}
> private void notifyNamenodeBlock(ExtendedBlock block, BlockStatus status,
> String delHint, String storageUuid, boolean isOnTransientStorage) {
> checkBlock(block);
> final ReceivedDeletedBlockInfo info = new ReceivedDeletedBlockInfo(
> block.getLocalBlock(), status, delHint);
> final DatanodeStorage storage = dn.getFSDataset().getStorage(storageUuid);
>
> // storage == null here because it's already removed earlier.
> for (BPServiceActor actor : bpServices) {
> actor.getIbrManager().notifyNamenodeBlock(info, storage,
> isOnTransientStorage);
> }
> } {code}
> so IBRs with a null storage are now pending.
> The reason why notifyNamenodeBlock can trigger on such blocks is up in
> DirectoryScanner#reconcile
> {code:java}
> public void reconcile() throws IOException {
>
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839957#comment-17839957
]
ASF GitHub Bot commented on HDFS-17488:
---
kokonguyen191 commented on PR #6759:
URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2071495428
Hi @ZanderXu @Hexiaoqiao, can you help me take a look whenever you're free?
Thanks!
> DN can fail IBRs with NPE when a volume is removed
> --
>
> Key: HDFS-17488
> URL: https://issues.apache.org/jira/browse/HDFS-17488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Felix N
>Assignee: Felix N
>Priority: Major
> Labels: pull-request-available
>
>
> Error logs
> {code:java}
> 2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830
> heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode
> (BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool
> BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid
> 1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246)
> at
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920)
> at java.lang.Thread.run(Thread.java:748) {code}
> The root cause is in BPOfferService#notifyNamenodeBlock, happens when it's
> called on a block belonging to a volume already removed prior. Because the
> volume was already removed
>
> {code:java}
> private void notifyNamenodeBlock(ExtendedBlock block, BlockStatus status,
> String delHint, String storageUuid, boolean isOnTransientStorage) {
> checkBlock(block);
> final ReceivedDeletedBlockInfo info = new ReceivedDeletedBlockInfo(
> block.getLocalBlock(), status, delHint);
> final DatanodeStorage storage = dn.getFSDataset().getStorage(storageUuid);
>
> // storage == null here because it's already removed earlier.
> for (BPServiceActor actor : bpServices) {
> actor.getIbrManager().notifyNamenodeBlock(info, storage,
> isOnTransientStorage);
> }
> } {code}
> so IBRs with a null storage are now pending.
> The reason why notifyNamenodeBlock can trigger on such blocks is up in
> DirectoryScanner#reconcile
> {code:java}
> public void reconcile() throws IOException {
> LOG.debug("reconcile start DirectoryScanning");
> scan();
> // If a volume is removed here after scan() already finished running,
> // diffs is stale and checkAndUpdate will run on a removed volume
> // HDFS-14476: run checkAndUpdate with batch to avoid holding the lock too
> // long
> int loopCount = 0;
> synchronized (diffs) {
> for (final Map.Entry entry : diffs.getEntries()) {
> dataset.checkAndUpdate(entry.getKey(), entry.getValue());
> ...
> } {code}
> Inside checkAndUpdate, memBlockInfo is null because all the block meta in
> memory is removed during the volume removal, but diskFile still exists. Then
> DataNode#notifyNamenodeDeletedBlock (and further down the line,
> notifyNamenodeBlock) is called on this block.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839929#comment-17839929
]
ASF GitHub Bot commented on HDFS-17488:
---
hadoop-yetus commented on PR #6759:
URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2071375244
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 118m 49s | | trunk passed |
| +1 :green_heart: | compile | 8m 15s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 190m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 57s | | the patch passed |
| +1 :green_heart: | compile | 4m 54s | | the patch passed |
| +1 :green_heart: | javac | 4m 54s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 25s | | the patch passed |
| +1 :green_heart: | mvnsite | 6m 07s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 207m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 9m 28s | | The patch does not
generate ASF License warnings. |
| | | 557m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6759 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 9dbcaf4dc965 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / d5a87e15db1c70fb0c3974832c75504ff184b0d1 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6759/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6759/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> DN can fail IBRs with NPE when a volume is removed
> --
>
> Key: HDFS-17488
> URL: https://issues.apache.org/jira/browse/HDFS-17488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Felix N
>Assignee: Felix N
>Priority: Major
> Labels: pull-request-available
>
>
> Error logs
> {code:java}
> 2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830
> heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode
> (BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool
> BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid
> 1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246)
> at
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920)
> at java.lang.Thread.run(Thread.java:748) {code}
> The root cause is in BPOfferService#notifyNamenodeBlock, happens when it's
>
[jira] [Commented] (HDFS-17484) Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually
[ https://issues.apache.org/jira/browse/HDFS-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839919#comment-17839919 ] ASF GitHub Bot commented on HDFS-17484: --- hfutatzhanghb commented on PR #6758: URL: https://github.com/apache/hadoop/pull/6758#issuecomment-2071305686 @Hexiaoqiao @zhangshuyan0 @haiyang1987 @tomscut Sir, could you please help me review this PR when you have free time? Thanks ahead. > Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when > they are not busy actually > - > > Key: HDFS-17484 > URL: https://issues.apache.org/jira/browse/HDFS-17484 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > > Currently, we have `dfs.namenode.redundancy.considerLoad` equals true by > default, and > dfs.namenode.redundancy.considerLoad.factor equals 2.0 by default. > Think about below situation. when we are doing stress test, we may deploy > hdfs client onto the datanode. So, this hdfs client will prefer to write to > its local datanode and increase this machine's load. Suppose we have 3 > datanodes, the load of them are as below: 5.0, 0.2, 0.3. > > The load equals to 5.0 will be excluded when choose datanodes for a block. > But actually, it is not slow node when load equals to 5.0 for a machine with > 80 cpu cores. > > So, we should better add a new configuration entry : > `dfs.namenode.redundancy.considerLoad.minLoad` to indicate the mininum factor > we will make considerLoad take effect. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[ https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839915#comment-17839915 ] ASF GitHub Bot commented on HDFS-17384: --- ZanderXu opened a new pull request, #6762: URL: https://github.com/apache/hadoop/pull/6762 We plan to merge HDFS-17384 to the trunk branch. This PR is used to review all changes in HDFS-17384. > [FGL] Replace the global lock with global FS Lock and global BM lock > > > Key: HDFS-17384 > URL: https://issues.apache.org/jira/browse/HDFS-17384 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: FGL > > First, we can replace the current global lock with two locks, global FS lock > and global BM lock. > The global FS lock is used to make directory tree-related operations > thread-safe. > The global BM lock is used to make block-related operations and DN-related > operations thread-safe. > > For some operations involving both directory tree and block or DN, the global > FS lock and the global BM lock are acquired. > > The lock order should be: > * The global FS lock > * The global BM lock > > There are some special requirements for this ticket. > * End-user can choose to use global lock or fine-grained lock through > configuration. > * Try not to modify the current implementation logic as much as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[ https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17384: -- Labels: FGL pull-request-available (was: FGL) > [FGL] Replace the global lock with global FS Lock and global BM lock > > > Key: HDFS-17384 > URL: https://issues.apache.org/jira/browse/HDFS-17384 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: FGL, pull-request-available > > First, we can replace the current global lock with two locks, global FS lock > and global BM lock. > The global FS lock is used to make directory tree-related operations > thread-safe. > The global BM lock is used to make block-related operations and DN-related > operations thread-safe. > > For some operations involving both directory tree and block or DN, the global > FS lock and the global BM lock are acquired. > > The lock order should be: > * The global FS lock > * The global BM lock > > There are some special requirements for this ticket. > * End-user can choose to use global lock or fine-grained lock through > configuration. > * Try not to modify the current implementation logic as much as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17484) Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually
[
https://issues.apache.org/jira/browse/HDFS-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839911#comment-17839911
]
ASF GitHub Bot commented on HDFS-17484:
---
hadoop-yetus commented on PR #6758:
URL: https://github.com/apache/hadoop/pull/6758#issuecomment-2071256591
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 00s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 89m 50s | | trunk passed |
| +1 :green_heart: | compile | 5m 55s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 148m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 49s | | the patch passed |
| +1 :green_heart: | compile | 3m 26s | | the patch passed |
| +1 :green_heart: | javac | 3m 26s | | the patch passed |
| -1 :x: | blanks | 0m 00s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6758/1/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 2m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 03s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 158m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 29s | | The patch does not
generate ASF License warnings. |
| | | 423m 09s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6758 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | MINGW64_NT-10.0-17763 be352c545e0b 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / b01d3bcf66844d7f7f66c04a496d8dcc39cddae6 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6758/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6758/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when
> they are not busy actually
> -
>
> Key: HDFS-17484
> URL: https://issues.apache.org/jira/browse/HDFS-17484
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.4.0
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Minor
> Labels: pull-request-available
>
> Currently, we have `dfs.namenode.redundancy.considerLoad` equals true by
> default, and
> dfs.namenode.redundancy.considerLoad.factor equals 2.0 by default.
> Think about below situation. when we are doing stress test, we may deploy
> hdfs client onto the datanode. So, this hdfs client will prefer to write to
> its local datanode and increase this machine's load. Suppose we have 3
> datanodes, the load of them are as below: 5.0, 0.2, 0.3.
>
> The load equals to 5.0 will be excluded when choose datanodes for a block.
> But actually, it is not slow node when load equals to 5.0 for a machine
[jira] [Commented] (HDFS-17457) [FGL] UTs support fine-grained locking
[ https://issues.apache.org/jira/browse/HDFS-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839909#comment-17839909 ] ASF GitHub Bot commented on HDFS-17457: --- ferhui commented on PR #6741: URL: https://github.com/apache/hadoop/pull/6741#issuecomment-2071256036 @ZanderXu Thanks for contribution. @kokonguyen191 Thanks for review. Merged. > [FGL] UTs support fine-grained locking > -- > > Key: HDFS-17457 > URL: https://issues.apache.org/jira/browse/HDFS-17457 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > [FGL] UTs support fine-grained locking -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17457) [FGL] UTs support fine-grained locking
[ https://issues.apache.org/jira/browse/HDFS-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839908#comment-17839908 ] ASF GitHub Bot commented on HDFS-17457: --- ferhui merged PR #6741: URL: https://github.com/apache/hadoop/pull/6741 > [FGL] UTs support fine-grained locking > -- > > Key: HDFS-17457 > URL: https://issues.apache.org/jira/browse/HDFS-17457 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > [FGL] UTs support fine-grained locking -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17457) [FGL] UTs support fine-grained locking
[
https://issues.apache.org/jira/browse/HDFS-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839793#comment-17839793
]
ASF GitHub Bot commented on HDFS-17457:
---
hadoop-yetus commented on PR #6741:
URL: https://github.com/apache/hadoop/pull/6741#issuecomment-2070306218
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 40 new or modified test files. |
_ HDFS-17384 Compile Tests _ |
| +0 :ok: | mvndep | 13m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 48s | | HDFS-17384 passed |
| +1 :green_heart: | compile | 8m 58s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 8m 24s | | HDFS-17384 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 2m 8s | | HDFS-17384 passed |
| +1 :green_heart: | mvnsite | 2m 3s | | HDFS-17384 passed |
| +1 :green_heart: | javadoc | 1m 56s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 16s | | HDFS-17384 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| -1 :x: | spotbugs | 0m 54s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6741/4/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-rbf in HDFS-17384 has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 20m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 8m 26s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 8m 26s | | the patch passed |
| +1 :green_heart: | compile | 8m 11s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 8m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 6s | | root: The patch generated
0 new + 356 unchanged - 1 fixed = 356 total (was 357) |
| +1 :green_heart: | mvnsite | 2m 0s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 11s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 198m 18s | | hadoop-hdfs in the patch
passed. |
| -1 :x: | unit | 29m 28s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6741/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | unit | 0m 39s | | hadoop-fs2img in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 375m 58s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination |
| | hadoop.hdfs.server.federation.router.TestRouterRpc |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6741/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6741 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux
[jira] [Commented] (HDFS-17484) Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually
[
https://issues.apache.org/jira/browse/HDFS-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839734#comment-17839734
]
ASF GitHub Bot commented on HDFS-17484:
---
hadoop-yetus commented on PR #6758:
URL: https://github.com/apache/hadoop/pull/6758#issuecomment-2069821953
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 8s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/3/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 2s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 252 unchanged
- 0 fixed = 254 total (was 252) |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 277m 15s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 437m 8s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6758 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux fafa9f8a14b5 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b01d3bcf66844d7f7f66c04a496d8dcc39cddae6 |
| Default
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839723#comment-17839723
]
ASF GitHub Bot commented on HDFS-17488:
---
hadoop-yetus commented on PR #6759:
URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2069730206
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 25s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 20s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 1s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 230 unchanged
- 0 fixed = 231 total (was 230) |
| +1 :green_heart: | mvnsite | 1m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 235m 27s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 388m 51s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6759/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6759 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 34352d4fc452 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d5a87e15db1c70fb0c3974832c75504ff184b0d1 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
[jira] [Commented] (HDFS-17484) Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually
[
https://issues.apache.org/jira/browse/HDFS-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839680#comment-17839680
]
ASF GitHub Bot commented on HDFS-17484:
---
hadoop-yetus commented on PR #6758:
URL: https://github.com/apache/hadoop/pull/6758#issuecomment-2069624972
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 17s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 47s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 2s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 251 unchanged
- 0 fixed = 253 total (was 251) |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 246m 54s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 404m 20s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.server.datanode.TestDataNodeReconfiguration |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6758 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 4e072519eda8 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5
20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Commented] (HDFS-17484) Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually
[
https://issues.apache.org/jira/browse/HDFS-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839679#comment-17839679
]
ASF GitHub Bot commented on HDFS-17484:
---
hadoop-yetus commented on PR #6758:
URL: https://github.com/apache/hadoop/pull/6758#issuecomment-2069616190
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 5m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 17s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 1s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 251 unchanged
- 0 fixed = 253 total (was 251) |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 244m 35s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 395m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6758/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6758 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux c90e17858d0b 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5
20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
[jira] [Commented] (HDFS-17451) RBF: fix spotbugs for redundant nullcheck of dns.
[ https://issues.apache.org/jira/browse/HDFS-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839611#comment-17839611 ] ASF GitHub Bot commented on HDFS-17451: --- ZanderXu commented on PR #6697: URL: https://github.com/apache/hadoop/pull/6697#issuecomment-2069092826 As I described in HDFS-17485, this spotbug should be fixed. If there is no input type DNs, `dnCache` will return an empty result, not null. So I thinks this modification is ok. > RBF: fix spotbugs for redundant nullcheck of dns. > - > > Key: HDFS-17451 > URL: https://issues.apache.org/jira/browse/HDFS-17451 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > > h2. Dodgy code Warnings > ||Code||Warning|| > |RCN|Redundant nullcheck of dns, which is known to be non-null in > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType)| > | |[Bug type RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (click for > details)|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6655/8/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html#RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE] > In class org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer > In method > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) > Value loaded from dns > Return value of > org.apache.hadoop.thirdparty.com.google.common.cache.LoadingCache.get(Object) > of type Object > Redundant null check at RouterRpcServer.java:[line 1093]| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17485) Fix SpotBug in RouterRpcServer.java
[ https://issues.apache.org/jira/browse/HDFS-17485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839608#comment-17839608 ] ASF GitHub Bot commented on HDFS-17485: --- ZanderXu closed pull request #6757: HDFS-17485. Remove redundant nullcheck in RouterRpcServer.java URL: https://github.com/apache/hadoop/pull/6757 > Fix SpotBug in RouterRpcServer.java > --- > > Key: HDFS-17485 > URL: https://issues.apache.org/jira/browse/HDFS-17485 > Project: Hadoop HDFS > Issue Type: Task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > Attachments: image-2024-04-22-15-02-33-725.png > > > !image-2024-04-22-15-02-33-725.png|width=1566,height=265! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17485) Fix SpotBug in RouterRpcServer.java
[
https://issues.apache.org/jira/browse/HDFS-17485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839596#comment-17839596
]
ASF GitHub Bot commented on HDFS-17485:
---
hadoop-yetus commented on PR #6757:
URL: https://github.com/apache/hadoop/pull/6757#issuecomment-2069029513
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 19m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| -1 :x: | spotbugs | 1m 22s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6757/1/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-rbf in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 39m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 22s | |
hadoop-hdfs-project/hadoop-hdfs-rbf generated 0 new + 0 unchanged - 1 fixed = 0
total (was 1) |
| +1 :green_heart: | shadedclient | 39m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 31m 38s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 195m 47s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6757/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6757 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f89a44debf13 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a270576ad84fb985927883e82f2da12f64114fe2 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
[jira] [Commented] (HDFS-17457) [FGL] UTs support fine-grained locking
[ https://issues.apache.org/jira/browse/HDFS-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839580#comment-17839580 ] ASF GitHub Bot commented on HDFS-17457: --- kokonguyen191 commented on code in PR #6741: URL: https://github.com/apache/hadoop/pull/6741#discussion_r1574451782 ## hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListOpenFiles.java: ## @@ -48,6 +48,7 @@ import org.apache.hadoop.hdfs.protocol.OpenFilesIterator; import org.apache.hadoop.hdfs.protocol.OpenFilesIterator.OpenFilesType; import org.apache.hadoop.hdfs.server.namenode.ha.HATestUtil; +import org.apache.hadoop.hdfs.server.namenode.fgl.FSNamesystemLockMode; Review Comment: `import org.apache.hadoop.hdfs.server.namenode.fgl.FSNamesystemLockMode;` should be above `import org.apache.hadoop.hdfs.server.namenode.ha.HATestUtil;`. Probably better to set up the import order in IDE and let the auto formatter handle it. > [FGL] UTs support fine-grained locking > -- > > Key: HDFS-17457 > URL: https://issues.apache.org/jira/browse/HDFS-17457 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > [FGL] UTs support fine-grained locking -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-17488:
--
Labels: pull-request-available (was: )
> DN can fail IBRs with NPE when a volume is removed
> --
>
> Key: HDFS-17488
> URL: https://issues.apache.org/jira/browse/HDFS-17488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Felix N
>Assignee: Felix N
>Priority: Major
> Labels: pull-request-available
>
>
> Error logs
> {code:java}
> 2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830
> heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode
> (BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool
> BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid
> 1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246)
> at
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920)
> at java.lang.Thread.run(Thread.java:748) {code}
> The root cause is in BPOfferService#notifyNamenodeBlock, happens when it's
> called on a block belonging to a volume already removed prior. Because the
> volume was already removed
>
> {code:java}
> private void notifyNamenodeBlock(ExtendedBlock block, BlockStatus status,
> String delHint, String storageUuid, boolean isOnTransientStorage) {
> checkBlock(block);
> final ReceivedDeletedBlockInfo info = new ReceivedDeletedBlockInfo(
> block.getLocalBlock(), status, delHint);
> final DatanodeStorage storage = dn.getFSDataset().getStorage(storageUuid);
>
> // storage == null here because it's already removed earlier.
> for (BPServiceActor actor : bpServices) {
> actor.getIbrManager().notifyNamenodeBlock(info, storage,
> isOnTransientStorage);
> }
> } {code}
> so IBRs with a null storage are now pending.
> The reason why notifyNamenodeBlock can trigger on such blocks is up in
> DirectoryScanner#reconcile
> {code:java}
> public void reconcile() throws IOException {
> LOG.debug("reconcile start DirectoryScanning");
> scan();
> // If a volume is removed here after scan() already finished running,
> // diffs is stale and checkAndUpdate will run on a removed volume
> // HDFS-14476: run checkAndUpdate with batch to avoid holding the lock too
> // long
> int loopCount = 0;
> synchronized (diffs) {
> for (final Map.Entry entry : diffs.getEntries()) {
> dataset.checkAndUpdate(entry.getKey(), entry.getValue());
> ...
> } {code}
> Inside checkAndUpdate, memBlockInfo is null because all the block meta in
> memory is removed during the volume removal, but diskFile still exists. Then
> DataNode#notifyNamenodeDeletedBlock (and further down the line,
> notifyNamenodeBlock) is called on this block.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839544#comment-17839544
]
ASF GitHub Bot commented on HDFS-17488:
---
kokonguyen191 opened a new pull request, #6759:
URL: https://github.com/apache/hadoop/pull/6759
### Description of PR
Error logs
```
2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830
heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode
(BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool
BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid
1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246)
at
org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920)
at java.lang.Thread.run(Thread.java:748)
```
The root cause is in `BPOfferService#notifyNamenodeBlock`, happens when it's
called on a block belonging to a volume already removed prior. Because the
volume was already removed
```java
private void notifyNamenodeBlock(ExtendedBlock block, BlockStatus status,
String delHint, String storageUuid, boolean isOnTransientStorage) {
checkBlock(block);
final ReceivedDeletedBlockInfo info = new ReceivedDeletedBlockInfo(
block.getLocalBlock(), status, delHint);
final DatanodeStorage storage = dn.getFSDataset().getStorage(storageUuid);
// storage == null here because it's already removed earlier.
for (BPServiceActor actor : bpServices) {
actor.getIbrManager().notifyNamenodeBlock(info, storage,
isOnTransientStorage);
}
}
```
so IBRs with a null storage are now pending.
The reason why notifyNamenodeBlock can trigger on such blocks is up in
DirectoryScanner#reconcile
```java
public void reconcile() throws IOException {
LOG.debug("reconcile start DirectoryScanning");
scan();
// If a volume is removed here after scan() already finished running,
// diffs is stale and checkAndUpdate will run on a removed volume
// HDFS-14476: run checkAndUpdate with batch to avoid holding the lock
too
// long
int loopCount = 0;
synchronized (diffs) {
for (final Map.Entry entry : diffs.getEntries()) {
dataset.checkAndUpdate(entry.getKey(), entry.getValue());
...
}
```
Inside `checkAndUpdate`, `memBlockInfo` is null because all the block meta
in memory is removed during the volume removal, but `diskFile` still exists.
Then `DataNode#notifyNamenodeDeletedBlock` (and further down the line,
`notifyNamenodeBlock`) is called on this block.
This patch effectively contains 2 parts:
* Preventing processing IBRs with null storage
* Preventing `FsDatasetImpl#checkAndUpdate` from running on a block
belonging to a removed storage
### How was this patch tested?
Unit tests. Partially on production cluster.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
> DN can fail IBRs with NPE when a volume is removed
> --
>
> Key: HDFS-17488
> URL: https://issues.apache.org/jira/browse/HDFS-17488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Felix N
>Assignee: Felix N
>Priority: Major
>
>
> Error logs
> {code:java}
> 2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830
> heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode
> (BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool
> BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid
> 1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246)
> at
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920)
> at java.lang.Thread.run(Thread.java:748) {code}
> The
[jira] [Updated] (HDFS-17484) Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually
[ https://issues.apache.org/jira/browse/HDFS-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17484: -- Labels: pull-request-available (was: ) > Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when > they are not busy actually > - > > Key: HDFS-17484 > URL: https://issues.apache.org/jira/browse/HDFS-17484 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > > Currently, we have `dfs.namenode.redundancy.considerLoad` equals true by > default, and > dfs.namenode.redundancy.considerLoad.factor equals 2.0 by default. > Think about below situation. when we are doing stress test, we may deploy > hdfs client onto the datanode. So, this hdfs client will prefer to write to > its local datanode and increase this machine's load. Suppose we have 3 > datanodes, the load of them are as below: 5.0, 0.2, 0.3. > > The load equals to 5.0 will be excluded when choose datanodes for a block. > But actually, it is not slow node when load equals to 5.0 for a machine with > 80 cpu cores. > > So, we should better add a new configuration entry : > `dfs.namenode.redundancy.considerLoad.minLoad` to indicate the mininum factor > we will make considerLoad take effect. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17484) Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually
[ https://issues.apache.org/jira/browse/HDFS-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839524#comment-17839524 ] ASF GitHub Bot commented on HDFS-17484: --- hfutatzhanghb opened a new pull request, #6758: URL: https://github.com/apache/hadoop/pull/6758 ### Description of PR Refer to HDFS-17484. Currently, we have `dfs.namenode.redundancy.considerLoad` equals true by default, and dfs.namenode.redundancy.considerLoad.factor equals 2.0 by default. Think about below situation. when we are doing stress test, we may deploy hdfs client onto the datanode. So, this hdfs client will prefer to write to its local datanode and increase this machine's load. Suppose we have 3 datanodes, the load of them are as below: 5.0, 0.2, 0.3. The load equals to 5.0 will be excluded when choose datanodes for a block. But actually, it is not slow node when load equals to 5.0 for a machine with 80 cpu cores. So, we should better add a new configuration entry : `dfs.namenode.redundancy.considerLoad.minLoad` to indicate the mininum factor we will make considerLoad take effect. ### How was this patch tested? Add an unit test. > Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when > they are not busy actually > - > > Key: HDFS-17484 > URL: https://issues.apache.org/jira/browse/HDFS-17484 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > > Currently, we have `dfs.namenode.redundancy.considerLoad` equals true by > default, and > dfs.namenode.redundancy.considerLoad.factor equals 2.0 by default. > Think about below situation. when we are doing stress test, we may deploy > hdfs client onto the datanode. So, this hdfs client will prefer to write to > its local datanode and increase this machine's load. Suppose we have 3 > datanodes, the load of them are as below: 5.0, 0.2, 0.3. > > The load equals to 5.0 will be excluded when choose datanodes for a block. > But actually, it is not slow node when load equals to 5.0 for a machine with > 80 cpu cores. > > So, we should better add a new configuration entry : > `dfs.namenode.redundancy.considerLoad.minLoad` to indicate the mininum factor > we will make considerLoad take effect. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
