[jira] [Resolved] (HDFS-17408) Reduce the number of quota calculations in FSDirRenameOp
[ https://issues.apache.org/jira/browse/HDFS-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuyan Zhang resolved HDFS-17408. - Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Target Version/s: 3.5.0 Resolution: Fixed > Reduce the number of quota calculations in FSDirRenameOp > > > Key: HDFS-17408 > URL: https://issues.apache.org/jira/browse/HDFS-17408 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: lei w >Assignee: lei w >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > During the execution of the rename operation, we first calculate the quota > for the source INode using verifyQuotaForRename, and at the same time, we > calculate the quota for the target INode. Subsequently, in > RenameOperation#removeSrc, RenameOperation#removeSrc4OldRename, and > RenameOperation#addSourceToDestination, the quota for the source directory is > calculated again. In exceptional cases, RenameOperation#restoreDst and > RenameOperation#restoreSource will also perform quota calculations for the > source and target directories. In fact, many of the quota calculations are > redundant and unnecessary, so we should optimize them away. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17408) Reduce the number of quota calculations in FSDirRenameOp
[ https://issues.apache.org/jira/browse/HDFS-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832997#comment-17832997 ] ASF GitHub Bot commented on HDFS-17408: --- zhangshuyan0 commented on PR #6653: URL: https://github.com/apache/hadoop/pull/6653#issuecomment-2030971312 Committed to trunk. Thanks for your contribution @ThinkerLei @Hexiaoqiao @dineshchitlangia . > Reduce the number of quota calculations in FSDirRenameOp > > > Key: HDFS-17408 > URL: https://issues.apache.org/jira/browse/HDFS-17408 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: lei w >Assignee: lei w >Priority: Major > Labels: pull-request-available > > During the execution of the rename operation, we first calculate the quota > for the source INode using verifyQuotaForRename, and at the same time, we > calculate the quota for the target INode. Subsequently, in > RenameOperation#removeSrc, RenameOperation#removeSrc4OldRename, and > RenameOperation#addSourceToDestination, the quota for the source directory is > calculated again. In exceptional cases, RenameOperation#restoreDst and > RenameOperation#restoreSource will also perform quota calculations for the > source and target directories. In fact, many of the quota calculations are > redundant and unnecessary, so we should optimize them away. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17408) Reduce the number of quota calculations in FSDirRenameOp
[ https://issues.apache.org/jira/browse/HDFS-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832996#comment-17832996 ] ASF GitHub Bot commented on HDFS-17408: --- zhangshuyan0 merged PR #6653: URL: https://github.com/apache/hadoop/pull/6653 > Reduce the number of quota calculations in FSDirRenameOp > > > Key: HDFS-17408 > URL: https://issues.apache.org/jira/browse/HDFS-17408 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: lei w >Assignee: lei w >Priority: Major > Labels: pull-request-available > > During the execution of the rename operation, we first calculate the quota > for the source INode using verifyQuotaForRename, and at the same time, we > calculate the quota for the target INode. Subsequently, in > RenameOperation#removeSrc, RenameOperation#removeSrc4OldRename, and > RenameOperation#addSourceToDestination, the quota for the source directory is > calculated again. In exceptional cases, RenameOperation#restoreDst and > RenameOperation#restoreSource will also perform quota calculations for the > source and target directories. In fact, many of the quota calculations are > redundant and unnecessary, so we should optimize them away. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17438) RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
[ https://issues.apache.org/jira/browse/HDFS-17438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832977#comment-17832977 ] ASF GitHub Bot commented on HDFS-17438: --- goiri commented on code in PR #6655: URL: https://github.com/apache/hadoop/pull/6655#discussion_r1546976890 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestNamenodeResolver.java: ## @@ -359,11 +359,12 @@ public void testRegistrationNamenodeSelection() FederationNamenodeServiceState.ACTIVE); // 1) ns0:nn0 - Standby (oldest) -// 2) ns0:nn1 - Standby (newest) -// 3) ns0:nn2 - Standby -// Verify the selected entry is the newest standby entry +// 2) ns0:nn1 - Standby +// 3) ns0:nn2 - Standby (newest) +// Verify the selected entry is the oldest standby entry assertTrue(namenodeResolver.registerNamenode(createNamenodeReport( NAMESERVICES[0], NAMENODES[0], HAServiceState.STANDBY))); +Thread.sleep(1500); Review Comment: We shouldn't be adding waits like this, can we add a GenericTestUtils#waitFor or similar? > RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority. > - > > Key: HDFS-17438 > URL: https://issues.apache.org/jira/browse/HDFS-17438 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > Attachments: HDFS-17438.001.patch > > > At present, when the status of all namenodes in an ns in the router is the > same, the namenode which is the newest reported will be placed at the top of > the cache. when the client accesses the ns through the router, it will first > access the namenode. > If multiple namenodes in this route are in an active state, or if there are > namenodes with multiple observer states, the existing logic is not a problem, > because the newest reported active or observer state namenode have a higher > probability of being true active or observer compared to the namenode that > reported active or observer state a long time ago. > Similarly, the newest reported namenode with a status of standby or > unavailable has a higher probability of being a standby or unavailable > namenode compared to the namenode reported with a status of standby or > unavailable a long time ago. Therefore, the newest nn reported as standby or > unavailable status should have a lower priority for access, the oldest nn > reported as standby or unavailable status should have a higher priority for > access. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17216) When distcp handle the small files, the bandwidth parameter will be invalid, resulting in serious overspeed behavior
[ https://issues.apache.org/jira/browse/HDFS-17216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832994#comment-17832994 ] ASF GitHub Bot commented on HDFS-17216: --- hiwangzhihui commented on PR #6138: URL: https://github.com/apache/hadoop/pull/6138#issuecomment-2030966943 > just missed this as it went in under hdfs/distcp rather than hadoop common distcp (why two? who knows?) > > made some basic comments. If you can address them as a followup (same jira id, mention me in the PR) then we can get this in and then into branch-3.4. > > please don't backport until then! @xiaojunxiang2023 Let's confirm and follow up on it. > When distcp handle the small files, the bandwidth parameter will be invalid, > resulting in serious overspeed behavior > > > Key: HDFS-17216 > URL: https://issues.apache.org/jira/browse/HDFS-17216 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp >Affects Versions: 3.3.4 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > Attachments: DiscpAnalyze.jpg > > > When distcp copies small files (file size slightly smaller than the > bandwidth), the throbber only starts to throb after 1 second, and the > throttled is specific to a single file. so the throbber becomes invalid, > causing distcp to fill the cluster bandwidth and crush production traffic, > which is a terrible thing. > Also, it takes time for files to set up the IO pipeline, so you shouldn't > test with very small files, which will slow the transfer, especially as > bandwidth kicks in, which will amplify the impact of small files on the rate -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17408) Reduce the number of quota calculations in FSDirRenameOp
[ https://issues.apache.org/jira/browse/HDFS-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832995#comment-17832995 ] ASF GitHub Bot commented on HDFS-17408: --- zhangshuyan0 commented on PR #6653: URL: https://github.com/apache/hadoop/pull/6653#issuecomment-2030969495 @ThinkerLei Thanks for your explanation. Agree with you. > Reduce the number of quota calculations in FSDirRenameOp > > > Key: HDFS-17408 > URL: https://issues.apache.org/jira/browse/HDFS-17408 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: lei w >Assignee: lei w >Priority: Major > Labels: pull-request-available > > During the execution of the rename operation, we first calculate the quota > for the source INode using verifyQuotaForRename, and at the same time, we > calculate the quota for the target INode. Subsequently, in > RenameOperation#removeSrc, RenameOperation#removeSrc4OldRename, and > RenameOperation#addSourceToDestination, the quota for the source directory is > calculated again. In exceptional cases, RenameOperation#restoreDst and > RenameOperation#restoreSource will also perform quota calculations for the > source and target directories. In fact, many of the quota calculations are > redundant and unnecessary, so we should optimize them away. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17216) When distcp handle the small files, the bandwidth parameter will be invalid, resulting in serious overspeed behavior
[ https://issues.apache.org/jira/browse/HDFS-17216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832941#comment-17832941 ] ASF GitHub Bot commented on HDFS-17216: --- steveloughran commented on PR #6138: URL: https://github.com/apache/hadoop/pull/6138#issuecomment-2030463783 just missed this as it went in under hdfs/distcp rather than hadoop common distcp (why two? who knows?) made some basic comments. If you can address them as a followup (same jira id, mention me in the PR) then we can get this in and then into branch-3.4. please don't backport until then! > When distcp handle the small files, the bandwidth parameter will be invalid, > resulting in serious overspeed behavior > > > Key: HDFS-17216 > URL: https://issues.apache.org/jira/browse/HDFS-17216 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp >Affects Versions: 3.3.4 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > Attachments: DiscpAnalyze.jpg > > > When distcp copies small files (file size slightly smaller than the > bandwidth), the throbber only starts to throb after 1 second, and the > throttled is specific to a single file. so the throbber becomes invalid, > causing distcp to fill the cluster bandwidth and crush production traffic, > which is a terrible thing. > Also, it takes time for files to set up the IO pipeline, so you shouldn't > test with very small files, which will slow the transfer, especially as > bandwidth kicks in, which will amplify the impact of small files on the rate -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17446) The DataNode adds a log to report its progress during addToReplicasMap execution.
[
https://issues.apache.org/jira/browse/HDFS-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832796#comment-17832796
]
ASF GitHub Bot commented on HDFS-17446:
---
xiaojunxiang2023 commented on code in PR #6685:
URL: https://github.com/apache/hadoop/pull/6685#discussion_r1546157919
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java:
##
@@ -749,6 +749,8 @@ void addToReplicasMap(ReplicaMap volumeMap, File dir,
lazyWriteReplicaMap, isFinalized, exceptions, subTaskQueue);
subTask.fork();
subTaskQueue.add(subTask);
+ } else if (isFinalized && ++scanBlockCnt % 1 == 0) {
+LOG.info((scanBlockCnt - 1) + " blocks (finalize state) have been
scanned.");
Review Comment:
> The log style is not easy to view.
I have optimize it:
![Uploading image.png…]()
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java:
##
@@ -749,6 +749,8 @@ void addToReplicasMap(ReplicaMap volumeMap, File dir,
lazyWriteReplicaMap, isFinalized, exceptions, subTaskQueue);
subTask.fork();
subTaskQueue.add(subTask);
+ } else if (isFinalized && ++scanBlockCnt % 1 == 0) {
+LOG.info((scanBlockCnt - 1) + " blocks (finalize state) have been
scanned.");
Review Comment:
> The log style is not easy to view.
I have optimize it:
> The DataNode adds a log to report its progress during addToReplicasMap
> execution.
> -
>
> Key: HDFS-17446
> URL: https://issues.apache.org/jira/browse/HDFS-17446
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: xiaojunxiang
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-03-28-23-53-29-057.png,
> image-2024-04-01-17-26-05-433.png
>
>
> Datanodes do not print logs when addToReplicasMap is just started, so we will
> not be aware of its progress, or even wait up to 4 hours for a large cluster.
> This is very confusing.
> A cluster is as follows:
> !image-2024-03-28-23-53-29-057.png!
>
> The optimized result is as follows:
> (Because Scan thread is ForkJoinPool, so the printing may be repeated, but it
> will affect the overall progress of the presentation
> )
> !image-2024-04-01-17-26-05-433.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17408) Reduce the number of quota calculations in FSDirRenameOp
[ https://issues.apache.org/jira/browse/HDFS-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832834#comment-17832834 ] ASF GitHub Bot commented on HDFS-17408: --- ThinkerLei commented on PR #6653: URL: https://github.com/apache/hadoop/pull/6653#issuecomment-2029711670 > @ThinkerLei Thanks for your reply. So what you mean is: If src directory does not have any storage policy set (that is, files under src use the default storage policy, ALL_DISK for example) and dst is set to ALL_SSD, then after renaming src to dst, the data blocks will be stored on the SSD. Do I understand correctly? The current logic is that if the block is not rebuilt, the data blocks will not be stored on the SSD. I'm not quite clear on the rationale behind this design. If necessary, we can open another issue to discuss it further. > Reduce the number of quota calculations in FSDirRenameOp > > > Key: HDFS-17408 > URL: https://issues.apache.org/jira/browse/HDFS-17408 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: lei w >Assignee: lei w >Priority: Major > Labels: pull-request-available > > During the execution of the rename operation, we first calculate the quota > for the source INode using verifyQuotaForRename, and at the same time, we > calculate the quota for the target INode. Subsequently, in > RenameOperation#removeSrc, RenameOperation#removeSrc4OldRename, and > RenameOperation#addSourceToDestination, the quota for the source directory is > calculated again. In exceptional cases, RenameOperation#restoreDst and > RenameOperation#restoreSource will also perform quota calculations for the > source and target directories. In fact, many of the quota calculations are > redundant and unnecessary, so we should optimize them away. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17408) Reduce the number of quota calculations in FSDirRenameOp
[ https://issues.apache.org/jira/browse/HDFS-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832836#comment-17832836 ] ASF GitHub Bot commented on HDFS-17408: --- ThinkerLei commented on PR #6653: URL: https://github.com/apache/hadoop/pull/6653#issuecomment-2029713202 > @ThinkerLei Thanks for your reply. So what you mean is: If src directory does not have any storage policy set (that is, files under src use the default storage policy, ALL_DISK for example) and dst is set to ALL_SSD, then after renaming src to dst, the data blocks will be stored on the SSD. Do I understand correctly? The current logic is that if the block is not rebuilt, the data blocks will not be stored on the SSD. I'm not quite clear on the rationale behind this design. If necessary, we can open another issue to discuss it further. > Reduce the number of quota calculations in FSDirRenameOp > > > Key: HDFS-17408 > URL: https://issues.apache.org/jira/browse/HDFS-17408 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: lei w >Assignee: lei w >Priority: Major > Labels: pull-request-available > > During the execution of the rename operation, we first calculate the quota > for the source INode using verifyQuotaForRename, and at the same time, we > calculate the quota for the target INode. Subsequently, in > RenameOperation#removeSrc, RenameOperation#removeSrc4OldRename, and > RenameOperation#addSourceToDestination, the quota for the source directory is > calculated again. In exceptional cases, RenameOperation#restoreDst and > RenameOperation#restoreSource will also perform quota calculations for the > source and target directories. In fact, many of the quota calculations are > redundant and unnecessary, so we should optimize them away. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17446) The DataNode adds a log to report its progress during addToReplicasMap execution.
[
https://issues.apache.org/jira/browse/HDFS-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832896#comment-17832896
]
ASF GitHub Bot commented on HDFS-17446:
---
hadoop-yetus commented on PR #6685:
URL: https://github.com/apache/hadoop/pull/6685#issuecomment-2030122801
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 23s | | trunk passed |
| +1 :green_heart: | compile | 2m 5s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 43m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 3s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 263m 44s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 424m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6685/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6685 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 148fd7f0fef4 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 150f9834c98b60a1d0013945927a007d22280259 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6685/9/testReport/ |
| Max. process+thread count | 3151 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6685/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> The DataNode adds a log to report its
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[
https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832778#comment-17832778
]
ASF GitHub Bot commented on HDFS-17444:
---
hadoop-yetus commented on PR #6694:
URL: https://github.com/apache/hadoop/pull/6694#issuecomment-2029430219
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 33s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 201m 16s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 301m 10s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6694 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8dee1c44b678 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bd16e6a9023dd0a74148f3436ddbdea72338993d |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/5/testReport/ |
| Max. process+thread count | 4450 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[
https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832806#comment-17832806
]
ASF GitHub Bot commented on HDFS-17444:
---
hadoop-yetus commented on PR #6694:
URL: https://github.com/apache/hadoop/pull/6694#issuecomment-2029527535
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 3s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 278m 11s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 420m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6694 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux fc139e1dfaa2 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 47bfef529f5f69a1ae5d0f2b430df35648c47deb |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/3/testReport/ |
| Max. process+thread count | 3366 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Add getJournalSyncerStatus jmx metrics,
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[
https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832883#comment-17832883
]
ASF GitHub Bot commented on HDFS-17444:
---
hadoop-yetus commented on PR #6694:
URL: https://github.com/apache/hadoop/pull/6694#issuecomment-2030056382
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 274m 41s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 416m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6694 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2561c830dc72 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 745dd1fd0c1d69030ec3ab0dd7365b731d382603 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/7/testReport/ |
| Max. process+thread count | 3148 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Add getJournalSyncerStatus jmx metrics,
[jira] [Commented] (HDFS-17446) The DataNode adds a log to report its progress during addToReplicasMap execution.
[
https://issues.apache.org/jira/browse/HDFS-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832888#comment-17832888
]
ASF GitHub Bot commented on HDFS-17446:
---
hadoop-yetus commented on PR #6685:
URL: https://github.com/apache/hadoop/pull/6685#issuecomment-2030082476
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 16s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 51m 29s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 275m 37s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6685/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 425m 18s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6685/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6685 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d3a96023bf53 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bc54efbe18c7f55f9d8ae1c96d654e34e6507f71 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6685/8/testReport/ |
| Max. process+thread count | 3278 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Updated] (HDFS-17446) The DataNode adds a log to report its progress during addToReplicasMap execution.
[ https://issues.apache.org/jira/browse/HDFS-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaojunxiang updated HDFS-17446: Description: Datanodes do not print logs when addToReplicasMap is just started, so we will not be aware of its progress, or even wait up to 4 hours for a large cluster. This is very confusing. A cluster is as follows: !image-2024-03-28-23-53-29-057.png! The optimized result is as follows: (Because Scan thread is ForkJoinPool, so the printing may be repeated, but it will affect the overall progress of the presentation ) !image-2024-04-01-17-26-05-433.png! was: Datanodes do not print logs when addToReplicasMap is just started, so we will not be aware of its progress, or even wait up to 4 hours for a large cluster. This is very confusing. A cluster is as follows: !image-2024-03-28-23-53-29-057.png! > The DataNode adds a log to report its progress during addToReplicasMap > execution. > - > > Key: HDFS-17446 > URL: https://issues.apache.org/jira/browse/HDFS-17446 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: xiaojunxiang >Priority: Critical > Labels: pull-request-available > Attachments: image-2024-03-28-23-53-29-057.png, > image-2024-04-01-17-26-05-433.png > > > Datanodes do not print logs when addToReplicasMap is just started, so we will > not be aware of its progress, or even wait up to 4 hours for a large cluster. > This is very confusing. > A cluster is as follows: > !image-2024-03-28-23-53-29-057.png! > > The optimized result is as follows: > (Because Scan thread is ForkJoinPool, so the printing may be repeated, but it > will affect the overall progress of the presentation > ) > !image-2024-04-01-17-26-05-433.png! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17446) The DataNode adds a log to report its progress during addToReplicasMap execution.
[
https://issues.apache.org/jira/browse/HDFS-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832782#comment-17832782
]
ASF GitHub Bot commented on HDFS-17446:
---
hiwangzhihui commented on code in PR #6685:
URL: https://github.com/apache/hadoop/pull/6685#discussion_r1546135795
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java:
##
@@ -749,6 +749,8 @@ void addToReplicasMap(ReplicaMap volumeMap, File dir,
lazyWriteReplicaMap, isFinalized, exceptions, subTaskQueue);
subTask.fork();
subTaskQueue.add(subTask);
+ } else if (isFinalized && ++scanBlockCnt % 1 == 0) {
+LOG.info((scanBlockCnt - 1) + " blocks (finalize state) have been
scanned.");
Review Comment:
Please take care of concurrency issues that lead to inaccurate counting.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java:
##
@@ -749,6 +749,8 @@ void addToReplicasMap(ReplicaMap volumeMap, File dir,
lazyWriteReplicaMap, isFinalized, exceptions, subTaskQueue);
subTask.fork();
subTaskQueue.add(subTask);
+ } else if (isFinalized && ++scanBlockCnt % 1 == 0) {
+LOG.info((scanBlockCnt - 1) + " blocks (finalize state) have been
scanned.");
Review Comment:
The log style is not easy to view.
> The DataNode adds a log to report its progress during addToReplicasMap
> execution.
> -
>
> Key: HDFS-17446
> URL: https://issues.apache.org/jira/browse/HDFS-17446
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: xiaojunxiang
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-03-28-23-53-29-057.png
>
>
> Datanodes do not print logs when addToReplicasMap is just started, so we will
> not be aware of its progress, or even wait up to 4 hours for a large cluster.
> This is very confusing.
> A cluster is as follows:
> !image-2024-03-28-23-53-29-057.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17446) The DataNode adds a log to report its progress during addToReplicasMap execution.
[ https://issues.apache.org/jira/browse/HDFS-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaojunxiang updated HDFS-17446: Attachment: image-2024-04-01-17-26-05-433.png > The DataNode adds a log to report its progress during addToReplicasMap > execution. > - > > Key: HDFS-17446 > URL: https://issues.apache.org/jira/browse/HDFS-17446 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: xiaojunxiang >Priority: Critical > Labels: pull-request-available > Attachments: image-2024-03-28-23-53-29-057.png, > image-2024-04-01-17-26-05-433.png > > > Datanodes do not print logs when addToReplicasMap is just started, so we will > not be aware of its progress, or even wait up to 4 hours for a large cluster. > This is very confusing. > A cluster is as follows: > !image-2024-03-28-23-53-29-057.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17446) The DataNode adds a log to report its progress during addToReplicasMap execution.
[
https://issues.apache.org/jira/browse/HDFS-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832794#comment-17832794
]
ASF GitHub Bot commented on HDFS-17446:
---
xiaojunxiang2023 commented on code in PR #6685:
URL: https://github.com/apache/hadoop/pull/6685#discussion_r1546155860
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java:
##
@@ -749,6 +749,8 @@ void addToReplicasMap(ReplicaMap volumeMap, File dir,
lazyWriteReplicaMap, isFinalized, exceptions, subTaskQueue);
subTask.fork();
subTaskQueue.add(subTask);
+ } else if (isFinalized && ++scanBlockCnt % 1 == 0) {
+LOG.info((scanBlockCnt - 1) + " blocks (finalize state) have been
scanned.");
Review Comment:
ok. I will fix it.
> The DataNode adds a log to report its progress during addToReplicasMap
> execution.
> -
>
> Key: HDFS-17446
> URL: https://issues.apache.org/jira/browse/HDFS-17446
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: xiaojunxiang
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-03-28-23-53-29-057.png,
> image-2024-04-01-17-26-05-433.png
>
>
> Datanodes do not print logs when addToReplicasMap is just started, so we will
> not be aware of its progress, or even wait up to 4 hours for a large cluster.
> This is very confusing.
> A cluster is as follows:
> !image-2024-03-28-23-53-29-057.png!
>
> The optimized result is as follows:
> (Because Scan thread is ForkJoinPool, so the printing may be repeated, but it
> will affect the overall progress of the presentation
> )
> !image-2024-04-01-17-26-05-433.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[
https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832852#comment-17832852
]
ASF GitHub Bot commented on HDFS-17444:
---
hadoop-yetus commented on PR #6694:
URL: https://github.com/apache/hadoop/pull/6694#issuecomment-2029801205
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 26s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 202m 52s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 293m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6694 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c2c0413f8033 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 745dd1fd0c1d69030ec3ab0dd7365b731d382603 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/6/testReport/ |
| Max. process+thread count | 3947 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Add getJournalSyncerStatus jmx metrics,
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[
https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832766#comment-17832766
]
ASF GitHub Bot commented on HDFS-17444:
---
hadoop-yetus commented on PR #6694:
URL: https://github.com/apache/hadoop/pull/6694#issuecomment-2029365198
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 35s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 199m 12s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 299m 50s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6694 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5a1da249503d 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 47bfef529f5f69a1ae5d0f2b430df35648c47deb |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/4/testReport/ |
| Max. process+thread count | 4236 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
