[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[ https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845777#comment-17845777 ] ASF GitHub Bot commented on HDFS-17509: --- hadoop-yetus commented on PR #6784: URL: https://github.com/apache/hadoop/pull/6784#issuecomment-2106577140 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 19s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 32m 33s | | trunk passed | | +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | | +1 :green_heart: | compile | 0m 20s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | | +1 :green_heart: | checkstyle | 0m 20s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 27s | | trunk passed | | +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | | +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | | +1 :green_heart: | spotbugs | 0m 49s | | trunk passed | | +1 :green_heart: | shadedclient | 19m 45s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 19s | | the patch passed | | +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | | +1 :green_heart: | javac | 0m 21s | | the patch passed | | +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | | +1 :green_heart: | javac | 0m 16s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 0m 12s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/7/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) | | +1 :green_heart: | mvnsite | 0m 19s | | the patch passed | | +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 | | +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | | +1 :green_heart: | spotbugs | 0m 49s | | the patch passed | | +1 :green_heart: | shadedclient | 19m 49s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 29m 7s | | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 26s | | The patch does not generate ASF License warnings. | | | | 111m 17s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/6784 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | Linux 10b17be7049b 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 61d08c583e427df3340a7afcd984ad69943a1675 | | Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/7/testReport/ | | Max. process+thread count | 3163 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output |
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[ https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845776#comment-17845776 ] ASF GitHub Bot commented on HDFS-17384: --- hadoop-yetus commented on PR #6762: URL: https://github.com/apache/hadoop/pull/6762#issuecomment-2106553639 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | -1 :x: | patch | 1m 02s | | https://github.com/apache/hadoop/pull/6762 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6762 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6762/3/console | | versions | git=2.44.0.windows.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > [FGL] Replace the global lock with global FS Lock and global BM lock > > > Key: HDFS-17384 > URL: https://issues.apache.org/jira/browse/HDFS-17384 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: FGL, pull-request-available > > First, we can replace the current global lock with two locks, global FS lock > and global BM lock. > The global FS lock is used to make directory tree-related operations > thread-safe. > The global BM lock is used to make block-related operations and DN-related > operations thread-safe. > > For some operations involving both directory tree and block or DN, the global > FS lock and the global BM lock are acquired. > > The lock order should be: > * The global FS lock > * The global BM lock > > There are some special requirements for this ticket. > * End-user can choose to use global lock or fine-grained lock through > configuration. > * Try not to modify the current implementation logic as much as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17506) [FGL] Performance for phase 1
[ https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845771#comment-17845771 ] ASF GitHub Bot commented on HDFS-17506: --- ferhui commented on PR #6806: URL: https://github.com/apache/hadoop/pull/6806#issuecomment-2106540855 Thanks for contribution. Merged. > [FGL] Performance for phase 1 > - > > Key: HDFS-17506 > URL: https://issues.apache.org/jira/browse/HDFS-17506 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Do some benchmark testing for phase 1. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17506) [FGL] Performance for phase 1
[ https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei resolved HDFS-17506. Resolution: Fixed > [FGL] Performance for phase 1 > - > > Key: HDFS-17506 > URL: https://issues.apache.org/jira/browse/HDFS-17506 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Do some benchmark testing for phase 1. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[ https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845768#comment-17845768 ] ASF GitHub Bot commented on HDFS-17518: --- ThinkerLei commented on code in PR #6809: URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597806803 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -3738,7 +3738,7 @@ boolean internalReleaseLease(Lease lease, String src, INodesInPath iip, NameNode.stateChangeLog.warn("BLOCK*" + " internalReleaseLease: All existing blocks are COMPLETE," + " lease removed, file " + src + " closed."); - return true; // closed! + return false; // closed! Review Comment: @vinayakumarb Thank you for your review. Let me explain the current logic. The logic I am modifying now is as follows: if the lease is recovered or the lease is reassigned, it will return false, just like the previous logic. Then, in the checkLeases method, if the return is false and needSync is false, needSync will be reset to true. Subsequently, the edits log will be flushed by leaseMonitor. This way, when RPCs such as recoverLease call the internalReleaseLease method, they can remain consistent with the original behavior. > In the lease monitor, if a file is closed, we should sync the editslog > -- > > Key: HDFS-17518 > URL: https://issues.apache.org/jira/browse/HDFS-17518 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > Labels: pull-request-available > > In the lease monitor, if a file is closed, method checklease will return > true, and then the edits log will not be sync. In my opinion, we should sync > the edits log to avoid not synchronizing the state to the standby NameNode > for a long time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[ https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845769#comment-17845769 ] ASF GitHub Bot commented on HDFS-17518: --- ThinkerLei commented on code in PR #6809: URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597807172 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java: ## @@ -626,7 +626,8 @@ private synchronized boolean checkLeases(Collection leasesToCheck) { } } // If a lease recovery happened, we need to sync later. Review Comment: @vinayakumarb Thank you for your review. Let me explain the current logic. The logic I am modifying now is as follows: if the lease is recovered or the lease is reassigned, it will return false, just like the previous logic. Then, in the checkLeases method, if the return is false and needSync is false, needSync will be reset to true. Subsequently, the edits log will be flushed by leaseMonitor. This way, when RPCs such as recoverLease call the internalReleaseLease method, they can remain consistent with the original behavior. > In the lease monitor, if a file is closed, we should sync the editslog > -- > > Key: HDFS-17518 > URL: https://issues.apache.org/jira/browse/HDFS-17518 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > Labels: pull-request-available > > In the lease monitor, if a file is closed, method checklease will return > true, and then the edits log will not be sync. In my opinion, we should sync > the edits log to avoid not synchronizing the state to the standby NameNode > for a long time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[ https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845767#comment-17845767 ] ASF GitHub Bot commented on HDFS-17518: --- ThinkerLei commented on code in PR #6809: URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597806803 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -3738,7 +3738,7 @@ boolean internalReleaseLease(Lease lease, String src, INodesInPath iip, NameNode.stateChangeLog.warn("BLOCK*" + " internalReleaseLease: All existing blocks are COMPLETE," + " lease removed, file " + src + " closed."); - return true; // closed! + return false; // closed! Review Comment: @vinayakumarb Thank you for your review. Let me explain the current logic. The logic I am modifying now is as follows: if the lease is recovered or the lease is reassigned, it will return false, just like the previous logic. Then, in the checkLeases method, if the return is false and needSync is false, needSync will be reset to true. Subsequently, the edits log will be flushed by leaseMonitor. This way, when RPCs such as recoverLease call the internalReleaseLease method, they can remain consistent with the original behavior. > In the lease monitor, if a file is closed, we should sync the editslog > -- > > Key: HDFS-17518 > URL: https://issues.apache.org/jira/browse/HDFS-17518 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > Labels: pull-request-available > > In the lease monitor, if a file is closed, method checklease will return > true, and then the edits log will not be sync. In my opinion, we should sync > the edits log to avoid not synchronizing the state to the standby NameNode > for a long time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[ https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845764#comment-17845764 ] ASF GitHub Bot commented on HDFS-17509: --- LiuGuH commented on code in PR #6784: URL: https://github.com/apache/hadoop/pull/6784#discussion_r1597789296 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java: ## @@ -1224,6 +1224,17 @@ public void testProxyConcatFile() throws Exception { String badPath = "/unknownlocation/unknowndir"; compareResponses(routerProtocol, nnProtocol, m, new Object[] {badPath, new String[] {routerFile}}); + +// Test when concat trg is a empty file Review Comment: Add test for a empty src file > RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file. > -- > > Key: HDFS-17509 > URL: https://issues.apache.org/jira/browse/HDFS-17509 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: liuguanghua >Priority: Minor > Labels: pull-request-available > > hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2 > When /tmp/merge is a empty file, this command will throw NPE via DFSRouter. > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[ https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845763#comment-17845763 ] ASF GitHub Bot commented on HDFS-17509: --- LiuGuH commented on code in PR #6784: URL: https://github.com/apache/hadoop/pull/6784#discussion_r1597788935 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java: ## @@ -1009,6 +1000,20 @@ public HdfsFileStatus getFileInfo(String src) throws IOException { return ret; } + public RemoteResult getFileRemoteResult(String path) + throws IOException { +rpcServer.checkOperation(NameNode.OperationCategory.READ); + +final List locations = rpcServer.getLocationsForPath(path, false, false); +RemoteMethod method = +new RemoteMethod("getFileInfo", new Class[] {String.class}, new RemoteParam()); +// Check for file information sequentially +RemoteResult result = Review Comment: OK > RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file. > -- > > Key: HDFS-17509 > URL: https://issues.apache.org/jira/browse/HDFS-17509 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: liuguanghua >Priority: Minor > Labels: pull-request-available > > hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2 > When /tmp/merge is a empty file, this command will throw NPE via DFSRouter. > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17397) Choose another DN as soon as possible, when encountering network issues
[ https://issues.apache.org/jira/browse/HDFS-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845760#comment-17845760 ] ASF GitHub Bot commented on HDFS-17397: --- daragu commented on PR #6591: URL: https://github.com/apache/hadoop/pull/6591#issuecomment-2106461328 LGTM > Choose another DN as soon as possible, when encountering network issues > --- > > Key: HDFS-17397 > URL: https://issues.apache.org/jira/browse/HDFS-17397 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: xleoken >Priority: Minor > Labels: pull-request-available > Attachments: hadoop.png > > > Choose another DN as soon as possible, when encountering network issues. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[ https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845742#comment-17845742 ] ASF GitHub Bot commented on HDFS-17518: --- vinayakumarb commented on code in PR #6809: URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597693783 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java: ## @@ -626,7 +626,8 @@ private synchronized boolean checkLeases(Collection leasesToCheck) { } } // If a lease recovery happened, we need to sync later. Review Comment: This is a nice hack. But this will not handle the case, where actual recovery of the file is triggered and lease is reassigned. Lease re-assignment also will have a edit log. This also should be synced as well. I would recommend you to change the return type of `internalReleaseLease()` to `ImmutablePair` to include both `completed` and `needSync` values. needSync will be true in both cases of file closed and lease re-assignment. > In the lease monitor, if a file is closed, we should sync the editslog > -- > > Key: HDFS-17518 > URL: https://issues.apache.org/jira/browse/HDFS-17518 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > Labels: pull-request-available > > In the lease monitor, if a file is closed, method checklease will return > true, and then the edits log will not be sync. In my opinion, we should sync > the edits log to avoid not synchronizing the state to the standby NameNode > for a long time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[ https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845739#comment-17845739 ] ASF GitHub Bot commented on HDFS-17518: --- hadoop-yetus commented on PR #6809: URL: https://github.com/apache/hadoop/pull/6809#issuecomment-2106325014 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 00s | | No case conflicting files found. | | +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not available. | | +0 :ok: | codespell | 0m 01s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 00s | | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 110m 34s | | trunk passed | | +1 :green_heart: | compile | 7m 49s | | trunk passed | | +1 :green_heart: | checkstyle | 6m 17s | | trunk passed | | +1 :green_heart: | mvnsite | 8m 58s | | trunk passed | | +1 :green_heart: | javadoc | 8m 27s | | trunk passed | | +1 :green_heart: | shadedclient | 181m 41s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 01s | | the patch passed | | +1 :green_heart: | compile | 4m 44s | | the patch passed | | +1 :green_heart: | javac | 4m 44s | | the patch passed | | +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 3m 22s | | the patch passed | | +1 :green_heart: | mvnsite | 5m 12s | | the patch passed | | +1 :green_heart: | javadoc | 4m 25s | | the patch passed | | +1 :green_heart: | shadedclient | 195m 47s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | asflicense | 7m 19s | | The patch does not generate ASF License warnings. | | | | 523m 52s | | | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6809 | | JIRA Issue | HDFS-17518 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | MINGW64_NT-10.0-17763 117457eea4e9 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys | | Build tool | maven | | Personality | /c/hadoop/dev-support/bin/hadoop.sh | | git revision | trunk / 6c1e9a137d17034a7f2b2a36b3b9ce0f6afb5ac9 | | Default Java | Azul Systems, Inc.-1.8.0_332-b09 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6809/2/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6809/2/console | | versions | git=2.44.0.windows.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > In the lease monitor, if a file is closed, we should sync the editslog > -- > > Key: HDFS-17518 > URL: https://issues.apache.org/jira/browse/HDFS-17518 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > Labels: pull-request-available > > In the lease monitor, if a file is closed, method checklease will return > true, and then the edits log will not be sync. In my opinion, we should sync > the edits log to avoid not synchronizing the state to the standby NameNode > for a long time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17438) RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
[ https://issues.apache.org/jira/browse/HDFS-17438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845730#comment-17845730 ] ASF GitHub Bot commented on HDFS-17438: --- goiri commented on code in PR #6655: URL: https://github.com/apache/hadoop/pull/6655#discussion_r1597672487 ## hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java: ## @@ -404,6 +404,28 @@ public static void waitFor(final Supplier check, } } + /** + * Wait at least {@code atLeastWaitForMillis} from start to end of the test. + * + * @param check the test to perform. + * @param atLeastWaitForMillis the minimum waiting time from the beginning + * to the end of the test. + * @throws InterruptedException if the method is interrupted while waiting. + */ + public static T atLeastWaitFor(final Supplier check, + long atLeastWaitForMillis) throws InterruptedException { +if (atLeastWaitForMillis < 0) { + atLeastWaitForMillis = 0; +} +long st = Time.monotonicNow(); +T result = check.get(); Review Comment: I don't think you are waiting at least, you are getting the value and then waiting. Either change the method name to execute and wait or actually wait after the sleep. > RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority. > - > > Key: HDFS-17438 > URL: https://issues.apache.org/jira/browse/HDFS-17438 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Major > Labels: pull-request-available > Attachments: HDFS-17438.001.patch > > > At present, when the status of all namenodes in an ns in the router is the > same, the namenode which is the newest reported will be placed at the top of > the cache. when the client accesses the ns through the router, it will first > access the namenode. > If multiple namenodes in this route are in an active state, or if there are > namenodes with multiple observer states, the existing logic is not a problem, > because the newest reported active or observer state namenode have a higher > probability of being true active or observer compared to the namenode that > reported active or observer state a long time ago. > Similarly, the newest reported namenode with a status of standby or > unavailable has a higher probability of being a standby or unavailable > namenode compared to the namenode reported with a status of standby or > unavailable a long time ago. Therefore, the newest nn reported as standby or > unavailable status should have a lower priority for access, the oldest nn > reported as standby or unavailable status should have a higher priority for > access. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[ https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845727#comment-17845727 ] ASF GitHub Bot commented on HDFS-17518: --- hadoop-yetus commented on PR #6809: URL: https://github.com/apache/hadoop/pull/6809#issuecomment-2106300111 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 00s | | No case conflicting files found. | | +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not available. | | +0 :ok: | codespell | 0m 01s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 00s | | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 87m 17s | | trunk passed | | +1 :green_heart: | compile | 5m 56s | | trunk passed | | +1 :green_heart: | checkstyle | 4m 41s | | trunk passed | | +1 :green_heart: | mvnsite | 6m 23s | | trunk passed | | +1 :green_heart: | javadoc | 6m 00s | | trunk passed | | +1 :green_heart: | shadedclient | 142m 05s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 30s | | the patch passed | | +1 :green_heart: | compile | 3m 22s | | the patch passed | | +1 :green_heart: | javac | 3m 22s | | the patch passed | | +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 2m 18s | | the patch passed | | +1 :green_heart: | mvnsite | 4m 01s | | the patch passed | | +1 :green_heart: | javadoc | 3m 28s | | the patch passed | | +1 :green_heart: | shadedclient | 155m 05s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | asflicense | 5m 13s | | The patch does not generate ASF License warnings. | | | | 411m 17s | | | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6809 | | JIRA Issue | HDFS-17518 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | MINGW64_NT-10.0-17763 b1f57066a6fa 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys | | Build tool | maven | | Personality | /c/hadoop/dev-support/bin/hadoop.sh | | git revision | trunk / 6c1e9a137d17034a7f2b2a36b3b9ce0f6afb5ac9 | | Default Java | Azul Systems, Inc.-1.8.0_332-b09 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6809/3/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6809/3/console | | versions | git=2.44.0.windows.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > In the lease monitor, if a file is closed, we should sync the editslog > -- > > Key: HDFS-17518 > URL: https://issues.apache.org/jira/browse/HDFS-17518 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > Labels: pull-request-available > > In the lease monitor, if a file is closed, method checklease will return > true, and then the edits log will not be sync. In my opinion, we should sync > the edits log to avoid not synchronizing the state to the standby NameNode > for a long time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17523) Add fine-grained locks metrics in DataSetLockManager
lei w created HDFS-17523: Summary: Add fine-grained locks metrics in DataSetLockManager Key: HDFS-17523 URL: https://issues.apache.org/jira/browse/HDFS-17523 Project: Hadoop HDFS Issue Type: Improvement Reporter: lei w Currently we use fine-grained locks to manage FsDataSetImpl. But we did not add lock-related metrics. In some cases, we actually need lock-holding information to understand the time-consuming lock-holding of a certain operation. Using this information, we can also optimize some long-term lock operations as early as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
[ https://issues.apache.org/jira/browse/HDFS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17522: -- Affects Version/s: 3.5.0 > JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection > -- > > Key: HDFS-17522 > URL: https://issues.apache.org/jira/browse/HDFS-17522 > Project: Hadoop HDFS > Issue Type: Bug > Components: journal-node >Affects Versions: 3.0.0-alpha1, 3.5.0 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > > [HDFS-10579 |https://issues.apache.org/jira/browse/HDFS-10579] has added > protection for NameNode and DataNode, but missing protection for JournalNode > web interfaces. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
[ https://issues.apache.org/jira/browse/HDFS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845706#comment-17845706 ] ASF GitHub Bot commented on HDFS-17522: --- hadoop-yetus commented on PR #6814: URL: https://github.com/apache/hadoop/pull/6814#issuecomment-2106252485 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 01s | | No case conflicting files found. | | +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not available. | | +0 :ok: | codespell | 0m 00s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 00s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 00s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 93m 21s | | trunk passed | | +1 :green_heart: | compile | 6m 17s | | trunk passed | | +1 :green_heart: | checkstyle | 5m 02s | | trunk passed | | +1 :green_heart: | mvnsite | 6m 54s | | trunk passed | | +1 :green_heart: | javadoc | 6m 32s | | trunk passed | | +1 :green_heart: | shadedclient | 153m 56s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 53s | | the patch passed | | +1 :green_heart: | compile | 3m 44s | | the patch passed | | +1 :green_heart: | javac | 3m 44s | | the patch passed | | -1 :x: | blanks | 0m 00s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6814/1/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | checkstyle | 2m 35s | | the patch passed | | +1 :green_heart: | mvnsite | 4m 35s | | the patch passed | | +1 :green_heart: | javadoc | 3m 41s | | the patch passed | | +1 :green_heart: | shadedclient | 165m 22s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | asflicense | 5m 42s | | The patch does not generate ASF License warnings. | | | | 440m 54s | | | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6814 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | MINGW64_NT-10.0-17763 53b71b4f2f76 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys | | Build tool | maven | | Personality | /c/hadoop/dev-support/bin/hadoop.sh | | git revision | trunk / 213e3fc0e16b2e7b1c9d5e86ff411a9762da7061 | | Default Java | Azul Systems, Inc.-1.8.0_332-b09 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6814/1/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6814/1/console | | versions | git=2.45.0.windows.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection > -- > > Key: HDFS-17522 > URL: https://issues.apache.org/jira/browse/HDFS-17522 > Project: Hadoop HDFS > Issue Type: Bug > Components: journal-node >Affects Versions: 3.0.0-alpha1 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > > [HDFS-10579 |https://issues.apache.org/jira/browse/HDFS-10579] has added > protection for NameNode and DataNode, but missing protection for JournalNode > web interfaces. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[ https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845674#comment-17845674 ] ASF GitHub Bot commented on HDFS-17518: --- ThinkerLei commented on code in PR #6809: URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597594907 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -3738,7 +3738,7 @@ boolean internalReleaseLease(Lease lease, String src, INodesInPath iip, NameNode.stateChangeLog.warn("BLOCK*" + " internalReleaseLease: All existing blocks are COMPLETE," + " lease removed, file " + src + " closed."); - return true; // closed! + return false; // closed! Review Comment: @vinayakumarb Thank you very much for your comment; it is very accurate. I have modified. Please review again. > In the lease monitor, if a file is closed, we should sync the editslog > -- > > Key: HDFS-17518 > URL: https://issues.apache.org/jira/browse/HDFS-17518 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > Labels: pull-request-available > > In the lease monitor, if a file is closed, method checklease will return > true, and then the edits log will not be sync. In my opinion, we should sync > the edits log to avoid not synchronizing the state to the standby NameNode > for a long time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[ https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845673#comment-17845673 ] ASF GitHub Bot commented on HDFS-17518: --- hadoop-yetus commented on PR #6809: URL: https://github.com/apache/hadoop/pull/6809#issuecomment-2106176232 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 15s | | https://github.com/apache/hadoop/pull/6809 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6809 | | JIRA Issue | HDFS-17518 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6809/3/console | | versions | git=2.34.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > In the lease monitor, if a file is closed, we should sync the editslog > -- > > Key: HDFS-17518 > URL: https://issues.apache.org/jira/browse/HDFS-17518 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > Labels: pull-request-available > > In the lease monitor, if a file is closed, method checklease will return > true, and then the edits log will not be sync. In my opinion, we should sync > the edits log to avoid not synchronizing the state to the standby NameNode > for a long time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
[ https://issues.apache.org/jira/browse/HDFS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845664#comment-17845664 ] ASF GitHub Bot commented on HDFS-17522: --- hiwangzhihui opened a new pull request, #6814: URL: https://github.com/apache/hadoop/pull/6814 …NS protection ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection > -- > > Key: HDFS-17522 > URL: https://issues.apache.org/jira/browse/HDFS-17522 > Project: Hadoop HDFS > Issue Type: Bug > Components: journal-node >Affects Versions: 3.0.0-alpha1 >Reporter: wangzhihui >Priority: Major > > [HDFS-10579 |https://issues.apache.org/jira/browse/HDFS-10579] has added > protection for NameNode and DataNode, but missing protection for JournalNode > web interfaces. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
[ https://issues.apache.org/jira/browse/HDFS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17522: -- Labels: pull-request-available (was: ) > JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection > -- > > Key: HDFS-17522 > URL: https://issues.apache.org/jira/browse/HDFS-17522 > Project: Hadoop HDFS > Issue Type: Bug > Components: journal-node >Affects Versions: 3.0.0-alpha1 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > > [HDFS-10579 |https://issues.apache.org/jira/browse/HDFS-10579] has added > protection for NameNode and DataNode, but missing protection for JournalNode > web interfaces. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
[ https://issues.apache.org/jira/browse/HDFS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhihui updated HDFS-17522: -- Description: [HDFS-10579 |https://issues.apache.org/jira/browse/HDFS-10579] has added protection for NameNode and DataNode, but missing protection for JournalNode web interfaces. > JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection > -- > > Key: HDFS-17522 > URL: https://issues.apache.org/jira/browse/HDFS-17522 > Project: Hadoop HDFS > Issue Type: Bug > Components: journal-node >Affects Versions: 3.0.0-alpha1 >Reporter: wangzhihui >Priority: Major > > [HDFS-10579 |https://issues.apache.org/jira/browse/HDFS-10579] has added > protection for NameNode and DataNode, but missing protection for JournalNode > web interfaces. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
wangzhihui created HDFS-17522: - Summary: JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection Key: HDFS-17522 URL: https://issues.apache.org/jira/browse/HDFS-17522 Project: Hadoop HDFS Issue Type: Bug Components: journal-node Affects Versions: 3.0.0-alpha1 Reporter: wangzhihui -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org