Re: [PR] YARN-11191. Fix potentional deadlock in GlobalScheduler refreshQueues… [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6768: URL: https://github.com/apache/hadoop/pull/6768#issuecomment-2081364250 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] YARN-11685. Create a config to enable/disable cgroup v2 functionality [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6770: URL: https://github.com/apache/hadoop/pull/6770#issuecomment-2081384515 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | -1 :x: | patch

[jira] [Commented] (HADOOP-19156) ZooKeeper based state stores use different ZK address configs

2024-04-28 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841638#comment-17841638 ] ASF GitHub Bot commented on HADOOP-19156: - hadoop-yetus commented on PR #6767: URL:

Re: [PR] HADOOP-19156. ZooKeeper based state stores use different ZK address c… [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6767: URL: https://github.com/apache/hadoop/pull/6767#issuecomment-2081478842 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HDFS-17500: Add missing operation name while authorizing create and completeFile operations [hadoop]

2024-04-28 Thread via GitHub
Hexiaoqiao commented on PR #6776: URL: https://github.com/apache/hadoop/pull/6776#issuecomment-2081485427 #1829 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] HDFS-17484. Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually. [hadoop]

2024-04-28 Thread via GitHub
Hexiaoqiao commented on PR #6758: URL: https://github.com/apache/hadoop/pull/6758#issuecomment-2081503955 How about turn off `dfs.namenode.redundancy.considerLoad` or turn up `dfs.namenode.redundancy.considerLoad.factor`. Thanks. -- This is an automated message from the Apache Git

[jira] [Commented] (HADOOP-19159) Fix hadoop-aws document for fs.s3a.committer.abort.pending.uploads

2024-04-28 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841654#comment-17841654 ] ASF GitHub Bot commented on HADOOP-19159: - hadoop-yetus commented on PR #6778: URL:

Re: [PR] HADOOP-19159. Fix document for fs.s3a.committer.abort.pending.uploads [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6778: URL: https://github.com/apache/hadoop/pull/6778#issuecomment-2081508572 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HADOOP-19156. ZooKeeper based state stores use different ZK address c… [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6767: URL: https://github.com/apache/hadoop/pull/6767#issuecomment-2081429516 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-19156) ZooKeeper based state stores use different ZK address configs

2024-04-28 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841628#comment-17841628 ] ASF GitHub Bot commented on HADOOP-19156: - hadoop-yetus commented on PR #6767: URL:

Re: [PR] HDFS-17502. Adjust the log format of the printStatistics() in FSEditLog.java [hadoop]

2024-04-28 Thread via GitHub
Hexiaoqiao closed pull request #6777: HDFS-17502. Adjust the log format of the printStatistics() in FSEditLog.java URL: https://github.com/apache/hadoop/pull/6777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] HDFS-17502. Adjust the log format of the printStatistics() in FSEditLog.java [hadoop]

2024-04-28 Thread via GitHub
Hexiaoqiao commented on PR #6777: URL: https://github.com/apache/hadoop/pull/6777#issuecomment-2081481103 Thanks @yuw1 for your contribution. Sorry, -1 from my side. Here we change the blank space to comma here, another PR change back to period, which is meaningless but cost resources.

Re: [PR] HDFS-17452 : DfsRouterAdmin RefreshCallQueue fails when authorization is enabled [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6779: URL: https://github.com/apache/hadoop/pull/6779#issuecomment-2081570914 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] HADOOP-19155. Fix TestZKSignerSecretProvider failing unit test [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6766: URL: https://github.com/apache/hadoop/pull/6766#issuecomment-2081601725 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HDFS-17497. The number of bytes of the last committed block should be calculated into the file length [hadoop]

2024-04-28 Thread via GitHub
Hexiaoqiao commented on PR #6765: URL: https://github.com/apache/hadoop/pull/6765#issuecomment-2081489723 Great catch, not review carefully, but I remember this have been discussed for long time. IIRC, client also check the file length through request DataNode which manage the uncomplete

Re: [PR] HDFS-17488. DN can fail IBRs with NPE when a volume is removed [hadoop]

2024-04-28 Thread via GitHub
kokonguyen191 commented on code in PR #6759: URL: https://github.com/apache/hadoop/pull/6759#discussion_r1582197257 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java: ## @@ -324,6 +325,10 @@ private void

[jira] [Commented] (HADOOP-19158) S3A: Support ByteBufferPositionedReadable through vector IO

2024-04-28 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841672#comment-17841672 ] ASF GitHub Bot commented on HADOOP-19158: - hadoop-yetus commented on PR #6773: URL:

Re: [PR] HADOOP-19158. S3A: Support ByteBufferPositionedReadable through vector IO [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6773: URL: https://github.com/apache/hadoop/pull/6773#issuecomment-2081538670 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HDFS-17500: Add missing operation name while authorizing create and completeFile operations [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6776: URL: https://github.com/apache/hadoop/pull/6776#issuecomment-2081547000 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

[jira] [Commented] (HADOOP-19156) ZooKeeper based state stores use different ZK address configs

2024-04-28 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841692#comment-17841692 ] ASF GitHub Bot commented on HADOOP-19156: - hadoop-yetus commented on PR #6767: URL:

Re: [PR] HDFS-17488. DN can fail IBRs with NPE when a volume is removed [hadoop]

2024-04-28 Thread via GitHub
kokonguyen191 commented on PR #6759: URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2081512148 Hi @Hexiaoqiao, thanks for the review. For your first question, `getPerStorageIBR(storage)` doesn't do a null check on `storage` and assigns a new `PerStorageIBG` object to `null`.

[PR] HDFS-17452 : DfsRouterAdmin RefreshCallQueue fails when authorization is enabled [hadoop]

2024-04-28 Thread via GitHub
AnanyaSingh2121 opened a new pull request, #6779: URL: https://github.com/apache/hadoop/pull/6779 ### Description of PR Adding the kerberos principal key for Router refreshCallQueue command ### How was this patch tested? On a federated hadoop cluster

[jira] [Commented] (HADOOP-19155) Fix TestZKSignerSecretProvider failing unit test

2024-04-28 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841693#comment-17841693 ] ASF GitHub Bot commented on HADOOP-19155: - hadoop-yetus commented on PR #6766: URL:

Re: [PR] HDFS-17502. Adjust the log format of the printStatistics() in FSEditLog.java [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6777: URL: https://github.com/apache/hadoop/pull/6777#issuecomment-2081516325 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HDFS-17501. Add a parameter "redirectByIPAddress" to WebHDFS [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6775: URL: https://github.com/apache/hadoop/pull/6775#issuecomment-2081518185 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HADOOP-19156. ZooKeeper based state stores use different ZK address c… [hadoop]

2024-04-28 Thread via GitHub
Hexiaoqiao commented on PR #6767: URL: https://github.com/apache/hadoop/pull/6767#issuecomment-2081488645 @liubin101 Thanks for your works. Strong +1 from my side. Just one concerns, we should consider to be compatible with the old configuration keys. Thanks. -- This is an automated

[jira] [Commented] (HADOOP-19156) ZooKeeper based state stores use different ZK address configs

2024-04-28 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841645#comment-17841645 ] ASF GitHub Bot commented on HADOOP-19156: - Hexiaoqiao commented on PR #6767: URL:

Re: [PR] HADOOP-19156. ZooKeeper based state stores use different ZK address c… [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6767: URL: https://github.com/apache/hadoop/pull/6767#issuecomment-2081593653 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HDFS-17488. DN can fail IBRs with NPE when a volume is removed [hadoop]

2024-04-28 Thread via GitHub
haiyang1987 commented on code in PR #6759: URL: https://github.com/apache/hadoop/pull/6759#discussion_r1582071262 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java: ## @@ -324,6 +325,10 @@ private void

Re: [PR] HDFS-17464. Improve some logs output in class FsDatasetImpl [hadoop]

2024-04-28 Thread via GitHub
haiyang1987 commented on code in PR #6724: URL: https://github.com/apache/hadoop/pull/6724#discussion_r1582072552 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java: ## @@ -2091,14 +2092,16 @@ public void

[jira] [Updated] (HADOOP-18594) ProxyUserAuthenticationFilter add properties 'hadoop.security.impersonation.provider.class' to enable load custom ImpersonationProvider class when start namenode

2024-04-28 Thread Xie Yi (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xie Yi updated HADOOP-18594: Priority: Major (was: Minor) > ProxyUserAuthenticationFilter add properties >

Re: [PR] HDFS-17464. Improve some logs output in class FsDatasetImpl [hadoop]

2024-04-28 Thread via GitHub
haiyang1987 commented on code in PR #6724: URL: https://github.com/apache/hadoop/pull/6724#discussion_r1582072571 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java: ## @@ -2028,7 +2028,8 @@ private

Re: [PR] HDFS-17464. Improve some logs output in class FsDatasetImpl [hadoop]

2024-04-28 Thread via GitHub
hfutatzhanghb commented on code in PR #6724: URL: https://github.com/apache/hadoop/pull/6724#discussion_r1582077047 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java: ## @@ -2091,14 +2092,16 @@ public void

Re: [PR] HDFS-17488. DN can fail IBRs with NPE when a volume is removed [hadoop]

2024-04-28 Thread via GitHub
Hexiaoqiao commented on PR #6759: URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2081497703 @kokonguyen191 Thanks for your report and contributions. Sorry didn't get this issue completely. As description, you mentioned that `storage == null` as following. I wonder why NPE

Re: [PR] HDFS-17497. The number of bytes of the last committed block should be calculated into the file length [hadoop]

2024-04-28 Thread via GitHub
ZanderXu commented on PR #6765: URL: https://github.com/apache/hadoop/pull/6765#issuecomment-2081766984 > IIRC, client also check the file length through request DataNode which manage the uncomplete block? Like other committed blocks, the client does not need to get the visible

Re: [PR] HDFS-17497. The number of bytes of the last committed block should be calculated into the file length [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6765: URL: https://github.com/apache/hadoop/pull/6765#issuecomment-2081636005 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HADOOP-19150: [ABFS] Fixing Test Code for ITestAbfsRestOperationException#testAuthFailException [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6756: URL: https://github.com/apache/hadoop/pull/6756#issuecomment-2081786711 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

[jira] [Commented] (HADOOP-19150) Test ITestAbfsRestOperationException#testAuthFailException is broken.

2024-04-28 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841810#comment-17841810 ] ASF GitHub Bot commented on HADOOP-19150: - hadoop-yetus commented on PR #6756: URL:

Re: [PR] HDFS-17456. Fix the dfsused statistics of datanode are incorrect when appending a file. [hadoop]

2024-04-28 Thread via GitHub
Hexiaoqiao commented on PR #6713: URL: https://github.com/apache/hadoop/pull/6713#issuecomment-2081850789 @fuchaohong Hi, IIUC the `dfsused` will be refreshed periodically. Do you meet any issues about this metric? Thanks. -- This is an automated message from the Apache Git Service. To

Re: [PR] HDFS-17384. [FGL] Replace the global lock with global FS Lock and global BM lock [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6762: URL: https://github.com/apache/hadoop/pull/6762#issuecomment-2081720138 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HDFS-17488. DN can fail IBRs with NPE when a volume is removed [hadoop]

2024-04-28 Thread via GitHub
hfutatzhanghb commented on PR #6759: URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2081770902 > @kokonguyen191 Thanks for your report and contributions. Sorry didn't get this issue completely. As description, you mentioned that `storage == null` as following. I wonder why

[jira] [Commented] (HADOOP-19156) ZooKeeper based state stores use different ZK address configs

2024-04-28 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841808#comment-17841808 ] ASF GitHub Bot commented on HADOOP-19156: - liubin101 commented on PR #6767: URL:

Re: [PR] HDFS-17488. DN can fail IBRs with NPE when a volume is removed [hadoop]

2024-04-28 Thread via GitHub
kokonguyen191 commented on code in PR #6759: URL: https://github.com/apache/hadoop/pull/6759#discussion_r1582488078 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java: ## @@ -324,6 +325,10 @@ private void

Re: [PR] HADOOP-19156. ZooKeeper based state stores use different ZK address c… [hadoop]

2024-04-28 Thread via GitHub
liubin101 commented on PR #6767: URL: https://github.com/apache/hadoop/pull/6767#issuecomment-2081785275 > @liubin101 Thanks for your works. Strong +1 from my side. Just one concerns, we should consider to be compatible with the old configuration keys. Thanks.

Re: [PR] HDFS-17458. Remove unnecessary BP lock in ReplicaMap. [hadoop]

2024-04-28 Thread via GitHub
Hexiaoqiao commented on PR #6717: URL: https://github.com/apache/hadoop/pull/6717#issuecomment-2081852396 @hfutatzhanghb As discussed above (which is also confirmed by @zhangshuyan0), It is unsafe to remove BP lock here, just close this PR. Please feel free to reopen it if there is one

Re: [PR] HDFS-17458. Remove unnecessary BP lock in ReplicaMap. [hadoop]

2024-04-28 Thread via GitHub
Hexiaoqiao closed pull request #6717: HDFS-17458. Remove unnecessary BP lock in ReplicaMap. URL: https://github.com/apache/hadoop/pull/6717 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] HDFS-13603: do not propagate ExecutionException and add maxRetries li… [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6774: URL: https://github.com/apache/hadoop/pull/6774#issuecomment-2081665080 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] YARN-11675. Update MemoryResourceHandler implementation for cgroup v2 support [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6760: URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2081688017 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HDFS-16993. Datanode supports configure TopN DatanodeNetworkCounts [hadoop]

2024-04-28 Thread via GitHub
huangzhaobo99 commented on code in PR #5597: URL: https://github.com/apache/hadoop/pull/5597#discussion_r1582490565 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java: ## @@ -2630,6 +2631,28 @@ public int

Re: [PR] HDFS-17458. Remove unnecessary BP lock in ReplicaMap. [hadoop]

2024-04-28 Thread via GitHub
hfutatzhanghb commented on PR #6717: URL: https://github.com/apache/hadoop/pull/6717#issuecomment-2081853953 > @hfutatzhanghb As discussed above (which is also confirmed by @zhangshuyan0), It is unsafe to remove BP lock here, just close this PR. Please feel free to reopen it if there is

Re: [PR] HDFS-17496. DataNode supports more fine-grained dataset lock based on blockid. [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6764: URL: https://github.com/apache/hadoop/pull/6764#issuecomment-2081668506 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HDFS-17488. DN can fail IBRs with NPE when a volume is removed [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6759: URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2081754060 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HDFS-17484. Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually. [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6758: URL: https://github.com/apache/hadoop/pull/6758#issuecomment-2081769086 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HDFS-17367. Add PercentUsed for Different StorageTypes in JMX [hadoop]

2024-04-28 Thread via GitHub
zhtttylz commented on PR #6735: URL: https://github.com/apache/hadoop/pull/6735#issuecomment-2081815007 Thanks to @slfan1989 and @haiyang1987 for reviewing and merging this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] YARN-11686. Correct traversing indexs when scheduling asynchronously using Capacity Scheduler [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6750: URL: https://github.com/apache/hadoop/pull/6750#issuecomment-2081901457 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] [DRAFT] Jackson 2.14 [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6761: URL: https://github.com/apache/hadoop/pull/6761#issuecomment-2081883978 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks

Re: [PR] HDFS-17476. fix: False positive "Observer Node is too far behind" due to long overflow. [hadoop]

2024-04-28 Thread via GitHub
hadoop-yetus commented on PR #6747: URL: https://github.com/apache/hadoop/pull/6747#issuecomment-2081933949 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| _ Prechecks