[GitHub] [hadoop] Hexiaoqiao commented on pull request #5330: HDFS-16898. Make write lock fine-grain in method processCommandFromActor

2023-02-02 Thread via GitHub
Hexiaoqiao commented on PR #5330: URL: https://github.com/apache/hadoop/pull/5330#issuecomment-1415226805 @hfutatzhanghb Thanks for your works here. Totally agree issue it caused, however I am concerned about if it is safe to move process command out of write lock. do you have any lock

[GitHub] [hadoop] hfutatzhanghb commented on a diff in pull request #5330: HDFS-16898. Make write lock fine-grain in method processCommandFromActor

2023-02-02 Thread via GitHub
hfutatzhanghb commented on code in PR #5330: URL: https://github.com/apache/hadoop/pull/5330#discussion_r1095415826 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java: ## @@ -679,16 +679,19 @@ boolean

[jira] [Commented] (HADOOP-18616) Java 11 JavaDoc fails due to missing package comments

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683691#comment-17683691 ] ASF GitHub Bot commented on HADOOP-18616: - hadoop-yetus commented on PR #5344: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5344: HADOOP-18616. Java 11 JavaDoc fails due to missing package comments

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5344: URL: https://github.com/apache/hadoop/pull/5344#issuecomment-1415091485 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5326: YARN-11425. [Federation] Router Supports SubClusterCleaner.

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5326: URL: https://github.com/apache/hadoop/pull/5326#issuecomment-1415055787 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5326: YARN-11425. [Federation] Router Supports SubClusterCleaner.

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5326: URL: https://github.com/apache/hadoop/pull/5326#issuecomment-1414990116 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5309: YARN-11419. Remove redundant exception capture in NMClientAsyncImpl and improve readability in ContainerShellWebSocket, etc

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5309: URL: https://github.com/apache/hadoop/pull/5309#issuecomment-1414980064 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5324: HDFS-16895. NamenodeHeartbeatService should use credentials of logged…

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5324: URL: https://github.com/apache/hadoop/pull/5324#issuecomment-1414955456 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5276: HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5276: URL: https://github.com/apache/hadoop/pull/5276#issuecomment-1414946803 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5276: HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5276: URL: https://github.com/apache/hadoop/pull/5276#issuecomment-1414944176 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5147: HDFS-16848. RBF: Improve StateStoreZooKeeperImpl performance

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5147: URL: https://github.com/apache/hadoop/pull/5147#issuecomment-1414923604 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4435: YARN-11178. Avoid CPU busy idling and resource wasting in DelegationTokenRenewerPoolTracker thread

2023-02-02 Thread via GitHub
slfan1989 commented on code in PR #4435: URL: https://github.com/apache/hadoop/pull/4435#discussion_r1095306416 ##

[GitHub] [hadoop] slfan1989 commented on pull request #4435: YARN-11178. Avoid CPU busy idling and resource wasting in DelegationTokenRenewerPoolTracker thread

2023-02-02 Thread via GitHub
slfan1989 commented on PR #4435: URL: https://github.com/apache/hadoop/pull/4435#issuecomment-1414736328 @LennonChin Thank you for your contribution! Is the unit test error caused by this pr? -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hadoop] hfutatzhanghb commented on pull request #5330: HDFS-16898. Make write lock fine-grain in method processCommandFromActor

2023-02-02 Thread via GitHub
hfutatzhanghb commented on PR #5330: URL: https://github.com/apache/hadoop/pull/5330#issuecomment-1414734286 > Thanks for the PR @hfutatzhanghb. Curious if you have any thread dumps or logs collected (before coming to this conclusion) and would like to share reg the issue. hi,

[jira] [Updated] (HADOOP-18616) Java 11 JavaDoc fails due to missing package comments

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18616: Labels: pull-request-available (was: ) > Java 11 JavaDoc fails due to missing package

[jira] [Commented] (HADOOP-18616) Java 11 JavaDoc fails due to missing package comments

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683631#comment-17683631 ] ASF GitHub Bot commented on HADOOP-18616: - snmvaughan opened a new pull request, #5344: URL:

[GitHub] [hadoop] snmvaughan opened a new pull request, #5344: HADOOP-18616. Java 11 JavaDoc fails due to missing package comments

2023-02-02 Thread via GitHub
snmvaughan opened a new pull request, #5344: URL: https://github.com/apache/hadoop/pull/5344 ### Description of PR Add JavaDoc comments to package-info.java to avoid errors resulting from the use of Hadoop annotations. ### How was this patch tested? Running JavaDoc

[jira] [Created] (HADOOP-18616) Java 11 JavaDoc fails due to missing package comments

2023-02-02 Thread Steve Vaughan (Jira)
Steve Vaughan created HADOOP-18616: -- Summary: Java 11 JavaDoc fails due to missing package comments Key: HADOOP-18616 URL: https://issues.apache.org/jira/browse/HADOOP-18616 Project: Hadoop Common

[GitHub] [hadoop] hfutatzhanghb commented on pull request #5276: HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath

2023-02-02 Thread via GitHub
hfutatzhanghb commented on PR #5276: URL: https://github.com/apache/hadoop/pull/5276#issuecomment-1414668844 hi, @goiri @tomscut I have changed the two fields from static to non-static, and add getter method to access them. -- This is an automated message from the Apache Git Service. To

[GitHub] [hadoop] howzi commented on a diff in pull request #5147: HDFS-16848. RBF: Improve StateStoreZooKeeperImpl performance

2023-02-02 Thread via GitHub
howzi commented on code in PR #5147: URL: https://github.com/apache/hadoop/pull/5147#discussion_r1095265796 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/store/driver/TestStateStoreZK.java: ## @@ -126,33 +131,75 @@ private void

[GitHub] [hadoop] howzi commented on a diff in pull request #5147: HDFS-16848. RBF: Improve StateStoreZooKeeperImpl performance

2023-02-02 Thread via GitHub
howzi commented on code in PR #5147: URL: https://github.com/apache/hadoop/pull/5147#discussion_r1095265373 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java: ## @@ -239,6 +239,18 @@ public class

[GitHub] [hadoop] tomscut commented on pull request #5334: HDFS-16902. Add Namenode status to BPServiceActor metrics and improve logging in offerservice

2023-02-02 Thread via GitHub
tomscut commented on PR #5334: URL: https://github.com/apache/hadoop/pull/5334#issuecomment-1414582148 Sorry, I didn't notice the problem with the header format until after the merge. I'll pay attention later. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hadoop] tomscut commented on pull request #5334: HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice

2023-02-02 Thread via GitHub
tomscut commented on PR #5334: URL: https://github.com/apache/hadoop/pull/5334#issuecomment-1414578474 Thanks @virajjasani for your contribution! Thanks @liuml07 and @slfan1989 for your review! -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hadoop] tomscut merged pull request #5334: HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice

2023-02-02 Thread via GitHub
tomscut merged PR #5334: URL: https://github.com/apache/hadoop/pull/5334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hadoop] tomscut commented on pull request #5334: HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice

2023-02-02 Thread via GitHub
tomscut commented on PR #5334: URL: https://github.com/apache/hadoop/pull/5334#issuecomment-1414574640 The failed unit tests are unrelated to the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop] virajjasani commented on a diff in pull request #5330: HDFS-16898. Make write lock fine-grain in method processCommandFromActor

2023-02-02 Thread via GitHub
virajjasani commented on code in PR #5330: URL: https://github.com/apache/hadoop/pull/5330#discussion_r1095229521 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java: ## @@ -679,16 +679,19 @@ boolean

[GitHub] [hadoop] virajjasani commented on pull request #5330: HDFS-16898. Make write lock fine-grain in method processCommandFromActor

2023-02-02 Thread via GitHub
virajjasani commented on PR #5330: URL: https://github.com/apache/hadoop/pull/5330#issuecomment-1414550630 Thanks for the PR @hfutatzhanghb. Curious if you have any thread dumps or logs collected (before coming to this conclusion) and would like to share reg the issue. -- This is an

[GitHub] [hadoop] hadoop-yetus commented on pull request #5342: HDFS-16904. Close webhdfs during the teardown

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5342: URL: https://github.com/apache/hadoop/pull/5342#issuecomment-1414549975 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Comment Edited] (HADOOP-18615) CryptoOutputStream::close leak when encrypted zones + quota exceptions

2023-02-02 Thread Colm Dougan (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683586#comment-17683586 ] Colm Dougan edited comment on HADOOP-18615 at 2/2/23 11:07 PM: --- PS - *

[jira] [Updated] (HADOOP-18615) CryptoOutputStream::close leak when encrypted zones + quota exceptions

2023-02-02 Thread Colm Dougan (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colm Dougan updated HADOOP-18615: - Description: {color:#172b4d}I would like to report an issue with a resource leak

[jira] [Updated] (HADOOP-18615) CryptoOutputStream::close leak when encrypted zones + quota exceptions

2023-02-02 Thread Colm Dougan (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colm Dougan updated HADOOP-18615: - Description: {color:#172b4d}I would like to report an issue with a resource leak

[jira] [Commented] (HADOOP-18615) CryptoOutputStream::close leak when encrypted zones + quota exceptions

2023-02-02 Thread Colm Dougan (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683586#comment-17683586 ] Colm Dougan commented on HADOOP-18615: -- PS - I was unsure how priorities are assigned so I

[jira] [Created] (HADOOP-18615) CryptoOutputStream::close leak when encrypted zones + quota exceptions

2023-02-02 Thread Colm Dougan (Jira)
Colm Dougan created HADOOP-18615: Summary: CryptoOutputStream::close leak when encrypted zones + quota exceptions Key: HADOOP-18615 URL: https://issues.apache.org/jira/browse/HADOOP-18615 Project:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5329: HDFS-16897. fix abundant Broken pipe exception in BlockSender

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5329: URL: https://github.com/apache/hadoop/pull/5329#issuecomment-1414384071 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18614) Ensure that the config writers are closed

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683567#comment-17683567 ] ASF GitHub Bot commented on HADOOP-18614: - hadoop-yetus commented on PR #5341: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5341: HADOOP-18614. Ensure that the config writers are closed

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5341: URL: https://github.com/apache/hadoop/pull/5341#issuecomment-1414381460 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5343: URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1414249776 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18612) Avoid mixing canonical and non-canonical when performing comparisons

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683550#comment-17683550 ] ASF GitHub Bot commented on HADOOP-18612: - hadoop-yetus commented on PR #5339: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5339: HADOOP-18612. Avoid mixing canonical and non-canonical when performing comparisons

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5339: URL: https://github.com/apache/hadoop/pull/5339#issuecomment-1414246965 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683549#comment-17683549 ] ASF GitHub Bot commented on HADOOP-18206: - virajjasani commented on PR #5315: URL:

[GitHub] [hadoop] virajjasani commented on pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future

2023-02-02 Thread via GitHub
virajjasani commented on PR #5315: URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1414240581 @steveloughran @jojochuang @Apache9 I have addressed all review comments so far. For the couple of IA.Public/IS.Evolving methods, instead of removing, I kept them with `@Deprecated`,

[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683546#comment-17683546 ] ASF GitHub Bot commented on HADOOP-18206: - virajjasani commented on code in PR #5315: URL:

[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future

2023-02-02 Thread via GitHub
virajjasani commented on code in PR #5315: URL: https://github.com/apache/hadoop/pull/5315#discussion_r1094964986 ## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/log/LogLevel.java: ## @@ -340,22 +337,14 @@ public void doGet(HttpServletRequest request,

[jira] [Commented] (HADOOP-18542) Azure Token provider requires tenant and client IDs despite being optional

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683544#comment-17683544 ] ASF GitHub Bot commented on HADOOP-18542: - steveloughran commented on PR #4262: URL:

[GitHub] [hadoop] steveloughran commented on pull request #4262: HADOOP-18542. Keep MSI tenant ID and client ID optional

2023-02-02 Thread via GitHub
steveloughran commented on PR #4262: URL: https://github.com/apache/hadoop/pull/4262#issuecomment-1414233381 that test_120_terasort(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort failure is happening because yarn wants the owner of a file as determined from getFileStatus to match

[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683542#comment-17683542 ] ASF GitHub Bot commented on HADOOP-18206: - virajjasani commented on code in PR #5315: URL:

[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future

2023-02-02 Thread via GitHub
virajjasani commented on code in PR #5315: URL: https://github.com/apache/hadoop/pull/5315#discussion_r1094962403 ## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceOperations.java: ## @@ -67,26 +66,6 @@ public static Exception

[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683541#comment-17683541 ] ASF GitHub Bot commented on HADOOP-18206: - virajjasani commented on code in PR #5315: URL:

[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future

2023-02-02 Thread via GitHub
virajjasani commented on code in PR #5315: URL: https://github.com/apache/hadoop/pull/5315#discussion_r1094072017 ## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceOperations.java: ## @@ -67,26 +66,6 @@ public static Exception

[jira] [Commented] (HADOOP-18602) Remove netty3 dependency

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683534#comment-17683534 ] ASF GitHub Bot commented on HADOOP-18602: - steveloughran commented on PR #5323: URL:

[jira] [Commented] (HADOOP-18351) S3A prefetching: Error logging during reads

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683533#comment-17683533 ] ASF GitHub Bot commented on HADOOP-18351: - steveloughran commented on code in PR #5274: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5323: HADOOP-18602. Remove netty3 dependency

2023-02-02 Thread via GitHub
steveloughran commented on PR #5323: URL: https://github.com/apache/hadoop/pull/5323#issuecomment-1414216443 thanks. looks like a useful but big backport -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5274: HADOOP-18351. Reduce excess logging of errors during S3A prefetching reads

2023-02-02 Thread via GitHub
steveloughran commented on code in PR #5274: URL: https://github.com/apache/hadoop/pull/5274#discussion_r1094943230 ## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/CachingBlockManager.java: ## @@ -302,7 +302,13 @@ public void

[GitHub] [hadoop] cnauroth commented on a diff in pull request #5335: YARN-11426. Improve YARN NodeLabel Memory Display.

2023-02-02 Thread via GitHub
cnauroth commented on code in PR #5335: URL: https://github.com/apache/hadoop/pull/5335#discussion_r1094945678 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/api/records/TestResource.java: ## @@ -42,4 +42,70 @@ void testCastToIntSafely()

[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683527#comment-17683527 ] ASF GitHub Bot commented on HADOOP-18596: - steveloughran commented on code in PR #5308: URL:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.

2023-02-02 Thread via GitHub
steveloughran commented on code in PR #5308: URL: https://github.com/apache/hadoop/pull/5308#discussion_r1094941352 ## hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm: ## @@ -631,14 +631,39 @@ hadoop distcp -update -numListstatusThreads 20 \ Because object stores

[jira] [Resolved] (HADOOP-18246) Remove lower limit on s3a prefetching/caching block size

2023-02-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18246. - Fix Version/s: 3.4.0 Resolution: Fixed > Remove lower limit on s3a

[jira] [Commented] (HADOOP-18246) Remove lower limit on s3a prefetching/caching block size

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683523#comment-17683523 ] ASF GitHub Bot commented on HADOOP-18246: - steveloughran merged PR #5120: URL:

[GitHub] [hadoop] steveloughran merged pull request #5120: HADOOP-18246. Reduce lower limit on s3a prefetching/caching block size to 1 byte.

2023-02-02 Thread via GitHub
steveloughran merged PR #5120: URL: https://github.com/apache/hadoop/pull/5120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5340: MAPREDUCE-7416. Upgrade Junit 4 to 5 in hadoop-mapreduce-client-shuffle

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5340: URL: https://github.com/apache/hadoop/pull/5340#issuecomment-1414179639 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] snmvaughan opened a new pull request, #5343: HDFS-16905. Provide default hadoop.log.dir for tests

2023-02-02 Thread via GitHub
snmvaughan opened a new pull request, #5343: URL: https://github.com/apache/hadoop/pull/5343 ### Description of PR Allow testing in other environments by providing a default setting for hadoop.log.dir ### How was this patch tested? Tested using the Hadoop development

[GitHub] [hadoop] snmvaughan opened a new pull request, #5342: HDFS-16904. Close webhdfs during the teardown

2023-02-02 Thread via GitHub
snmvaughan opened a new pull request, #5342: URL: https://github.com/apache/hadoop/pull/5342 ### Description of PR The teardown for the tests shutdown the cluster, but leaves HDFS open. ### How was this patch tested? Tested using the Hadoop development environment Docker

[jira] [Updated] (HADOOP-18614) Ensure that the config writers are closed

2023-02-02 Thread Steve Vaughan (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Vaughan updated HADOOP-18614: --- Summary: Ensure that the config writers are closed (was: Ensure that the config writer

[jira] [Commented] (HADOOP-18614) Ensure that the config writer are closed

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683494#comment-17683494 ] ASF GitHub Bot commented on HADOOP-18614: - snmvaughan opened a new pull request, #5341: URL:

[jira] [Updated] (HADOOP-18614) Ensure that the config writer are closed

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18614: Labels: pull-request-available (was: ) > Ensure that the config writer are closed >

[GitHub] [hadoop] snmvaughan opened a new pull request, #5341: HADOOP-18614. Ensure that the config writer are closed

2023-02-02 Thread via GitHub
snmvaughan opened a new pull request, #5341: URL: https://github.com/apache/hadoop/pull/5341 ### Description of PR Use AutoCloseable to ensure that the config writers are closed between tests. ### How was this patch tested? Tested using the Hadoop development environment

[jira] [Created] (HADOOP-18614) Ensure that the config writer are closed

2023-02-02 Thread Steve Vaughan (Jira)
Steve Vaughan created HADOOP-18614: -- Summary: Ensure that the config writer are closed Key: HADOOP-18614 URL: https://issues.apache.org/jira/browse/HADOOP-18614 Project: Hadoop Common Issue

[GitHub] [hadoop] ashutoshcipher opened a new pull request, #5340: MAPREDUCE-7416. Upgrade Junit 4 to 5 in hadoop-mapreduce-client-shuffle

2023-02-02 Thread via GitHub
ashutoshcipher opened a new pull request, #5340: URL: https://github.com/apache/hadoop/pull/5340 ### Description of PR Upgrade Junit 4 to 5 in hadoop-mapreduce-client-shuffle JIRA - MAPREDUCE-7416 ### For code changes: - [X] Does the title or this PR starts with

[jira] [Created] (HADOOP-18613) Upgrade ZooKeeper to version 3.8.1

2023-02-02 Thread Tamas Penzes (Jira)
Tamas Penzes created HADOOP-18613: - Summary: Upgrade ZooKeeper to version 3.8.1 Key: HADOOP-18613 URL: https://issues.apache.org/jira/browse/HADOOP-18613 Project: Hadoop Common Issue Type:

[jira] [Commented] (HADOOP-18612) Avoid mixing canonical and non-canonical when performing comparisons

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683470#comment-17683470 ] ASF GitHub Bot commented on HADOOP-18612: - snmvaughan opened a new pull request, #5339: URL:

[jira] [Updated] (HADOOP-18612) Avoid mixing canonical and non-canonical when performing comparisons

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18612: Labels: pull-request-available (was: ) > Avoid mixing canonical and non-canonical when

[jira] [Updated] (HADOOP-18612) Avoid mixing canonical and non-canonical when performing comparisons

2023-02-02 Thread Steve Vaughan (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Vaughan updated HADOOP-18612: --- Description: The test mixes canonical and non-canonical paths and then perform

[GitHub] [hadoop] snmvaughan opened a new pull request, #5339: HADOOP-18612. Avoid mixing canonical and non-canonical when performing comparisons

2023-02-02 Thread via GitHub
snmvaughan opened a new pull request, #5339: URL: https://github.com/apache/hadoop/pull/5339 ### Description of PR The test mixes canonical and non-canonical paths and then perform comparisons. We can avoid unexpected failures by ensuring that comparisons are always made against

[jira] [Created] (HADOOP-18612) Avoid mixing canonical and non-canonical when performing comparisons

2023-02-02 Thread Steve Vaughan (Jira)
Steve Vaughan created HADOOP-18612: -- Summary: Avoid mixing canonical and non-canonical when performing comparisons Key: HADOOP-18612 URL: https://issues.apache.org/jira/browse/HADOOP-18612 Project:

[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683457#comment-17683457 ] ASF GitHub Bot commented on HADOOP-18596: - hadoop-yetus commented on PR #5308: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5308: URL: https://github.com/apache/hadoop/pull/5308#issuecomment-1413886297 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18611) AbstractSTestS3AHugeFiles put request assert is using wrong iostatistic

2023-02-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683401#comment-17683401 ] Steve Loughran commented on HADOOP-18611: - stack {code} [ERROR] Failures: [ERROR]

[jira] [Created] (HADOOP-18611) AbstractSTestS3AHugeFiles put request assert is using wrong iostatistic

2023-02-02 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-18611: --- Summary: AbstractSTestS3AHugeFiles put request assert is using wrong iostatistic Key: HADOOP-18611 URL: https://issues.apache.org/jira/browse/HADOOP-18611

[jira] [Commented] (HADOOP-17451) Intermittent failure of S3A tests which make assertions on statistics/IOStatistics

2023-02-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683394#comment-17683394 ] Steve Loughran commented on HADOOP-17451: - This fix is incorrect as when the stream goes to

[GitHub] [hadoop] hadoop-yetus commented on pull request #5335: YARN-11426. Improve YARN NodeLabel Memory Display.

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5335: URL: https://github.com/apache/hadoop/pull/5335#issuecomment-1413619060 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] steveloughran commented on pull request #4415: MAPREDUCE-7386. Maven parallel builds (skipping tests) fail

2023-02-02 Thread via GitHub
steveloughran commented on PR #4415: URL: https://github.com/apache/hadoop/pull/4415#issuecomment-1413611054 @ndimiduk woo? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hadoop] ndimiduk commented on pull request #4415: MAPREDUCE-7386. Maven parallel builds (skipping tests) fail

2023-02-02 Thread via GitHub
ndimiduk commented on PR #4415: URL: https://github.com/apache/hadoop/pull/4415#issuecomment-1413593119 Woo! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[jira] [Commented] (HADOOP-18246) Remove lower limit on s3a prefetching/caching block size

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683352#comment-17683352 ] ASF GitHub Bot commented on HADOOP-18246: - sauraank commented on PR #5120: URL:

[GitHub] [hadoop] sauraank commented on pull request #5120: HADOOP-18246. Reduce lower limit on s3a prefetching/caching block size to 1 byte.

2023-02-02 Thread via GitHub
sauraank commented on PR #5120: URL: https://github.com/apache/hadoop/pull/5120#issuecomment-1413577047 Hey @steveloughran , please review this PR about reducing the lower limit on prefetching/caching block size. Reduced it to 1 byte. Thanks. -- This is an automated message from the

[jira] [Commented] (HADOOP-18351) S3A prefetching: Error logging during reads

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683351#comment-17683351 ] ASF GitHub Bot commented on HADOOP-18351: - sauraank commented on PR #5274: URL:

[GitHub] [hadoop] sauraank commented on pull request #5274: HADOOP-18351. Reduce excess logging of errors during S3A prefetching reads

2023-02-02 Thread via GitHub
sauraank commented on PR #5274: URL: https://github.com/apache/hadoop/pull/5274#issuecomment-1413574674 Hey @steveloughran ! Can you please review it. Tested the logging of Errors by throwing errors in case of both prefetch and read. Thanks -- This is an automated message from the

[GitHub] [hadoop] ZanderXu commented on a diff in pull request #5147: HDFS-16848. RBF: Improve StateStoreZooKeeperImpl performance

2023-02-02 Thread via GitHub
ZanderXu commented on code in PR #5147: URL: https://github.com/apache/hadoop/pull/5147#discussion_r1094369908 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/store/driver/TestStateStoreZK.java: ## @@ -126,33 +131,75 @@ private

[GitHub] [hadoop] hadoop-yetus commented on pull request #5334: HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5334: URL: https://github.com/apache/hadoop/pull/5334#issuecomment-1413537599 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18610) ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS

2023-02-02 Thread Haifeng Chen (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1768#comment-1768 ] Haifeng Chen commented on HADOOP-18610: --- The preview patch is based Hadoop 3.3.1 we are using. It

[GitHub] [hadoop] hadoop-yetus commented on pull request #5335: YARN-11426. Improve YARN NodeLabel Memory Display.

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5335: URL: https://github.com/apache/hadoop/pull/5335#issuecomment-1413527028 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5338: [HDFS-16903]. Fix javadoc of LightWeightResizableGSet class

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5338: URL: https://github.com/apache/hadoop/pull/5338#issuecomment-1413517328 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5272: YARN-11217. [Federation] Add dumpSchedulerLogs REST APIs for Router.

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5272: URL: https://github.com/apache/hadoop/pull/5272#issuecomment-1413478935 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5334: HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5334: URL: https://github.com/apache/hadoop/pull/5334#issuecomment-1413448254 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5276: HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath

2023-02-02 Thread via GitHub
hadoop-yetus commented on PR #5276: URL: https://github.com/apache/hadoop/pull/5276#issuecomment-1413351259 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.

2023-02-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683260#comment-17683260 ] ASF GitHub Bot commented on HADOOP-18596: - mehakmeet commented on code in PR #5308: URL:

[GitHub] [hadoop] mehakmeet commented on a diff in pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.

2023-02-02 Thread via GitHub
mehakmeet commented on code in PR #5308: URL: https://github.com/apache/hadoop/pull/5308#discussion_r1094170778 ## hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java: ## @@ -50,6 +50,7 @@ import

[jira] [Updated] (HADOOP-18610) ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS

2023-02-02 Thread Haifeng Chen (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HADOOP-18610: -- Attachment: HADOOP-18610-preview.patch > ABFS OAuth2 Token Provider to support Azure