[GitHub] [hadoop] hfutatzhanghb commented on pull request #5937: HDFS-17150. EC: Fix the bug of failed lease recovery.

2023-08-14 Thread via GitHub
hfutatzhanghb commented on PR #5937: URL: https://github.com/apache/hadoop/pull/5937#issuecomment-1678450434 LGTM. +1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hadoop] Hexiaoqiao commented on pull request #5925: HDFS-17141. Optimize the default parameters for the FileUtil.isRegularFile() method

2023-08-14 Thread via GitHub
Hexiaoqiao commented on PR #5925: URL: https://github.com/apache/hadoop/pull/5925#issuecomment-1678411955 > Because the current parameter setting of this method is not optimal Sorry, the core improvement here is that change `isRegularFile(file, true);` to `isRegularFile(file,

[GitHub] [hadoop] Hexiaoqiao commented on pull request #5924: HDFS-17140. Optimize the BPOfferService.reportBadBlocks() method

2023-08-14 Thread via GitHub
Hexiaoqiao commented on PR #5924: URL: https://github.com/apache/hadoop/pull/5924#issuecomment-1678407443 OK, let's wait if any other comments here, if not we could check in this changes. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hadoop] Hexiaoqiao commented on a diff in pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incompl

2023-08-14 Thread via GitHub
Hexiaoqiao commented on code in PR #5855: URL: https://github.com/apache/hadoop/pull/5855#discussion_r1294151058 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java: ## @@ -2957,6 +2957,22 @@ public boolean

[GitHub] [hadoop] Hexiaoqiao commented on a diff in pull request #5941: HDFS-17154. EC: Fix bug in updateBlockForPipeline after failover.

2023-08-14 Thread via GitHub
Hexiaoqiao commented on code in PR #5941: URL: https://github.com/apache/hadoop/pull/5941#discussion_r1294147667 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -5950,8 +5952,26 @@ LocatedBlock

[GitHub] [hadoop] zhangshuyan0 commented on a diff in pull request #5941: HDFS-17154. EC: Fix bug in updateBlockForPipeline after failover.

2023-08-14 Thread via GitHub
zhangshuyan0 commented on code in PR #5941: URL: https://github.com/apache/hadoop/pull/5941#discussion_r1294132630 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -5950,8 +5952,26 @@ LocatedBlock

[GitHub] [hadoop] zhangshuyan0 commented on pull request #5937: HDFS-17150. EC: Fix the bug of failed lease recovery.

2023-08-14 Thread via GitHub
zhangshuyan0 commented on PR #5937: URL: https://github.com/apache/hadoop/pull/5937#issuecomment-1678346266 @Hexiaoqiao @haiyang1987 Some comments have been added. Please take a check when you have free time. -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Commented] (HADOOP-18845) Add ability to configure ConnectionTTL of http connections while creating S3 Client.

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754336#comment-17754336 ] ASF GitHub Bot commented on HADOOP-18845: - hadoop-yetus commented on PR #5948: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5948: HADOOP-18845. Add ability to configure s3 connection ttl

2023-08-14 Thread via GitHub
hadoop-yetus commented on PR #5948: URL: https://github.com/apache/hadoop/pull/5948#issuecomment-1678208383 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5905: [YARN-11421] Graceful Decommission ignores launched containers and gets deactivated before timeout

2023-08-14 Thread via GitHub
hadoop-yetus commented on PR #5905: URL: https://github.com/apache/hadoop/pull/5905#issuecomment-1678177289 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754288#comment-17754288 ] ASF GitHub Bot commented on HADOOP-18820: - mukund-thakur commented on PR #5872: URL:

[GitHub] [hadoop] mukund-thakur commented on pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-08-14 Thread via GitHub
mukund-thakur commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1678172812 S3AFileSystem.getObjectMetadata() is already broken because of the new response type. Any application using this will have to update its code. S3AInternals sounds good to me.

[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754265#comment-17754265 ] ASF GitHub Bot commented on HADOOP-18184: - hadoop-yetus commented on PR #5832: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.

2023-08-14 Thread via GitHub
hadoop-yetus commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1678098374 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18845) Add ability to configure ConnectionTTL of http connections while creating S3 Client.

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754264#comment-17754264 ] ASF GitHub Bot commented on HADOOP-18845: - mukund-thakur commented on PR #5948: URL:

[GitHub] [hadoop] mukund-thakur commented on pull request #5948: HADOOP-18845. Add ability to configure s3 connection ttl

2023-08-14 Thread via GitHub
mukund-thakur commented on PR #5948: URL: https://github.com/apache/hadoop/pull/5948#issuecomment-1678080422 Ran tests again us-west-1. All good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[jira] [Updated] (HADOOP-18845) Add ability to configure ConnectionTTL of http connections while creating S3 Client.

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18845: Labels: pull-request-available (was: ) > Add ability to configure ConnectionTTL of http

[jira] [Commented] (HADOOP-18845) Add ability to configure ConnectionTTL of http connections while creating S3 Client.

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754257#comment-17754257 ] ASF GitHub Bot commented on HADOOP-18845: - mukund-thakur opened a new pull request, #5948: URL:

[GitHub] [hadoop] mukund-thakur opened a new pull request, #5948: HADOOP-18845. Add ability to configure s3 connection ttl

2023-08-14 Thread via GitHub
mukund-thakur opened a new pull request, #5948: URL: https://github.com/apache/hadoop/pull/5948 ### Description of PR Introducing fs.s3a.connection.ttl configuration which can be configured to expire old HTTP connection after specified period of time. ### How was this patch

[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754231#comment-17754231 ] ASF GitHub Bot commented on HADOOP-18184: - virajjasani commented on PR #5832: URL:

[GitHub] [hadoop] virajjasani commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.

2023-08-14 Thread via GitHub
virajjasani commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1677902355 > also a timeout in ITestS3APrefetchingLruEviction even after including this commit https://github.com/apache/hadoop/commit/90793e1bce58e0d12234d080a26515d8a81d82ed ? --

[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754225#comment-17754225 ] ASF GitHub Bot commented on HADOOP-18184: - steveloughran commented on PR #5832: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.

2023-08-14 Thread via GitHub
steveloughran commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1677888683 testing: some failures. also a timeout in ITestS3APrefetchingLruEviction which I think shows a test in need of some tuning -- This is an automated message from the Apache Git

[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754223#comment-17754223 ] ASF GitHub Bot commented on HADOOP-18184: - steveloughran commented on code in PR #5832: URL:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.

2023-08-14 Thread via GitHub
steveloughran commented on code in PR #5832: URL: https://github.com/apache/hadoop/pull/5832#discussion_r1293843650 ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/prefetch/ITestS3APrefetchingCacheFiles.java: ## @@ -0,0 +1,173 @@ +/* + * Licensed to the Apache

[GitHub] [hadoop] hadoop-yetus commented on pull request #5332: YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup

2023-08-14 Thread via GitHub
hadoop-yetus commented on PR #5332: URL: https://github.com/apache/hadoop/pull/5332#issuecomment-1677697519 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18154) S3A Authentication to support WebIdentity

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754161#comment-17754161 ] ASF GitHub Bot commented on HADOOP-18154: - boblozano commented on PR #4070: URL:

[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754153#comment-17754153 ] ASF GitHub Bot commented on HADOOP-18184: - steveloughran commented on code in PR #5832: URL:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.

2023-08-14 Thread via GitHub
steveloughran commented on code in PR #5832: URL: https://github.com/apache/hadoop/pull/5832#discussion_r1293641249 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3ACachingInputStream.java: ## @@ -76,36 +79,75 @@ public S3ACachingInputStream(

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754149#comment-17754149 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on PR #5872: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-08-14 Thread via GitHub
steveloughran commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1677551297 HADOOP-18820. SDK v1 cut: S3AInternals move "public" s3 client api calls out of S3AFS and into a new S3AInternals interface which is implemented by a non-static inner class.

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754106#comment-17754106 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on PR #5872: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-08-14 Thread via GitHub
steveloughran commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1677351774 one more "little" bit of redesign. i want to move those entry points we expose from s3afs (client, credentials, getObjectMetadata() into a new S3AInternals API, which is what you

[GitHub] [hadoop] hadoop-yetus commented on pull request #5947: HDFS-17158. Show the rate of metrics in EC recovery task.

2023-08-14 Thread via GitHub
hadoop-yetus commented on PR #5947: URL: https://github.com/apache/hadoop/pull/5947#issuecomment-1677283353 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] slfan1989 commented on pull request #5515: YARN-11037. Add configurable logic to split resource request to the least loaded SC.

2023-08-14 Thread via GitHub
slfan1989 commented on PR #5515: URL: https://github.com/apache/hadoop/pull/5515#issuecomment-1677246013 @goiri Can you help review this PR again? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [hadoop] slfan1989 commented on pull request #5945: YARN-10218. [GPG] Support HTTPS in GPG.

2023-08-14 Thread via GitHub
slfan1989 commented on PR #5945: URL: https://github.com/apache/hadoop/pull/5945#issuecomment-1677246944 @goiri Can you help review this pr? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[jira] [Commented] (HADOOP-18393) Hadoop 3.3.2 has CVEs coming from dependencies

2023-08-14 Thread Xiping Zhang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754045#comment-17754045 ] Xiping Zhang commented on HADOOP-18393: --- [~ste...@apache.org] That's okay. Thank you for your

[jira] [Commented] (HADOOP-18393) Hadoop 3.3.2 has CVEs coming from dependencies

2023-08-14 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754036#comment-17754036 ] Steve Loughran commented on HADOOP-18393: - don't recall, sorry. > Hadoop 3.3.2 has CVEs

[GitHub] [hadoop] haiyang1987 commented on a diff in pull request #5937: HDFS-17150. EC: Fix the bug of failed lease recovery.

2023-08-14 Thread via GitHub
haiyang1987 commented on code in PR #5937: URL: https://github.com/apache/hadoop/pull/5937#discussion_r1293312263 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -3802,16 +3803,26 @@ boolean

[GitHub] [hadoop] hadoop-yetus commented on pull request #5930: HDFS-17143. Optimize the logic for reconfigure ReadStrategy enable for Namenode.

2023-08-14 Thread via GitHub
hadoop-yetus commented on PR #5930: URL: https://github.com/apache/hadoop/pull/5930#issuecomment-1677017162 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18393) Hadoop 3.3.2 has CVEs coming from dependencies

2023-08-14 Thread Xiping Zhang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753965#comment-17753965 ] Xiping Zhang commented on HADOOP-18393: --- [~ste...@apache.org] Excuse me, for CVE-2021-33036 ,

[GitHub] [hadoop] hadoop-yetus commented on pull request #5515: YARN-11037. Add configurable logic to split resource request to the least loaded SC.

2023-08-14 Thread via GitHub
hadoop-yetus commented on PR #5515: URL: https://github.com/apache/hadoop/pull/5515#issuecomment-1676904345 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] huangzhaobo99 commented on pull request #5930: HDFS-17143. Optimize the logic for reconfigure ReadStrategy enable for Namenode.

2023-08-14 Thread via GitHub
huangzhaobo99 commented on PR #5930: URL: https://github.com/apache/hadoop/pull/5930#issuecomment-1676897661 Hi @slfan1989, Could you help review this when you have time? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hadoop] hadoop-yetus commented on pull request #5944: YARN-11537. [Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues.

2023-08-14 Thread via GitHub
hadoop-yetus commented on PR #5944: URL: https://github.com/apache/hadoop/pull/5944#issuecomment-1676891644 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] Tre2878 commented on pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block rep

2023-08-14 Thread via GitHub
Tre2878 commented on PR #5855: URL: https://github.com/apache/hadoop/pull/5855#issuecomment-1676829536 @Hexiaoqiao Thank you for your patient guidance. Now All checks have passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hadoop] Hexiaoqiao commented on a diff in pull request #5941: HDFS-17154. EC: Fix bug in updateBlockForPipeline after failover.

2023-08-14 Thread via GitHub
Hexiaoqiao commented on code in PR #5941: URL: https://github.com/apache/hadoop/pull/5941#discussion_r1293059482 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -5950,8 +5952,26 @@ LocatedBlock

[GitHub] [hadoop] zhangshuyan0 commented on a diff in pull request #5937: HDFS-17150. EC: Fix the bug of failed lease recovery.

2023-08-14 Thread via GitHub
zhangshuyan0 commented on code in PR #5937: URL: https://github.com/apache/hadoop/pull/5937#discussion_r1293027160 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -3802,16 +3803,26 @@ boolean

[GitHub] [hadoop] YuanbenWang opened a new pull request, #5947: HDFS-17158. Show the rate of metrics in EC recovery task.

2023-08-14 Thread via GitHub
YuanbenWang opened a new pull request, #5947: URL: https://github.com/apache/hadoop/pull/5947 JIRA: [HDFS-17158](https://issues.apache.org/jira/browse/HDFS-17158). Show the rate of metrics in EC recovery task. -- This is an automated message from the Apache Git Service. To respond to the