[GitHub] [hadoop] Tre2878 commented on pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block rep

2023-08-17 Thread via GitHub
Tre2878 commented on PR #5855: URL: https://github.com/apache/hadoop/pull/5855#issuecomment-1683193008 @Hexiaoqiao The unit test error should have nothing to do with this change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hadoop] chunyiyang commented on pull request #5951: HDFS-17156. Client may receive old state ID which will lead to inconsistent reads.

2023-08-17 Thread via GitHub
chunyiyang commented on PR #5951: URL: https://github.com/apache/hadoop/pull/5951#issuecomment-1683172861 Thanks for updating the title and merging it! @tasanuma -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[jira] [Commented] (HADOOP-18845) Add ability to configure ConnectionTTL of http connections while creating S3 Client.

2023-08-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755710#comment-17755710 ] ASF GitHub Bot commented on HADOOP-18845: - hadoop-yetus commented on PR #5948: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5948: HADOOP-18845. Add ability to configure s3 connection ttl

2023-08-17 Thread via GitHub
hadoop-yetus commented on PR #5948: URL: https://github.com/apache/hadoop/pull/5948#issuecomment-1683044464 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()

2023-08-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755683#comment-17755683 ] ASF GitHub Bot commented on HADOOP-18184: - hadoop-yetus commented on PR #5832: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.

2023-08-17 Thread via GitHub
hadoop-yetus commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1682858231 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] simbadzina commented on pull request #5951: HDFS-17156. Client may receive old state ID which will lead to inconsistent reads.

2023-08-17 Thread via GitHub
simbadzina commented on PR #5951: URL: https://github.com/apache/hadoop/pull/5951#issuecomment-1682710012 Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hadoop] tasanuma commented on pull request #5951: HDFS-17156. Client may receive old state ID which will lead to inconsistent reads.

2023-08-17 Thread via GitHub
tasanuma commented on PR #5951: URL: https://github.com/apache/hadoop/pull/5951#issuecomment-1682709152 Yes, I just did it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hadoop] simbadzina commented on pull request #5951: HDFS-17156. Client may receive old state ID which will lead to inconsistent reads.

2023-08-17 Thread via GitHub
simbadzina commented on PR #5951: URL: https://github.com/apache/hadoop/pull/5951#issuecomment-1682701064 Thank you. @tasanuma could you please also backport this into branch-3.3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hadoop] tasanuma commented on pull request #5951: HDFS-17156. Client may receive old state ID which will lead to inconsistent reads.

2023-08-17 Thread via GitHub
tasanuma commented on PR #5951: URL: https://github.com/apache/hadoop/pull/5951#issuecomment-1682643278 Merged. Thanks for your contribution, @chunyiyang! Thanks for your review and providing the unit test, @simbadzina! -- This is an automated message from the Apache Git Service. To

[GitHub] [hadoop] tasanuma merged pull request #5951: HDFS-17156. Client may receive old state ID which will lead to inconsistent reads.

2023-08-17 Thread via GitHub
tasanuma merged PR #5951: URL: https://github.com/apache/hadoop/pull/5951 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5958: YARN-11551. RM format-conf-store should delete all the content of ZKConfigStore

2023-08-17 Thread via GitHub
hadoop-yetus commented on PR #5958: URL: https://github.com/apache/hadoop/pull/5958#issuecomment-1682627104 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18531) assertion failure in ITestS3APrefetchingInputStream

2023-08-17 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755627#comment-17755627 ] Steve Loughran commented on HADOOP-18531: - aaha, seen this again nearby {code}

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-08-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755625#comment-17755625 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on PR #5872: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-08-17 Thread via GitHub
steveloughran commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1682615704 @mukund-thakur @mehakmeet you happy with this? we are, and once it's in I'm ready to rebase and merge to trunk -- This is an automated message from the Apache Git Service. To

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-08-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755622#comment-17755622 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on PR #5872: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-08-17 Thread via GitHub
steveloughran commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1682611004 tested s3 london at scale; got the usual huge file multipart failure. turn off for now? also saw a transitive failure of ITestS3APrefetchingInputStream; seen that before.

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-08-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755620#comment-17755620 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on PR #5872: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-08-17 Thread via GitHub
steveloughran commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1682609014 OK, I added the setConf() feature. Means thing is potentially useful. V1 handlers get an error, as seen in the test ``` 2023-08-17 17:24:42,849 [setup] WARN

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-08-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755612#comment-17755612 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on PR #5872: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-08-17 Thread via GitHub
steveloughran commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1682558300 latest patch removed the redundant method *and* updates the auditing docs to cover the migration to the new option, plus any other updates needed. Note: storediag updated to print

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-08-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755611#comment-17755611 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on code in PR #5872: URL:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-08-17 Thread via GitHub
steveloughran commented on code in PR #5872: URL: https://github.com/apache/hadoop/pull/5872#discussion_r1297436980 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/V2Migration.java: ## @@ -28,83 +28,47 @@ /** * This class provides utility methods

[jira] [Commented] (HADOOP-18328) S3A supports S3 on Outposts

2023-08-17 Thread Yuting Chen (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755606#comment-17755606 ] Yuting Chen commented on HADOOP-18328: -- I ran all integration tests with  *mvn -Dparallel-tests

[GitHub] [hadoop] hadoop-yetus commented on pull request #5332: YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup

2023-08-17 Thread via GitHub
hadoop-yetus commented on PR #5332: URL: https://github.com/apache/hadoop/pull/5332#issuecomment-1682463928 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-08-17 Thread via GitHub
steveloughran commented on code in PR #5872: URL: https://github.com/apache/hadoop/pull/5872#discussion_r1297276732 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/V2Migration.java: ## @@ -28,83 +28,47 @@ /** * This class provides utility methods

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-08-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755568#comment-17755568 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on code in PR #5872: URL:

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-08-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755566#comment-17755566 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on code in PR #5872: URL:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-08-17 Thread via GitHub
steveloughran commented on code in PR #5872: URL: https://github.com/apache/hadoop/pull/5872#discussion_r1297275400 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java: ## @@ -998,7 +999,7 @@ private void bindAWSClient(URI name, boolean

[GitHub] [hadoop] hadoop-yetus commented on pull request #5888: HDFS-17121. BPServiceActor to provide new thread to handle FBR

2023-08-17 Thread via GitHub
hadoop-yetus commented on PR #5888: URL: https://github.com/apache/hadoop/pull/5888#issuecomment-1682311673 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5888: HDFS-17121. BPServiceActor to provide new thread to handle FBR

2023-08-17 Thread via GitHub
hadoop-yetus commented on PR #5888: URL: https://github.com/apache/hadoop/pull/5888#issuecomment-1682286799 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] ferdelyi commented on pull request #5958: YARN-11551. RM format-conf-store should delete all the content of ZKConfigStore

2023-08-17 Thread via GitHub
ferdelyi commented on PR #5958: URL: https://github.com/apache/hadoop/pull/5958#issuecomment-1682224503 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hadoop] brumi1024 opened a new pull request, #5958: YARN-11551. RM format-conf-store should delete all the content of ZKConfigStore

2023-08-17 Thread via GitHub
brumi1024 opened a new pull request, #5958: URL: https://github.com/apache/hadoop/pull/5958 ### Description of PR To easily overcome the issue mentioned in [YARN-11075](https://issues.apache.org/jira/browse/YARN-11075) format-conf-store should delete everything under

[GitHub] [hadoop] hadoop-yetus commented on pull request #5956: HDFS-17161: Adding test for StripedBlockReader#createBlockReader leak…

2023-08-17 Thread via GitHub
hadoop-yetus commented on PR #5956: URL: https://github.com/apache/hadoop/pull/5956#issuecomment-1682117418 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5956: HDFS-17161: Adding test for StripedBlockReader#createBlockReader leak…

2023-08-17 Thread via GitHub
hadoop-yetus commented on PR #5956: URL: https://github.com/apache/hadoop/pull/5956#issuecomment-1682114560 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Created] (HADOOP-18853) AWS SDK V2 - Integrate new transfer manager

2023-08-17 Thread Ahmar Suhail (Jira)
Ahmar Suhail created HADOOP-18853: - Summary: AWS SDK V2 - Integrate new transfer manager Key: HADOOP-18853 URL: https://issues.apache.org/jira/browse/HADOOP-18853 Project: Hadoop Common

[GitHub] [hadoop] hadoop-yetus commented on pull request #5957: YARN-6537. Consider running RM tests against the Router.

2023-08-17 Thread via GitHub
hadoop-yetus commented on PR #5957: URL: https://github.com/apache/hadoop/pull/5957#issuecomment-1681997984 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18850) Enable dual-layer server-side encryption with AWS KMS keys (DSSE-KMS)

2023-08-17 Thread Viraj Jasani (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755406#comment-17755406 ] Viraj Jasani commented on HADOOP-18850: --- [~ste...@apache.org] are you in favor of this before v2

[jira] [Commented] (HADOOP-18850) Enable dual-layer server-side encryption with AWS KMS keys (DSSE-KMS)

2023-08-17 Thread Viraj Jasani (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755394#comment-17755394 ] Viraj Jasani commented on HADOOP-18850: --- only recently HADOOP-18832 bumped sdk bundle to

[jira] [Comment Edited] (HADOOP-18850) Enable dual-layer server-side encryption with AWS KMS keys (DSSE-KMS)

2023-08-17 Thread Viraj Jasani (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755392#comment-17755392 ] Viraj Jasani edited comment on HADOOP-18850 at 8/17/23 7:13 AM: it

[jira] [Commented] (HADOOP-18850) Enable dual-layer server-side encryption with AWS KMS keys (DSSE-KMS)

2023-08-17 Thread Viraj Jasani (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755392#comment-17755392 ] Viraj Jasani commented on HADOOP-18850: --- SSEAlgorithm added DSSE as part of 1.12.488 release:

[jira] [Commented] (HADOOP-18852) S3ACachingInputStream.ensureCurrentBuffer(): lazy seek means all reads look like random IO

2023-08-17 Thread Viraj Jasani (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755385#comment-17755385 ] Viraj Jasani commented on HADOOP-18852: --- {quote}for other reads, we may want a bigger prefech

[jira] [Commented] (HADOOP-18852) S3ACachingInputStream.ensureCurrentBuffer(): lazy seek means all reads look like random IO

2023-08-17 Thread Viraj Jasani (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755384#comment-17755384 ] Viraj Jasani commented on HADOOP-18852: --- {quote}also, if a read is in a block other than the

[GitHub] [hadoop] slfan1989 opened a new pull request, #5957: YARN-6537. Consider running RM tests against the Router.

2023-08-17 Thread via GitHub
slfan1989 opened a new pull request, #5957: URL: https://github.com/apache/hadoop/pull/5957 ### Description of PR JIRA: YARN-6537. Consider running RM tests against the Router. ### How was this patch tested? ### For code changes: - [ ] Does the title or

[GitHub] [hadoop] chunyiyang commented on pull request #5951: HDFS-17156. mapreduce job encounters java.io.IOException

2023-08-17 Thread via GitHub
chunyiyang commented on PR #5951: URL: https://github.com/apache/hadoop/pull/5951#issuecomment-1681667372 This PR has passed all the checks, but seems like I don't have the permission to merge. Could someone help to merge this if possible? Thanks in advance! -- This is an automated