[GitHub] [hadoop] hadoop-yetus commented on pull request #5875: HDFS-17115. HttpFS Add Support getErasureCodeCodecs API

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5875: URL: https://github.com/apache/hadoop/pull/5875#issuecomment-1649141065 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] saxenapranav commented on a diff in pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.

2023-07-24 Thread via GitHub
saxenapranav commented on code in PR #5881: URL: https://github.com/apache/hadoop/pull/5881#discussion_r1272322497 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestTracingContext.java: ## @@ -270,4 +273,71 @@ fileSystemId,

[GitHub] [hadoop] hadoop-yetus commented on pull request #5878: HDFS-17030 Limit wait time for getHAServiceState in ObserverReadProxyProvider

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5878: URL: https://github.com/apache/hadoop/pull/5878#issuecomment-1648959263 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5878: HDFS-17030 Limit wait time for getHAServiceState in ObserverReadProxyProvider

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5878: URL: https://github.com/apache/hadoop/pull/5878#issuecomment-1648957700 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746730#comment-17746730 ] ASF GitHub Bot commented on HADOOP-18805: - virajjasani commented on PR #5887: URL:

[GitHub] [hadoop] virajjasani commented on pull request #5887: HADOOP-18805. S3A prefetch tests to work with small files

2023-07-24 Thread via GitHub
virajjasani commented on PR #5887: URL: https://github.com/apache/hadoop/pull/5887#issuecomment-1648848349 us-west-2: `$ mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch` ``` [INFO]

[GitHub] [hadoop] Tre2878 commented on pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete block rep

2023-07-24 Thread via GitHub
Tre2878 commented on PR #5855: URL: https://github.com/apache/hadoop/pull/5855#issuecomment-1648848258 > `storageInfo.blockReportCount`在第 2932 行更新,这意味着`storageInfo.getBlockReportCount() > 0`只有在第 2924 行成功处理该存储的第一个报告后,该情况才成立。 > >

[jira] [Commented] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746722#comment-17746722 ] ASF GitHub Bot commented on HADOOP-18805: - virajjasani commented on PR #5887: URL:

[GitHub] [hadoop] virajjasani commented on pull request #5887: HADOOP-18805. S3A prefetch tests to work with small files

2023-07-24 Thread via GitHub
virajjasani commented on PR #5887: URL: https://github.com/apache/hadoop/pull/5887#issuecomment-1648836765 @steveloughran we need HADOOP-18246 (commit: https://github.com/apache/hadoop/commit/22f6d55b71e53fc25f72e18e89a2a647ff8c0dca) on branch-3.3. i have tested by including that commit

[jira] [Commented] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746721#comment-17746721 ] ASF GitHub Bot commented on HADOOP-18805: - hadoop-yetus commented on PR #5887: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5887: HADOOP-18805. S3A prefetch tests to work with small files

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5887: URL: https://github.com/apache/hadoop/pull/5887#issuecomment-1648835263 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] prashantpogde commented on pull request #5885: HDFS-17120. Support snapshot diff based copylisting for flat paths.

2023-07-24 Thread via GitHub
prashantpogde commented on PR #5885: URL: https://github.com/apache/hadoop/pull/5885#issuecomment-1648796679 @umamaheswararao Can you please take a look ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop] hadoop-yetus commented on pull request #5860: HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5860: URL: https://github.com/apache/hadoop/pull/5860#issuecomment-1648777434 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746684#comment-17746684 ] ASF GitHub Bot commented on HADOOP-18805: - virajjasani commented on PR #5887: URL:

[GitHub] [hadoop] virajjasani commented on pull request #5887: HADOOP-18805. S3A prefetch tests to work with small files

2023-07-24 Thread via GitHub
virajjasani commented on PR #5887: URL: https://github.com/apache/hadoop/pull/5887#issuecomment-1648746764 Some relevant failures, will get to them: ``` [ERROR] testStatusProbesAfterClosingStream(org.apache.hadoop.fs.s3a.ITestS3APrefetchingInputStream) Time elapsed: 0.34 s <<<

[GitHub] [hadoop] virajjasani opened a new pull request, #5887: HADOOP-18805. S3A prefetch tests to work with small files

2023-07-24 Thread via GitHub
virajjasani opened a new pull request, #5887: URL: https://github.com/apache/hadoop/pull/5887 branch-3.3 backport of PR: #5851 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[jira] [Commented] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746674#comment-17746674 ] ASF GitHub Bot commented on HADOOP-18805: - virajjasani opened a new pull request, #5887: URL:

[GitHub] [hadoop] simbadzina commented on pull request #5860: HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled.

2023-07-24 Thread via GitHub
simbadzina commented on PR #5860: URL: https://github.com/apache/hadoop/pull/5860#issuecomment-1648705296 > @simbadzina, > > To trigger a rebuild, we can create empty commit. That is more explicit. Something like the following. > > ``` >git commit --allow-empty

[GitHub] [hadoop] xinglin commented on pull request #5860: HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled.

2023-07-24 Thread via GitHub
xinglin commented on PR #5860: URL: https://github.com/apache/hadoop/pull/5860#issuecomment-1648697443 @simbadzina, To trigger a rebuild, we can create empty commit. That is more explicit. Something like the following. ``` git commit --allow-empty -m "An empty

[jira] [Resolved] (HADOOP-18823) Add Labeler Github Action.

2023-07-24 Thread Ayush Saxena (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18823. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add Labeler

[jira] [Commented] (HADOOP-18823) Add Labeler Github Action.

2023-07-24 Thread Ayush Saxena (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746661#comment-17746661 ] Ayush Saxena commented on HADOOP-18823: --- Committed to trunk. Thanx [~hexiaoqiao] for the

[jira] [Updated] (HADOOP-12677) DecompressorStream throws IndexOutOfBoundsException when calling skip(long)

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-12677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-12677: Labels: pull-request-available (was: ) > DecompressorStream throws

[jira] [Commented] (HADOOP-12677) DecompressorStream throws IndexOutOfBoundsException when calling skip(long)

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-12677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746660#comment-17746660 ] ASF GitHub Bot commented on HADOOP-12677: - jojochuang opened a new pull request, #5886: URL:

[GitHub] [hadoop] jojochuang opened a new pull request, #5886: HADOOP-12677. DecompressorStream throws IndexOutOfBoundsException when calling skip(long)

2023-07-24 Thread via GitHub
jojochuang opened a new pull request, #5886: URL: https://github.com/apache/hadoop/pull/5886 ### Description of PR Cast DecompressorStream.skip() properly. ### How was this patch tested? Unit test ### For code changes: - [ X] Does the title or this PR starts with

[GitHub] [hadoop] xinglin commented on pull request #5880: HDFS-17118 Fixed a couple checkstyle warnings in TestObserverReadProxyProvider

2023-07-24 Thread via GitHub
xinglin commented on PR #5880: URL: https://github.com/apache/hadoop/pull/5880#issuecomment-1648685769 Hi @goiri, Can you take a look at this PR? Fix some checkstyle warnings introduced in HDFS-17030. The uncheck cast error was not detected during the builds when merging into

[GitHub] [hadoop] xinglin commented on pull request #5878: HDFS-17030 Limit wait time for getHAServiceState in ObserverReadProxyProvider

2023-07-24 Thread via GitHub
xinglin commented on PR #5878: URL: https://github.com/apache/hadoop/pull/5878#issuecomment-1648682794 Hi @goiri, This PR is coming. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hadoop] hadoop-yetus commented on pull request #5860: HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5860: URL: https://github.com/apache/hadoop/pull/5860#issuecomment-1648679047 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] ayushtkn merged pull request #5874: HADOOP-18823. Add Labeler Github Action.

2023-07-24 Thread via GitHub
ayushtkn merged PR #5874: URL: https://github.com/apache/hadoop/pull/5874 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (HADOOP-18823) Add Labeler Github Action.

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746650#comment-17746650 ] ASF GitHub Bot commented on HADOOP-18823: - ayushtkn merged PR #5874: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5885: HDFS-17120. Support snapshot diff based copylisting for flat paths.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5885: URL: https://github.com/apache/hadoop/pull/5885#issuecomment-1648595933 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5884: YARN-11535: Snakeyaml should be excluded from jackson-dataformat-yaml-2.12.7 as it may cause transitive dependency issue.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5884: URL: https://github.com/apache/hadoop/pull/5884#issuecomment-1648589852 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746631#comment-17746631 ] ASF GitHub Bot commented on HADOOP-18820: - hadoop-yetus commented on PR #5872: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1648531582 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746630#comment-17746630 ] ASF GitHub Bot commented on HADOOP-18805: - virajjasani commented on PR #5851: URL:

[GitHub] [hadoop] virajjasani commented on pull request #5851: HADOOP-18805. S3A prefetch tests to work with small files

2023-07-24 Thread via GitHub
virajjasani commented on PR #5851: URL: https://github.com/apache/hadoop/pull/5851#issuecomment-1648531160 sure thing, will create 3.3 backport PR and test it. Thank you for merging the PR! -- This is an automated message from the Apache Git Service. To respond to the message,

[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746629#comment-17746629 ] ASF GitHub Bot commented on HADOOP-18184: - steveloughran commented on code in PR #5832: URL:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.

2023-07-24 Thread via GitHub
steveloughran commented on code in PR #5832: URL: https://github.com/apache/hadoop/pull/5832#discussion_r1272698192 ## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/CachingBlockManager.java: ## @@ -298,16 +306,26 @@ public void

[GitHub] [hadoop] steveloughran commented on pull request #5867: HADOOP-18817.Upgrade version of aws-java-sdk-bundle to 1.12.368 avoid verify error.

2023-07-24 Thread via GitHub
steveloughran commented on PR #5867: URL: https://github.com/apache/hadoop/pull/5867#issuecomment-1648525939 @gp1314 didn't get the end of your comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[jira] [Commented] (HADOOP-18817) Upgrade version of aws-java-sdk-bundle to 1.12.368 avoid verify error.

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746627#comment-17746627 ] ASF GitHub Bot commented on HADOOP-18817: - steveloughran commented on PR #5867: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5878: HDFS-17030 Limit wait time for getHAServiceState in ObserverReadProxyProvider

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5878: URL: https://github.com/apache/hadoop/pull/5878#issuecomment-1648468525 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5825: HDFS-17072. DFSAdmin: add a triggerDirectoryScanner command.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5825: URL: https://github.com/apache/hadoop/pull/5825#issuecomment-1648466106 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] sadanand48 opened a new pull request, #5885: HDFS-17120. Support snapshot diff based copylisting for flat paths.

2023-07-24 Thread via GitHub
sadanand48 opened a new pull request, #5885: URL: https://github.com/apache/hadoop/pull/5885 ### Description of PR Currently for Diff-based copyListing that is used during the distcpSync step of an incremental copy by default the SimpleCopyListing implementation is used. In it's

[GitHub] [hadoop] hadoop-yetus commented on pull request #5871: YARN-11520. Support capacity vector for AQCv2 dynamic templates.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5871: URL: https://github.com/apache/hadoop/pull/5871#issuecomment-1648439396 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5871: YARN-11520. Support capacity vector for AQCv2 dynamic templates.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5871: URL: https://github.com/apache/hadoop/pull/5871#issuecomment-1648437872 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746593#comment-17746593 ] ASF GitHub Bot commented on HADOOP-18805: - steveloughran commented on PR #5851: URL:

[jira] [Commented] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746592#comment-17746592 ] ASF GitHub Bot commented on HADOOP-18805: - steveloughran merged PR #5851: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5851: HADOOP-18805. S3A prefetch tests to work with small files

2023-07-24 Thread via GitHub
steveloughran commented on PR #5851: URL: https://github.com/apache/hadoop/pull/5851#issuecomment-1648407783 merged to trunk; now I will get to rebase my unbuffer() patch atop it. Can you do the homework of a cherrypick and retest onto branch-3? thanks -- This is an automated

[GitHub] [hadoop] steveloughran merged pull request #5851: HADOOP-18805. S3A prefetch tests to work with small files

2023-07-24 Thread via GitHub
steveloughran merged PR #5851: URL: https://github.com/apache/hadoop/pull/5851 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5860: HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5860: URL: https://github.com/apache/hadoop/pull/5860#issuecomment-1648397789 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18073) Upgrade AWS SDK to v2

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746583#comment-17746583 ] ASF GitHub Bot commented on HADOOP-18073: - mukund-thakur commented on code in PR #5858: URL:

[GitHub] [hadoop] mukund-thakur commented on a diff in pull request #5858: HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK V2.(#5707)

2023-07-24 Thread via GitHub
mukund-thakur commented on code in PR #5858: URL: https://github.com/apache/hadoop/pull/5858#discussion_r1272603815 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java: ## @@ -1215,213 +1079,6 @@ public static void deleteWithWarning(FileSystem fs,

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746573#comment-17746573 ] ASF GitHub Bot commented on HADOOP-18820: - ahmarsuhail commented on code in PR #5872: URL:

[GitHub] [hadoop] ahmarsuhail commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-07-24 Thread via GitHub
ahmarsuhail commented on code in PR #5872: URL: https://github.com/apache/hadoop/pull/5872#discussion_r1272548061 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/adapter/AwsV1BindingSupport.java: ## @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software

[GitHub] [hadoop] hadoop-yetus commented on pull request #5877: HDFS-17117. BlockManager#processMisReplicatesAsync can periodically print reconstructionQueuesInitProgress

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5877: URL: https://github.com/apache/hadoop/pull/5877#issuecomment-1648366182 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] susheelgupta7 opened a new pull request, #5884: YARN-11535: Snakeyaml should be excluded from jackson-dataformat-yaml-2.12.7 as it may cause transitive dependency issue.

2023-07-24 Thread via GitHub
susheelgupta7 opened a new pull request, #5884: URL: https://github.com/apache/hadoop/pull/5884 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g.

[GitHub] [hadoop] hadoop-yetus commented on pull request #5880: HDFS-17118 Fixed a couple checkstyle warnings in TestObserverReadProxyProvider

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5880: URL: https://github.com/apache/hadoop/pull/5880#issuecomment-1648333656 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] xinglin commented on a diff in pull request #5860: HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled.

2023-07-24 Thread via GitHub
xinglin commented on code in PR #5860: URL: https://github.com/apache/hadoop/pull/5860#discussion_r1272550379 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java: ## @@ -264,7 +264,7 @@ public

[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746534#comment-17746534 ] ASF GitHub Bot commented on HADOOP-18184: - ahmarsuhail commented on code in PR #5832: URL:

[GitHub] [hadoop] ahmarsuhail commented on a diff in pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.

2023-07-24 Thread via GitHub
ahmarsuhail commented on code in PR #5832: URL: https://github.com/apache/hadoop/pull/5832#discussion_r1270694111 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3ARemoteInputStream.java: ## @@ -409,27 +455,54 @@ protected String getOffsetStr(long

[jira] [Created] (HADOOP-18824) ZKDelegationTokenSecretManager causes ArithmeticException due to improper numRetries value checking

2023-07-24 Thread ConfX (Jira)
ConfX created HADOOP-18824: -- Summary: ZKDelegationTokenSecretManager causes ArithmeticException due to improper numRetries value checking Key: HADOOP-18824 URL: https://issues.apache.org/jira/browse/HADOOP-18824

[jira] [Commented] (HADOOP-18073) Upgrade AWS SDK to v2

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746522#comment-17746522 ] ASF GitHub Bot commented on HADOOP-18073: - ahmarsuhail commented on code in PR #5858: URL:

[GitHub] [hadoop] ahmarsuhail commented on a diff in pull request #5858: HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK V2.(#5707)

2023-07-24 Thread via GitHub
ahmarsuhail commented on code in PR #5858: URL: https://github.com/apache/hadoop/pull/5858#discussion_r1272491279 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java: ## @@ -1215,213 +1079,6 @@ public static void deleteWithWarning(FileSystem fs,

[jira] [Commented] (HADOOP-18073) Upgrade AWS SDK to v2

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746521#comment-17746521 ] ASF GitHub Bot commented on HADOOP-18073: - ahmarsuhail commented on code in PR #5858: URL:

[GitHub] [hadoop] ahmarsuhail commented on a diff in pull request #5858: HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK V2.(#5707)

2023-07-24 Thread via GitHub
ahmarsuhail commented on code in PR #5858: URL: https://github.com/apache/hadoop/pull/5858#discussion_r1272491279 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java: ## @@ -1215,213 +1079,6 @@ public static void deleteWithWarning(FileSystem fs,

[jira] [Commented] (HADOOP-18073) Upgrade AWS SDK to v2

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746517#comment-17746517 ] ASF GitHub Bot commented on HADOOP-18073: - mukund-thakur commented on code in PR #5858: URL:

[GitHub] [hadoop] mukund-thakur commented on a diff in pull request #5858: HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK V2.(#5707)

2023-07-24 Thread via GitHub
mukund-thakur commented on code in PR #5858: URL: https://github.com/apache/hadoop/pull/5858#discussion_r1272471538 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java: ## @@ -1215,213 +1079,6 @@ public static void deleteWithWarning(FileSystem fs,

[GitHub] [hadoop] simbadzina commented on a diff in pull request #5860: HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled.

2023-07-24 Thread via GitHub
simbadzina commented on code in PR #5860: URL: https://github.com/apache/hadoop/pull/5860#discussion_r1272445455 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java: ## @@ -264,7 +264,7 @@ public

[GitHub] [hadoop] simbadzina commented on a diff in pull request #5860: HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled.

2023-07-24 Thread via GitHub
simbadzina commented on code in PR #5860: URL: https://github.com/apache/hadoop/pull/5860#discussion_r1272444972 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java: ## @@ -1928,9 +1928,14 @@ public

[GitHub] [hadoop] brumi1024 commented on a diff in pull request #5870: YARN-11522. Update the documentation with the YARN-11000 changes.

2023-07-24 Thread via GitHub
brumi1024 commented on code in PR #5870: URL: https://github.com/apache/hadoop/pull/5870#discussion_r1272376852 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md: ## @@ -64,8 +64,14 @@ The `CapacityScheduler` supports the following

[GitHub] [hadoop] YuanbenWang commented on pull request #5869: HDFS-17113. Reconfig transfer and write bandwidth for datanode.

2023-07-24 Thread via GitHub
YuanbenWang commented on PR #5869: URL: https://github.com/apache/hadoop/pull/5869#issuecomment-1648083803 @ayushtkn @Hexiaoqiao Hello! Could you please help review this pr? Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hadoop] hadoop-yetus commented on pull request #5883: YARN-11521. Parameterize tests runs with legacy/non-legacy queue-mode.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5883: URL: https://github.com/apache/hadoop/pull/5883#issuecomment-1648055527 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] symious commented on pull request #5863: HADOOP-18816. Rebuild exceptions on Client side

2023-07-24 Thread via GitHub
symious commented on PR #5863: URL: https://github.com/apache/hadoop/pull/5863#issuecomment-1647980381 This feature can lead to many use cases. For example, when returning a standbyException, the current active Namenode can be included in the error, eliminating the need for the client-side

[jira] [Commented] (HADOOP-18816) Rebuild Exceptions on Client side to get genuine exceptions

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746473#comment-17746473 ] ASF GitHub Bot commented on HADOOP-18816: - symious commented on PR #5863: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5881: URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1647973890 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] ferhui commented on pull request #5848: MAPREDUCE-7442. Exception message is not intusive when accessing the WebUI

2023-07-24 Thread via GitHub
ferhui commented on PR #5848: URL: https://github.com/apache/hadoop/pull/5848#issuecomment-1647940260 Merged. @liangyu-1 Thanks again. @slfan1989 Thanks for review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hadoop] hadoop-yetus commented on pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5881: URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1647941330 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] ferhui merged pull request #5848: MAPREDUCE-7442. Exception message is not intusive when accessing the WebUI

2023-07-24 Thread via GitHub
ferhui merged PR #5848: URL: https://github.com/apache/hadoop/pull/5848 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete bloc

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5855: URL: https://github.com/apache/hadoop/pull/5855#issuecomment-1647893413 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Updated] (HADOOP-18759) [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout failures

2023-07-24 Thread Anuj Modi (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuj Modi updated HADOOP-18759: --- Summary: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout failures

[jira] [Updated] (HADOOP-18759) [ABFS][Backoff-Optimization] Have a Linear retry policy for connection timeout failures

2023-07-24 Thread Anuj Modi (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuj Modi updated HADOOP-18759: --- Description: Today when a request fails with connection timeout, it falls back into the loop for

[GitHub] [hadoop] hfutatzhanghb commented on a diff in pull request #5876: HDFS-17116. Reset startupTime and enterSafeModeTime if check time interval is negative during router safe mode exit check

2023-07-24 Thread via GitHub
hfutatzhanghb commented on code in PR #5876: URL: https://github.com/apache/hadoop/pull/5876#discussion_r1272217666 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterSafemode.java: ## @@ -141,6 +141,31 @@ public void

[jira] [Commented] (HADOOP-18817) Upgrade version of aws-java-sdk-bundle to 1.12.368 avoid verify error.

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746432#comment-17746432 ] ASF GitHub Bot commented on HADOOP-18817: - gp1314 closed pull request #5867:

[GitHub] [hadoop] gp1314 closed pull request #5867: HADOOP-18817.Upgrade version of aws-java-sdk-bundle to 1.12.368 avoid verify error.

2023-07-24 Thread via GitHub
gp1314 closed pull request #5867: HADOOP-18817.Upgrade version of aws-java-sdk-bundle to 1.12.368 avoid verify error. URL: https://github.com/apache/hadoop/pull/5867 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hadoop] hadoop-yetus commented on pull request #5825: HDFS-17072. DFSAdmin: add a triggerDirectoryScanner command.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5825: URL: https://github.com/apache/hadoop/pull/5825#issuecomment-1647811078 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] ayushtkn commented on a diff in pull request #5860: HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled.

2023-07-24 Thread via GitHub
ayushtkn commented on code in PR #5860: URL: https://github.com/apache/hadoop/pull/5860#discussion_r1272175747 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java: ## @@ -1928,9 +1928,14 @@ public

[GitHub] [hadoop] ayushtkn merged pull request #5882: HDFS-17119. RBF:Logger fix for StateStoreMySQLImpl

2023-07-24 Thread via GitHub
ayushtkn merged PR #5882: URL: https://github.com/apache/hadoop/pull/5882 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hadoop] haiyang1987 commented on pull request #5877: HDFS-17117. BlockManager#processMisReplicatesAsync can periodically print reconstructionQueuesInitProgress

2023-07-24 Thread via GitHub
haiyang1987 commented on PR #5877: URL: https://github.com/apache/hadoop/pull/5877#issuecomment-1647712508 > Please fix the checkstyle, other changes look good to me. Thanks @tomscut help review it , i will update later. -- This is an automated message from the Apache Git Service.

[GitHub] [hadoop] haiyang1987 commented on a diff in pull request #5877: HDFS-17117. BlockManager#processMisReplicatesAsync can periodically print reconstructionQueuesInitProgress

2023-07-24 Thread via GitHub
haiyang1987 commented on code in PR #5877: URL: https://github.com/apache/hadoop/pull/5877#discussion_r1272112992 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java: ## @@ -261,4 +261,11 @@ public interface

[GitHub] [hadoop] haiyang1987 commented on a diff in pull request #5877: HDFS-17117. BlockManager#processMisReplicatesAsync can periodically print reconstructionQueuesInitProgress

2023-07-24 Thread via GitHub
haiyang1987 commented on code in PR #5877: URL: https://github.com/apache/hadoop/pull/5877#discussion_r1272112235 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java: ## @@ -3891,6 +3891,8 @@ private void

[GitHub] [hadoop] haiyang1987 closed pull request #5877: HDFS-17117. BlockManager#processMisReplicatesAsync can periodically print reconstructionQueuesInitProgress

2023-07-24 Thread via GitHub
haiyang1987 closed pull request #5877: HDFS-17117. BlockManager#processMisReplicatesAsync can periodically print reconstructionQueuesInitProgress URL: https://github.com/apache/hadoop/pull/5877 -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hadoop] haiyang1987 commented on pull request #5877: HDFS-17117. BlockManager#processMisReplicatesAsync can periodically print reconstructionQueuesInitProgress

2023-07-24 Thread via GitHub
haiyang1987 commented on PR #5877: URL: https://github.com/apache/hadoop/pull/5877#issuecomment-1647694291 > Please fix the checkstyle, other changes look good to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hadoop] hadoop-yetus commented on pull request #5880: HDFS-17118 Fixed a couple checkstyle warnings in TestObserverReadProxyProvider

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5880: URL: https://github.com/apache/hadoop/pull/5880#issuecomment-1647658969 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5878: HDFS-17030 Limit wait time for getHAServiceState in ObserverReadProxyProvider

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5878: URL: https://github.com/apache/hadoop/pull/5878#issuecomment-1647634685 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5878: HDFS-17030 Limit wait time for getHAServiceState in ObserverReadProxyProvider

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5878: URL: https://github.com/apache/hadoop/pull/5878#issuecomment-1647632331 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] tomicooler opened a new pull request, #5883: YARN-11521. Parameterize tests runs with legacy/non-legacy queue-mode.

2023-07-24 Thread via GitHub
tomicooler opened a new pull request, #5883: URL: https://github.com/apache/hadoop/pull/5883 ### Description of PR NOTE: I cherry-picked YARN-11539, will rebase later when that gets merged to trunk. ### How was this patch tested? Tested locally, Yetus will verify it.

[GitHub] [hadoop] hadoop-yetus commented on pull request #5863: HADOOP-18816. Rebuild exceptions on Client side

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5863: URL: https://github.com/apache/hadoop/pull/5863#issuecomment-1647593960 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18816) Rebuild Exceptions on Client side to get genuine exceptions

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746321#comment-17746321 ] ASF GitHub Bot commented on HADOOP-18816: - hadoop-yetus commented on PR #5863: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5879: Blocks on IN_MAINTENANCE DNs should be sorted properly in LocatedBlocks.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5879: URL: https://github.com/apache/hadoop/pull/5879#issuecomment-1647579208 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5862: YARN-11536. [Federation] Router CLI Supports Batch Save the SubClusterPolicyConfiguration Of Queues.

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5862: URL: https://github.com/apache/hadoop/pull/5862#issuecomment-1647573283 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5882: HDFS-17119. RBF:Logger fix for StateStoreMySQLImpl

2023-07-24 Thread via GitHub
hadoop-yetus commented on PR #5882: URL: https://github.com/apache/hadoop/pull/5882#issuecomment-1647519901 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

  1   2   >