[GitHub] [hadoop] hadoop-yetus commented on pull request #5130: YARN-11371. [Federation] Refactor FederationInterceptorREST#createNewApplication\submitApplication Use FederationActionRetry.

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5130: URL: https://github.com/apache/hadoop/pull/5130#issuecomment-1313239096 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] hadoop-yetus commented on pull request #4892: YARN-11226. [Federation] Add createNewReservation, submitReservation, updateReservation, deleteReservation REST APIs for Router.

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #4892: URL: https://github.com/apache/hadoop/pull/4892#issuecomment-1313293079 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] hadoop-yetus commented on pull request #4717: YARN-6946. Upgrade JUnit from 4 to 5 in hadoop-yarn-common

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #4717: URL: https://github.com/apache/hadoop/pull/4717#issuecomment-1313374551 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] dannycjones commented on a diff in pull request #5110: HADOOP-18183. s3a audit logs to publish range start/end of GET requests in audit header

2022-11-14 Thread GitBox
dannycjones commented on code in PR #5110: URL: https://github.com/apache/hadoop/pull/5110#discussion_r1021284548 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/audit/impl/LoggingAuditor.java: ## @@ -230,6 +240,26 @@ private class LoggingAuditSpan extends Abs

[jira] [Commented] (HADOOP-18183) s3a audit logs to publish range start/end of GET requests in audit header

2022-11-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633643#comment-17633643 ] ASF GitHub Bot commented on HADOOP-18183: - dannycjones commented on code in PR

[GitHub] [hadoop] dannycjones commented on pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2022-11-14 Thread GitBox
dannycjones commented on PR #5054: URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1313413338 We'd love to take a look but this week is pretty busy for our team. I should have more time next week, will see if I can do earlier but I want to be realistic :') -- This is an auto

[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocatoin

2022-11-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633649#comment-17633649 ] ASF GitHub Bot commented on HADOOP-18399: - dannycjones commented on PR #5054: U

[GitHub] [hadoop] hadoop-yetus commented on pull request #5111: YARN-11350. [Federation] Router Support DelegationToken With ZK.(Old Version)

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5111: URL: https://github.com/apache/hadoop/pull/5111#issuecomment-1313421471 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] hadoop-yetus commented on pull request #5123: HDFS-16837. [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespaces

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5123: URL: https://github.com/apache/hadoop/pull/5123#issuecomment-1313421729 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] hadoop-yetus commented on pull request #5111: YARN-11350. [Federation] Router Support DelegationToken With ZK.(Old Version)

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5111: URL: https://github.com/apache/hadoop/pull/5111#issuecomment-1313467014 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] hadoop-yetus commented on pull request #5131: YARN-11350. [Federation] Router Support DelegationToken With ZK.

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5131: URL: https://github.com/apache/hadoop/pull/5131#issuecomment-1313480954 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[jira] [Updated] (HADOOP-18521) ABFS ReadBufferManager buffer sharing across concurrent HTTP requests

2022-11-14 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18521: Description: AbfsInputStream.close() can trigger the return of buffers used for active p

[GitHub] [hadoop] hadoop-yetus commented on pull request #5130: YARN-11371. [Federation] Refactor FederationInterceptorREST#createNewApplication\submitApplication Use FederationActionRetry.

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5130: URL: https://github.com/apache/hadoop/pull/5130#issuecomment-1313554413 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[jira] [Commented] (HADOOP-18501) [ABFS]: Partial Read should add to throttling metric

2022-11-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633752#comment-17633752 ] ASF GitHub Bot commented on HADOOP-18501: - hadoop-yetus commented on PR #5109:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5109: HADOOP-18501: ABFS: Partial read should add to throttling data

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5109: URL: https://github.com/apache/hadoop/pull/5109#issuecomment-1313599905 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] zhengchenyu opened a new pull request, #5132: HDFS-16830. [SBN READ] dfsrouter transmit state id according to clie…

2022-11-14 Thread GitBox
zhengchenyu opened a new pull request, #5132: URL: https://github.com/apache/hadoop/pull/5132 ### Description of PR For now, dfsrouter transmit the state of all nameservice. In this PR, dfsrouter only transmit state id according to client's demand. ### How was this patch tested

[GitHub] [hadoop] zhengchenyu commented on pull request #5132: HDFS-16830. [SBN READ] dfsrouter transmit state id according to clie…

2022-11-14 Thread GitBox
zhengchenyu commented on PR #5132: URL: https://github.com/apache/hadoop/pull/5132#issuecomment-1313633460 @simbadzina @omalley @ZanderXu Can you please review this PR? It is unnecessary to transmit all state id. -- This is an automated message from the Apache Git Service. To respond to

[jira] [Created] (HADOOP-18528) ABFS: Disable readAhead/prefetching by default

2022-11-14 Thread Mehakmeet Singh (Jira)
Mehakmeet Singh created HADOOP-18528: Summary: ABFS: Disable readAhead/prefetching by default Key: HADOOP-18528 URL: https://issues.apache.org/jira/browse/HADOOP-18528 Project: Hadoop Common

[GitHub] [hadoop] snvijaya opened a new pull request, #5133: Draft change - Prefetch corruption

2022-11-14 Thread GitBox
snvijaya opened a new pull request, #5133: URL: https://github.com/apache/hadoop/pull/5133 A ReadBuffer with a valid Buffer assigned to it can be in certain states when stream is closed, and with the above change, I am trying to address it as below : 1. Is in QueueReadAheadList - No chan

[GitHub] [hadoop] snvijaya commented on pull request #5117: HADOOP-18521. ABFS ReadBufferManager must not reuse in-progress buffers

2022-11-14 Thread GitBox
snvijaya commented on PR #5117: URL: https://github.com/apache/hadoop/pull/5117#issuecomment-1313734822 Hi @steveloughran, Wanted to get your opinion on below change as a possible replacement for this change : [https://github.com/apache/hadoop/pull/5133](url) A ReadBuffer with a va

[jira] [Commented] (HADOOP-18521) ABFS ReadBufferManager buffer sharing across concurrent HTTP requests

2022-11-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633807#comment-17633807 ] ASF GitHub Bot commented on HADOOP-18521: - snvijaya commented on PR #5117: URL:

[GitHub] [hadoop] slfan1989 commented on pull request #4450: YARN-11183. Federation: Remove outdated ApplicationHomeSubCluster in …

2022-11-14 Thread GitBox
slfan1989 commented on PR #4450: URL: https://github.com/apache/hadoop/pull/4450#issuecomment-1313870319 @zhengchenyu Thank you very much for your answer, from my personal point of view, I think the `Dispatcher` and `StateMachine` model is designed to solve high throughput message processin

[GitHub] [hadoop] hadoop-yetus commented on pull request #5133: Draft change - Prefetch corruption

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5133: URL: https://github.com/apache/hadoop/pull/5133#issuecomment-1313937909 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] mehakmeet opened a new pull request, #5134: HADOOP-18528. Disable readAhead/prefetching by default

2022-11-14 Thread GitBox
mehakmeet opened a new pull request, #5134: URL: https://github.com/apache/hadoop/pull/5134 ### Description of PR Setting readAhead config to false by default. ### How was this patch tested? Region: `us-west-2` `mvn -Dparallel-tests=abfs -DtestsThreadCount=8 -Dscale cl

[jira] [Commented] (HADOOP-18528) ABFS: Disable readAhead/prefetching by default

2022-11-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633892#comment-17633892 ] ASF GitHub Bot commented on HADOOP-18528: - mehakmeet opened a new pull request,

[jira] [Updated] (HADOOP-18528) ABFS: Disable readAhead/prefetching by default

2022-11-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18528: Labels: pull-request-available (was: ) > ABFS: Disable readAhead/prefetching by default

[GitHub] [hadoop] steveloughran commented on pull request #5134: HADOOP-18528. Disable readAhead/prefetching by default

2022-11-14 Thread GitBox
steveloughran commented on PR #5134: URL: https://github.com/apache/hadoop/pull/5134#issuecomment-1313985228 where do you test it? + @snvijaya @anmolanmol1234 @pranavsaxena-microsoft what do you all think? this is us disabling the readahead so we can get on with the real fix wi

[jira] [Commented] (HADOOP-18528) ABFS: Disable readAhead/prefetching by default

2022-11-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633911#comment-17633911 ] ASF GitHub Bot commented on HADOOP-18528: - steveloughran commented on PR #5134:

[GitHub] [hadoop] steveloughran commented on pull request #5117: HADOOP-18521. ABFS ReadBufferManager must not reuse in-progress buffers

2022-11-14 Thread GitBox
steveloughran commented on PR #5117: URL: https://github.com/apache/hadoop/pull/5117#issuecomment-1313988205 see also #5134 which is the "disable readahead" patch for 3.3.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[jira] [Commented] (HADOOP-18521) ABFS ReadBufferManager buffer sharing across concurrent HTTP requests

2022-11-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633912#comment-17633912 ] ASF GitHub Bot commented on HADOOP-18521: - steveloughran commented on PR #5117:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5133: Draft change - Prefetch corruption

2022-11-14 Thread GitBox
steveloughran commented on code in PR #5133: URL: https://github.com/apache/hadoop/pull/5133#discussion_r1021743476 ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferWorker.java: ## @@ -84,4 +93,16 @@ public void run() { } }

[GitHub] [hadoop] hadoop-yetus commented on pull request #5131: YARN-11350. [Federation] Router Support DelegationToken With ZK.

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5131: URL: https://github.com/apache/hadoop/pull/5131#issuecomment-1314040098 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] simbadzina commented on pull request #5123: HDFS-16837. [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespaces

2022-11-14 Thread GitBox
simbadzina commented on PR #5123: URL: https://github.com/apache/hadoop/pull/5123#issuecomment-1314094492 Code looks good to me. Do you know why there are these new Spotbugs violations? -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [hadoop] brumi1024 commented on pull request #5116: YARN-11369. Commons.compress throws an IllegalArgumentException with …

2022-11-14 Thread GitBox
brumi1024 commented on PR #5116: URL: https://github.com/apache/hadoop/pull/5116#issuecomment-1314132283 @ashutoshcipher do you have a specific way in mind? The behaviour change introduced in 1.21 was that if someone with a large UID creates some files and tries to use the lib with no BigNu

[GitHub] [hadoop] li-leyang opened a new pull request, #5135: HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized

2022-11-14 Thread GitBox
li-leyang opened a new pull request, #5135: URL: https://github.com/apache/hadoop/pull/5135 HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized ### Description of PR ### How was this patch tested? ### For code changes

[GitHub] [hadoop] hadoop-yetus commented on pull request #5134: HADOOP-18528. Disable readAhead/prefetching by default

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5134: URL: https://github.com/apache/hadoop/pull/5134#issuecomment-1314278623 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[jira] [Commented] (HADOOP-18528) ABFS: Disable readAhead/prefetching by default

2022-11-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17634021#comment-17634021 ] ASF GitHub Bot commented on HADOOP-18528: - hadoop-yetus commented on PR #5134:

[GitHub] [hadoop] tomscut commented on pull request #5123: HDFS-16837. [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespaces

2022-11-14 Thread GitBox
tomscut commented on PR #5123: URL: https://github.com/apache/hadoop/pull/5123#issuecomment-1314588275 > Code looks good to me. Do you know why there are these new Spotbugs violations? Because of this change, and we did not config configure filter. `hadoop-hdfs-project/hadoop-hdfs-

[GitHub] [hadoop] li-leyang commented on pull request #5135: HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized

2022-11-14 Thread GitBox
li-leyang commented on PR #5135: URL: https://github.com/apache/hadoop/pull/5135#issuecomment-1314589306 @omalley @virajith can you please take a look as well? thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hadoop] hadoop-yetus commented on pull request #5104: YARN-11158. Support (Create/Renew/Cancel) DelegationToken API's for Federation.

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5104: URL: https://github.com/apache/hadoop/pull/5104#issuecomment-1314681157 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] zhengchenyu commented on pull request #4450: YARN-11183. Federation: Remove outdated ApplicationHomeSubCluster in …

2022-11-14 Thread GitBox
zhengchenyu commented on PR #4450: URL: https://github.com/apache/hadoop/pull/4450#issuecomment-1314709006 @slfan1989 In AsyncDispatcher event execute in sequence. If one operation is slow, the size of queue will increase, resourcemanager will work in bad performance. We can't do any high c

[GitHub] [hadoop] hadoop-yetus commented on pull request #5135: HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5135: URL: https://github.com/apache/hadoop/pull/5135#issuecomment-1314721562 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] hadoop-yetus commented on pull request #5131: YARN-11350. [Federation] Router Support DelegationToken With ZK.

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5131: URL: https://github.com/apache/hadoop/pull/5131#issuecomment-1314769675 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] hadoop-yetus commented on pull request #5126: YARN-11370. [Federation] Refactor MemoryFederationStateStore code.

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5126: URL: https://github.com/apache/hadoop/pull/5126#issuecomment-1314863246 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] hadoop-yetus commented on pull request #5057: YARN-11359. [Federation] Routing admin invocations transparently to multiple RMs.

2022-11-14 Thread GitBox
hadoop-yetus commented on PR #5057: URL: https://github.com/apache/hadoop/pull/5057#issuecomment-1314902607 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex