[jira] [Commented] (HADOOP-17276) Extend CallerContext to make it include many items

2023-02-22 Thread Takanobu Asanuma (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692550#comment-17692550 ] Takanobu Asanuma commented on HADOOP-17276: --- I added 3.3.5 to fix versions. > Extend

[jira] [Updated] (HADOOP-17276) Extend CallerContext to make it include many items

2023-02-22 Thread Takanobu Asanuma (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HADOOP-17276: -- Fix Version/s: 3.3.5 > Extend CallerContext to make it include many items >

[jira] [Updated] (HADOOP-18345) Enhance client protocol to propagate last seen state IDs for multiple nameservices.

2023-02-22 Thread Takanobu Asanuma (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HADOOP-18345: -- Fix Version/s: (was: 3.3.5) > Enhance client protocol to propagate last seen

[jira] [Commented] (HADOOP-18345) Enhance client protocol to propagate last seen state IDs for multiple nameservices.

2023-02-22 Thread Takanobu Asanuma (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692548#comment-17692548 ] Takanobu Asanuma commented on HADOOP-18345: --- It is only in trunk. I corrected the fixed

[jira] [Commented] (HADOOP-18632) ABFS: Customize and optimize timeouts made based on each separate request

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692542#comment-17692542 ] ASF GitHub Bot commented on HADOOP-18632: - sreeb-msft commented on code in PR #5399: URL:

[GitHub] [hadoop] sreeb-msft commented on a diff in pull request #5399: HADOOP-18632: [ABFS] Customize and optimize timeouts made based on each separate request

2023-02-22 Thread via GitHub
sreeb-msft commented on code in PR #5399: URL: https://github.com/apache/hadoop/pull/5399#discussion_r1115320710 ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/TimeoutOptimizer.java: ## @@ -0,0 +1,227 @@ +/** + * Licensed to the Apache

[jira] [Commented] (HADOOP-18632) ABFS: Customize and optimize timeouts made based on each separate request

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692536#comment-17692536 ] ASF GitHub Bot commented on HADOOP-18632: - sreeb-msft commented on code in PR #5399: URL:

[GitHub] [hadoop] sreeb-msft commented on a diff in pull request #5399: HADOOP-18632: [ABFS] Customize and optimize timeouts made based on each separate request

2023-02-22 Thread via GitHub
sreeb-msft commented on code in PR #5399: URL: https://github.com/apache/hadoop/pull/5399#discussion_r1115313395 ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/TimeoutOptimizer.java: ## @@ -0,0 +1,227 @@ +/** + * Licensed to the Apache

[GitHub] [hadoop] tasanuma commented on pull request #5371: HDFS-18324. Fix race condition in closing IPC connections.

2023-02-22 Thread via GitHub
tasanuma commented on PR #5371: URL: https://github.com/apache/hadoop/pull/5371#issuecomment-1441289106 Seems HDFS-18324 is wrong, we should update the title to HDFS-16853 or HADOOP-18324, shouldn't we? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hadoop] hadoop-yetus commented on pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5397: URL: https://github.com/apache/hadoop/pull/5397#issuecomment-1441280753 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692516#comment-17692516 ] ASF GitHub Bot commented on HADOOP-18399: - hadoop-yetus commented on PR #5054: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5054: URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1441271453 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18550) Web UI has no session controller.There is a risk of session hijacking.

2023-02-22 Thread rickboker (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692513#comment-17692513 ] rickboker commented on HADOOP-18550: [~brahmareddy] yes I have a plane to reslove it. > Web UI has

[GitHub] [hadoop] hadoop-yetus commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5380: URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1441258209 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5397: URL: https://github.com/apache/hadoop/pull/5397#issuecomment-1441251613 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5397: URL: https://github.com/apache/hadoop/pull/5397#issuecomment-1441216333 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Resolved] (HADOOP-18582) No need to clean tmp files in distcp direct mode

2023-02-22 Thread Ayush Saxena (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18582. --- Hadoop Flags: Reviewed Resolution: Fixed > No need to clean tmp files in distcp

[GitHub] [hadoop] hadoop-yetus commented on pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5397: URL: https://github.com/apache/hadoop/pull/5397#issuecomment-1441210715 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692484#comment-17692484 ] ASF GitHub Bot commented on HADOOP-18399: - virajjasani commented on PR #5054: URL:

[GitHub] [hadoop] virajjasani commented on pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread via GitHub
virajjasani commented on PR #5054: URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1441203078 > -1 ❌ mvninstall 33m 29s [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/branch-mvninstall-root.txt)

[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692434#comment-17692434 ] ASF GitHub Bot commented on HADOOP-18399: - hadoop-yetus commented on PR #5054: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5054: URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1441113957 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18631) Migrate Async appenders to log4j properties

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692427#comment-17692427 ] ASF GitHub Bot commented on HADOOP-18631: - hadoop-yetus commented on PR #5418: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5418: HADOOP-18631. Migrate Async appenders to log4j properties

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5418: URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1441096436 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18631) Migrate Async appenders to log4j properties

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692426#comment-17692426 ] ASF GitHub Bot commented on HADOOP-18631: - hadoop-yetus commented on PR #5418: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5418: HADOOP-18631. Migrate Async appenders to log4j properties

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5418: URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1441085608 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] slfan1989 commented on pull request #5335: YARN-11426. Improve YARN NodeLabel Memory Display.

2023-02-22 Thread via GitHub
slfan1989 commented on PR #5335: URL: https://github.com/apache/hadoop/pull/5335#issuecomment-1441019375 @ayushtkn Can you help review this pr? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hadoop] slfan1989 commented on pull request #5394: YARN-5604. [Federation] Add versioning for FederationStateStore.

2023-02-22 Thread via GitHub
slfan1989 commented on PR #5394: URL: https://github.com/apache/hadoop/pull/5394#issuecomment-1441018320 @goiri Can you help review this pr again? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hadoop] slfan1989 commented on pull request #5126: YARN-11370. [Federation] Refactor MemoryFederationStateStore code.

2023-02-22 Thread via GitHub
slfan1989 commented on PR #5126: URL: https://github.com/apache/hadoop/pull/5126#issuecomment-1441017546 @goiri Thank you very much for your help in reviewing the code! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hadoop] rdingankar commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
rdingankar commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1115082559 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java: ## @@ -1122,6 +1124,7 @@ public void copyBlock(final

[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692393#comment-17692393 ] ASF GitHub Bot commented on HADOOP-18399: - virajjasani commented on PR #5054: URL:

[GitHub] [hadoop] virajjasani commented on pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread via GitHub
virajjasani commented on PR #5054: URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1440881795 Ran the tests with `scale` and `prefetch` profiles against `us-west-2`: ``` $mvn -Dparallel-tests -DtestsThreadCount=8 -Dprefetch -Dscale clean verify ``` Test

[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692390#comment-17692390 ] ASF GitHub Bot commented on HADOOP-18399: - virajjasani commented on code in PR #5054: URL:

[GitHub] [hadoop] virajjasani commented on a diff in pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread via GitHub
virajjasani commented on code in PR #5054: URL: https://github.com/apache/hadoop/pull/5054#discussion_r1115017372 ## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java: ## @@ -176,7 +179,8 @@ private Entry

[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692389#comment-17692389 ] ASF GitHub Bot commented on HADOOP-18399: - virajjasani commented on code in PR #5054: URL:

[GitHub] [hadoop] virajjasani commented on a diff in pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-02-22 Thread via GitHub
virajjasani commented on code in PR #5054: URL: https://github.com/apache/hadoop/pull/5054#discussion_r1115016369 ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingCacheFiles.java: ## @@ -0,0 +1,124 @@ +/* + * Licensed to the Apache Software

[GitHub] [hadoop] rdingankar commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
rdingankar commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1115011334 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java: ## @@ -1936,4 +1936,20 @@ public static boolean isParentEntry(final String

[GitHub] [hadoop] rdingankar commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
rdingankar commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114847865 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java: ## @@ -1122,6 +1124,7 @@ public void copyBlock(final

[GitHub] [hadoop] omalley merged pull request #5346: HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used.

2023-02-22 Thread via GitHub
omalley merged PR #5346: URL: https://github.com/apache/hadoop/pull/5346 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5426: MAPREDUCE-7434. Fix ShuffleHandler tests.

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5426: URL: https://github.com/apache/hadoop/pull/5426#issuecomment-1440849204 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] goiri merged pull request #5126: YARN-11370. [Federation] Refactor MemoryFederationStateStore code.

2023-02-22 Thread via GitHub
goiri merged PR #5126: URL: https://github.com/apache/hadoop/pull/5126 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5427: Bump derby from 10.10.2.0 to 10.14.2.0 in /hadoop-project

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5427: URL: https://github.com/apache/hadoop/pull/5427#issuecomment-1440756424 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5424: HDFS-16931. Observer nn delete blocks asynchronously when tail OP_DEL…

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5424: URL: https://github.com/apache/hadoop/pull/5424#issuecomment-1440749446 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Updated] (HADOOP-18582) No need to clean tmp files in distcp direct mode

2023-02-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18582: Description: it not necessary to do `cleanupTempFiles`  while ditcp commit job in direct 

[jira] [Commented] (HADOOP-18582) No need to clean tmp files in distcp direct mode

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692338#comment-17692338 ] ASF GitHub Bot commented on HADOOP-18582: - steveloughran commented on PR #5409: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5409: HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp.

2023-02-22 Thread via GitHub
steveloughran commented on PR #5409: URL: https://github.com/apache/hadoop/pull/5409#issuecomment-1440684847 merged; will note on jira that both patches are needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hadoop] mkuchenbecker commented on a diff in pull request #5322: HDFS-16896 clear ignoredNodes list when we clear deadnode list on ref…

2023-02-22 Thread via GitHub
mkuchenbecker commented on code in PR #5322: URL: https://github.com/apache/hadoop/pull/5322#discussion_r1114855356 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java: ## @@ -197,6 +197,15 @@ private void clearLocalDeadNodes() {

[jira] [Commented] (HADOOP-18582) No need to clean tmp files in distcp direct mode

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692336#comment-17692336 ] ASF GitHub Bot commented on HADOOP-18582: - steveloughran commented on PR #5409: URL:

[jira] [Commented] (HADOOP-18582) No need to clean tmp files in distcp direct mode

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692335#comment-17692335 ] ASF GitHub Bot commented on HADOOP-18582: - steveloughran merged PR #5409: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5409: HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp.

2023-02-22 Thread via GitHub
steveloughran commented on PR #5409: URL: https://github.com/apache/hadoop/pull/5409#issuecomment-1440680482 thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hadoop] steveloughran merged pull request #5409: HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp.

2023-02-22 Thread via GitHub
steveloughran merged PR #5409: URL: https://github.com/apache/hadoop/pull/5409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Resolved] (HADOOP-18535) Implement token storage solution based on MySQL

2023-02-22 Thread Owen O'Malley (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved HADOOP-18535. Fix Version/s: 3.3.6 3.4.0 Resolution: Fixed Thanks, Hector!

[GitHub] [hadoop] rdingankar commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
rdingankar commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114852429 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java: ## @@ -1936,4 +1936,20 @@ public static boolean isParentEntry(final String

[GitHub] [hadoop] rdingankar commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
rdingankar commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114847865 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java: ## @@ -1122,6 +1124,7 @@ public void copyBlock(final

[GitHub] [hadoop] mkuchenbecker commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
mkuchenbecker commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114838006 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java: ## @@ -1936,4 +1936,20 @@ public static boolean isParentEntry(final String

[GitHub] [hadoop] mkuchenbecker commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
mkuchenbecker commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114836649 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java: ## @@ -1122,6 +1124,7 @@ public void copyBlock(final

[jira] [Commented] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692321#comment-17692321 ] ASF GitHub Bot commented on HADOOP-18639: - hadoop-yetus commented on PR #5425: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5425: HADOOP-18639 DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5425: URL: https://github.com/apache/hadoop/pull/5425#issuecomment-1440644469 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] mkuchenbecker commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads

2023-02-22 Thread via GitHub
mkuchenbecker commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114833478 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java: ## @@ -1936,4 +1936,20 @@ public static boolean isParentEntry(final String

[jira] [Commented] (HADOOP-18535) Implement token storage solution based on MySQL

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692316#comment-17692316 ] ASF GitHub Bot commented on HADOOP-18535: - omalley closed pull request #5325: HADOOP-18535.

[jira] [Commented] (HADOOP-18535) Implement token storage solution based on MySQL

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692315#comment-17692315 ] ASF GitHub Bot commented on HADOOP-18535: - omalley commented on PR #5325: URL:

[GitHub] [hadoop] dependabot[bot] opened a new pull request, #5427: Bump derby from 10.10.2.0 to 10.14.2.0 in /hadoop-project

2023-02-22 Thread via GitHub
dependabot[bot] opened a new pull request, #5427: URL: https://github.com/apache/hadoop/pull/5427 Bumps derby from 10.10.2.0 to 10.14.2.0. [![Dependabot compatibility

[GitHub] [hadoop] omalley closed pull request #5325: HADOOP-18535. Implement token storage solution based on MySQL

2023-02-22 Thread via GitHub
omalley closed pull request #5325: HADOOP-18535. Implement token storage solution based on MySQL URL: https://github.com/apache/hadoop/pull/5325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hadoop] omalley commented on pull request #5325: HADOOP-18535. Implement token storage solution based on MySQL

2023-02-22 Thread via GitHub
omalley commented on PR #5325: URL: https://github.com/apache/hadoop/pull/5325#issuecomment-1440598019 +1 LGTM (I committed with minor touchups.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [hadoop] hadoop-yetus commented on pull request #5426: MAPREDUCE-7434. Fix ShuffleHandler tests.

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5426: URL: https://github.com/apache/hadoop/pull/5426#issuecomment-1440554689 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5426: MAPREDUCE-7434. Fix ShuffleHandler tests.

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5426: URL: https://github.com/apache/hadoop/pull/5426#issuecomment-1440413098 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692263#comment-17692263 ] ASF GitHub Bot commented on HADOOP-18639: - hadoop-yetus commented on PR #5425: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5425: HADOOP-18639 DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5425: URL: https://github.com/apache/hadoop/pull/5425#issuecomment-1440331866 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18631) Migrate Async appenders to log4j properties

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692256#comment-17692256 ] ASF GitHub Bot commented on HADOOP-18631: - virajjasani commented on PR #5418: URL:

[GitHub] [hadoop] virajjasani commented on pull request #5418: HADOOP-18631. Migrate Async appenders to log4j properties

2023-02-22 Thread via GitHub
virajjasani commented on PR #5418: URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1440310580 Will fix the spotbugs warning -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2

2023-02-22 Thread Viraj Jasani (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692255#comment-17692255 ] Viraj Jasani commented on HADOOP-16206: --- For sub-task HADOOP-18631, PR is ready for review. A

[GitHub] [hadoop] tomicooler opened a new pull request, #5426: MAPREDUCE-7434. Fix ShuffleHandler tests.

2023-02-22 Thread via GitHub
tomicooler opened a new pull request, #5426: URL: https://github.com/apache/hadoop/pull/5426 ### Description of PR Fix the user for the ShuffleHandler tests. ### How was this patch tested? I tested manually with the docker environment: `./start-build-env.sh` The 3

[jira] [Updated] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.

2023-02-22 Thread Sejin Hwang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sejin Hwang updated HADOOP-18639: - Description: YARN NodeManager's deletion service has two types of deletion tasks: the

[jira] [Updated] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.

2023-02-22 Thread Sejin Hwang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sejin Hwang updated HADOOP-18639: - Description: YARN NodeManager's deletion service has two types of deletion tasks: the

[jira] [Commented] (HADOOP-18565) AWS SDK V2 - Complete outstanding items

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692216#comment-17692216 ] ASF GitHub Bot commented on HADOOP-18565: - hadoop-yetus commented on PR #5421: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5421: HADOOP-18565. Completes outstanding items for the SDK V2 upgrade.

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5421: URL: https://github.com/apache/hadoop/pull/5421#issuecomment-1440116271 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18631) Migrate Async appenders to log4j properties

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692209#comment-17692209 ] ASF GitHub Bot commented on HADOOP-18631: - hadoop-yetus commented on PR #5418: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5418: HADOOP-18631. Migrate Async appenders to log4j properties

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5418: URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1440080838 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18631) Migrate Async appenders to log4j properties

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692204#comment-17692204 ] ASF GitHub Bot commented on HADOOP-18631: - hadoop-yetus commented on PR #5418: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5418: HADOOP-18631. Migrate Async appenders to log4j properties

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #5418: URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1440062836 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Updated] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18639: Labels: pull-request-available (was: ) > DockerContainerDeletionTask is not removed

[jira] [Commented] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692199#comment-17692199 ] ASF GitHub Bot commented on HADOOP-18639: - Sejin-Hwang opened a new pull request, #5425: URL:

[GitHub] [hadoop] Sejin-Hwang opened a new pull request, #5425: HADOOP-18639 DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed

2023-02-22 Thread via GitHub
Sejin-Hwang opened a new pull request, #5425: URL: https://github.com/apache/hadoop/pull/5425 ### Description of PR When the DockerContainerDeletionTask is completed, it should be deleted from the NodeManager's statestore, just like the FileDeletionTask. JIRA :

[jira] [Updated] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.

2023-02-22 Thread Sejin Hwang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sejin Hwang updated HADOOP-18639: - Description: YARN NodeManager has two types of deletion tasks: the FileDeletionTask for

[jira] [Created] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.

2023-02-22 Thread Sejin Hwang (Jira)
Sejin Hwang created HADOOP-18639: Summary: DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed. Key: HADOOP-18639 URL:

[GitHub] [hadoop] liuaer opened a new pull request, #5424: HDFS-16931. Observer nn delete blocks asynchronously when tail OP_DEL…

2023-02-22 Thread via GitHub
liuaer opened a new pull request, #5424: URL: https://github.com/apache/hadoop/pull/5424 In our HDFS cluster that holds hundreds of millions of metadata, all of which is EC data. When Observer NN replays the OP_DELETE operation which deletes a large directory, involving tens of millions of

[jira] [Commented] (HADOOP-18258) Merging of S3A Audit Logs

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692168#comment-17692168 ] ASF GitHub Bot commented on HADOOP-18258: - hadoop-yetus commented on PR #4383: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #4383: HADOOP-18258. Merging of S3A Audit Logs

2023-02-22 Thread via GitHub
hadoop-yetus commented on PR #4383: URL: https://github.com/apache/hadoop/pull/4383#issuecomment-1439940590 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692149#comment-17692149 ] ASF GitHub Bot commented on HADOOP-18636: - steveloughran merged PR #5412: URL:

[GitHub] [hadoop] steveloughran merged pull request #5412: HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion

2023-02-22 Thread via GitHub
steveloughran merged PR #5412: URL: https://github.com/apache/hadoop/pull/5412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692147#comment-17692147 ] ASF GitHub Bot commented on HADOOP-18636: - steveloughran commented on PR #5412: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5412: HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion

2023-02-22 Thread via GitHub
steveloughran commented on PR #5412: URL: https://github.com/apache/hadoop/pull/5412#issuecomment-1439886887 thanks. That checkwrite is really a safety check about whether a dir is *still* writable; the allocator precreates all parent dirs whenever the config changes merging --

[jira] (HADOOP-17063) S3A deleteObjects hanging/retrying forever

2023-02-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17063 ] Steve Loughran deleted comment on HADOOP-17063: - was (Author: ste...@apache.org): assuming this is in dir marker deletion, how about we disable retries in that case -now that recent

[jira] [Commented] (HADOOP-17881) S3A DeleteOperation to parallelize POSTing of bulk deletes

2023-02-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692140#comment-17692140 ] Steve Loughran commented on HADOOP-17881: - aa bit of shuffling on a page would be good; maybe

[GitHub] [hadoop] steveloughran commented on pull request #5295: YARN-11404. Add junit5 dependency to hadoop-mapreduce-client-app to fix few unit test failure

2023-02-22 Thread via GitHub
steveloughran commented on PR #5295: URL: https://github.com/apache/hadoop/pull/5295#issuecomment-1439816394 ``` Caused by: java.lang.ClassNotFoundException: org.junit.jupiter.api.Assertions at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at

[jira] [Commented] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692120#comment-17692120 ] ASF GitHub Bot commented on HADOOP-18636: - steveloughran commented on PR #5412: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5412: HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion

2023-02-22 Thread via GitHub
steveloughran commented on PR #5412: URL: https://github.com/apache/hadoop/pull/5412#issuecomment-1439805659 to completely clear: that temp dir creation already happens on the first invocation of getLocalPathForWrite, this just adds it to every call (this is cheap if the path exists()).

[jira] [Commented] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692119#comment-17692119 ] ASF GitHub Bot commented on HADOOP-18636: - steveloughran commented on PR #5412: URL:

[jira] [Commented] (HADOOP-18565) AWS SDK V2 - Complete outstanding items

2023-02-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692118#comment-17692118 ] ASF GitHub Bot commented on HADOOP-18565: - ahmarsuhail commented on code in PR #5421: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5412: HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion

2023-02-22 Thread via GitHub
steveloughran commented on PR #5412: URL: https://github.com/apache/hadoop/pull/5412#issuecomment-1439803446 Currently `LocalDirAllocator.getLocalPathForWrite()` will create all target dirs if absent on first invocation or if the config is changed via `confChanged()` On later calls,

  1   2   >