[GitHub] [hadoop] iwasakims commented on pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
iwasakims commented on pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#issuecomment-755926955 > * The implementation has lots of code redundancy. > * It is inefficient in the setup and tearing down. The large percentage of time execution is exhausted by starting

[GitHub] [hadoop] iwasakims commented on pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
iwasakims commented on pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#issuecomment-755926211 Thank for working on this, @amahussein. LGTM overall pending some nits. While it is too late here, it is hard to follow which part of the TestDistributedShell is

[GitHub] [hadoop] iwasakims commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
iwasakims commented on a change in pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#discussion_r553140533 ## File path:

[GitHub] [hadoop] iwasakims commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
iwasakims commented on a change in pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#discussion_r553140343 ## File path:

[GitHub] [hadoop] hadoop-yetus commented on pull request #2578: [HDFS-15754] Add DataNode packet metrics

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2578: URL: https://github.com/apache/hadoop/pull/2578#issuecomment-755923450 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[GitHub] [hadoop] hadoop-yetus commented on pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#issuecomment-755917599 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[GitHub] [hadoop] sunchao commented on a change in pull request #2578: [HDFS-15754] Add DataNode packet metrics

2021-01-06 Thread GitBox
sunchao commented on a change in pull request #2578: URL: https://github.com/apache/hadoop/pull/2578#discussion_r553128834 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java ## @@ -161,6 +163,53 @@

[jira] [Work logged] (HADOOP-17433) Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17433?focusedWorklogId=532280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532280 ] ASF GitHub Bot logged work on HADOOP-17433: --- Author: ASF GitHub Bot

[GitHub] [hadoop] mukund-thakur commented on pull request #2600: HADOOP-17433. Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole.

2021-01-06 Thread GitBox
mukund-thakur commented on pull request #2600: URL: https://github.com/apache/hadoop/pull/2600#issuecomment-755902810 LGTM +1 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [hadoop] amahussein commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
amahussein commented on a change in pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#discussion_r553108878 ## File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java

[GitHub] [hadoop] amahussein commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
amahussein commented on a change in pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#discussion_r553108800 ## File path:

[GitHub] [hadoop] amahussein commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
amahussein commented on a change in pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#discussion_r553108729 ## File path:

[GitHub] [hadoop] hadoop-yetus commented on pull request #2603: Merge pull request #1 from apache/trunk

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2603: URL: https://github.com/apache/hadoop/pull/2603#issuecomment-755870419 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[jira] [Work logged] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17408?focusedWorklogId=532251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532251 ] ASF GitHub Bot logged work on HADOOP-17408: --- Author: ASF GitHub Bot

[GitHub] [hadoop] hadoop-yetus commented on pull request #2601: HADOOP-17408. Optimize NetworkTopology sorting block locations.

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2601: URL: https://github.com/apache/hadoop/pull/2601#issuecomment-755863757 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[GitHub] [hadoop] zml4518079 opened a new pull request #2603: Merge pull request #1 from apache/trunk

2021-01-06 Thread GitBox
zml4518079 opened a new pull request #2603: URL: https://github.com/apache/hadoop/pull/2603 sychronizing code ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding

[jira] [Work logged] (HADOOP-17452) Upgrade guice to 4.2.3

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17452?focusedWorklogId=532245=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532245 ] ASF GitHub Bot logged work on HADOOP-17452: --- Author: ASF GitHub Bot

[GitHub] [hadoop] aajisaka commented on pull request #2582: HADOOP-17452. Upgrade Guice to 4.2.3

2021-01-06 Thread GitBox
aajisaka commented on pull request #2582: URL: https://github.com/apache/hadoop/pull/2582#issuecomment-755850969 > Let me check if there are any test failures due to this change. Created #2602 This is an automated

[jira] [Work logged] (HADOOP-17452) Upgrade guice to 4.2.3

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17452?focusedWorklogId=532244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532244 ] ASF GitHub Bot logged work on HADOOP-17452: --- Author: ASF GitHub Bot

[GitHub] [hadoop] aajisaka opened a new pull request #2602: Test PR for HADOOP-17452

2021-01-06 Thread GitBox
aajisaka opened a new pull request #2602: URL: https://github.com/apache/hadoop/pull/2602 Added a commit to run all the YARN unit tests for #2582 This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hadoop] iwasakims commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
iwasakims commented on a change in pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#discussion_r553079537 ## File path:

[GitHub] [hadoop] iwasakims commented on pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
iwasakims commented on pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#issuecomment-755846099 Is the fix of o.a.h.tools.dynamometer.Cliet related to TestDistributedShell? It should be addressed in another JIRA if not.

[jira] [Work logged] (HADOOP-17452) Upgrade guice to 4.2.3

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17452?focusedWorklogId=532228=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532228 ] ASF GitHub Bot logged work on HADOOP-17452: --- Author: ASF GitHub Bot

[GitHub] [hadoop] hadoop-yetus commented on pull request #2582: HADOOP-17452. Upgrade Guice to 4.2.3

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2582: URL: https://github.com/apache/hadoop/pull/2582#issuecomment-755829152 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[GitHub] [hadoop] hadoop-yetus commented on pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#issuecomment-755824076 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[GitHub] [hadoop] amahussein commented on pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
amahussein commented on pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#issuecomment-755797255 > > @goiri the code changes fix {{testDSShellWithEnforceExecutionType}}. > > The problem with the test was that it launches two containers executing cmd `date`. Apparently

[GitHub] [hadoop] sunchao commented on a change in pull request #2578: [HDFS-15754] Add DataNode packet metrics

2021-01-06 Thread GitBox
sunchao commented on a change in pull request #2578: URL: https://github.com/apache/hadoop/pull/2578#discussion_r55296 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java ## @@ -690,4 +695,20 @@

[GitHub] [hadoop] goiri commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
goiri commented on a change in pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#discussion_r553013460 ## File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java

[GitHub] [hadoop] goiri commented on pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
goiri commented on pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#issuecomment-755773457 > @goiri the code changes fix {{testDSShellWithEnforceExecutionType}}. > The problem with the test was that it launches two containers executing cmd `date`. Apparently the two

[GitHub] [hadoop] amahussein commented on pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
amahussein commented on pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#issuecomment-755767916 CC: @iwasakims . This includes a fix to `testDSShellWithEnforceExecutionType` This is an automated message

[GitHub] [hadoop] amahussein edited a comment on pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
amahussein edited a comment on pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#issuecomment-755765315 @goiri the code changes fix {{testDSShellWithEnforceExecutionType}}. The problem with the test was that it launches two containers executing cmd `date`. Apparently

[GitHub] [hadoop] amahussein commented on pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
amahussein commented on pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#issuecomment-755765315 @goiri the code changes fix {{testDSShellWithOpportunisticContainers}}. The problem with the test was that it launches two containers executing cmd `date`. Apparently the

[GitHub] [hadoop] amahussein commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
amahussein commented on a change in pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#discussion_r553003268 ## File path:

[GitHub] [hadoop] goiri commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell

2021-01-06 Thread GitBox
goiri commented on a change in pull request #2581: URL: https://github.com/apache/hadoop/pull/2581#discussion_r553001971 ## File path:

[jira] [Commented] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations

2021-01-06 Thread Ahmed Hussein (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260090#comment-17260090 ] Ahmed Hussein commented on HADOOP-17408: Thanks [~Jim_Brennan] for the feedback. I created a

[GitHub] [hadoop] amahussein opened a new pull request #2601: HADOOP-17408. Optimize NetworkTopology sorting block locations.

2021-01-06 Thread GitBox
amahussein opened a new pull request #2601: URL: https://github.com/apache/hadoop/pull/2601 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g.

[jira] [Work logged] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17408?focusedWorklogId=532174=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532174 ] ASF GitHub Bot logged work on HADOOP-17408: --- Author: ASF GitHub Bot

[jira] [Work logged] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17408?focusedWorklogId=532171=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532171 ] ASF GitHub Bot logged work on HADOOP-17408: --- Author: ASF GitHub Bot

[GitHub] [hadoop] amahussein closed pull request #2514: HADOOP-17408. Optimize NetworkTopology while sorting of block locations.

2021-01-06 Thread GitBox
amahussein closed pull request #2514: URL: https://github.com/apache/hadoop/pull/2514 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[jira] [Work logged] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17408?focusedWorklogId=532169=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532169 ] ASF GitHub Bot logged work on HADOOP-17408: --- Author: ASF GitHub Bot

[GitHub] [hadoop] hadoop-yetus commented on pull request #2514: HADOOP-17408. Optimize NetworkTopology while sorting of block locations.

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2514: URL: https://github.com/apache/hadoop/pull/2514#issuecomment-755756961 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[jira] [Work logged] (HADOOP-17414) Magic committer files don't have the count of bytes written collected by spark

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17414?focusedWorklogId=532087=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532087 ] ASF GitHub Bot logged work on HADOOP-17414: --- Author: ASF GitHub Bot

[GitHub] [hadoop] liuml07 commented on pull request #2530: HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark

2021-01-06 Thread GitBox
liuml07 commented on pull request #2530: URL: https://github.com/apache/hadoop/pull/2530#issuecomment-755566048 Will take a look this week. Thanks! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hadoop] hadoop-yetus commented on pull request #2598: HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStre…

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2598: URL: https://github.com/apache/hadoop/pull/2598#issuecomment-755522663 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[jira] [Work logged] (HADOOP-17404) ABFS: Piggyback flush on Append calls for short writes

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17404?focusedWorklogId=532058=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532058 ] ASF GitHub Bot logged work on HADOOP-17404: --- Author: ASF GitHub Bot

[GitHub] [hadoop] DadanielZ merged pull request #2509: HADOOP-17404. ABFS: Small write - Merge append and flush

2021-01-06 Thread GitBox
DadanielZ merged pull request #2509: URL: https://github.com/apache/hadoop/pull/2509 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[jira] [Work logged] (HADOOP-17404) ABFS: Piggyback flush on Append calls for short writes

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17404?focusedWorklogId=532053=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532053 ] ASF GitHub Bot logged work on HADOOP-17404: --- Author: ASF GitHub Bot

[GitHub] [hadoop] hadoop-yetus commented on pull request #2509: HADOOP-17404. ABFS: Small write - Merge append and flush

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2509: URL: https://github.com/apache/hadoop/pull/2509#issuecomment-755484937 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[jira] [Work logged] (HADOOP-17337) NetworkBinding has a runtime class dependency on a third-party shaded class

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17337?focusedWorklogId=532042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532042 ] ASF GitHub Bot logged work on HADOOP-17337: --- Author: ASF GitHub Bot

[GitHub] [hadoop] cwensel commented on pull request #2599: HADOOP-17337. NetworkBinding has a runtime dependency on shaded httpclient

2021-01-06 Thread GitBox
cwensel commented on pull request #2599: URL: https://github.com/apache/hadoop/pull/2599#issuecomment-755469136 @steveloughran without actually running it, - what would the fallback behavior be if the shaded artifact isn't found leaving the default SSL factory? - should the

[jira] [Work logged] (HADOOP-17433) Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17433?focusedWorklogId=532026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532026 ] ASF GitHub Bot logged work on HADOOP-17433: --- Author: ASF GitHub Bot

[GitHub] [hadoop] hadoop-yetus commented on pull request #2600: HADOOP-17433. Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole.

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2600: URL: https://github.com/apache/hadoop/pull/2600#issuecomment-755457753 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[jira] [Work logged] (HADOOP-17337) NetworkBinding has a runtime class dependency on a third-party shaded class

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17337?focusedWorklogId=532015=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532015 ] ASF GitHub Bot logged work on HADOOP-17337: --- Author: ASF GitHub Bot

[GitHub] [hadoop] hadoop-yetus commented on pull request #2599: HADOOP-17337. NetworkBinding has a runtime dependency on shaded httpclient

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2599: URL: https://github.com/apache/hadoop/pull/2599#issuecomment-755449206 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[jira] [Work logged] (HADOOP-17414) Magic committer files don't have the count of bytes written collected by spark

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17414?focusedWorklogId=531997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531997 ] ASF GitHub Bot logged work on HADOOP-17414: --- Author: ASF GitHub Bot

[GitHub] [hadoop] sunchao commented on pull request #2530: HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark

2021-01-06 Thread GitBox
sunchao commented on pull request #2530: URL: https://github.com/apache/hadoop/pull/2530#issuecomment-755439979 > @sunchao @liuml07 could either of you take a look @ this? Yes I can help on this PR, sorry for the delay. Will spend some time this week.

[jira] [Work logged] (HADOOP-16080) hadoop-aws does not work with hadoop-client-api

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16080?focusedWorklogId=531993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531993 ] ASF GitHub Bot logged work on HADOOP-16080: --- Author: ASF GitHub Bot

[GitHub] [hadoop] sunchao commented on pull request #2575: HADOOP-16080. hadoop-aws does not work with hadoop-client-api

2021-01-06 Thread GitBox
sunchao commented on pull request #2575: URL: https://github.com/apache/hadoop/pull/2575#issuecomment-755439216 > some of the tests are parameterized to do test runs with/without dynamoDB. They shouldn't be run if the -Ddynamo option wasn't set, but what has inevitably happened is that

[jira] [Work logged] (HADOOP-17414) Magic committer files don't have the count of bytes written collected by spark

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17414?focusedWorklogId=531989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531989 ] ASF GitHub Bot logged work on HADOOP-17414: --- Author: ASF GitHub Bot

[GitHub] [hadoop] steveloughran commented on pull request #2530: HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark

2021-01-06 Thread GitBox
steveloughran commented on pull request #2530: URL: https://github.com/apache/hadoop/pull/2530#issuecomment-755437582 @sunchao @liuml07 could either of you take a look @ this? This is an automated message from the Apache Git

[jira] [Work logged] (HADOOP-17404) ABFS: Piggyback flush on Append calls for short writes

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17404?focusedWorklogId=531980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531980 ] ASF GitHub Bot logged work on HADOOP-17404: --- Author: ASF GitHub Bot

[GitHub] [hadoop] snvijaya commented on pull request #2509: HADOOP-17404. ABFS: Small write - Merge append and flush

2021-01-06 Thread GitBox
snvijaya commented on pull request #2509: URL: https://github.com/apache/hadoop/pull/2509#issuecomment-755430384 Thanks @DadanielZ. Have addressed the review comment. Kindly request your review. Latest test results: HNS-OAuth [INFO] Results:

[jira] [Work logged] (HADOOP-17404) ABFS: Piggyback flush on Append calls for short writes

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17404?focusedWorklogId=531979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531979 ] ASF GitHub Bot logged work on HADOOP-17404: --- Author: ASF GitHub Bot

[GitHub] [hadoop] snvijaya commented on a change in pull request #2509: HADOOP-17404. ABFS: Small write - Merge append and flush

2021-01-06 Thread GitBox
snvijaya commented on a change in pull request #2509: URL: https://github.com/apache/hadoop/pull/2509#discussion_r552811022 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java ## @@ -55,6 +55,7 @@ public

[jira] [Work logged] (HADOOP-17451) intermittent failure of S3A tests which make assertions on statistics/IOStatistics

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17451?focusedWorklogId=531977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531977 ] ASF GitHub Bot logged work on HADOOP-17451: --- Author: ASF GitHub Bot

[GitHub] [hadoop] hadoop-yetus commented on pull request #2594: HADOOP-17451. IOStatistics test failures in S3A code.

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2594: URL: https://github.com/apache/hadoop/pull/2594#issuecomment-755427098 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[jira] [Updated] (HADOOP-17433) Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17433: Labels: pull-request-available (was: ) > Skipping network I/O in S3A getFileStatus(/)

[jira] [Work logged] (HADOOP-17433) Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17433?focusedWorklogId=531959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531959 ] ASF GitHub Bot logged work on HADOOP-17433: --- Author: ASF GitHub Bot

[GitHub] [hadoop] steveloughran commented on pull request #2600: HADOOP-17433. Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole.

2021-01-06 Thread GitBox
steveloughran commented on pull request #2600: URL: https://github.com/apache/hadoop/pull/2600#issuecomment-755411529 FYI @mukund-thakur @bgaborg This is an automated message from the Apache Git Service. To respond to the

[jira] [Work logged] (HADOOP-17433) Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17433?focusedWorklogId=531958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531958 ] ASF GitHub Bot logged work on HADOOP-17433: --- Author: ASF GitHub Bot

[GitHub] [hadoop] steveloughran opened a new pull request #2600: HADOOP-17433. Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole.

2021-01-06 Thread GitBox
steveloughran opened a new pull request #2600: URL: https://github.com/apache/hadoop/pull/2600 ran ITestAssumeRole against s3 ireland in IDE and then from cli with/without s3guard This is an automated message from the

[jira] [Commented] (HADOOP-13845) s3a to instrument duration of HTTP calls

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259848#comment-17259848 ] Steve Loughran commented on HADOOP-13845: - HADOOP-17271 does this for HEAD and LIST requests,

[jira] [Work logged] (HADOOP-17337) NetworkBinding has a runtime class dependency on a third-party shaded class

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17337?focusedWorklogId=531949=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531949 ] ASF GitHub Bot logged work on HADOOP-17337: --- Author: ASF GitHub Bot

[GitHub] [hadoop] steveloughran commented on pull request #2599: HADOOP-17337. NetworkBinding has a runtime dependency on shaded httpclient

2021-01-06 Thread GitBox
steveloughran commented on pull request #2599: URL: https://github.com/apache/hadoop/pull/2599#issuecomment-755398014 testing: done the unit tests but not a full live run (which it will need) @cwensel -this look good to you?

[jira] [Work logged] (HADOOP-17337) NetworkBinding has a runtime class dependency on a third-party shaded class

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17337?focusedWorklogId=531947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531947 ] ASF GitHub Bot logged work on HADOOP-17337: --- Author: ASF GitHub Bot

[jira] [Updated] (HADOOP-17337) NetworkBinding has a runtime class dependency on a third-party shaded class

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17337: Labels: pull-request-available (was: ) > NetworkBinding has a runtime class dependency

[GitHub] [hadoop] steveloughran opened a new pull request #2599: HADOOP-17337. NetworkBinding has a runtime dependency on shaded httpclient

2021-01-06 Thread GitBox
steveloughran opened a new pull request #2599: URL: https://github.com/apache/hadoop/pull/2599 Adds another class with the dependencies, uses reflection to load and invoke that. Change-Id: Iaad9ede15dc6ac3240cba3dfa80c79825dbd007c

[jira] [Resolved] (HADOOP-17312) S3AInputStream to be resilient to faiures in abort(); translate AWS Exceptions

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17312. - Fix Version/s: 3.3.1 Resolution: Duplicate > S3AInputStream to be resilient to

[jira] [Resolved] (HADOOP-16133) S3A statistic collection underrecords bytes written in helper threads

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16133. - Fix Version/s: 3.3.1 Resolution: Done > S3A statistic collection underrecords

[jira] [Commented] (HADOOP-16618) increase the default number of threads and http connections in S3A

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259830#comment-17259830 ] Steve Loughran commented on HADOOP-16618: - also: socket buffer sizes > increase the default

[jira] [Commented] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259829#comment-17259829 ] Steve Loughran commented on HADOOP-13704: - should really fix this rather than repeated open

[jira] [Resolved] (HADOOP-16468) S3AFileSystem.getContentSummary() to use listFiles(recursive)

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16468. - Fix Version/s: hadoop-13704 Resolution: Duplicate >

[jira] [Resolved] (HADOOP-17359) [Hadoop-Tools]S3A MultiObjectDeleteException after uploading a file

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17359. - Resolution: Cannot Reproduce well, if you can't reproduce it and nobody else can,

[GitHub] [hadoop] touchida commented on pull request #2585: HDFS-15759. EC: Verify EC reconstruction correctness on DataNode

2021-01-06 Thread GitBox
touchida commented on pull request #2585: URL: https://github.com/apache/hadoop/pull/2585#issuecomment-755382135 Failed unit tests: * org.apache.hadoop.hdfs.TestMultipleNNPortQOP.testMultipleNNPortOverwriteDownStream * This failure is obviously unrelated to this PR. I filed it in

[GitHub] [hadoop] touchida commented on pull request #2598: HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStre…

2021-01-06 Thread GitBox
touchida commented on pull request #2598: URL: https://github.com/apache/hadoop/pull/2598#issuecomment-755361761 The above build seems to have been aborted when I converted this PR to draft. This is an automated message from

[jira] [Commented] (HADOOP-17437) Update Hadoop Documentation with a new AWS Credential Provider used with EKS

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259760#comment-17259760 ] Steve Loughran commented on HADOOP-17437: - yes, do that. Maybe actually start a section or doc

[jira] [Updated] (HADOOP-17444) ADLFS: Update SDK version from 2.3.6 to 2.3.9

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17444: Resolution: Fixed Status: Resolved (was: Patch Available) Fixed in 3.4; wil

[jira] [Work logged] (HADOOP-17444) ADLFS: Update SDK version from 2.3.6 to 2.3.9

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17444?focusedWorklogId=531903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531903 ] ASF GitHub Bot logged work on HADOOP-17444: --- Author: ASF GitHub Bot

[GitHub] [hadoop] steveloughran commented on pull request #2551: HADOOP-17444. ADLS Gen1: Updating gen1 SDK version from 2.3.6 to 2.3.9

2021-01-06 Thread GitBox
steveloughran commented on pull request #2551: URL: https://github.com/apache/hadoop/pull/2551#issuecomment-755334143 +1; merged to trunk. Not pulled in to -3.3, but only because the asf repo seems to be out of sync and so I can't easily do the cherrypick @bilaharith could you do a

[jira] [Work logged] (HADOOP-17444) ADLFS: Update SDK version from 2.3.6 to 2.3.9

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17444?focusedWorklogId=531901=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531901 ] ASF GitHub Bot logged work on HADOOP-17444: --- Author: ASF GitHub Bot

[GitHub] [hadoop] steveloughran merged pull request #2551: HADOOP-17444. ADLS Gen1: Updating gen1 SDK version from 2.3.6 to 2.3.9

2021-01-06 Thread GitBox
steveloughran merged pull request #2551: URL: https://github.com/apache/hadoop/pull/2551 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[jira] [Commented] (HADOOP-17458) S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259749#comment-17259749 ] Steve Loughran commented on HADOOP-17458: - HADOOP-17312 shows a similar-but-different error

[GitHub] [hadoop] hadoop-yetus commented on pull request #2598: HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStre…

2021-01-06 Thread GitBox
hadoop-yetus commented on pull request #2598: URL: https://github.com/apache/hadoop/pull/2598#issuecomment-755322085 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0

[jira] [Commented] (HADOOP-17458) S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259743#comment-17259743 ] Steve Loughran commented on HADOOP-17458: - {code} [ERROR]

[jira] [Created] (HADOOP-17458) S3A to treat "dkClientException: Data read has a different length than the expected" as EOFException

2021-01-06 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17458: --- Summary: S3A to treat "dkClientException: Data read has a different length than the expected" as EOFException Key: HADOOP-17458 URL:

[jira] [Updated] (HADOOP-17458) S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17458: Summary: S3A to treat "SdkClientException: Data read has a different length than the

[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259742#comment-17259742 ] Steve Loughran commented on HADOOP-17338: - now, I just got the error {{Data read has a

[jira] [Commented] (HADOOP-17456) S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259732#comment-17259732 ] Steve Loughran commented on HADOOP-17456: - need to assert on count of bulk delete calls, and

[jira] [Commented] (HADOOP-17455) [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259728#comment-17259728 ] Steve Loughran commented on HADOOP-17455: - cause is the assert was probing the wrong metric:

[jira] [Issue Comment Deleted] (HADOOP-17456) S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17456: Comment: was deleted (was: cause is the assert was probing the wrong metric: #of objects

  1   2   >