[jira] [Work logged] (HADOOP-1719) Improve the utilization of shuffle copier threads
[ https://issues.apache.org/jira/browse/HADOOP-1719?focusedWorklogId=479391=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479391 ] ASF GitHub Bot logged work on HADOOP-1719: -- Author: ASF GitHub Bot Created on: 06/Sep/20 06:42 Start Date: 06/Sep/20 06:42 Worklog Time Spent: 10m Work Description: bilaharith edited a comment on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-687710639 **Driver test results using accounts in Canary** mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t... [INFO] [ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 64 [INFO] Results: [INFO] [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t... [INFO] [ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 24 [INFO] Results: [INFO] [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16 NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t... [INFO] [ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 245 [INFO] Results: [INFO] [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16 The above error is tracked under the JIRA: https://issues.apache.org/jira/browse/HADOOP-17160 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 479391) Time Spent: 0.5h (was: 20m) > Improve the utilization of shuffle copier threads > - > > Key: HADOOP-1719 > URL: https://issues.apache.org/jira/browse/HADOOP-1719 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Devaraj Das >Assignee: Amar Kamat >Priority: Major > Labels: pull-request-available > Fix For: 0.16.0 > > Attachments: 1719.1.patch, 1719.patch, 1719.patch, HADOOP-1719.patch, > HADOOP-1719.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > In the current design, the scheduling of copies is done and the scheduler > (the main loop in fetchOutputs) won't schedule anything until it hears back > from at least one of the copier threads. Due to this, the main loop won't > query the TaskTracker asking for new map locations and may not be using all > the copiers effectively. This may not be an issue for small-sized map > outputs, where at steady state, the frequency of such notifications is > frequent. > Ideally, we should schedule all what we can, and, depending on how busy we > currently are, query the tasktracker for more map locations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-1719) Improve the utilization of shuffle copier threads
[ https://issues.apache.org/jira/browse/HADOOP-1719?focusedWorklogId=479390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479390 ] ASF GitHub Bot logged work on HADOOP-1719: -- Author: ASF GitHub Bot Created on: 06/Sep/20 06:40 Start Date: 06/Sep/20 06:40 Worklog Time Spent: 10m Work Description: bilaharith commented on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-687710639 **Driver test results using accounts in Central India** mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t... [INFO] [ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 64 [INFO] Results: [INFO] [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t... [INFO] [ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 24 [INFO] Results: [INFO] [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16 NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t... [INFO] [ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 245 [INFO] Results: [INFO] [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 479390) Time Spent: 20m (was: 10m) > Improve the utilization of shuffle copier threads > - > > Key: HADOOP-1719 > URL: https://issues.apache.org/jira/browse/HADOOP-1719 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Devaraj Das >Assignee: Amar Kamat >Priority: Major > Labels: pull-request-available > Fix For: 0.16.0 > > Attachments: 1719.1.patch, 1719.patch, 1719.patch, HADOOP-1719.patch, > HADOOP-1719.patch > > Time Spent: 20m > Remaining Estimate: 0h > > In the current design, the scheduling of copies is done and the scheduler > (the main loop in fetchOutputs) won't schedule anything until it hears back > from at least one of the copier threads. Due to this, the main loop won't > query the TaskTracker asking for new map locations and may not be using all > the copiers effectively. This may not be an issue for small-sized map > outputs, where at steady state, the frequency of such notifications is > frequent. > Ideally, we should schedule all what we can, and, depending on how busy we > currently are, query the tasktracker for more map locations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-1719) Improve the utilization of shuffle copier threads
[ https://issues.apache.org/jira/browse/HADOOP-1719?focusedWorklogId=479389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479389 ] ASF GitHub Bot logged work on HADOOP-1719: -- Author: ASF GitHub Bot Created on: 06/Sep/20 06:38 Start Date: 06/Sep/20 06:38 Worklog Time Spent: 10m Work Description: bilaharith opened a new pull request #2278: URL: https://github.com/apache/hadoop/pull/2278 ADLS Gen 2 supports accounts with and without hierarchical namespace support. ABFS driver supports various authorization mechanisms like OAuth, haredKey, Shared Access Signature. The integration tests need to be executed against accounts with and without hierarchical namespace support using various authorization mechanisms. Currently the developer has to manually run the tests with different combinations of configurations. The expectation is to automate these runs with different combinations. The PR introduces a shell script with which the developer can specify the configuration variants and get different combinations of tests executed. The script runtests.sh contain template for 3 combinations of tests. In case any new flags or properties are introduced with the code change, add the combinations with the possible configurations into the runtests.sh. Adding a combination of tests involves setting the variable scenario (ex: HNS -OAuth) and specifying the specific configurations for the particular combination with 2 arrays namely properties and values. Specify the property names within the array properties and corresponding values in the values array. The property and value is determined by the array index. The value for the property mentioned at index 1 of array properties should be specified at index 1 of the array values. Call the function runtestwithconfs once the 3 values mentioned are set. Now the script runtests.sh is ready to be ran. Once the tests are completed, logs will be present in the directory testlogs. A consolidated test results will be present in the file Test-$starttime-Results .log, $startname will be the start time of the test. Similarly, the full test report can be found in individual log files, for each of the scenarios with the file name Test-$starttime-Logs-$scenario. Please attach the consolidates test results from the file Test-$starttime-Results.log into the respective PRs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 479389) Remaining Estimate: 0h Time Spent: 10m > Improve the utilization of shuffle copier threads > - > > Key: HADOOP-1719 > URL: https://issues.apache.org/jira/browse/HADOOP-1719 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Devaraj Das >Assignee: Amar Kamat >Priority: Major > Fix For: 0.16.0 > > Attachments: 1719.1.patch, 1719.patch, 1719.patch, HADOOP-1719.patch, > HADOOP-1719.patch > > Time Spent: 10m > Remaining Estimate: 0h > > In the current design, the scheduling of copies is done and the scheduler > (the main loop in fetchOutputs) won't schedule anything until it hears back > from at least one of the copier threads. Due to this, the main loop won't > query the TaskTracker asking for new map locations and may not be using all > the copiers effectively. This may not be an issue for small-sized map > outputs, where at steady state, the frequency of such notifications is > frequent. > Ideally, we should schedule all what we can, and, depending on how busy we > currently are, query the tasktracker for more map locations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
