[
https://issues.apache.org/jira/browse/HADOOP-1719?focusedWorklogId=479391&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479391
]
ASF GitHub Bot logged work on HADOOP-1719:
------------------------------------------
Author: ASF GitHub Bot
Created on: 06/Sep/20 06:42
Start Date: 06/Sep/20 06:42
Worklog Time Spent: 10m
Work Description: bilaharith edited a comment on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-687710639
**Driver test results using accounts in Canary**
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
HNS-OAuth
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 »
TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 64
[INFO] Results:
[INFO]
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24
HNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 »
TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16
NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 »
TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 245
[INFO] Results:
[INFO]
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16
The above error is tracked under the JIRA:
https://issues.apache.org/jira/browse/HADOOP-17160
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 479391)
Time Spent: 0.5h (was: 20m)
> Improve the utilization of shuffle copier threads
> -------------------------------------------------
>
> Key: HADOOP-1719
> URL: https://issues.apache.org/jira/browse/HADOOP-1719
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Devaraj Das
> Assignee: Amar Kamat
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.16.0
>
> Attachments: 1719.1.patch, 1719.patch, 1719.patch, HADOOP-1719.patch,
> HADOOP-1719.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In the current design, the scheduling of copies is done and the scheduler
> (the main loop in fetchOutputs) won't schedule anything until it hears back
> from at least one of the copier threads. Due to this, the main loop won't
> query the TaskTracker asking for new map locations and may not be using all
> the copiers effectively. This may not be an issue for small-sized map
> outputs, where at steady state, the frequency of such notifications is
> frequent.
> Ideally, we should schedule all what we can, and, depending on how busy we
> currently are, query the tasktracker for more map locations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
