abstractdog commented on a change in pull request #152:
URL: https://github.com/apache/tez/pull/152#discussion_r732664840
##########
File path:
tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java
##########
@@ -1793,80 +1793,107 @@ public TaskAttemptStateInternal
transition(TaskAttemptImpl attempt, TaskAttemptE
MultipleArcTransition<TaskAttemptImpl, TaskAttemptEvent,
TaskAttemptStateInternal> {
@Override
- public TaskAttemptStateInternal transition(TaskAttemptImpl attempt,
+ public TaskAttemptStateInternal transition(TaskAttemptImpl sourceAttempt,
TaskAttemptEvent event) {
TaskAttemptEventOutputFailed outputFailedEvent =
(TaskAttemptEventOutputFailed) event;
- TezEvent tezEvent = outputFailedEvent.getInputFailedEvent();
- TezTaskAttemptID failedDestTaId =
tezEvent.getSourceInfo().getTaskAttemptID();
- InputReadErrorEvent readErrorEvent =
(InputReadErrorEvent)tezEvent.getEvent();
+ TezEvent inputFailedEvent = outputFailedEvent.getInputFailedEvent();
+ TezTaskAttemptID failedDestTaId =
inputFailedEvent.getSourceInfo().getTaskAttemptID();
+
+ InputReadErrorEvent readErrorEvent =
(InputReadErrorEvent)inputFailedEvent.getEvent();
int failedInputIndexOnDestTa = readErrorEvent.getIndex();
- if (readErrorEvent.getVersion() != attempt.getID().getId()) {
- throw new TezUncheckedException(attempt.getID()
+
+ if (readErrorEvent.getVersion() != sourceAttempt.getID().getId()) {
+ throw new TezUncheckedException(sourceAttempt.getID()
+ " incorrectly blamed for read error from " + failedDestTaId
+ " at inputIndex " + failedInputIndexOnDestTa + " version"
+ readErrorEvent.getVersion());
}
- LOG.info(attempt.getID()
- + " blamed for read error from " + failedDestTaId
- + " at inputIndex " + failedInputIndexOnDestTa);
- long time = attempt.clock.getTime();
- Long firstErrReportTime =
attempt.uniquefailedOutputReports.get(failedDestTaId);
+ // source host: where the data input is supposed to come from
+ String sHost = sourceAttempt.getNodeId().getHost();
+ // destination: where the data is tried to be fetched to
+ String dHost = readErrorEvent.getDestinationLocalhostName();
+
+ LOG.info("{} (on {}) blamed for read error from {} (on {}) at inputIndex
{}", sourceAttempt.getID(),
+ sHost, failedDestTaId, dHost, failedInputIndexOnDestTa);
+
+ boolean tooManyDownstreamHostsBlamedTheSameUpstreamHost = false;
+ Map<String, Set<String>> downstreamBlamingHosts =
sourceAttempt.getVertex().getDownstreamBlamingHosts();
Review comment:
[ContainerLauncherContext.getNumNodes
](https://github.com/apache/tez/blob/58fca8bb77d63c1ca6e6eb400eb60588159d6ae0/tez-dag/src/main/java/org/apache/tez/dag/app/ContainerLauncherContextImpl.java#L108)seems
to be okay, because it's updated at runtime: every time when a task is
allocated, the node is marked as seen
fortunately, it should work for LLAP without a Hive change, because
[LlapTaskSchedulerService.scheduleTask](https://github.com/apache/hive/blob/5bd9a14e20e1c218c5292b8c37d5d5acf721541d/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java)
takes care of updating it via TaskSchedulerManager.taskAllocated (like
DagAwareYarnTaskScheduler and YarnTaskSchedulerService in Tez)
```
getContext().taskAllocated(taskInfo.task, taskInfo.clientCookie, container);
```
so it doesn't reflect all the nodes, instead, the ones that are being used,
I think I can use this as a max, and use a hostFailureFraction as
reportingDownstreamHosts / numNodes
let me know if it doesn't make sense, I'll try to experiment a bit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
