[
https://issues.apache.org/jira/browse/TEZ-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207102#comment-14207102
]
Siddharth Seth commented on TEZ-14:
-----------------------------------
Couple of other comments.
- The TaskAttempt sending an event directly to the vertex (about the attempt)
is avoidable. Instead, this should route via the Task. Ideally, just inform the
speculator directly as Hitesh mentioned - and short circuit the queue delays.
- Instead of relying on TaskAttemptStatus - we could just rely on querying the
status from the task itself. Especially the case if the TaskAttemptStatus
instance is being saved anywhere in the speculator, since that can be changed
by new status reports.
- DAGImpl - getTask - should return Task instead of TaskImpl. Returning
TaskImpl makes unit testing and mock difficult.
- Nit: ta.reportedStatus.progress - Set to 1.0f instead of 1
- The TODO about the race needs more explanation
- The if (task.leafVertex) test is removed from the KillTransition. Does this
need to be added elsewhere - after taking speculation into account ? Also, the
same check exists for the FailTransition. Does that need to change ?
- DataStatistics.updateStatisitcs never used
- {code}+ // This race conditon is okay. If we skip a speculation attempt we
+ // should have tried because the event that lowers the number of
+ // containers needed to zero hasn't come through,{code}
The comment seems invalid since we're not dealing with container requests.
- speculationValue(taskId, time) - send task as well instead of looking it up
again (read-lock)
- Progress tracking is of almost no use since it's left up to Processors, and I
don't think we mandate this being set. The speculator may be simpler dealing
with just completions.
- As Hitesh mentioned, docs within the speculator on what it's trying to do
would be useful.
- Nit: mapperStatistics rename to taskStatistics
> Support for speculation of slow tasks
> -------------------------------------
>
> Key: TEZ-14
> URL: https://issues.apache.org/jira/browse/TEZ-14
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-14.1.patch, TEZ-14.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
