[
https://issues.apache.org/jira/browse/TEZ-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14216642#comment-14216642
]
Bikas Saha commented on TEZ-14:
-------------------------------
Its 1 per physically running task not all tasks, to be clear.
Sending them via the central dispatcher is not going to solve the event
overload/backlog problem. Synchronous methods calls are not a good solution
since they will still be blocking the central dispatcher in updated attempts
state machine. So the time spent on the central dispatcher would not change,
besides resulting in a bunch of sync calls.
Perhaps I was not clear. I am not saying that TEZ-93 be completed now. I am
saying that the code to enable TEZ-93 be written now. Allow entities to send
events without having to worry about mapping which dispatcher is for which
event. The central dispatcher can offload to different dispatchers (something
that I think is already supported in the YARN library). So the speculator event
would go to the speculator dispatcher while the remaining events would continue
to go through the central dispatcher like they do today. That would have very
little risk while keeping the code clean everywhere else.
Thats why I proposed that we dont bloat this patch any further and commit this.
Then follow up with handling the event backlog/pressure issue. IMO, this seems
like a reasonable break-up of work items. Hence I am looking for an agreement
on this and moving forward to complete these and other jiras related to
speculation.
> Support for speculation of slow tasks
> -------------------------------------
>
> Key: TEZ-14
> URL: https://issues.apache.org/jira/browse/TEZ-14
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-14.1.patch, TEZ-14.2.patch, TEZ-14.3.patch,
> TEZ-14.4.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
