[
https://issues.apache.org/jira/browse/TEZ-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234784#comment-14234784
]
Bikas Saha edited comment on TEZ-1789 at 12/4/14 11:27 PM:
-----------------------------------------------------------
Attaching a patch based on the approach outlined in TEZ-14.
An instance of AsyncDispatcher is created for speculation events. Its event
handler is passed to the central dispatcher as the handler for speculator
events. So the central dispatcher delegates these events to the dispatcher for
speculator events. This new dispatcher calls the real handler which passes the
events to the speculator. This makes event handling transparent to the
sender/receiver. They continue to send to the same dispatcher that in turn
multiplexes to others. This patch shows that this works as designed (there is
no event specific dispatch code in TaskAttemptImpl). So this methodology could
be extended to delegate Attempt, Task, Vertex, VertexManager events to separate
dispatchers so that they dont block each other. Which events get routed to
which dispatcher can be changed without having to change the senders/receivers.
Existing speculation tests pass and I have manually verified in the debugger
that the event flow in those tests are delegating as designed.
[~sseth] [~hitesh] [~jeagles] [~zjffdu] Please review.
was (Author: bikassaha):
Attaching a patch based on the approach outlined in TEZ-14.
An instance of AsyncDispatcher is created for speculation events. Its event
handler is passed to the central dispatcher as the handler for speculator
events. So the central dispatcher delegates these events to the dispatcher for
speculator events. This new dispatcher calls the real handler which passes the
events to the speculator. This makes event handling transparent to the
sender/receiver. They continue to send to the same dispatcher that in turn
multiplexes to others. This patch shows that this works as designed (not event
specific changes in TaskAttemptImpl). So this methodology could be extended to
delegate Attempt, Task, Vertex, VertexManager events to separate dispatchers so
that they dont block each other. Which events get routed to which dispatcher
can be changed without having to change the senders/receivers.
Existing speculation tests pass and I have manually verified in the debugger
that the event flow in those tests are delegating as designed.
[~sseth] [~hitesh] [~jeagles] [~zjffdu] Please review.
> Move speculator processing off the central dispatcher
> -----------------------------------------------------
>
> Key: TEZ-1789
> URL: https://issues.apache.org/jira/browse/TEZ-1789
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Attachments: TEZ-1789.1.patch
>
>
> There may be a large number of such events and speculators may have expensive
> logic while handling them. Moving off the central dispatcher will keep this
> off critical execution path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
