[
https://issues.apache.org/jira/browse/TEZ-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14236553#comment-14236553
]
Siddharth Seth commented on TEZ-1789:
-------------------------------------
Took a quick look over the patch. Mostly looked good.
- Instead of Vertex.handleSpeculatorEvent. Should this just be
Vertex.getSpeculator() - followed by a handlEvent on the speculator itself.
- Is it easy to extend AsyncDispatcher to setup a custom thread name (without
re-implementing the whole thing here)
In terms of multiple speculators, I agree we'll need our own version of
AsynDispatcher (or an equivalent from something like guava) when we want to
split processing contexts across different threads.
At the moment, the events would still end up going on to a single queue (the
central dispatcher), then moving out to the new dispatcher
(SpeculatorDispatcher) - where they're actually processed. A custom dispatcher
could easily set up multiple queues if clear separation of contexts is known up
front.
What we've done in the past to take processing off a thread is to setup queues
in the local event handler - SchedulerEvents, ContainerLaunchEvents for example
- these components have their own queues and threads. This approach ends up
using the queueing and threading mechanism from a new dispatcher - which I
think is also fine.
> Move speculator processing off the central dispatcher
> -----------------------------------------------------
>
> Key: TEZ-1789
> URL: https://issues.apache.org/jira/browse/TEZ-1789
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-1789.1.patch
>
>
> There may be a large number of such events and speculators may have expensive
> logic while handling them. Moving off the central dispatcher will keep this
> off critical execution path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
