[
https://issues.apache.org/jira/browse/TEZ-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikas Saha updated TEZ-1867:
----------------------------
Attachment: TEZ-1867.1.patch
Attaching patch.
Brings AsyncDispatcher code in from YARN into tez-common and modifies it.
1) Names for dispatcher threads
2) Support for registering event handlers that use their own delegated
dispatchers. Removed code from DAGAppMaster that handled this in a special case.
3) Main event handler immediately delegates to the delegating dispatcher if one
exists. Or else it falls back to its own queue to enqueue them.
4) Test added.
Rest of the code is oblivious to this and continues to use the Dispatcher
interface to send events.
Speculator events run on their delegated dispatcher currently. Now its trivial
to add other events on their own delegated dispatchers.
Will follow up separately to move away from using a thread to using an executor
service which will allow higher concurrency for events that can take advantage
of it. This is why the code is not inheriting from YARN AsyncDispatcher because
most of the concurrency code will be changing.
[~hitesh] [~sseth] [~zjffdu] Please review.
> Create new central dispatcher for Tez AM
> ----------------------------------------
>
> Key: TEZ-1867
> URL: https://issues.apache.org/jira/browse/TEZ-1867
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-1867.1.patch
>
>
> The dispatcher should be able to federate events across multiple event
> specific dispatchers for AM control plane scalability. Add dispatcher
> specific thread names for easier debugging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
