[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265238#comment-14265238
]
Bikas Saha commented on TEZ-1897:
---------------------------------
Its less about 1-2 sec gain but more about unrelated processing not blocking
important stuff. Delays and queueing tend to have multiplicative effects
globally instead of additive. Additionally, user code eg. user-written vertex
manager could take a long time to make some optimizations. Not everything needs
to move off but that could be an ideal starting point.
Given that events are the primary means of communication over the control
plane, vertex manager logic could be invoked upon receipt of an event instead
of direct calls made by the vertex (like it happens right now). The new
dispatcher could be used to route those events and a higher concurrency would
help handle 1000s of vertices without creating 1000s of threads. The dispatcher
becomes a custom wrapper around a generic executor service to tie Tez events to
the executor service mechanism. This will have to take care of a few other
things like ordering etc. and can be discussed in TEZ-1914. Very likely
multiple executors will only be used for vertex managers. Other entities like
tasks, attempts and vertices would continue to have single threaded
dispatchers, which ensures the ordering but would separate entity specific
processing to the respective entities.
> Allow higher concurrency in AsyncDispatcher
> -------------------------------------------
>
> Key: TEZ-1897
> URL: https://issues.apache.org/jira/browse/TEZ-1897
> Project: Apache Tez
> Issue Type: Task
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch
>
>
> Currently, it processes events on a single thread. For events that can be
> executed in parallel, e.g. vertex manager events, allowing higher concurrency
> may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
