[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265547#comment-14265547
]
Siddharth Seth commented on TEZ-1897:
-------------------------------------
bq. Its less about 1-2 sec gain but more about unrelated processing not
blocking important stuff. Delays and queueing tend to have multiplicative
effects globally instead of additive. Additionally, user code eg. user-written
vertex manager could take a long time to make some optimizations. Not
everything needs to move off but that could be an ideal starting point.
User code definitely needs to move off. That's TEZ-1914, and I believe there's
one more jira for the committers moving off the dispatcher. I'm not sure
whether a dispatcher is the best way to achieve multiple threads within a VM
though, rather than just setting up threads directly. InputInitializers have
their own manager, which setup threads to take care of offloading work.
In terms of the rest of the flow, even when multiplicative - I'm not sure it's
longer than 1-2 seconds for large queries (an example is TaskAttempt messages
not reaching before all Task messages are processed - even though there's no
reason to block them).
I don't see any problems with changing the dispatcher to support multiple
threads in this manner - just unsure on how it'll eventually be used - that
would be TEZ-1914 I presume.
> Allow higher concurrency in AsyncDispatcher
> -------------------------------------------
>
> Key: TEZ-1897
> URL: https://issues.apache.org/jira/browse/TEZ-1897
> Project: Apache Tez
> Issue Type: Task
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch
>
>
> Currently, it processes events on a single thread. For events that can be
> executed in parallel, e.g. vertex manager events, allowing higher concurrency
> may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
