[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527068#comment-14527068 ] Siddharth Seth commented on TEZ-1897: - Looks like this went in with concurrentDispatchers enabled. Can you please undo that bit. Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 0.7.0 Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch, TEZ-1897.7.patch, TEZ-1897.8.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527081#comment-14527081 ] Bikas Saha commented on TEZ-1897: - Thanks for catching it. My bad. Fixed. commit 5218f481dba2a26c3aa5dd8f69285ab9da419dd1 Author: Bikas Saha bi...@apache.org Date: Mon May 4 12:05:39 2015 -0700 TEZ-1897 addendum to turn off by default . Create a concurrent version of AsyncDispatcher (bikas) Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 0.7.0 Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch, TEZ-1897.7.patch, TEZ-1897.8.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523499#comment-14523499
]
Bikas Saha commented on TEZ-1897:
-
bq. AsyncDispatcherConcurrent(String name, int numThreads) { super(name)
instead of super(dispatcher)
bq. final LinkedBlockingQueueEvent queue;; - Double ;
bq. In AsyncDispatcher - the error checking code for previously registered
dispatchers
bq. aitForDrained.wait(1000);, LO
Fixed
bq. At the same place - do the threads need to
Cannot interrupt executorservice. After waiting shutdownNow is called which
should take care of this. In any case, drained code patch is not used and we
should probably remove it separately.
bq. serviceStop / serviceStart don't need to invoke super.
Thats the way other CompositeServices are written and seems to be the correct
thing to do since this method overrides super.serviceStart().
bq. here's a lot of code duplication between AsyncDispatcher and
AsyncDispatcherConcu
I chose to do so, so that the concurrent version can replace the existing
version in a future release. The concurrent version with thread size set to 1
is the same as the existing version. Manual thread creation is replaced by
using a single thread in the threadpool. So there is no need to maintain 2
classes.
thanks for the reviews. Unless there are further comments, will commit by EOD.
Create a concurrent version of AsyncDispatcher
--
Key: TEZ-1897
URL: https://issues.apache.org/jira/browse/TEZ-1897
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch,
TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch
Currently, it processes events on a single thread. For events that can be
executed in parallel, e.g. vertex manager events, allowing higher concurrency
may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523641#comment-14523641 ] Bikas Saha commented on TEZ-1897: - From what I see - AbstractService#start()-serviceStart()-end up calling AsyncDispatcher-serviceStart() since it overrides CompositeService#serviceStart(). So the derived class must call its parents serviceStart(). MRAppMaster is also written that way. Thanks! We can remove this draining code and other unused code when we remove AsyncDispatcher. Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch, TEZ-1897.7.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523659#comment-14523659
]
TezQA commented on TEZ-1897:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729796/TEZ-1897.7.patch
against master revision c96eed3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/602//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/602//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/602//console
This message is automatically generated.
Create a concurrent version of AsyncDispatcher
--
Key: TEZ-1897
URL: https://issues.apache.org/jira/browse/TEZ-1897
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch,
TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch, TEZ-1897.7.patch
Currently, it processes events on a single thread. For events that can be
executed in parallel, e.g. vertex manager events, allowing higher concurrency
may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522300#comment-14522300 ] Siddharth Seth commented on TEZ-1897: - Rest looks good to me. Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522298#comment-14522298
]
Siddharth Seth commented on TEZ-1897:
-
Mostly minor comments.
- AsyncDispatcherConcurrent(String name, int numThreads) { super(name)
instead of super(dispatcher)
- final LinkedBlockingQueueEvent queue;; - Double ;
- serviceStop / serviceStart don't need to invoke super. These will be invoked
via the stop() / start() methods automatically.
- In AsyncDispatcher - the error checking code for previously registered
dispatchers (/* check to see if we have a listener registered */) can be
common across all register methods. registerAndCreateDispatcher(without
#threads) is missing a check on ConcurrentDispatchers
- waitForDrained.wait(1000);, LOG.info(Waiting for AsyncDispatcher to
drain.); - log line before the wait ?
- At the same place - do the threads need to be interrupted ? Otherwise they'll
always wait for the 1000ms if the queue is already empty.
- There's changes in TaskImpl, Task, Vertex - which are unrelated to this.
That's likely adding to perf gains. There's a separate jira for this - will see
if the scope of that is no longer valid after theses changes.
Major bit:
There's a lot of code duplication between AsyncDispatcher and
AsyncDispatcherConcurrent - GenericEventHandler, MultiAttemptListener and a
bunch more.The register functions seem a little unnecessary since noone
registers directly with this. Not sure if you want to fix this here or in a
follow up.
Create a concurrent version of AsyncDispatcher
--
Key: TEZ-1897
URL: https://issues.apache.org/jira/browse/TEZ-1897
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch,
TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch
Currently, it processes events on a single thread. For events that can be
executed in parallel, e.g. vertex manager events, allowing higher concurrency
may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518254#comment-14518254 ] Bikas Saha commented on TEZ-1897: - bq. Rename registerAndCreateDispatcher to registerAndCreateConcurrentDispatcher ? bq. Rename registerWithExistingDispatcher to registerWithExistingConcurrentDispatcher ? The types in the arguments should be enough to differentiate and also provide compile time check. Adding it to the names would be redundant. Also, once this code is exercised and stabilized then we may use this as the only dispatcher. bq. Javadoc for AsyncDispatcherConcurrent is the same as AsyncDispatcher Fixed bq. Precondition numThreads should be at least 2 ? I think 1 is the same as AysnDispatcher This should be able to replace the legacy AsyncDispatcher and run all events on a single thread in the threadpool. So its allowed. bq. TaskAttemptImpl.java Some places still use getTask().getVertex(), can change to getVertex directly. Fixed Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518493#comment-14518493
]
TezQA commented on TEZ-1897:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12728962/TEZ-1897.6.patch
against master revision 5b2f011.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/567//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/567//console
This message is automatically generated.
Create a concurrent version of AsyncDispatcher
--
Key: TEZ-1897
URL: https://issues.apache.org/jira/browse/TEZ-1897
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch,
TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch
Currently, it processes events on a single thread. For events that can be
executed in parallel, e.g. vertex manager events, allowing higher concurrency
may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14517978#comment-14517978 ] Siddharth Seth commented on TEZ-1897: - Quick glance. This should be disabled by default. Will look in detail a little later. Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14517996#comment-14517996 ] Bikas Saha commented on TEZ-1897: - the patch enabled it by default to get the jenkins run with it. It will be off by default. Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516643#comment-14516643
]
Jeff Zhang commented on TEZ-1897:
-
Comments:
* Rename registerAndCreateDispatcher to registerAndCreateConcurrentDispatcher ?
Rename registerWithExistingDispatcher to
registerWithExistingConcurrentDispatcher ?
* Javadoc for AsyncDispatcherConcurrent is the same as AsyncDispatcher
* No concurentDispatcher checking in registerAndCreateDispatcher(Class?
extends Enum eventType,
EventHandler handler, String dispatcherName)
* Precondition numThreads should be at least 2 ? I think 1 is the same as
AysnDispatcher
{code}
public AsyncDispatcherConcurrent registerAndCreateDispatcher(Class? extends
Enum eventType,
EventHandler handler, String dispatcherName, int numThreads)
{code}
* TaskAttemptImpl.java
Some places still use getTask().getVertex(), can change to getVertex directly.
Create a concurrent version of AsyncDispatcher
--
Key: TEZ-1897
URL: https://issues.apache.org/jira/browse/TEZ-1897
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch,
TEZ-1897.4.patch, TEZ-1897.5.patch
Currently, it processes events on a single thread. For events that can be
executed in parallel, e.g. vertex manager events, allowing higher concurrency
may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516578#comment-14516578 ] Bikas Saha commented on TEZ-1897: - Sorry about that. Deleted the wrong one. Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516528#comment-14516528 ] Bikas Saha commented on TEZ-1897: - Folks, please review. I would like to get this into 0.7. Thanks! Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.5.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516530#comment-14516530 ] Jeff Zhang commented on TEZ-1897: - Which one is the latest patch ? Saw 2 TEZ-1897.5.patch and both 5 days ago. Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.5.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510002#comment-14510002 ] Bikas Saha commented on TEZ-1897: - [~zjffdu] [~hitesh] Please continue your reviews with the latest patch. Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.5.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507588#comment-14507588
]
TezQA commented on TEZ-1897:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727292/TEZ-1897.4.patch
against master revision 9faa54c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.TestMockDAGAppMaster
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/514//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/514//artifact/patchprocess/newPatchFindbugsWarningstez-common.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/514//console
This message is automatically generated.
Create a concurrent version of AsyncDispatcher
--
Key: TEZ-1897
URL: https://issues.apache.org/jira/browse/TEZ-1897
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch,
TEZ-1897.4.patch
Currently, it processes events on a single thread. For events that can be
executed in parallel, e.g. vertex manager events, allowing higher concurrency
may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508247#comment-14508247 ] Bikas Saha commented on TEZ-1897: - The last patch builds on the previous patch to actually use the concurrent dispatcher to run Task and TaskAttempt events concurrently. There is a configuration to turn this on or off and when it is turned off the code runs exactly the same path as it does today. So this change is very safe. In order to keep things sane, events for a given Task and its attempts are serialized on the same thread. This is done by using a serializing hash determined from the TezTaskID. Different tasks run on different threads. That takes care of a lot of locking issues. Next, Vertex has reference to DAG, Task has reference to Vertex and Attempt has reference to Task and Vertex. This helps remove unnecessary locking issues and delays that occur when they are accessed from the AppContext to get dag/vertex/task etc. and then look-up into their internal maps. This change would be beneficial in general by reducing lock contention compared to today. Added a simulation test that runs 5 tasks at 1000 concurrency which runs up to 30% faster with the change than without. The patch has the config turned on for patch test execution. This will be off by default and is marked private so only advanced users can try this for large clusters where we can get 10-20K running tasks concurrently. Please review. Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.5.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508132#comment-14508132
]
TezQA commented on TEZ-1897:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727349/TEZ-1897.5.patch
against master revision 9faa54c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/516//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/516//console
This message is automatically generated.
Create a concurrent version of AsyncDispatcher
--
Key: TEZ-1897
URL: https://issues.apache.org/jira/browse/TEZ-1897
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch,
TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.5.patch
Currently, it processes events on a single thread. For events that can be
executed in parallel, e.g. vertex manager events, allowing higher concurrency
may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
