[
https://issues.apache.org/jira/browse/TEZ-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355925#comment-14355925
]
Bikas Saha commented on TEZ-2191:
---------------------------------
The patch extracts the improvements made in the MockDAGAppMaster simulation
code from the patch in TEZ-776, so that it can be used in other jiras for tests
and measurements.
Changes
1) Support for higher concurrency and containers to do the heartbeat
2) Support for invoking the real heartbeat method to exercise the real code
paths
3) Changes to set the correct working dir for the temp files produced so that
they end up in the target dir and get deleted on a clean build. Else they
accumulate in /tmp
4) Changing to an event based model to improve the efficiency of large
simulations.
5) Support for sending events and counters. After this patch is committed, I
will change the event code to use the delegate pattern in TEZ-776.
Changes to other tests for the above
Added a test to verify global/perVertex/perLogicalEdge counter aggregation.
Existing tests using the MockDAGAppMaster verify the changes.
Using this I have been able to measure the memory usage when enabling per
logical IO counters at large scale. It would be useful to commit this to enable
that work and others to use the better simulation code.
Changes in DAGAppMaster are unrelated and minor. The lock is not necessary
since its updated once (under DAGAppMaster lock) and is thread safe for
visibility. The null check for execService during shutdown handles the case
when init fails for some reason.
[~hitesh] [~rajesh.balamohan] Please review. Thanks!
> Simulation improvements to MockDAGAppMaster
> -------------------------------------------
>
> Key: TEZ-2191
> URL: https://issues.apache.org/jira/browse/TEZ-2191
> Project: Apache Tez
> Issue Type: Task
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-2191.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
