[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344691#comment-14344691
]
Bikas Saha commented on TEZ-776:
--------------------------------
Attached new revision (2) of the patch. It re-writes the simulation code to
work in parallel and also to reduce cpu cycles from the simulation so that
profiles and cpu usage reflect the main code. Profiles are too big to attach
and are available here - http://1drv.ms/17J7Q9f. They show that the cpu
consumption is dominated by the creation of the exploded dm events. The
simulation also captures the worst case scenario where the AM is only routing
events without any breathing room provided by the tasks actually spending time
doing real work. Running the simulation consumes 200-400% cpu but the AM is not
always bottle-necked on event routing. The central dispatcher thread is also on
the critical path for the state machine orchestrations. In real life, the cpu
consumption is far lower (30-40% as viewed on top) and not observably different
with or without the change. Tried the patch with a few real jobs on the
cluster. The patch also has some temporary code to measure the cpu usage of the
AM in order to capture aggregate usage in case eye-balling top stats may have
missed something. For a job with 2 large joins taking ~1300s to complete on a
20 node cluster the cpu times reported are 487640 (with on demand routing) and
526340 (with existing routing). The numbers are close to each other and reflect
the inner loop cost observation made earlier based on the profiles. At this
point, there seems to be sufficient data to show that the CPU usage of on
demand routing is comparable to existing cpu usage.
The patch updates the API and allows for the existing API to be used by
changing configuration. This allows introduction of the API in 0.7 for mixed
use based on availability of updated user code in Hive and other projects who
have custom edge plugins. APIs can be removed in 0.8. The patch does not add
new javadoc or made the new API optional or added deprecation annotations. It
includes improvements to the simulation code, bug fixes and test fixes. All
tests pass and as mentioned earlier the memory usage on the AM is essentially
controlled from a framework point of view. I believe the patch is ready for
review.
Since this is a large change I would appreciate reviews/feedback from more
people. [~rajesh.balamohan] [~hitesh] [~sseth] and other watchers on the jira,
your attention would be very helpful.
To help with this, please start with the APIs on the edge plugins for on-demand
routing and then move on to Edge.java+VertexImpl.java to see how the new APIs
are used to route events on demand to tasks. That should cover the essence of
the change and is not a large amount of code.
The new APIs have a somewhat orthogonal change and that is to remove the
visibility of the routed events from the edge plugins to protect the payload
from a security point of view. IOs and edge plugins are user code that may not
have a common origin. Hence, data movement event payloads should in theory be
visible only to the IOs and no other (potentially third party) user code.
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch,
> TEZ-776.ondemand.patch, events-problem-solutions.txt
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
