[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325647#comment-14325647
]
Bikas Saha commented on TEZ-776:
--------------------------------
Yes. ConfigureEvent could be sent always. Perhaps resuse of
InputDataInformationEvent.
Yes. They dont need to go to the same targetIndex. Which is why I am not
exploring this option in priority.
Yes. on demand routing would trade memory for cpu overhead.
>From what I understand 3 is figuring out a compact way to encode the routing
>table. So I am not sure how it is going to reduce the overhead of storing the
>exploded events (reference overhead) or the events themselves (after exploding
>them). Seemed like it depends on composite event routing and ondemand routing.
>So its like a mix of subsets of 1 and 2 with the encoding used to reduce the
>routing cpu overhead?
Moving routing into vertex plugins instead of edge plugins is shifting
responsibilities architecturally. Not sure how it solves the event overhead or
reference overhead. Making a hack in the shuffle plugin is probably not a good
idea. Users can be extending it or using their own for shuffles when they want
their own version of auto-reduce.
TezEvents are wrappers around data movement events. They are replicated only
for composite dm events since they are exploded. tezevent store identical
src/dest metainfo for the exploded events. Not sure about the math of reducing
the ref count from 16 to 2-3. Not replicating them and sharing them across dm
events may cause race conditions during rpc serialization.
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: events-problem-solutions.txt
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
