[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355938#comment-14355938
]
Siddharth Seth commented on TEZ-776:
------------------------------------
One option would be to make these changes now, and agree upon moving storage
into the EdgePlugin in a month or so (I can take this up) - before maybe a
0.7.1, and definitely before a 0.8 release. What gets in the way of this is
users having to change their edge plugins twice (Hive has one, not sure about
Pig, Cascading and others).
[~bikassaha] - I'm not sure if the patch is an incompatible change or not, but
is it possible to make these changes in a way that external EdgeManagers (Hive
has one, not sure about Pig and others) don't need to change. One option for
this would be to use the existing APIs for a default implementation of the new
methods - and pick the relevant target task. Fairly inefficient though.
The reason I say this is that the API changes in this patch and API changes for
moving event storage into the EdgePlugin - will not be the same. If we can
make this in a compatible manner, and maybe add notes on the APIs saying
they'll be removed soon - in a month or so, once EdgePlugin storage changes are
made, downstream projects would need to change their implementation only once.
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch,
> TEZ-776.ondemand.3.patch, TEZ-776.ondemand.4.patch, TEZ-776.ondemand.5.patch,
> TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png,
> With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png,
> events-problem-solutions.txt, with_patch_jmc_output_of_AM.png,
> without_patch_jmc_output_of_AM.png
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
