[ https://issues.apache.org/jira/browse/YARN-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Hussein reassigned YARN-9597: ----------------------------------- Assignee: Ahmed Hussein > Memory efficiency in speculator > -------------------------------- > > Key: YARN-9597 > URL: https://issues.apache.org/jira/browse/YARN-9597 > Project: Hadoop YARN > Issue Type: Improvement > Reporter: Ahmed Hussein > Assignee: Ahmed Hussein > Priority: Minor > > The data structures in speculator and runtime-estimator are bloating. Data > elements such as (taskID, TA-ID, task stats, tasks speculated, tasks > finished..etc) are added to the concurrent maps but never removed. > For long running jobs, there are couple of issues: > # memory leakage: the speculator memory usage increases over time. > # performance: keeping large structures in the heap affects the performance > due to locality and cache misses. > *Suggested Fixes:* > - When a TA transitions to {{MoveContainerToSucceededFinishingTransition}}, > the TA notifies the speculator. The latter handles the event by cleaning the > internal structure accordingly. > - When a task transitions is failed/killed, the speculator is notified to > clean the internal data structure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org