[ https://issues.apache.org/jira/browse/YARN-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946028#comment-15946028 ]
Jason Lowe commented on YARN-6406: ---------------------------------- I haven't dug into YARN-6040, but in general I'm a big +1 for having the RM aggressively remove bookkeeping entries that aren't necessary to improve lookup/iteration performance in addition to reducing the heap pressure. That was the whole idea behind YARN-5540. I don't see why we would need to keep scheduler keys or requests around once there are no more containers to allocate for them. > Garbage Collect unused SchedulerRequestKeys > ------------------------------------------- > > Key: YARN-6406 > URL: https://issues.apache.org/jira/browse/YARN-6406 > Project: Hadoop YARN > Issue Type: Improvement > Reporter: Arun Suresh > Assignee: Arun Suresh > > YARN-5540 introduced some optimizations to remove satisfied SchedulerKeys > from the AppScheduleingInfo. It looks like after YARN-6040, > ScedulerRequestKeys are removed only if the Application sends a 0 > numContainers requests. While earlier, the outstanding schedulerKeys were > also remove as soon as a container is allocated as well. > An additional optimization we were hoping to include is to remove the > ResourceRequests itself once the numContainers == 0, since we see in our > clusters that the RM heap space consumption increases drastically due to a > large number of ResourceRequests with 0 num containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org