[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves resolved SPARK-21243. ----------------------------------- Resolution: Fixed Fix Version/s: 2.3.0 > Limit the number of maps in a single shuffle fetch > -------------------------------------------------- > > Key: SPARK-21243 > URL: https://issues.apache.org/jira/browse/SPARK-21243 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.1.0, 2.1.1 > Reporter: Dhruve Ashar > Assignee: Dhruve Ashar > Priority: Minor > Fix For: 2.3.0 > > > Right now spark can limit the # of parallel fetches and also limits the > amount of data in one fetch, but one fetch to a host could be for 100's of > blocks. In one instance we saw 450+ blocks. When you have 100's of those and > 1000's of reducers fetching that becomes a lot of metadata and can run the > Node Manager out of memory. We should add a config to limit the # of maps per > fetch to reduce the load on the NM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org