[ https://issues.apache.org/jira/browse/SPARK-21424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-21424. ------------------------------- Resolution: Not A Problem > Blockmanger has memory problems in high concurrent scenes > --------------------------------------------------------- > > Key: SPARK-21424 > URL: https://issues.apache.org/jira/browse/SPARK-21424 > Project: Spark > Issue Type: Improvement > Components: Block Manager > Affects Versions: 2.1.1 > Reporter: yiming.xu > > * Blockmanger removeBroadcast call entries in high concurrent scenes will > copy a lot of hashmap, producing many tuple2, we should redesign the > blockmanager.infos. > * In the use of spark to do the application, will also submit a lot of the > job, this time to produce a lot of boardcost, but after the completion of the > task, clean up the boardcost need to copy the BlockInfoManager.infos traverse > to get their own need to clear the block, which is very Unreasonable design, > resulting in the generation of a large number of tuple2. The more tasks, the > higher the memory occupied, seriously affected the jvm performance.We should > design a new program, can make the block clean up do not need to copy so much > data, but also To ensure that multi-threaded at the same time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org