SaintBacchus created SPARK-6584: ----------------------------------- Summary: Provide ExecutorPrefixTaskLocation to support the rdd which can be aware of partition's executor location. Key: SPARK-6584 URL: https://issues.apache.org/jira/browse/SPARK-6584 Project: Spark Issue Type: Sub-task Affects Versions: 1.4.0 Reporter: SaintBacchus
The function *RDD.getPreferredLocations* can only be set the host awareness prefer locations. If some *RDD* wants to be scheduled by executor(such as BlockRDD), spark can do nothing for this. So I want to provide *ExecutorPrefixTaskLocation* to support the rdd which can be aware of partition's executor location. This mechanism can avoid data transfor in the case of many executor in the same host. I think it's very useful especially for *SparkStreaming* since the *Receriver* save data into the *BlockManger* and then become a BlockRDD -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org