[
https://issues.apache.org/jira/browse/SPARK-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-18078.
----------------------------------
Resolution: Incomplete
> Add option for customize zipPartition task preferred locations
> --------------------------------------------------------------
>
> Key: SPARK-18078
> URL: https://issues.apache.org/jira/browse/SPARK-18078
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Weichen Xu
> Priority: Minor
> Labels: bulk-closed
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> `RDD.zipPartitions` task preferred locations strategy will use the
> intersection of corresponding zipped partitions locations, if the
> intersection is null, it use union of these locations.
> but in special case, I want to customize the task preferred locations for
> better performance. A typical case is in spark-tfocus *LinopMatrixAdjoint*
> operator: a distributed matrix(DMatrix) multiplying a distributed
> vector(DVector), it use RDD.zipPartitions (DMatrix and DVector RDD must be
> partitioned in the same way beforehand).
> https://github.com/databricks/spark-tfocs/blob/master/src/main/scala/org/apache/spark/mllib/optimization/tfocs/fs/dvector/vector/LinopMatrixAdjoint.scala
> Usually, the `DMatrix` RDD will be much larger than `DVector` RDD, we hope
> the zipPartition task always locates on `DMatrix` partition's location. it
> will get better data locality than the default preferred location strategy.
> I think it make sense to add an option for this.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
