[jira] [Updated] (SPARK-18078) Add option for customize zipPartition task preferred locations
[ https://issues.apache.org/jira/browse/SPARK-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18078: - Labels: bulk-closed (was: ) > Add option for customize zipPartition task preferred locations > -- > > Key: SPARK-18078 > URL: https://issues.apache.org/jira/browse/SPARK-18078 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Weichen Xu >Priority: Minor > Labels: bulk-closed > Original Estimate: 24h > Remaining Estimate: 24h > > `RDD.zipPartitions` task preferred locations strategy will use the > intersection of corresponding zipped partitions locations, if the > intersection is null, it use union of these locations. > but in special case, I want to customize the task preferred locations for > better performance. A typical case is in spark-tfocus *LinopMatrixAdjoint* > operator: a distributed matrix(DMatrix) multiplying a distributed > vector(DVector), it use RDD.zipPartitions (DMatrix and DVector RDD must be > partitioned in the same way beforehand). > https://github.com/databricks/spark-tfocs/blob/master/src/main/scala/org/apache/spark/mllib/optimization/tfocs/fs/dvector/vector/LinopMatrixAdjoint.scala > Usually, the `DMatrix` RDD will be much larger than `DVector` RDD, we hope > the zipPartition task always locates on `DMatrix` partition's location. it > will get better data locality than the default preferred location strategy. > I think it make sense to add an option for this. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18078) Add option for customize zipPartition task preferred locations
[ https://issues.apache.org/jira/browse/SPARK-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-18078: --- Priority: Minor (was: Major) > Add option for customize zipPartition task preferred locations > -- > > Key: SPARK-18078 > URL: https://issues.apache.org/jira/browse/SPARK-18078 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Weichen Xu >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > `RDD.zipPartitions` task preferred locations strategy will use the > intersection of corresponding zipped partitions locations, if the > intersection is null, it use union of these locations. > but in special case, I want to customize the task preferred locations for > better performance. A typical case is in spark-tfocus *LinopMatrixAdjoint* > operator: a distributed matrix(DMatrix) multiplying a distributed > vector(DVector), it use RDD.zipPartitions (DMatrix and DVector RDD must be > partitioned in the same way beforehand). > https://github.com/databricks/spark-tfocs/blob/master/src/main/scala/org/apache/spark/mllib/optimization/tfocs/fs/dvector/vector/LinopMatrixAdjoint.scala > Usually, the `DMatrix` RDD will be much larger than `DVector` RDD, we hope > the zipPartition task always locates on `DMatrix` partition's location. it > will get better data locality than the default preferred location strategy. > I think it make sense to add an option for this. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18078) Add option for customize zipPartition task preferred locations
[ https://issues.apache.org/jira/browse/SPARK-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-18078: --- Description: `RDD.zipPartitions` task preferred locations strategy will use the intersection of corresponding zipped partitions locations, if the intersection is null, it use union of these locations. but in special case, I want to customize the task preferred locations for better performance. A typical case is in spark-tfocus *LinopMatrixAdjoint* operator: a distributed matrix(DMatrix) multiplying a distributed vector(DVector), it use RDD.zipPartitions (DMatrix and DVector RDD must be partitioned in the same way beforehand). https://github.com/databricks/spark-tfocs/blob/master/src/main/scala/org/apache/spark/mllib/optimization/tfocs/fs/dvector/vector/LinopMatrixAdjoint.scala Usually, the `DMatrix` RDD will be much larger than `DVector` RDD, we hope the zipPartition task always locates on `DMatrix` partition's location. it will get better data locality than the default preferred location strategy. I think it make sense to add an option for this. was: `RDD.zipPartitions` task preferred locations strategy will use the intersection of corresponding zipped partitions locations, if the intersection is null, it use union of these locations. but in special case, I want to customize the task preferred locations for better performance. A typical case is in spark-tfocus *LinopMatrixAdjoint* operator: a distributed matrix(DMatrix) multiplying a distributed vector(DVector), it use RDD.zipPartitions (DMatrix and DVector RDD must be partitioned in the same way beforehand). https://github.com/WeichenXu123/spark-tfocs/blob/master/src/main/scala/org/apache/spark/mllib/optimization/tfocs/fs/dvector/vector/LinopMatrixAdjoint.scala Usually, the `DMatrix` RDD will be much larger than `DVector` RDD, we hope the zipPartition task always locates on `DMatrix` partition's location. it will get better data locality than the default preferred location strategy. I think it make sense to add an option for this. > Add option for customize zipPartition task preferred locations > -- > > Key: SPARK-18078 > URL: https://issues.apache.org/jira/browse/SPARK-18078 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Weichen Xu > Original Estimate: 24h > Remaining Estimate: 24h > > `RDD.zipPartitions` task preferred locations strategy will use the > intersection of corresponding zipped partitions locations, if the > intersection is null, it use union of these locations. > but in special case, I want to customize the task preferred locations for > better performance. A typical case is in spark-tfocus *LinopMatrixAdjoint* > operator: a distributed matrix(DMatrix) multiplying a distributed > vector(DVector), it use RDD.zipPartitions (DMatrix and DVector RDD must be > partitioned in the same way beforehand). > https://github.com/databricks/spark-tfocs/blob/master/src/main/scala/org/apache/spark/mllib/optimization/tfocs/fs/dvector/vector/LinopMatrixAdjoint.scala > Usually, the `DMatrix` RDD will be much larger than `DVector` RDD, we hope > the zipPartition task always locates on `DMatrix` partition's location. it > will get better data locality than the default preferred location strategy. > I think it make sense to add an option for this. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18078) Add option for customize zipPartition task preferred locations
[ https://issues.apache.org/jira/browse/SPARK-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-18078: --- Description: `RDD.zipPartitions` task preferred locations strategy will use the intersection of corresponding zipped partitions locations, if the intersection is null, it use union of these locations. but in special case, I want to customize the task preferred locations for better performance. A typical case is in spark-tfocus *LinopMatrixAdjoint* operator: a distributed matrix(DMatrix) multiplying a distributed vector(DVector), it use RDD.zipPartitions (DMatrix and DVector RDD must be partitioned in the same way beforehand). https://github.com/WeichenXu123/spark-tfocs/blob/master/src/main/scala/org/apache/spark/mllib/optimization/tfocs/fs/dvector/vector/LinopMatrixAdjoint.scala Usually, the `DMatrix` RDD will be much larger than `DVector` RDD, we hope the zipPartition task always locates on `DMatrix` partition's location. it will get better data locality than the default preferred location strategy. I think it make sense to add an option for this. was: `RDD.zipPartitions` task preferred locations strategy will use the intersection of corresponding zipped partitions locations, if the intersection is null, it use union of these locations. but in special case, I want to customize the task preferred locations for better performance. A typical case is in spark-tfocus: a distributed matrix(DMatrix) multiply a vector(DVector), it use RDD.zipPartitions. https://github.com/WeichenXu123/spark-tfocs/blob/master/src/main/scala/org/apache/spark/mllib/optimization/tfocs/DVectorFunctions.scala Usually, the `DMatrix` RDD will be much larger than `DVector` RDD, we hope the zipPartition task always locates on `DMatrix` partition's location. it will get better data locality than the default preferred location strategy. I think it make sense to add an option for this. > Add option for customize zipPartition task preferred locations > -- > > Key: SPARK-18078 > URL: https://issues.apache.org/jira/browse/SPARK-18078 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Weichen Xu > Original Estimate: 24h > Remaining Estimate: 24h > > `RDD.zipPartitions` task preferred locations strategy will use the > intersection of corresponding zipped partitions locations, if the > intersection is null, it use union of these locations. > but in special case, I want to customize the task preferred locations for > better performance. A typical case is in spark-tfocus *LinopMatrixAdjoint* > operator: a distributed matrix(DMatrix) multiplying a distributed > vector(DVector), it use RDD.zipPartitions (DMatrix and DVector RDD must be > partitioned in the same way beforehand). > https://github.com/WeichenXu123/spark-tfocs/blob/master/src/main/scala/org/apache/spark/mllib/optimization/tfocs/fs/dvector/vector/LinopMatrixAdjoint.scala > Usually, the `DMatrix` RDD will be much larger than `DVector` RDD, we hope > the zipPartition task always locates on `DMatrix` partition's location. it > will get better data locality than the default preferred location strategy. > I think it make sense to add an option for this. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
