[
https://issues.apache.org/jira/browse/SPARK-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15441782#comment-15441782
]
Darren Fu commented on SPARK-12394:
-----------------------------------
Good news, Tejas!
I think SMB join is a required feature to make SparkSQL to compete with
traditional MPP databases (such as Teradata and DB2) on large table join. I
have benchmarked that Hive did well in such map-side join, and Spark should
support that as well.
I noticed that there is a bucketBy() method implemented in Spark 2.0:
https://github.com/apache/spark/blob/v2.0.0/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L168-L171
What is used for now?
> Support writing out pre-hash-partitioned data and exploit that in join
> optimizations to avoid shuffle (i.e. bucketing in Hive)
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-12394
> URL: https://issues.apache.org/jira/browse/SPARK-12394
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Reporter: Reynold Xin
> Assignee: Nong Li
> Fix For: 2.0.0
>
> Attachments: BucketedTables.pdf
>
>
> In many cases users know ahead of time the columns that they will be joining
> or aggregating on. Ideally they should be able to leverage this information
> and pre-shuffle the data so that subsequent queries do not require a shuffle.
> Hive supports this functionality by allowing the user to define buckets,
> which are hash partitioning of the data based on some key.
> - Allow the user to specify a set of columns when caching or writing out data
> - Allow the user to specify some parallelism
> - Shuffle the data when writing / caching such that its distributed by these
> columns
> - When planning/executing a query, use this distribution to avoid another
> shuffle when reading, assuming the join or aggregation is compatible with the
> columns specified
> - Should work with existing save modes: append, overwrite, etc
> - Should work at least with all Hadoops FS data sources
> - Should work with any data source when caching
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
