[
https://issues.apache.org/jira/browse/HIVE-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-10673:
------------------------------
Attachment: HIVE-10673.10.patch
Patch v10:
- Rebase with trunk, looks like some methods in GenTezUtils were converted to
static
- When selecting distributed hash join, the join operator should get
OpTraits/stats set
- For the issue regarding the flattened expressions in the vectorized
rowObjectInspector, change the workaround to un-flatten the object inspector
during JoinUtil.getObjectInspectorsFromEvaluators(). This is still a bit of a
workaround, but only requires a change in 1 place, rather than the 2 changes
needed in the previous solution (having to modify the column names during
vectorized MapJoinOperator, as well as when generating the vectorized
rowObjectInspector in VectorizedBatchUtil)
- In the reducer, only the big table's input source should be vectorized
> Dynamically partitioned hash join for Tez
> -----------------------------------------
>
> Key: HIVE-10673
> URL: https://issues.apache.org/jira/browse/HIVE-10673
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning, Query Processor
> Reporter: Jason Dere
> Assignee: Jason Dere
> Attachments: HIVE-10673.1.patch, HIVE-10673.10.patch,
> HIVE-10673.2.patch, HIVE-10673.3.patch, HIVE-10673.4.patch,
> HIVE-10673.5.patch, HIVE-10673.6.patch, HIVE-10673.7.patch,
> HIVE-10673.8.patch, HIVE-10673.9.patch
>
>
> Some analysis of shuffle join queries by [~mmokhtar]/[~gopalv] found about
> 2/3 of the CPU was spent during sorting/merging.
> While this does not work for MR, for other execution engines (such as Tez),
> it is possible to create a reduce-side join that uses unsorted inputs in
> order to eliminate the sorting, which may be faster than a shuffle join. To
> join on unsorted inputs, we can use the hash join algorithm to perform the
> join in the reducer. This will require the small tables in the join to fit in
> the reducer/hash table for this to work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
