[
https://issues.apache.org/jira/browse/HIVE-16328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-16328:
----------------------------
Attachment: HIVE-16328.2.patch
Discussed with Xuefu & Jimmy offline. I think instead of rejecting branches
with RS, we can reject branches that contain either JOIN or LATERAL_VIEW, which
could potentially increase output data size. We mark those as big table
branches, and, if we have more than one such branch, the JOIN op cannot be
turned to a MAPJOIN op. This opens up more opportunities for optimizations.
Attaching patch v2 which realizes the above idea.
> HoS: more aggressive mapjoin optimization when
> hive.spark.use.file.size.for.mapjoin is true
> -------------------------------------------------------------------------------------------
>
> Key: HIVE-16328
> URL: https://issues.apache.org/jira/browse/HIVE-16328
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
> Reporter: Chao Sun
> Assignee: Chao Sun
> Attachments: HIVE-16328.1.patch, HIVE-16328.2.patch
>
>
> In HIVE-15489, when {{hive.spark.use.file.size.for.mapjoin}} is set to true,
> and if the JOIN op has any upstream RS operator, then we will stop converting
> the JOIN op to MAPJOIN op.
> However, this is overly conservative. A better solution is to treat the
> branch that has upstream RS as the big table and check if all other branches
> are map-only AND can fit in hash table size.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
