[
https://issues.apache.org/jira/browse/TAJO-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14530189#comment-14530189
]
Jihoon Son commented on TAJO-1553:
----------------------------------
Hi guys.
I got a new idea while working on this issue.
We should consider the following two cases to handle join plans with small
inputs.
* The size of one of two inputs is smaller than the given threshold (T1).
* The sum of the size of all inputs is smaller than the given threshold (T2).
_Note: T1 and T2 may have different values._
Obviously, in the first case, join execution can be improved by using broadcast
join algorithm.
However, I think that the second case is little different.
Let me suppose that T1 and T2 have the same value. Under this assumption, both
inputs of the join plan will be broadcasted according to the rule I described
in this issue. Obviously, the broadcast of two inputs unnecessarily increases
processing overhead. The best plan will be executing the join operation in a
single node where the larger input is stored in. This is not broadcast join,
but one-phase join execution.
I think that the one-phase execution can be applied for all kinds of operators.
If you agree, I'll separate the optimization for one-phase execution as another
issue.
> Improve broadcast join planning
> -------------------------------
>
> Key: TAJO-1553
> URL: https://issues.apache.org/jira/browse/TAJO-1553
> Project: Tajo
> Issue Type: Improvement
> Components: distributed query plan, planner/optimizer
> Reporter: Jihoon Son
> Assignee: Jihoon Son
> Fix For: 0.11.0
>
>
> The global engine generates a logical plan, and then marks some parts of the
> plan as broadcast plan which means that they and their input will be
> broadcasted to all workers.
> Currently, broadcast parts are identified according to some rigid and
> hard-coded rules. This will limit the broadcast opportunities in many cases.
> So, in this issue, I propose refactoring the broadcast planner to be more
> general.
> Broadcast parts can be identified recursively.
> * A leaf node will be broadcasted if its input size does not exceed the
> pre-defined threshold.
> * An intermediate node will be broadcasted if it has at least one broadcast
> child.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
