[
https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116894#comment-16116894
]
Volodymyr Vysotskyi commented on DRILL-1162:
--------------------------------------------
To take into account this pessimistic scenario when build side has much larger
rows count than the probe side I propose to use also {{getMaxRowCount()}}
method in {{SwapHashJoinVisitor}} when Drill takes a decision to swap join
inputs. If {{getMaxRowCount()}} for build side is greater than
{{getMaxRowCount()}} for probe side and greater than some critical value (c1),
then Drill should swap join inputs. Also if {{getMaxRowCount()}} for probe side
is greater than the same critical value c1, Drill should not swap join inputs
when it compares {{getRows()}} values.
The value of the option {{exec.max_hash_table_size}} may be used as this
critical value c1.
So then Drill should not fail with OOM for the query from Jira description and
for the query from my previous comment.
The plan will be changed to non-optimal when all these conditions are satisfied:
* joining by PK;
* the {{getRows()}} for build side is less than the {{getRows()}} for probe
side;
* the {{getMaxRowCount()}} for build side is greater than the
{{getMaxRowCount()}} for probe side;
* the {{getMaxRowCount()}} for build side is greater than
{{exec.max_hash_table_size}};
or when all these conditions are satisfied:
* joining by PK;
* the {{getMaxRowCount()}} for probe side is greater than critical value;
* the {{getRows()}} for build side is greater than the {{getRows()}} for probe
side;
For all other cases, the plan should not degrade with these changes.
[~jni], [~paul-rogers] am I on the right way?
> 25 way join ended up with OOM
> -----------------------------
>
> Key: DRILL-1162
> URL: https://issues.apache.org/jira/browse/DRILL-1162
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow, Query Planning & Optimization
> Reporter: Rahul Challapalli
> Assignee: Volodymyr Vysotskyi
> Priority: Critical
> Fix For: Future
>
> Attachments: error.log, oom_error.log
>
>
> git.commit.id.abbrev=e5c2da0
> The below query results in 0 results being returned
> {code:sql}
> select count(*) from `lineitem1.parquet` a
> inner join `part.parquet` j on a.l_partkey = j.p_partkey
> inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey
> inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey
> inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey
> = m.ps_suppkey
> inner join `customer.parquet` n on k.o_custkey = n.c_custkey
> inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey
> inner join `lineitem2.parquet` c on a.l_partkey = c.l_partkey
> inner join `lineitem2.parquet` d on a.l_suppkey = d.l_suppkey
> inner join `lineitem2.parquet` e on a.l_extendedprice = e.l_extendedprice
> inner join `lineitem2.parquet` f on a.l_comment = f.l_comment
> inner join `lineitem2.parquet` g on a.l_shipdate = g.l_shipdate
> inner join `lineitem2.parquet` h on a.l_commitdate = h.l_commitdate
> inner join `lineitem2.parquet` i on a.l_receiptdate = i.l_receiptdate
> inner join `lineitem2.parquet` o on a.l_receiptdate = o.l_receiptdate
> inner join `lineitem2.parquet` p on a.l_receiptdate = p.l_receiptdate
> inner join `lineitem2.parquet` q on a.l_receiptdate = q.l_receiptdate
> inner join `lineitem2.parquet` r on a.l_receiptdate = r.l_receiptdate
> inner join `lineitem2.parquet` s on a.l_receiptdate = s.l_receiptdate
> inner join `lineitem2.parquet` t on a.l_receiptdate = t.l_receiptdate
> inner join `lineitem2.parquet` u on a.l_receiptdate = u.l_receiptdate
> inner join `lineitem2.parquet` v on a.l_receiptdate = v.l_receiptdate
> inner join `lineitem2.parquet` w on a.l_receiptdate = w.l_receiptdate
> inner join `lineitem2.parquet` x on a.l_receiptdate = x.l_receiptdate;
> {code}
> However when we remove the last 'inner join' and run the query it returns
> '716372534'. Since the last inner join is similar to the one's before it, it
> should match some records and return the data appropriately.
> The logs indicated that it actually returned 0 results. Attached the log file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
