Tapas created SPARK-37811:
-----------------------------
Summary: Broadcast Join throws HintErrorLogger for joins with
multiple tables
Key: SPARK-37811
URL: https://issues.apache.org/jira/browse/SPARK-37811
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.1.2
Reporter: Tapas
Following query throws HintLoggerError Warnings in v3.1.2.
{code:java}
// code placeholder
val Query = " SELECT /*+ BROADCASTJOIN(L1, L2, L3) */ " +
" FROM L1 " +
" INNER JOIN L2 ON L2.id = L1.id " +
" INNER JOIN L3 ON L3.id = L1.id " +
" LEFT JOIN L4 ON L4.id = L1.id AND L4.idx = L2.idx AND L4.time
BETWEEN L3.time1 AND L3.time2 "
{code}
Following is the warning it thorws during runtime:
{code:java}
WARN HintErrorLogger: Count not find relation 'L1' specified in hint
'BROADCASTJOIN(L1,L2,L3)'
WARN HintErrorLogger: Count not find relation 'L2' specified in hint
'BROADCASTJOIN(L1,L2,L3)'
WARN HintErrorLogger: Count not find relation 'L3' specified in hint
'BROADCASTJOIN(L1,L2,L3)'{code}
The same query didn't have any warnings in v2.4.7. I am not entirely sure if
this is inherently not broadcasting the three small tables ({{{}L1, L2, L3{}}})
when doing a Left Join with a bigger table (L4).
I have set {{autoBroadcastJoinThreshold = 4G }} which is way bigger than
L1+L2+L3.
Let me know if you need more info.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
