Seonggon Namgung created HIVE-28833:
---------------------------------------
Summary: Do not perform SharedWorkOptimization if the merge
results in an excessively heavy MapJoin workload.
Key: HIVE-28833
URL: https://issues.apache.org/jira/browse/HIVE-28833
Project: Hive
Issue Type: Improvement
Reporter: Seonggon Namgung
SharedWorkOptimization merges two TableScan operators, which is equivalent to
merging two Map vertices into one. Suppose we are merging two TableScan
operators, and the corresponding Map vertices contain MapJoin operators. In
this case, merging the two TableScan operators results in a large Map vertex
that contains both MapJoin operators, which could lead to high memory pressure.
To avoid potential reliability issues, we should not perform SWO if the merge
results in an excessively heavy MapJoin workload. This might be done by
evaluating the number and/or the size of the MapJoin operators before merging.
This issue was first observed with an increase in OOM errors in the CI
environment after applying HIVE-26986. As a temporary solution, HIVE-28832
disabled SWO in the affected query file. This fix should be rolled back once
the underlying issue is resolved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
