[
https://issues.apache.org/jira/browse/HIVE-28833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-28833:
----------------------------------
Labels: hive-4.1.1-must (was: hive-4.1.0-must)
> Do not perform SharedWorkOptimization if the merge results in an excessively
> heavy MapJoin workload.
> ----------------------------------------------------------------------------------------------------
>
> Key: HIVE-28833
> URL: https://issues.apache.org/jira/browse/HIVE-28833
> Project: Hive
> Issue Type: Improvement
> Reporter: Seonggon Namgung
> Assignee: Seonggon Namgung
> Priority: Major
> Labels: hive-4.1.1-must
>
> SharedWorkOptimization merges two TableScan operators, which is equivalent to
> merging two Map vertices into one. Suppose we are merging two TableScan
> operators, and the corresponding Map vertices contain MapJoin operators. In
> this case, merging the two TableScan operators results in a large Map vertex
> that contains both MapJoin operators, which could lead to high memory
> pressure.
> To avoid potential reliability issues, we should not perform SWO if the merge
> results in an excessively heavy MapJoin workload. This might be done by
> evaluating the number and/or the size of the MapJoin operators before merging.
> This issue was first observed with an increase in OOM errors in the CI
> environment after applying HIVE-26986. As a temporary solution, HIVE-28832
> disabled SWO in the affected query file. This fix should be rolled back once
> the underlying issue is resolved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
