[
https://issues.apache.org/jira/browse/HIVE-28833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17939218#comment-17939218
]
Seonggon Namgung edited comment on HIVE-28833 at 3/28/25 11:52 AM:
-------------------------------------------------------------------
I checked that SharedWorkOptimizer already considers the size of the MapJoin
input data, so it seems that the size of MapJoin input data is not proper to
detect a heavy MapJoin workload.
An alternative method to prevent the merge is to check the number of MapJoin
operators. My initial approach was to avoid independent MapJoin operators,
i.e., to exclude the following pattern from the operator plan:
{code:java}
TS - MapJoin - ...
- MapJoin - ...{code}
However, I realized this could lead to suboptimal query plans. For instance, I
tested this approach on TPC-DS queries and observed that queries 1, 23a, 23b,
24a, 24b, 88, and 90 were affected. Among them, only query 23a showed
improvement after applying the patch (350.58s → 326.33s), while some queries
became slower. Specifically, query 24a increased from 76.65s to 98.36s, and
query 88 increased from 23.14s to 84.13s.
Therefore, I conclude that detecting and preventing a heavy MapJoin workload is
challenging to implement, and I currently have no further ideas. It would be
appreciated if someone could share any idea or address this issue.
was (Author: JIRAUSER298608):
I checked that SharedWorkOptimizer already considers the size of the MapJoin
input data, so it seems that the size of MapJoin input data is not proper to
detect a heavy MapJoin workload.
An alternative method to prevent the merge is to check the number of MapJoin
operators. My initial approach was to avoid independent MapJoin operators,
i.e., to exclude the following pattern from the operator plan:
TS-MapJoin-...-RS
-MapJoin-...-RS
However, I realized this could lead to suboptimal query plans. For instance, I
tested this approach on TPC-DS queries and observed that queries 1, 23a, 23b,
24a, 24b, 88, and 90 were affected. Among them, only query 23a showed
improvement after applying the patch (350.58s → 326.33s), while some queries
became slower. Specifically, query 24a increased from 76.65s to 98.36s, and
query 88 increased from 23.14s to 84.13s.
Therefore, I conclude that detecting and preventing a heavy MapJoin workload is
challenging to implement, and I currently have no further ideas. It would be
appreciated if someone could share any idea or address this issue.
> Do not perform SharedWorkOptimization if the merge results in an excessively
> heavy MapJoin workload.
> ----------------------------------------------------------------------------------------------------
>
> Key: HIVE-28833
> URL: https://issues.apache.org/jira/browse/HIVE-28833
> Project: Hive
> Issue Type: Improvement
> Reporter: Seonggon Namgung
> Assignee: Seonggon Namgung
> Priority: Major
> Labels: hive-4.1.0-must
>
> SharedWorkOptimization merges two TableScan operators, which is equivalent to
> merging two Map vertices into one. Suppose we are merging two TableScan
> operators, and the corresponding Map vertices contain MapJoin operators. In
> this case, merging the two TableScan operators results in a large Map vertex
> that contains both MapJoin operators, which could lead to high memory
> pressure.
> To avoid potential reliability issues, we should not perform SWO if the merge
> results in an excessively heavy MapJoin workload. This might be done by
> evaluating the number and/or the size of the MapJoin operators before merging.
> This issue was first observed with an increase in OOM errors in the CI
> environment after applying HIVE-26986. As a temporary solution, HIVE-28832
> disabled SWO in the affected query file. This fix should be rolled back once
> the underlying issue is resolved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
