[
https://issues.apache.org/jira/browse/HIVE-29124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18018136#comment-18018136
]
Denys Kuzmenko commented on HIVE-29124:
---------------------------------------
Merged to master
Thanks for the fix, [~zhengchenyu] and [~abstractdog] for the review!
> Avoid committing files when a task is aborted even though some source has
> completed
> -----------------------------------------------------------------------------------
>
> Key: HIVE-29124
> URL: https://issues.apache.org/jira/browse/HIVE-29124
> Project: Hive
> Issue Type: Bug
> Reporter: Chenyu Zheng
> Assignee: Chenyu Zheng
> Priority: Major
> Labels: pull-request-available
>
> I found that when the task is almost completed (more precisely, when the
> source has been processed), but not closed, if an exception is thrown at this
> time, it may cause the file to be committed incorrectly.
> Look at
> [https://github.com/apache/hive/blob/9b07c5c7136863ae1eb469e7a3c11357299d2ea1/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java#L349].
> abort may be set from true to false. It is not reasonable. The correct logic
> is that as long as abort is set to true at one place, abort should always be
> true, then do not commit.
> When I tried to reproduce this bug, I found that only
> [https://github.com/apache/hive/blob/9b07c5c7136863ae1eb469e7a3c11357299d2ea1/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java#L369]
> caused the problem. I initially thought that the problem would occur at
> [https://github.com/apache/hive/blob/9b07c5c7136863ae1eb469e7a3c11357299d2ea1/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java#L356]
> because the reduce op's abort was set to true in the abort. However,
> dummyOps was not properly aborted. Here, dummyOps should also be aborted.
> Therefore, the issue only occurs when dummyOps is used, such as in mapjoin.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
