[
https://issues.apache.org/jira/browse/HIVE-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423488#comment-13423488
]
Kevin Wilfong commented on HIVE-3302:
-------------------------------------
Uploaded a diff here https://reviews.facebook.net/D4395
> Race condition in query plan for merging at the end of a query
> --------------------------------------------------------------
>
> Key: HIVE-3302
> URL: https://issues.apache.org/jira/browse/HIVE-3302
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.10.0
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Priority: Critical
>
> In the query plan that's used to merge files at the end of a query, the
> dependency tree looks something like:
> MoveTask(1)
> / \
> ...ConditionalTask MoveTask(2)...
> \ /
> MergeTask
> Here MoveTask(1) moves the partition data to a temporary location, and
> MoveTask(2) moves it to the final location.
> However if there are dynamic partitions generated and some of these
> partitions are merged and others are moved, the dependency tree is changed at
> runtime to:
> ...ConditionalTask MoveTask(2)...
> \ /
> MergeTask
> \
> MoveTask(1)
> This produces a race condition between the two MoveTasks where if MoveTask(2)
> runs before MoveTask(1) the partitions moved by MoveTask(1) will get moved to
> an intermediate location and never moved to the final location. In this case
> those partitions are quietly lost.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
