[
https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788956#comment-17788956
]
Chenyu Zheng commented on HIVE-23354:
-------------------------------------
[~jfs] [~pvary] [~kuczoram] [~nareshpr]
Hi, I know we disable speculative execution for removeTempOrDuplicateFiles in
this issue.
I found some problem about "Hive on tez" when tez speculative execution is
enable. By my debug, I found some problem, and submit the issue HIVE-25561 and
HIVE-27899. And I explain the reason in the two issues.
_"That might be problematic when there are speculative execution on the
way, and the original execution is finished, but the newest/speculative
execution is still running"_
I think after HIVE-25561 and HIVE-27899, this is not a problem, at least on tez.
After HIVE-25561, If the original execution is killed, the generated file will
not be commtited, the generated will be a temp file. removeTempOrDuplicateFiles
will delete the temp file.
After HIVE-25561, if the original execution is finished with success state, the
generated file will be committed. the speculative task attempt will stuck until
received kill signal, will never trigger to commit file, so the file generated
by speculative task is tmp file, will be deleted by removeTempOrDuplicateFiles.
What do I think? Can we enable the speculative execution? After all,
speculative execution is crucial in large-scale production environments.
Looking forward to your reply!
> Remove file size sanity checking from compareTempOrDuplicateFiles
> -----------------------------------------------------------------
>
> Key: HIVE-23354
> URL: https://issues.apache.org/jira/browse/HIVE-23354
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: John Sherman
> Assignee: John Sherman
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch,
> HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch,
> HIVE-23354.6.patch, HIVE-23354.7.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010]
> compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to
> determine which file(s) to keep.
> I've seen instances where this function throws an exception due to the fact
> that the newer attemptId file size is less than the older attemptId (thus
> failing the query).
> I think this assumption is faulty, due to various factors such as file
> compression and the order in which values are written. It may be prudent to
> trust that the newest attemptId is in fact the best choice.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
