[
https://issues.apache.org/jira/browse/HIVE-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438172#comment-15438172
]
Sergey Shelukhin edited comment on HIVE-14636 at 8/26/16 12:20 AM:
-------------------------------------------------------------------
After some discussions and looking at code, it looks like for now we will write
a manifest into the FS to replace the FSOp-specific commit logic.
MoveTask will go off these manifests and delete them
There's no way to pass the payload from FSOp running inside tasks back to the
client where MoveTask lives without involving some intermediate storage.
As a side note, it appears that Hive does not actually support speculative
execution provided by Tez, so we never call canCommit. Update: Hive actually
just chooses the biggest file among different task attempts, so this can still
be coordinated by choosing the biggest file among those in manifests.
Retroactive failures are impossible for final outputs, and at any rate it
doesn't seem like they should happen against persistent FS if Hive could write
final outputs from intermediate stages - otherwise it would mean HDFS/S3/... is
losing data.
So \*waves hands vigorously\* manifests will not collide the same way the
existing FSOP moves do not collide.
cc [~gopalv] [~sseth] [~ashutoshc]
was (Author: sershe):
After some discussions and looking at code, it looks like for now we will write
a manifest into the FS to replace the FSOp-specific commit logic.
MoveTask will go off these manifests and delete them
There's no way to pass the payload from FSOp running inside tasks back to the
client where MoveTask lives without involving some intermediate storage.
As a side note, it appears that Hive does not actually support speculative
execution provided by Tez, so we never call canCommit.
Retroactive failures are impossible for final outputs, and at any rate it
doesn't seem like they should happen against persistent FS if Hive could write
final outputs from intermediate stages - otherwise it would mean HDFS/S3/... is
losing data.
So \*waves hands vigorously\* manifests will not collide the same way the
existing FSOP moves do not collide.
cc [~gopalv] [~sseth] [~ashutoshc]
> pass information from FSOP/TezTask to MoveTask to take care of speculative
> execution and failed tasks
> -----------------------------------------------------------------------------------------------------
>
> Key: HIVE-14636
> URL: https://issues.apache.org/jira/browse/HIVE-14636
> Project: Hive
> Issue Type: Sub-task
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
