[
https://issues.apache.org/jira/browse/HIVE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634332#comment-15634332
]
Sahil Takiar commented on HIVE-15121:
-------------------------------------
I'm hoping it should be a similar to HIVE-14270. I think there is a way to find
the last MR job in a plan. The {{MapredWork.isFinalMapRed}} method should work
here.
> Last MR job in Hive should be able to write to a different scratch directory
> ----------------------------------------------------------------------------
>
> Key: HIVE-15121
> URL: https://issues.apache.org/jira/browse/HIVE-15121
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
> Reporter: Sahil Takiar
>
> Hive should be able to configure all intermediate MR jobs to write to HDFS,
> but the final MR job to write to S3.
> This will be useful for implementing parallel renames on S3. The idea is that
> for a mutli-job query, all intermediate MR jobs write to HDFS, and then the
> final job writes to S3. Writing to HDFS should be faster than writing to S3,
> so it makes more sense to write intermediate data to HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
