[
https://issues.apache.org/jira/browse/HIVE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180458#comment-16180458
]
wangzhihao commented on HIVE-15121:
-----------------------------------
Hi [~stakiar]
Should not [this statement |
https://github.com/apache/hive/blob/aa7edfefe20cede2c37d8c8b6c864c3b6039923f/ql/src/java/org/apache/hadoop/hive/ql/Context.java#L497
] be
{code:java}
if (!isFinalJob && BlobStorageUtils.areOptimizationsEnabled(conf)) {
{code}
We should use HDFS staging dir when optimization is enabled and the MR is not
final job. But current logic will call getMRTmpPath() code when
areOptimizationsEnabled() return false.
> Last MR job in Hive should be able to write to a different scratch directory
> ----------------------------------------------------------------------------
>
> Key: HIVE-15121
> URL: https://issues.apache.org/jira/browse/HIVE-15121
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Fix For: 2.3.0
>
> Attachments: HIVE-15121.1.patch, HIVE-15121.2.patch,
> HIVE-15121.3.patch, HIVE-15121.patch, HIVE-15121.WIP.1.patch,
> HIVE-15121.WIP.2.patch, HIVE-15121.WIP.patch
>
>
> Hive should be able to configure all intermediate MR jobs to write to HDFS,
> but the final MR job to write to S3.
> This will be useful for implementing parallel renames on S3. The idea is that
> for a multi-job query, all intermediate MR jobs write to HDFS, and then the
> final job writes to S3. Writing to HDFS should be faster than writing to S3,
> so it makes more sense to write intermediate data to HDFS.
> The advantage is that any copying of data that needs to be done from the
> scratch directory to the final table directory can be done server-side,
> within the blobstore. The MoveTask simply renames data from the scratch
> directory to the final table location, which should translate to a
> server-side COPY request. This way HiveServer2 doesn't have to actually copy
> any data, it just tells the blobstore to do all the work.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
