[
https://issues.apache.org/jira/browse/HIVE-26758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Zhang updated HIVE-26758:
----------------------------
Description:
The query results are staged in stagingdir that is relative to the destination
path <destination_dir>/<staging_dir>/
It used to be able to change hive.exec.stagingdir for a different location, but
that is lost during blobstorage optimzation HIVE-17620.
HIVE-15215 mentioned the possibility of using scratch for staging when write to
S3 but it was long time ago and no activity.
This is to allow final job to use hive.exec.scratchdir as the interim jobs,
with a configuration
hive.use.scratchdir.for.staging
This is useful for cross Filesystem, user can use local source filesystem
instead of remote filesystem for the staging.
was:
The query results are staged in stagingdir that is relative to the destination
path <destination_dir>/<staging_dir>/
It used to be able to change hive.exec.stagingdir for a different location, but
that is lost during blobstorage optimzation HIVE-17620.
This is to allow final job to use hive.exec.scratchdir as the interim jobs,
with a configuration
hive.use.scratchdir_for_staging
This is useful for cross Filesystem, user can use local source filesystem
instead of remote filesystem for the staging.
main change:
for dynamic partitions that has static partition it was
<destination_dir>/<static_partition>/<staging_dir>/<dynamic_partition>
changes to
<destination_dir>/<staging_dir>/<static_partition>/<dynamic_partition>
or in case of \{hive.use.scratchdir_for_staging}
<scratch_dir>/<static_partition>/<dynamic_partition>
the change is due to that hive relies on parsing the path to discover
partitions.
> Allow use scratchdir for staging final job
> ------------------------------------------
>
> Key: HIVE-26758
> URL: https://issues.apache.org/jira/browse/HIVE-26758
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
> Affects Versions: 4.0.0-alpha-2
> Reporter: Yi Zhang
> Assignee: Yi Zhang
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The query results are staged in stagingdir that is relative to the
> destination path <destination_dir>/<staging_dir>/
> It used to be able to change hive.exec.stagingdir for a different location,
> but that is lost during blobstorage optimzation HIVE-17620.
> HIVE-15215 mentioned the possibility of using scratch for staging when write
> to S3 but it was long time ago and no activity.
>
> This is to allow final job to use hive.exec.scratchdir as the interim jobs,
> with a configuration
> hive.use.scratchdir.for.staging
> This is useful for cross Filesystem, user can use local source filesystem
> instead of remote filesystem for the staging.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
