[
https://issues.apache.org/jira/browse/IMPALA-15103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18089384#comment-18089384
]
ASF subversion and git services commented on IMPALA-15103:
----------------------------------------------------------
Commit 33d9560f52717994dc7948756c571f65e6338df2 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=33d9560f5 ]
IMPALA-15103: Hive configuration should honor the HDFS_REPLICATION env variable
This changes hive-site.xml.py to use the HDFS_REPLICATION
environment variable to set Hive's configuration for
HDFS replication. This makes setting HDFS_REPLICATION=1
apply to Hive tables. This is particularly useful when
using larger scale factors of TPC-H/TPC-DS to avoid excess
disk space use. Using this on a standard dataload is
fairly functional, but it will break some planner tests.
Testing:
- Ran bin/single_node_perf_run.py with this setting and
monitored disk space usage
Change-Id: Iceeaed41eb5b3797459c0a3035d159d903453eeb
Reviewed-on: http://gerrit.cloudera.org:8080/24461
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Minicluster Hive should respect the HDFS_REPLICATION env variable
> -----------------------------------------------------------------
>
> Key: IMPALA-15103
> URL: https://issues.apache.org/jira/browse/IMPALA-15103
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> HDFS replication defaults to 3, which dramatically increases disk space for
> tables. We have an existing HDFS_REPLICATION environment variable, but Hive
> does not use it in its configurations. We should change Hive's configurations
> to respect this environment variable. This is particularly useful for
> perf-AB-test jobs where we are loading large scale TPC-H or TPC-DS and want
> to avoid the disk space use.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
