Ottomata has submitted this change and it was merged. Change subject: Some fixes for Hive in CDH 5.3.1 ......................................................................
Some fixes for Hive in CDH 5.3.1 - Hive needs hive.exec.stagingdir set explicitly as a workaround for https://issues.cloudera.org/browse/DISTRO-681 - parquet.compression needs to be configurable in order to set a default compression for Parquet tables. Change-Id: I65e60044b80020e805c7992ae589db285190bd81 --- M manifests/hive.pp M manifests/hive/defaults.pp M templates/hive/hive-site.xml.erb 3 files changed, 33 insertions(+), 1 deletion(-) Approvals: Ottomata: Looks good to me, approved jenkins-bot: Verified diff --git a/manifests/hive.pp b/manifests/hive.pp index 27411d5..d91cb80 100644 --- a/manifests/hive.pp +++ b/manifests/hive.pp @@ -26,9 +26,12 @@ # Only set these if your root user cannot issue database # commands without a different username and password. # Default: undef -# $variable_substitute_depth - The maximum replacements the substitution engine will do. Default: undef +# $variable_substitute_depth - The maximum replacements the substitution engine will do. Default: undef # # $auxpath - Additional path to pass to hive. Default: undef +# $parquet_compression - Compression type for parquet-format to use. It will +# ignore mapreduce_output_compession_codec. Set this to +# one of UNCOMPRESSED, SNAPPY, GZIP. Default: undef # $exec_parallel_thread_number - Number of jobs at most can be executed in parallel. # Set this to 0 to disable parallel execution. # $optimize_skewjoin - Enable or disable skew join optimization. @@ -66,6 +69,7 @@ $variable_substitute_depth = $cdh::hive::defaults::variable_substitute_depth, $auxpath = $cdh::hive::defaults::auxpath, + $parquet_compressions = $cdh::hive::defaults::parquet_compression, $exec_parallel_thread_number = $cdh::hive::defaults::exec_parallel_thread_number, $optimize_skewjoin = $cdh::hive::defaults::optimize_skewjoin, diff --git a/manifests/hive/defaults.pp b/manifests/hive/defaults.pp index 2b33ee0..ba1e851 100644 --- a/manifests/hive/defaults.pp +++ b/manifests/hive/defaults.pp @@ -21,6 +21,8 @@ # Until Hive 0.12.0 this can only be a single path. See HIVE-2269. $auxpath = undef + $parquet_compression = undef + $exec_parallel_thread_number = 8 # set this to 0 to disable hive.exec.parallel $optimize_skewjoin = false $skewjoin_key = 10000 diff --git a/templates/hive/hive-site.xml.erb b/templates/hive/hive-site.xml.erb index 2df725a..b9f94a6 100644 --- a/templates/hive/hive-site.xml.erb +++ b/templates/hive/hive-site.xml.erb @@ -127,6 +127,21 @@ </property> <property> + <name>hive.exec.stagingdir</name> + <value>/tmp/hive-staging</value> + <description> + Directory name that will be created inside table locations in order + to support HDFS encryption.This is the replacement of ${hive.exec.scratchdir} + for all Hive statements that deals with HDFS encryption zones. ${hive.exec.scratchdir} + is still used for other temporary, such as job plans. + NOTE: This is being set explicitly as a work around for + https://issues.cloudera.org/browse/DISTRO-681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel + See also: http://community.cloudera.com/t5/Batch-SQL-Apache-Hive/CDH-5-3-Hive-staging-directory-has-wrong-default-value/td-p/23585 + </description> + </property> + + + <property> <name>hive.error.on.empty.partition</name> <description>Whether to throw an exception if dynamic partition insert generates empty results.</description> <value>true</value> @@ -281,4 +296,15 @@ <value><%= @auxpath %></value> </property> <% end -%> + +<% if @parquet_compression -%> + <!-- Default parquet output compressions codec to use. --> + <!-- See: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_parquet.html --> + <property> + <name>parquet.compression</name> + <value><%= @parquet_compression %></value> + <description>The valid options for compression are: UNCOMPRESSED, GZIP, SNAPPY</description> + </property> +<% end -%> + </configuration> \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/190827 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I65e60044b80020e805c7992ae589db285190bd81 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet/cdh Gerrit-Branch: master Gerrit-Owner: Ottomata <[email protected]> Gerrit-Reviewer: Ottomata <[email protected]> Gerrit-Reviewer: jenkins-bot <> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
