Ottomata has submitted this change and it was merged.
Change subject: Adding classes to install hive, pig and sqoop. (very simple!)
......................................................................
Adding classes to install hive, pig and sqoop. (very simple!)
Hive Server and Hive Metastore puppetiztaion will come in a separate commit.
Adding and fixing some tests.
Change-Id: Iaa7769ce39e7002edb2b2476df3f850c611b4e6b
---
M TODO.md
A manifests/hive.pp
A manifests/pig.pp
A manifests/sqoop.pp
A templates/pig/pig.properties.erb
M tests/Makefile
M tests/historyserver.pp
A tests/hive.pp
M tests/jobtracker.pp
A tests/pig.pp
M tests/resourcemanager.pp
A tests/sqoop.pp
12 files changed, 119 insertions(+), 3 deletions(-)
Approvals:
Ori.livneh: Looks good to me, but someone else must approve
Akosiaris: Looks good to me, approved
jenkins-bot: Verified
diff --git a/TODO.md b/TODO.md
index 2fb0842..9dac694 100644
--- a/TODO.md
+++ b/TODO.md
@@ -11,4 +11,16 @@
- Make log4j.properties more configurable.
- Support Secondary NameNode.
- Support High Availability NameNode.
-- Make JMX ports configurable.
\ No newline at end of file
+- Make JMX ports configurable.
+
+## Hive
+- Hive Server + Hive Metastore
+
+## Oozie
+
+## Hue
+
+## HBase
+
+## Zookeeper
+
diff --git a/manifests/hive.pp b/manifests/hive.pp
new file mode 100644
index 0000000..5bac968
--- /dev/null
+++ b/manifests/hive.pp
@@ -0,0 +1,10 @@
+# == Class cdh4::hive
+#
+# Installs Hive packages (needed for Hive Client).
+# Use cdh4::hive::server to install and set up a Hive server.
+#
+class cdh4::hive {
+ package { 'hive':
+ ensure => 'installed',
+ }
+}
\ No newline at end of file
diff --git a/manifests/pig.pp b/manifests/pig.pp
new file mode 100644
index 0000000..a78243d
--- /dev/null
+++ b/manifests/pig.pp
@@ -0,0 +1,14 @@
+# == Class cdh4::pig
+#
+# Installs and configures Apache Pig.
+#
+class cdh4::pig {
+ package { 'pig':
+ ensure => 'installed',
+ }
+
+ file { '/etc/pig/conf/pig.properties':
+ content => template('cdh4/pig/pig.properties.erb'),
+ require => Package['pig'],
+ }
+}
diff --git a/manifests/sqoop.pp b/manifests/sqoop.pp
new file mode 100644
index 0000000..bb93f19
--- /dev/null
+++ b/manifests/sqoop.pp
@@ -0,0 +1,16 @@
+# == Class cdh4::sqoop
+# Installs Sqoop
+class cdh4::sqoop {
+ package { ['sqoop', 'libmysql-java']:
+ ensure => 'installed',
+ }
+
+ # symlink the mysql-connector-java.jar that is installed by
+ # libmysql-java into /usr/lib/sqoop/lib
+
+ file { '/usr/lib/sqoop/lib/mysql-connector-java.jar':
+ ensure => 'link',
+ target => '/usr/share/java/mysql-connector-java.jar',
+ require => [Package['sqoop'], Package['libmysql-java']],
+ }
+}
\ No newline at end of file
diff --git a/templates/pig/pig.properties.erb b/templates/pig/pig.properties.erb
new file mode 100644
index 0000000..ea1ff44
--- /dev/null
+++ b/templates/pig/pig.properties.erb
@@ -0,0 +1,54 @@
+# Pig configuration file. All values can be overwritten by command line
arguments.
+# see bin/pig -help
+
+# log4jconf log4j configuration file
+# log4jconf=./conf/log4j.properties
+
+# brief logging (no timestamps)
+brief=false
+
+# clustername, name of the hadoop jobtracker. If no port is defined port 50020
will be used.
+#cluster
+
+#debug level, INFO is default
+debug=INFO
+
+# a file that contains pig script
+#file=
+
+# load jarfile, colon separated
+#jar=
+
+#verbose print all log messages to screen (default to print only INFO and
above to screen)
+verbose=false
+
+#exectype local|mapreduce, mapreduce is default
+#exectype=mapreduce
+# hod realted properties
+#ssh.gateway
+#hod.expect.root
+#hod.expect.useensure => installed
+#hod.command
+#hod.config.dir
+#hod.param
+
+
+#Do not spill temp files smaller than this size (bytes)
+pig.spill.size.threshold=5000000
+#EXPERIMENT: Activate garbage collection when spilling a file bigger than this
size (bytes)
+#This should help reduce the number of files being spilled.
+pig.spill.gc.activation.size=40000000
+
+
+######################
+# Everything below this line is Yahoo specific. Note that I've made
+# (almost) no changes to the lines above to make merging in from Apache
+# easier. Any values I don't want from above I override below.
+#
+# This file is configured for use with HOD on the production clusters. If you
+# want to run pig with a static cluster you will need to remove everything
+# below this line and set the cluster value (above) to the
+# hostname and port of your job tracker.
+
+exectype=mapreduce
+log.file=
diff --git a/tests/Makefile b/tests/Makefile
index d3fa013..b1acb3b 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -1,4 +1,4 @@
-MANIFESTS=datanode.po defaults.po hadoop.po historyserver.po jobtracker.po
Makefile master.po namenode.po nodemanager.po resourcemanager.po tasktracker.po
worker.po
+MANIFESTS=datanode.po defaults.po hadoop.po historyserver.po hive.po
jobtracker.po Makefile master.po namenode.po nodemanager.po pig.po
resourcemanager.po sqoop.po tasktracker.po worker.po
TESTS_DIR=$(dir $(CURDIR))
MODULE_DIR=$(TESTS_DIR:/=)
MODULES_DIR=$(dir $(MODULE_DIR))
diff --git a/tests/historyserver.pp b/tests/historyserver.pp
index 8e7e6a3..5df020b 100644
--- a/tests/historyserver.pp
+++ b/tests/historyserver.pp
@@ -1,8 +1,10 @@
-#
+#
class { '::cdh4::hadoop':
namenode_hostname => 'localhost',
dfs_name_dir => '/var/lib/hadoop/name',
}
+# historyserver requires namenode
+include cdh4::hadoop::namenode
include cdh4::hadoop::historyserver
diff --git a/tests/hive.pp b/tests/hive.pp
new file mode 100644
index 0000000..05d63da
--- /dev/null
+++ b/tests/hive.pp
@@ -0,0 +1,2 @@
+
+include cdh4::hive
\ No newline at end of file
diff --git a/tests/jobtracker.pp b/tests/jobtracker.pp
index 0fc4211..34aa74a 100644
--- a/tests/jobtracker.pp
+++ b/tests/jobtracker.pp
@@ -6,4 +6,6 @@
dfs_name_dir => '/var/lib/hadoop/name',
}
+# jobtracker requires namenode
+include cdh4::hadoop::namenode
include cdh4::hadoop::jobtracker
diff --git a/tests/pig.pp b/tests/pig.pp
new file mode 100644
index 0000000..62b4904
--- /dev/null
+++ b/tests/pig.pp
@@ -0,0 +1 @@
+include cdh4::pig
\ No newline at end of file
diff --git a/tests/resourcemanager.pp b/tests/resourcemanager.pp
index 3b8e74a..a168001 100644
--- a/tests/resourcemanager.pp
+++ b/tests/resourcemanager.pp
@@ -5,4 +5,6 @@
dfs_name_dir => '/var/lib/hadoop/name',
}
+# resourcemanager requires namenode
+include cdh4::hadoop::namenode
include cdh4::hadoop::resourcemanager
diff --git a/tests/sqoop.pp b/tests/sqoop.pp
new file mode 100644
index 0000000..2df828e
--- /dev/null
+++ b/tests/sqoop.pp
@@ -0,0 +1 @@
+include cdh4::sqoop
\ No newline at end of file
--
To view, visit https://gerrit.wikimedia.org/r/65267
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaa7769ce39e7002edb2b2476df3f850c611b4e6b
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet/cdh4
Gerrit-Branch: master
Gerrit-Owner: Ottomata <[email protected]>
Gerrit-Reviewer: Akosiaris <[email protected]>
Gerrit-Reviewer: Faidon <[email protected]>
Gerrit-Reviewer: Ori.livneh <[email protected]>
Gerrit-Reviewer: Ottomata <[email protected]>
Gerrit-Reviewer: jenkins-bot
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
