Repository: bigtop Updated Branches: refs/heads/master 362ca725e -> 6fd647e9c
BIGTOP-1914. improve puppet README.md file Signed-off-by: Evans Ye <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/6fd647e9 Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/6fd647e9 Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/6fd647e9 Branch: refs/heads/master Commit: 6fd647e9c5a5f7929f36482dbf780ea2ce61bae6 Parents: 362ca72 Author: Martin Bukatovic <[email protected]> Authored: Thu Jul 2 13:41:55 2015 +0200 Committer: Evans Ye <[email protected]> Committed: Sat Jul 4 18:49:41 2015 +0000 ---------------------------------------------------------------------- bigtop-deploy/puppet/README.md | 112 +++++++++++++++++------------------- 1 file changed, 52 insertions(+), 60 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/bigtop/blob/6fd647e9/bigtop-deploy/puppet/README.md ---------------------------------------------------------------------- diff --git a/bigtop-deploy/puppet/README.md b/bigtop-deploy/puppet/README.md index 634a820..1e9c68a 100644 --- a/bigtop-deploy/puppet/README.md +++ b/bigtop-deploy/puppet/README.md @@ -5,102 +5,89 @@ BigTop is now using Puppet 3.x (BIGTOP-1047)! Installing and starting hadoop services is non-trivial, and for this reason bigtop - packages puppet instrumentation for the various ecosystem components, which works - synergistically with bigtop produced software packages. The puppet classes for bigtop deployment setup and deploy hadoop services. - This includes tasks such as: - service installation - pointing slaves to masters (i.e. regionservers, nodemanagers to their respective master) - starting the services -The mode of puppet is masterless : there is no fancy coordination happening behind the scenes. +The mode of puppet is *masterless*: there is no fancy coordination happening behind the scenes. Puppet has a notion of a configuration directory, called config. +When running puppet apply, note that puppet's confdir is *underneath* the `--confdir` value. +For example: If you have `site.csv` in `/etc/puppet/config`, +Then you should use `--confdir=/etc/puppet` , and puppet finds the config dir underneath. -When running puppet apply, note that puppet's confdir is *underneath* the --confdir value. - -For example: - -If you have site.csv in /etc/puppet/config, - -Then you should use --confdir=/etc/puppet , and puppet finds the config dir underneath. - -As an end to end example, you can follow the vagrant-puppet-vm recipes to see how to set up - +As an end to end example, you can follow the `vagrant-puppet-vm` recipes to see how to set up a puppet managed bigtop hadoop installation. Those examples are gauranteed to work and - serve as a pedagogical round trip to the way bigtop integrates packaging, deployment, and - testing all into one package. ## Debugging -If in any case, you need to debug these recipes, you can add notify("...") statements into +If in any case, you need to debug these recipes, you can add `notify("...")` statements into the puppet scripts. In time, we will add more logging and debugging to these recipes. Feel free to submit - -a patch for this ! +a patch for this! ## Configuration -As above, we defined a confdir (i.e. /etc/puppet/) which has a config/ directory in it. - -The heart of puppet is the manifests file. This file ( manifests/init.pp ) +As above, we defined a confdir (i.e. `/etc/puppet/`) which has a `config/` directory in it. -expects configuration to live in hiera as specified by $confdir/hiera.yaml. An example -hiera.yaml as well as hiera configuration yaml files are provided with the bigtop classes. They +The heart of puppet is the manifests file. This file (`manifests/init.pp`) +expects configuration to live in hiera as specified by `$confdir/hiera.yaml`. An example +`hiera.yaml` as well as hiera configuration yaml files are provided with the bigtop classes. They basically take the form: -<pre> +``` key: value -</pre> +``` with syntactic variations for hashes and arrays. Please consult the excellent puppet and hiera documentation for details. -All configuration is done via such key value assignments in hierdata/site.yaml. Any options -not defined there will revert to a default value defined in hieradata/cluster.yaml, with the +All configuration is done via such key value assignments in `hierdata/site.yaml`. Any options +not defined there will revert to a default value defined in `hieradata/cluster.yaml`, with the following exceptions (which are required): -* bigtop::hadoop\_head\_node: must be set to the FQDN of the name node of your - cluster (which will also become its job tracker and gateway) +* `bigtop::hadoop_head_node`: must be set to the FQDN of the name node of your + cluster (which will also become its job tracker and gateway) -* bigtop::bigtop\_repo\_uri: uri of a repository containing packages for - hadoop as built by Bigtop. +* `bigtop::bigtop_repo_uri`: uri of a repository containing packages for + hadoop as built by Bigtop. -$confdir is the directory that puppet will look into for its configuration. On most systems, -this will be either /etc/puppet/ or /etc/puppetlabs/puppet/. You may override this value by -specifying --confdir=path/to/config/dir on the puppet command line. +`$confdir` is the directory that puppet will look into for its configuration. On most systems, +this will be either `/etc/puppet/` or `/etc/puppetlabs/puppet/`. You may override this value by +specifying `--confdir=path/to/config/dir` on the puppet command line. -cluster.yaml also serves as an example what parameters can be set and how they usually interact +`cluster.yaml` also serves as an example what parameters can be set and how they usually interact between modules. You can instruct the recipes to install ssh-keys for user hdfs to enable passwordless login across the cluster. This is for test purposes only, so by default the option is turned off. -Files such as ssh-keys are imported from the master using the puppet:/// URL scheme. For this +Files such as ssh-keys are imported from the master using the `puppet:///` URL scheme. For this to work, fileserver has to be enabled on the puppet master, the files module enabled and access allowed in auth.conf. fileserver.conf should therefore contain e.g.: -<pre> +``` [files] path /etc/puppet/files allow * -</pre> +``` No changes are required to the default puppet 3 auth.conf. For other options that may be set here, look for class parameters in the modules' manifests/init.pp files. Any class parameter can be used as a hiera key if prefixed with the module and class namespace. Module hue's server class will look for its parameter rm_host as -hue::server::rm_host in hiera. -Note that if hadoop::hadoop\_storage\_dirs is left unset, puppet will attempt to guess which +`hue::server::rm_host` in hiera. +Note that if `hadoop::hadoop_storage_dirs` is left unset, puppet will attempt to guess which directories to use. ## Usage @@ -109,13 +96,18 @@ directories to use. - Make sure puppet is installed - Make sure all the required puppet modules are installed: -<pre> -gradle toolchain-puppetmodules (if you already have JAVA installed) +``` +gradle toolchain-puppetmodules # if you already have JAVA installed +``` + or + +``` puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::puppet-modules" -</pre> +``` This will install the following module(s) for you: + * [puppet stdlib module](https://forge.puppetlabs.com/puppetlabs/stdlib) * [puppet apt module](https://forge.puppetlabs.com/puppetlabs/apt) on Ubuntu, Debian only @@ -124,11 +116,11 @@ Bigtop toolchan can take care of that for you, so just be aware of it. And run the following on those nodes: -<pre> +``` # cp bigtop-deploy/puppet/hiera.yaml /etc/puppet # mkdir -p /etc/puppet/hieradata # rsync -a --delete bigtop-deploy/puppet/hieradata/bigtop/ /etc/puppet/hieradata/bigtop/ -# cat > /etc/puppet/hieradata/site.yaml << EOF +# cat > /etc/puppet/hieradata/site.yaml <<EOF # bigtop::hadoop_head_node: "hadoopmaster.example.com" # hadoop::hadoop_storage_dirs: # - "/data/1" @@ -136,13 +128,13 @@ And run the following on those nodes: # bigtop::bigtop_repo_uri: "http://mirror.example.com/path/to/mirror/"; # EOF # puppet apply -d --modulepath="bigtop-deploy/puppet/modules:/etc/puppet/modules" bigtop-deploy/puppet/manifests/site.pp -</pre> +``` When ignite-hadoop accelerator is deployed the client configs are placed under -/etc/hadoop/ignite.client.conf. All one needs to do to run Mapreduce jobs on ignite-hadoop grid -is to set HADOOP_CONF_DIR=/etc/hadoop/ignite.client.conf in the client session. +`/etc/hadoop/ignite.client.conf`. All one needs to do to run Mapreduce jobs on ignite-hadoop grid +is to set `HADOOP_CONF_DIR=/etc/hadoop/ignite.client.conf` in the client session. -# Passwords +## Passwords These classes are mostly used for regression testing. For ease of use they contain insecure default passwords in a number of places. If you intend to use @@ -150,10 +142,10 @@ them in production environments, make sure to track down all those places and set proper passwords. This can be done using the corresponding hiera settings. Some of these (but almost certainly not all!) are: -<pre> +``` hadoop::common_hdfs::hadoop_http_authentication_signature_secret hadoop::httpfs::secret -</pre> +``` ## Automatic password generation @@ -175,11 +167,11 @@ with this approach at the moment: trocla needs to be installed on the master only. To do so, run the following: -<pre> +``` # gem install trocla # puppet module install duritong/trocla # puppet apply -e "class { 'trocla::config': manage_dependencies => false }" -</pre> +``` The trocla ruby gem pulls in highline, moneta and bcrypt. The bcrypt gem needs ruby development packages (ruby.h) and a compiler. @@ -187,9 +179,9 @@ ruby development packages (ruby.h) and a compiler. Alternatively you can install your distributions' binary packages *before* running gem install. On Debian those packages can be installed as follows: -<pre> +``` apt-get install ruby-highline ruby-moneta ruby-bcrypt -</pre> +``` This installation process is expected to get easier once operation system packages have been created. This is actively underway for Debian. See Debian @@ -198,14 +190,14 @@ bugs #777761 and #777906 for progress. After installing the trocla gem as outlined above, the following test should work: -<pre> +``` # puppet apply -e "file { '/tmp/test': content => trocla("test", "plain") }" # cat /tmp/test puGNOX-G%zYDKHet -</pre> +``` Now, automatic password generation can be activated in site.yaml using -<pre> +``` hadoop::generate_secrets: true -</pre> +```
