[
https://issues.apache.org/jira/browse/BIGTOP-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Slawski updated BIGTOP-1689:
----------------------------------
Attachment: BIGTOP-1689.1.patch
Attached is a patch that will enable merging in arbitrary configuration from
site.yaml into any xml template in puppet. The bigtop-util module is
reintroduced which includes a custom puppet function *merge_hadoop_config* that
merges site.xml content with overrides.
h4. Testing Done
h5. Unit Tests
Running unit tests are describe in the *README.mb* under *bigtop-util*.
{code}
$ cd bigtop-deploy/puppet/modules/bigtop-util
$ bundle install --path vendor/bundle
$ bundle exec rake spec
....
Finished in 0.10186 seconds
4 examples, 0 failures
{code}
h5. Sanity Verification
Included the following in site.yaml. I've verified the corresponding site files
were correctly merged.
{code}
hadoop::common_hdfs::hdfs_site_overrides:
dfs.namenode.fs-limits.min-block-size: 1000
hadoop::common_yarn::yarn_site_overrides:
yarn.log-aggregation-enable: "false"
yarn.scheduler.maximum-allocation-mb: 11520
yarn.resourcemanager.resource-tracker.client.thread-count: 64
yarn.resourcemanager.client.thread-count: 64
yarn.scheduler.minimum-allocation-mb: 256
yarn.nodemanager.resource.cpu-vcores: 8
yarn.nodemanager.resource.memory-mb: 11520
hadoop-hive::client::hive_site_overrides:
datanucleus.fixedDatastore: "true"
{code}
> puppet: Allow merging arbitrary site configuration
> --------------------------------------------------
>
> Key: BIGTOP-1689
> URL: https://issues.apache.org/jira/browse/BIGTOP-1689
> Project: Bigtop
> Issue Type: Improvement
> Components: deployment
> Affects Versions: 0.8.0
> Reporter: Peter Slawski
> Assignee: Peter Slawski
> Attachments: BIGTOP-1689.1.patch
>
>
> Puppet should be flexible in allowing arbitrary configuration name value
> pairs to be merged into a given site.xml file that was generated from a
> template.
> For example, the following could be included in site.yaml which would add a
> configuration entry for hadoop.tmp.dir in core-site.xml:
> {code}
> hadoop::common_hdfs::hadoop_core_site_overrides:
> "hadoop.tmp.dir": "/mnt/var/lib/hadoop/tmp"
> {code}
> This could be implemented as a puppet custom-function taking in the output of
> template:
> {code}
> file {
> "/etc/hadoop/conf/core-site.xml":
> content => merge_site(template('hadoop/core-site.xml'),
> $hadoop_core_site_overrides)
> require => [Package["hadoop"]],
> }
> {code}
> Perhaps another approach would be to have site.xml templates be created from
> a single map of name value pairs. The merge would happen before the file
> content is generated from the template.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
