[
https://issues.apache.org/jira/browse/BIGTOP-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14056728#comment-14056728
]
Richard Pelavin edited comment on BIGTOP-1047 at 7/9/14 8:45 PM:
-----------------------------------------------------------------
As a first step in the task of refactoring the puppet modules to handle puppet
>=3.x, wanted to start with a design approach for handling attribute settings.
Key goal is using defaults, but to make it easy to override any attribute that
can appear in any hadoop config file. The current puppet implementation had a
nice way to deal with this using the Puppet extlookup function. Changes are
needed to handle Puppet >=3.x which no longer supports dynamic scoping and to
take advantage of Puppet functionality, such as Hiera that provides more
convenient capabilities. Will shortly propose an approach that takes the Hiera
design pattern and will support Hiera, but also will support other
representations that encode the same information.
The approach I am proposing to embark on is to separate this into a number of
sub-task
- Propose a global namespace for describing hadoop config attributes that
will be applicable to a wide range of hadoop topologies with various mixtures
of services; I will look through all the related work in Bigtop and leverage
anything done here already.
- Put in a simple mechanism that better integrates
-- the way Linux packages include reference config files, and
-- templates (e.g Erubis, Jinja2) used by config management systems
- that tend to override and ignore what is packaged in the Linux package. This
seems like a unique opportunity for a project that handles both packaging and
configuration to better align the two. I will elaborate shortly, but this would
entail putting inside the packages themselves either templates or a simple meta
description of what parameters can be possibly set and their defaults. An
important consideration is making this so it can be incrementally and
gracefully folded in; so a key design goal would be to handle for example just
certain config fragments that are typically parametrized and falling back on
existing approach of having a sample config files with defaults that you copy
over. As a reference point will look at Augeas, which has both some successes
and some challenges
- Implement using this approach first HDFS and refine approach before moving
on to the other services. Yarn we be treated second.
was (Author: rnp):
As a first step in the task of refactoring the puppet modules to handle puppet
>=3.x, wanted to start with a design approach for handling attribute settings.
Key goal is using defaults, but to make it easy to override any attribute that
can appear in any hadoop config file. The current puppet implementation had a
nice way to deal with this using the Puppet extlookup function. Changes are
needed to handle Puppet >=3.x which no longer supports dynamic scoping and to
take advantage of Puppet functionality, such as Hiera that provides more
convenient capabilities. Will shortly propose an approach that takes the Hiera
design pattern and will support Hiera, but also will support other
representations that encode the same information.
The approach I am proposing to embark on is to separate this into a number of
sub-task
- Propose a global namespace for describing hadoop config attributes that
will be applicable to a wide range of hadoop topologies with various mixtures
of services; I will look through all the related work in Bigtop and leverage
anything done here already.
- Put in a simple mechanism that better integrates
-- the way Linux packages include reference config files, and
-- templates (e.g Erubis, Jinja2) used by config management systems
that tend to override and ignore what is packaged in the Linux package. This
seems like a unique opportunity for a project that handles both packaging and
configuration to better align the two. I will elaborate shortly, but this would
entail putting inside the packages themselves either templates or a simple meta
description of what parameters can be possibly set and their defaults. An
important consideration is making this so it can be incrementally and
gracefully folded in; so a key design goal would be to handle for example just
certain config fragments that are typically parametrized and falling back on
existing approach of having a sample config files with defaults that you copy
over. As a reference point will look at Augeas, which has both some successes
and some challenges
- Implement using this approach first HDFS and refine approach before moving
on to the other services. Yarn we be treated second.
> Support Puppet 3.x
> ------------------
>
> Key: BIGTOP-1047
> URL: https://issues.apache.org/jira/browse/BIGTOP-1047
> Project: Bigtop
> Issue Type: Improvement
> Components: Deployment
> Affects Versions: 0.6.0
> Reporter: Andrey Klochkov
> Attachments: BIGTOP-1047--n2.patch, BIGTOP-1047.patch
>
>
> Currently bigtop-deploy/puppet module supports Puppet 2.x only. In
> particular, this is because of using dynamic scoping which was deprecated
> since 2.5 and not supported in 3.x. This task is to get rid of dynamic
> scoping to make bigtop-deploy/puppet working with Puppet 3.x.
> More on dynamic scoping in Puppet:
> http://docs.puppetlabs.com/guides/scope_and_puppet.html
--
This message was sent by Atlassian JIRA
(v6.2#6252)
