[
https://issues.apache.org/jira/browse/BIGTOP-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354586#comment-14354586
]
Michael Weiser commented on BIGTOP-1746:
----------------------------------------
At least in the Puppet manifests there are only two places where the current
role concept is implemented:
- manifests/cluster.pp decides what daemons to put on which box
- hieradata/bigtop/cluster.yaml governs what to put into their config files
cluster.yaml can easily be adjusted and overridden so it doesn't force the
concept of head node and frontend into the config files any more. So the main
point of attack from my point of view is cluster.pp. Unfortunately it also
implements some dependencies between modules and some add-on logic, such as
running init-hdfs.sh. Basically I would suggest moving these dependencies into
their respective modules and then just throwing away cluster.pp. After that,
the classes could be included directly using hiera_include with the hiera
lookup hierachy or possibly an ENC or facts governing which roles a machine has.
I have a setup where I've basically done that. I have changes I was already
planning to propose for merging that move dependencies mostly into the hadoop
module. That would render cluster.pp quite empty already. I also have a concept
for assigning roles to nodes via hiera. This, however, is a bit convoluted and
would need streamlining for inclusion in mainline BigTop. In the most basic
case, classes such as hadoop::namenode can just be assigned to nodes directly
such as this:
manifests/site.pp:
{noformat}
hiera_include("classes")
{noformat}
hiera.yaml:
{noformat}
---
:yaml:
:datadir: /etc/puppet/hieradata
:hierarchy:
- "node/%{::fqdn}"
- site
- bigtop/cluster
{noformat}
hieradata/node/node1.do.main.yaml:
{noformat}
---
classes:
- hadoop::namenode
- hadoop-zookeeper::server
{noformat}
> Introduce the concept of roles in bigtop cluster deployment
> -----------------------------------------------------------
>
> Key: BIGTOP-1746
> URL: https://issues.apache.org/jira/browse/BIGTOP-1746
> Project: Bigtop
> Issue Type: New Feature
> Components: deployment
> Reporter: vishnu gajendran
> Labels: features
> Fix For: 0.9.0
>
>
> Currently, during cluster deployment, puppet categorizes nodes as head_node,
> worker_nodes, gateway_nodes, standy_node based on user specified info. This
> functionality gives user control over picking up a particular node as
> head_node, standy_node, gateway_node and rest others as worker_nodes. But, I
> woulld like to have more fine-grained control on which deamons should run on
> which node. For example, I do not want to run namenode, datanode on the same
> node. This functionality can be introduced with the concept of roles. Each
> node can be assigned a set of role. For example, Node A can be assigned
> ["namenode", "resourcemanager"] roles. Node B can be assigned ["datanode",
> "nodemanager"] and Node C can be assigned ["nodemanager", "hadoop-client"].
> Now, each node will only run the specified daemons. Prerequisite for this
> kind of deployment is that each node should be given the necessary
> configurations that it needs to know. For example, each datanode should know
> which is the namenode etc... This functionality will allow users to customize
> the cluster deployment according to their needs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
