[
https://issues.apache.org/jira/browse/HADOOP-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605538#action_12605538
]
Hemanth Yamijala commented on HADOOP-3579:
------------------------------------------
HADOOP-3479 asks for implementing configuration support for a Hadoop resource
manager (HADOOP-3421). Part of this configuration includes administrative
queues of jobs. Each such queue would have its own set of properties. It would
be ideal to reuse the configuration of Hadoop, so that administrators see a
consistent experience in configuring Hadoop. From a development standpoint,
there would be consistency in the APIs, as well as reuse of code for various
features.
To derive these benefits, Hadoop needs to support the following at a minimum:
- Grouping of related properties (for e.g. all properties related to a queue
are grouped)
- Having multiple instances of a group (for e.g. there can be multiple
instances of a queue)
- APIs for listing groups, their instances, and properties for a group's
instance.
One proposal is to do the following:
- Add a <group> element that can appear at the same level as a <property>
element in the current Hadoop configuration.
- The group element can have the following format:
{code:xml}
<group>
<name>hadoop.rm.queue</name>
<instance>
<name>research</name>
<property>
<name>hadoop.rm.queue.max-capacity</name>
<value>100</value>
</property>
</instance>
</group>
{code}
Then we could add APIs such as:
{code}
Set<String> getGroups();
Set<String> getInstances(String groupname);
String get(String groupname, String instancename, String propertyname);
{code}
The behavior of the following features needs to be defined for groups:
- Loading configuration from multiple resources
- Final parameters
- Variable substitution
We could define that a property of a group's instance from a resource that is
loaded latter overrides an existing value, unless it is marked final. The
granularity is at the level of a (group, instance, property) tuple.
For variable substitution, we should define a new syntax to refer to variables
in the instance of a group, something like ${groupname[instance].propertyname}
The existing configuration API can be retained as is, because there can be
properties that do not belong to any group, as they exist today in
hadoop-default and hadoop-site.
> Support property groups in Hadoop configuration
> -----------------------------------------------
>
> Key: HADOOP-3579
> URL: https://issues.apache.org/jira/browse/HADOOP-3579
> Project: Hadoop Core
> Issue Type: Improvement
> Components: conf
> Reporter: Hemanth Yamijala
>
> Hadoop configuration is currently a list of key and value pairs. There are
> some use-cases to support configuring groups of related properties. There
> could also be multiple instances of such groups. The issue is for adding
> support for such configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
