[
https://issues.apache.org/jira/browse/YARN-140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468128#comment-13468128
]
Ahmed Radwan commented on YARN-140:
-----------------------------------
Thanks Arun,
Let carify my understanding of the issue a little bit:
- We already have the capacity-scheduler.xml (which contains the default confs)
packaged in the hadoop-yarn-server-resourcemanager.<VERSION>.jar, so it is
available in the classpath, and when the user sets the scheduler class to CS in
yarn-site.xml, everything works fine and the default configutrations in
capacity-scheduler.xml are picked and used.
The *problem* happens, when the user tries to provide his own
capacity-scheduler.xml and places it in the conf directory. Since it has the
same name as the one in the jar, then it replaces the one in the classpath and
since CapacitySchedulerConfiguration constructor loads a single resource
(capacity-scheduler.xml), then the loaded configuration will be only the ones
from the user's capacity-scheduler.xml, and no defaults are loaded from the
file packaged in the jar.
by default confs, I mean stuff like:
{code}
<property>
<name>yarn.scheduler.capacity.root.capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.maximum-applications</name>
<value>10000</value>
</property>
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>
</property>
{code}
which are all lost and the user need to explicitly add them again to his xml,
or otherwise face errors like the one I mentioned above in the description.
So to address this problem, my attached patch modified the
CapacitySchedulerConfiguration constructor to load two resources instead of
one, so the default confs can be preserved and optionally overridden by the
user confs.
One other option is to provide a template conf with our distribution and
populate this file with all the defaults (basically a copy of the one in the
package), and then instruct users to use this file, but I think this option is
less convenient as it requires manual preservation of these defaults by the
user and knoweledge of what to change vs what to keep, etc.
> Add capacity-scheduler-default.xml to provide a default set of configurations
> for the capacity scheduler.
> ---------------------------------------------------------------------------------------------------------
>
> Key: YARN-140
> URL: https://issues.apache.org/jira/browse/YARN-140
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: YARN-140.patch
>
>
> When setting up the capacity scheduler users are faced with problems like:
> {code}
> FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
> starting ResourceManager
> java.lang.IllegalArgumentException: Illegal capacity of -1 for queue root
> {code}
> Which basically arises from missing basic configurations, which in many
> cases, there is no need to explicitly provide, and a default configuration
> will be sufficient. For example, to address the error above, the user need to
> add a capacity of 100 to the root queue.
> So, we need to add a capacity-scheduler-default.xml, this will be helpful to
> provide the basic set of default configurations required to run the capacity
> scheduler. The user can still override existing configurations or provide new
> ones in capacity-scheduler.xml. This is similar to *-default.xml vs
> *-site.xml for yarn, core, mapred, hdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
