Hello, I am trying to administrate a Spark Hadoop CLuster with the open source Hadoop distribution. We would like to set up a Capacity Scheduler with one queue per each user. My yarn-site.xml has the following lines:
<property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> </property> And This is my capacity-scheduler.xml <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><configuration> <property> <name>yarn.scheduler.capacity.maximum-applications</name> <value>10000</value> <description> Maximum number of applications that can be pending and running. </description> </property> <property> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name> <value>0.1</value> <description> Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications. </description> </property> <property> <name>yarn.scheduler.capacity.root.default.capacity</name> <value>60</value> </property> <property> <name>yarn.scheduler.capacity.resource-calculator</name> <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value> <description> The ResourceCalculator implementation to be used to compare Resources in the scheduler. The default i.e. DefaultResourceCalculator only uses Memory while DominantResourceCalculator uses dominant-resource to compare multi-dimensional resources such as Memory, CPU etc. </description> </property> <property> <name>yarn.scheduler.capacity.root.queues</name> <value>admin,users</value> <description> The queues at the this level (root is the root queue). </description> </property> <property> <name>yarn.scheduler.capacity.root.users.auto-create-child-queue.enabled</name> <value>true</value> <description> Allow auto creation of leaf queue. </description> </property> <property> <name>yarn.scheduler.capacity.root.admin.capacity</name> <value>10</value> <description> The queues at the this level is for admin (10% usage of the cluster). </description> </property> <property> <name>yarn.scheduler.capacity.root.users.capacity</name> <value>90</value> <description> The queues at the this level is for users (90% usage of the cluster) </description> </property> <property> <name>yarn.scheduler.capacity.queue-mappings</name> <value>u:fratnasamy:admin</value> <description>Specify user who can submit to admin queue</description> </property> <property> <name>yarn.scheduler.capacity.queue-mappings</name> <value>u:%user:users.%user</value> <description>Maps users to queues with the same name as user</description> </property> <property> <name>yarn.scheduler.capacity.queue-mappings-override.enable</name> <value>false</value> <description> If a queue mapping is present, will it override the value specified by the user? This can be used by administrators to place jobs in queues that are different than the one specified by the user. The default is false. </description> </property> <property> <name>yarn.scheduler.capacity.application.fail-fast</name> <value>false</value> <description> Whether RM should fail during recovery if previous applications' queue is no longer valid. </description> </property></configuration> However, when i try to run the pi.py example (or any other examples), I run into this error message : Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1632944726077_0002 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.users.fratnasamy already has 0 applications, Would anyone know how to troubleshoot this issue? *Fritz Ratnasamy* Data Scientist Information Technology The University of Chicago Booth School of Business 5807 S. Woodlawn Chicago, Illinois 60637 Phone: +(1) 773-834-4556
