[
https://issues.apache.org/jira/browse/MYRIAD-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934543#comment-14934543
]
DarinJ commented on MYRIAD-139:
-------------------------------
Swapnil I indicated in the comments that I did in fact set
yarn.scheduler.minimum-allocation-mb (and vcores ). Then when flexing up when
using the capacity scheduler the second exception occurs. The relevant bit
repeated here:
RuntimeException: Invalid resource scheduler memory allocation configuration,
yarn.scheduler.minimum-allocation-mb=0,
yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
0, max should be no smaller than min.
It occurs for capacity but not fair scheduler. This was vanilla hadoop 2.6 and
2.7 (I'm verifying both versions).
> CapacityScheduler doesn't work with Fine Grained Scaling
> --------------------------------------------------------
>
> Key: MYRIAD-139
> URL: https://issues.apache.org/jira/browse/MYRIAD-139
> Project: Myriad
> Issue Type: Bug
> Components: Executor, Scheduler
> Reporter: DarinJ
> Assignee: Swapnil Daingade
>
> When trying to flex up a zero profile NM you get the following error:
> {quote}
> 15/09/21 20:31:19 FATAL nodemanager.NodeManager: Error starting NodeManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN
> signal from
> Resourcemanager ,Registration of NodeManager failed, Message from
> ResourceManager: NodeManager from doesn't satis
> fy minimum allocations, Sending SHUTDOWN signal to the NodeManager.
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:197)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:264)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)
> Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved
> SHUTDOWN signal from Resourcemanager ,Registration of NodeManager
> failed, Message from ResourceManager: NodeManager from doesn't satisfy
> minimum allocations, Sending SHUTDOWN sign
> al to the NodeManager.
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:265)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:191)
> ... 6 more
> 15/09/21 20:31:19 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NodeManager at
> ************************************************************/
> {quote}
> This can be fixed by setting
> yarn.scheduler.minimum-allocation-vcores and
> yarn.scheduler.minimum-allocation-mb to 0 in yarn-site.xml.
> At this point com.ebay.myriad.scheduler.yarn.MyriadFairScheduler works as
> expected. However, when start the resource manager with
> com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler we get:
> {quote}
> RuntimeException: Invalid resource scheduler memory allocation configuration,
> yarn.scheduler.minimum-allocation-mb=0,
> yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
> 0, max should be no smaller than min.
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Invalid resource
> scheduler memory allocation configuration,
> yarn.scheduler.minimum-allocation-mb=0,
> yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
> 0, max should be no smaller than min.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.validateConf(CapacityScheduler.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:285)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:324)
> at
> com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler.serviceInit(MyriadCapacityScheduler.java:52)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:558)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:989)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:255)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
> 15/09/21 21:42:26 INFO resourcemanager.ResourceManager: Transitioning to
> standby state
> 15/09/21 21:42:26 INFO resourcemanager.ResourceManager: Transitioned to
> standby state
> 15/09/21 21:42:26 FATAL resourcemanager.ResourceManager: Error starting
> ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Invalid resource
> scheduler memory allocation configuration,
> yarn.scheduler.minimum-allocation-mb=0,
> yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
> 0, max should be no smaller than min.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.validateConf(CapacityScheduler.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:285)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:324)
> at
> com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler.serviceInit(MyriadCapacityScheduler.java:52)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:558)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:989)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:255)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
> 15/09/21 21:42:26 INFO resourcemanager.ResourceManager: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down ResourceManager at
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
