[
https://issues.apache.org/jira/browse/MYRIAD-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DarinJ updated MYRIAD-139:
--------------------------
Description:
When trying to flex up a zero profile NM you get the following error:
{quote}
15/09/21 20:31:19 FATAL nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN
signal from
Resourcemanager ,Registration of NodeManager failed, Message from
ResourceManager: NodeManager from doesn't satis
fy minimum allocations, Sending SHUTDOWN signal to the NodeManager.
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:197)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:264)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved
SHUTDOWN signal from Resourcemanager ,Registration of NodeManager
failed, Message from ResourceManager: NodeManager from doesn't satisfy
minimum allocations, Sending SHUTDOWN sign
al to the NodeManager.
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:265)
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:191)
... 6 more
15/09/21 20:31:19 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at
************************************************************/
{quote}
This can be fixed by setting
yarn.scheduler.minimum-allocation-vcores and
yarn.scheduler.minimum-allocation-mb to 0 in yarn-site.xml.
At this point com.ebay.myriad.scheduler.yarn.MyriadFairScheduler works as
expected. However, when start the resource manager with
com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler we get:
{quote}
RuntimeException: Invalid resource scheduler memory allocation configuration,
yarn.scheduler.minimum-allocation-mb=0,
yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
0, max should be no smaller than min.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Invalid resource
scheduler memory allocation configuration,
yarn.scheduler.minimum-allocation-mb=0,
yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
0, max should be no smaller than min.
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.validateConf(CapacityScheduler.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:285)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:324)
at
com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler.serviceInit(MyriadCapacityScheduler.java:52)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:558)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:989)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:255)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
15/09/21 21:42:26 INFO resourcemanager.ResourceManager: Transitioning to
standby state
15/09/21 21:42:26 INFO resourcemanager.ResourceManager: Transitioned to standby
state
15/09/21 21:42:26 FATAL resourcemanager.ResourceManager: Error starting
ResourceManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Invalid resource
scheduler memory allocation configuration,
yarn.scheduler.minimum-allocation-mb=0,
yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
0, max should be no smaller than min.
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.validateConf(CapacityScheduler.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:285)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:324)
at
com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler.serviceInit(MyriadCapacityScheduler.java:52)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:558)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:989)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:255)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
15/09/21 21:42:26 INFO resourcemanager.ResourceManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at
{quote}
was:
When trying to flex up a zero profile NM you get the following error:
15/09/21 20:31:19 FATAL nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN
signal from
Resourcemanager ,Registration of NodeManager failed, Message from
ResourceManager: NodeManager from doesn't satis
fy minimum allocations, Sending SHUTDOWN signal to the NodeManager.
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:197)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:264)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved
SHUTDOWN signal from Resourcemanager ,Registration of NodeManager
failed, Message from ResourceManager: NodeManager from doesn't satisfy
minimum allocations, Sending SHUTDOWN sign
al to the NodeManager.
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:265)
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:191)
... 6 more
15/09/21 20:31:19 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at
************************************************************/
This can be fixed by setting
yarn.scheduler.minimum-allocation-vcores and
yarn.scheduler.minimum-allocation-mb to 0 in yarn-site.xml.
At this point com.ebay.myriad.scheduler.yarn.MyriadFairScheduler works as
expected. However, when start the resource manager with
com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler we get:
untimeException: Invalid resource scheduler memory allocation configuration,
yarn.scheduler.minimum-allocation-mb=0,
yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
0, max should be no smaller than min.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Invalid resource
scheduler memory allocation configuration,
yarn.scheduler.minimum-allocation-mb=0,
yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
0, max should be no smaller than min.
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.validateConf(CapacityScheduler.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:285)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:324)
at
com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler.serviceInit(MyriadCapacityScheduler.java:52)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:558)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:989)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:255)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
15/09/21 21:42:26 INFO resourcemanager.ResourceManager: Transitioning to
standby state
15/09/21 21:42:26 INFO resourcemanager.ResourceManager: Transitioned to standby
state
15/09/21 21:42:26 FATAL resourcemanager.ResourceManager: Error starting
ResourceManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Invalid resource
scheduler memory allocation configuration,
yarn.scheduler.minimum-allocation-mb=0,
yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
0, max should be no smaller than min.
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.validateConf(CapacityScheduler.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:285)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:324)
at
com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler.serviceInit(MyriadCapacityScheduler.java:52)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:558)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:989)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:255)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
15/09/21 21:42:26 INFO resourcemanager.ResourceManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at
> CapacityScheduler doesn't work with Fine Grained Scaling
> --------------------------------------------------------
>
> Key: MYRIAD-139
> URL: https://issues.apache.org/jira/browse/MYRIAD-139
> Project: Myriad
> Issue Type: Bug
> Components: Executor, Scheduler
> Reporter: DarinJ
>
> When trying to flex up a zero profile NM you get the following error:
> {quote}
> 15/09/21 20:31:19 FATAL nodemanager.NodeManager: Error starting NodeManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN
> signal from
> Resourcemanager ,Registration of NodeManager failed, Message from
> ResourceManager: NodeManager from doesn't satis
> fy minimum allocations, Sending SHUTDOWN signal to the NodeManager.
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:197)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:264)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)
> Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved
> SHUTDOWN signal from Resourcemanager ,Registration of NodeManager
> failed, Message from ResourceManager: NodeManager from doesn't satisfy
> minimum allocations, Sending SHUTDOWN sign
> al to the NodeManager.
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:265)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:191)
> ... 6 more
> 15/09/21 20:31:19 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NodeManager at
> ************************************************************/
> {quote}
> This can be fixed by setting
> yarn.scheduler.minimum-allocation-vcores and
> yarn.scheduler.minimum-allocation-mb to 0 in yarn-site.xml.
> At this point com.ebay.myriad.scheduler.yarn.MyriadFairScheduler works as
> expected. However, when start the resource manager with
> com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler we get:
> {quote}
> RuntimeException: Invalid resource scheduler memory allocation configuration,
> yarn.scheduler.minimum-allocation-mb=0,
> yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
> 0, max should be no smaller than min.
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Invalid resource
> scheduler memory allocation configuration,
> yarn.scheduler.minimum-allocation-mb=0,
> yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
> 0, max should be no smaller than min.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.validateConf(CapacityScheduler.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:285)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:324)
> at
> com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler.serviceInit(MyriadCapacityScheduler.java:52)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:558)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:989)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:255)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
> 15/09/21 21:42:26 INFO resourcemanager.ResourceManager: Transitioning to
> standby state
> 15/09/21 21:42:26 INFO resourcemanager.ResourceManager: Transitioned to
> standby state
> 15/09/21 21:42:26 FATAL resourcemanager.ResourceManager: Error starting
> ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Invalid resource
> scheduler memory allocation configuration,
> yarn.scheduler.minimum-allocation-mb=0,
> yarn.scheduler.maximum-allocation-mb=8192, min and max should be greater than
> 0, max should be no smaller than min.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.validateConf(CapacityScheduler.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:285)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:324)
> at
> com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler.serviceInit(MyriadCapacityScheduler.java:52)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:558)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:989)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:255)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
> 15/09/21 21:42:26 INFO resourcemanager.ResourceManager: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down ResourceManager at
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
