[
https://issues.apache.org/jira/browse/SLIDER-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gour Saha updated SLIDER-1201:
------------------------------
Description:
Slider currently by default normalizes all resource requests which exceeds max
allowed in the cluster.
{code}
2016-11-02 07:24:47,722 [main] WARN state.AppState - Resource requirements of
COMMAND_LOGGER normalized from <memory:186368, vCores:1> to <memory:143616,
vCores:1>
{code}
This behavior should be made configurable. It will allow app owners to choose
if they want normalization and the app to continue running with reduced
resources (as default), OR if they do not want normalization and want the app
to fail right away, rather than several days later when it actually tries to
use as much memory it had requested, but never got. This causes a debugging
nightmare as well, trying to figure out why an app was even killed by YARN
(under false presumption that the app did not exceed its requested resource).
h5. Solution:
A new property _*yarn.resource.normalization.enabled*_ is being introduced
which needs to be set in the app's resources definition. By default it is
"true" (which was the behavior all along, to keep backward compatibility for
all existing apps). To disable normalization, set it to "false" for the desired
component of the app. An example is below -
{code}
{
"schema": "http://example.org/specification/v2.0.0";,
"metadata": {
},
"global": {
},
"components": {
"COMMAND_LOGGER": {
"yarn.role.priority": "1",
"yarn.component.instances": "1",
"yarn.resource.normalization.enabled": "false",
"yarn.memory": "186368",
"yarn.vcores": "1"
},
"slider-appmaster": {
}
}
}
{code}
was:
Slider currently by default normalizes all resource requests which exceeds max
allowed in the cluster.
{code}
2016-11-02 07:24:47,722 [main] WARN state.AppState - Resource requirements of
COMMAND_LOGGER normalized from <memory:186368, vCores:1> to <memory:143616,
vCores:1>
{code}
This behavior should be made configurable. It will allow app owners to choose
if they want normalization and the app to continue running with reduced
resources (as default), OR if they do not want normalization and want the app
to fail right away, rather than several days later when it actually tries to
use as much memory it had requested, but never got. This causes a debugging
nightmare as well, trying to figure out why the app was even killed by YARN
when it exceeded the resource usage.
A new property _*yarn.resource.normalization.enabled*_ is being introduced
which needs to be set in the app's resources definition. By default it is
"true" (which was the behavior all along, to keep backward compatibility for
all existing apps). To disable normalization, set it to "false" for the desired
component of the app. An example is below -
{code}
{
"schema": "http://example.org/specification/v2.0.0";,
"metadata": {
},
"global": {
},
"components": {
"COMMAND_LOGGER": {
"yarn.role.priority": "1",
"yarn.component.instances": "1",
"yarn.resource.normalization.enabled": "false",
"yarn.memory": "186368",
"yarn.vcores": "1"
},
"slider-appmaster": {
}
}
}
{code}
> Slider should make resource normalization configurable for app-owners
> ---------------------------------------------------------------------
>
> Key: SLIDER-1201
> URL: https://issues.apache.org/jira/browse/SLIDER-1201
> Project: Slider
> Issue Type: Bug
> Components: appmaster
> Affects Versions: Slider 0.91
> Reporter: Gour Saha
> Assignee: Gour Saha
> Fix For: Slider 1.0.0
>
> Attachments: SLIDER-1201.01.patch
>
>
> Slider currently by default normalizes all resource requests which exceeds
> max allowed in the cluster.
> {code}
> 2016-11-02 07:24:47,722 [main] WARN state.AppState - Resource requirements
> of COMMAND_LOGGER normalized from <memory:186368, vCores:1> to
> <memory:143616, vCores:1>
> {code}
> This behavior should be made configurable. It will allow app owners to choose
> if they want normalization and the app to continue running with reduced
> resources (as default), OR if they do not want normalization and want the app
> to fail right away, rather than several days later when it actually tries to
> use as much memory it had requested, but never got. This causes a debugging
> nightmare as well, trying to figure out why an app was even killed by YARN
> (under false presumption that the app did not exceed its requested resource).
> h5. Solution:
> A new property _*yarn.resource.normalization.enabled*_ is being introduced
> which needs to be set in the app's resources definition. By default it is
> "true" (which was the behavior all along, to keep backward compatibility for
> all existing apps). To disable normalization, set it to "false" for the
> desired component of the app. An example is below -
> {code}
> {
> "schema": "http://example.org/specification/v2.0.0";,
> "metadata": {
> },
> "global": {
> },
> "components": {
> "COMMAND_LOGGER": {
> "yarn.role.priority": "1",
> "yarn.component.instances": "1",
> "yarn.resource.normalization.enabled": "false",
> "yarn.memory": "186368",
> "yarn.vcores": "1"
> },
> "slider-appmaster": {
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
