[jira] [Comment Edited] (YARN-6156) AM blacklisting to consider node label partition
[
https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867376#comment-15867376
]
Varun Saxena edited comment on YARN-6156 at 2/15/17 7:29 AM:
-
For unmanaged AM as AM blacklisting is not done because AM is not launched by
RM, it is equivalent to AM blacklisting being disabled for this app. Hence we
do not need to create a SimpleBlacklistManager object in
RMAppImpl#createNewAttempt. We can create DisabledBlacklistManager object
instead.
This change has also been made as part of this JIRA.
was (Author: varun_saxena):
For unmanaged AM as AM blacklisting is not done, it is equivalent to
blacklisting being disabled for this app. Hence we do not need to create a
SimpleBlacklistManager object in RMAppImpl#createNewAttempt. We can create
DisabledBlacklistManager object instead.
This change has also been made as part of this JIRA.
> AM blacklisting to consider node label partition
>
>
> Key: YARN-6156
> URL: https://issues.apache.org/jira/browse/YARN-6156
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch,
> YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch,
> YARN-6156.0006.patch
>
>
> As per the current implementation for AM blacklisting, disabling is based on
> complete cluster resource.
> Incase of partitioned cluster though, nodes applicable for AM labels only
> should be considered as total nodes.
> Additionally for unmanaged AM as AM blacklisting is not done as AM is not
> launched by RM, it is equivalent to AM blacklisting being disabled for this
> app. Hence we do not need to create a SimpleBlacklistManager object in
> RMAppImpl#createNewAttempt.
> We can create DisabledBlacklistManager object instead.
> {code}
> BlacklistManager currentAMBlacklist;
> if (currentAttempt != null) {
> currentAMBlacklist = currentAttempt.getAMBlacklist();
> } else {
> if (amBlacklistingEnabled) {
> currentAMBlacklist = new SimpleBlacklistManager(
> scheduler.getNumClusterNodes(), blacklistDisableThreshold);
> } else {
> currentAMBlacklist = new DisabledBlacklistManager();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6156) AM blacklisting to consider node label partition
[
https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-6156:
---
Description:
As per the current implementation for AM blacklisting, disabling is based on
complete cluster resource.
Incase of partitioned cluster though, nodes applicable for AM labels only
should be considered as total nodes.
Additionally for unmanaged AM as AM blacklisting is not done as AM is not
launched by RM, it is equivalent to blacklisting being disabled for this app.
Hence we do not need to create a SimpleBlacklistManager object in
RMAppImpl#createNewAttempt.
We can create DisabledBlacklistManager object instead.
{code}
BlacklistManager currentAMBlacklist;
if (currentAttempt != null) {
currentAMBlacklist = currentAttempt.getAMBlacklist();
} else {
if (amBlacklistingEnabled) {
currentAMBlacklist = new SimpleBlacklistManager(
scheduler.getNumClusterNodes(), blacklistDisableThreshold);
} else {
currentAMBlacklist = new DisabledBlacklistManager();
}
}
{code}
was:
As per the current implementation for AM blacklisting, disabling is based on
complete cluster resource.
Incase of partitioned cluster though, nodes applicable for AM labels only
should be considered as total nodes.
Additionally for unmanaged AM as AM blacklisting is not done, it is equivalent
to blacklisting being disabled for this app. Hence we do not need to create a
SimpleBlacklistManager object in RMAppImpl#createNewAttempt.
We can create DisabledBlacklistManager object instead.
{code}
BlacklistManager currentAMBlacklist;
if (currentAttempt != null) {
currentAMBlacklist = currentAttempt.getAMBlacklist();
} else {
if (amBlacklistingEnabled) {
currentAMBlacklist = new SimpleBlacklistManager(
scheduler.getNumClusterNodes(), blacklistDisableThreshold);
} else {
currentAMBlacklist = new DisabledBlacklistManager();
}
}
{code}
> AM blacklisting to consider node label partition
>
>
> Key: YARN-6156
> URL: https://issues.apache.org/jira/browse/YARN-6156
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch,
> YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch,
> YARN-6156.0006.patch
>
>
> As per the current implementation for AM blacklisting, disabling is based on
> complete cluster resource.
> Incase of partitioned cluster though, nodes applicable for AM labels only
> should be considered as total nodes.
> Additionally for unmanaged AM as AM blacklisting is not done as AM is not
> launched by RM, it is equivalent to blacklisting being disabled for this app.
> Hence we do not need to create a SimpleBlacklistManager object in
> RMAppImpl#createNewAttempt.
> We can create DisabledBlacklistManager object instead.
> {code}
> BlacklistManager currentAMBlacklist;
> if (currentAttempt != null) {
> currentAMBlacklist = currentAttempt.getAMBlacklist();
> } else {
> if (amBlacklistingEnabled) {
> currentAMBlacklist = new SimpleBlacklistManager(
> scheduler.getNumClusterNodes(), blacklistDisableThreshold);
> } else {
> currentAMBlacklist = new DisabledBlacklistManager();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6156) AM blacklisting to consider node label partition
[
https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-6156:
---
Description:
As per the current implementation for AM blacklisting, disabling is based on
complete cluster resource.
Incase of partitioned cluster though, nodes applicable for AM labels only
should be considered as total nodes.
Additionally for unmanaged AM as AM blacklisting is not done, it is equivalent
to blacklisting being disabled for this app. Hence we do not need to create a
SimpleBlacklistManager object in RMAppImpl#createNewAttempt.
We can create DisabledBlacklistManager object instead.
{code}
BlacklistManager currentAMBlacklist;
if (currentAttempt != null) {
currentAMBlacklist = currentAttempt.getAMBlacklist();
} else {
if (amBlacklistingEnabled) {
currentAMBlacklist = new SimpleBlacklistManager(
scheduler.getNumClusterNodes(), blacklistDisableThreshold);
} else {
currentAMBlacklist = new DisabledBlacklistManager();
}
}
{code}
was:
As per the current implementation for AM blacklisting, disabling is based on
complete cluster resource.
Incase of partitioned cluster though, nodes applicable for AM labels only
should be considered as total nodes.
Additionally for unmanaged AM as AM blacklisting is not done, it is equivalent
to blacklisting being disabled for this app. Hence we do not need to create a
SimpleBlacklistManager object in RMAppImpl#createNewAttempt.
We can create DisableBlacklistManager object instead.
{code}
BlacklistManager currentAMBlacklist;
if (currentAttempt != null) {
currentAMBlacklist = currentAttempt.getAMBlacklist();
} else {
if (amBlacklistingEnabled) {
currentAMBlacklist = new SimpleBlacklistManager(
scheduler.getNumClusterNodes(), blacklistDisableThreshold);
} else {
currentAMBlacklist = new DisabledBlacklistManager();
}
}
{code}
> AM blacklisting to consider node label partition
>
>
> Key: YARN-6156
> URL: https://issues.apache.org/jira/browse/YARN-6156
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch,
> YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch,
> YARN-6156.0006.patch
>
>
> As per the current implementation for AM blacklisting, disabling is based on
> complete cluster resource.
> Incase of partitioned cluster though, nodes applicable for AM labels only
> should be considered as total nodes.
> Additionally for unmanaged AM as AM blacklisting is not done, it is
> equivalent to blacklisting being disabled for this app. Hence we do not need
> to create a SimpleBlacklistManager object in RMAppImpl#createNewAttempt.
> We can create DisabledBlacklistManager object instead.
> {code}
> BlacklistManager currentAMBlacklist;
> if (currentAttempt != null) {
> currentAMBlacklist = currentAttempt.getAMBlacklist();
> } else {
> if (amBlacklistingEnabled) {
> currentAMBlacklist = new SimpleBlacklistManager(
> scheduler.getNumClusterNodes(), blacklistDisableThreshold);
> } else {
> currentAMBlacklist = new DisabledBlacklistManager();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6156) AM blacklisting to consider node label partition
[
https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-6156:
---
Description:
As per the current implementation for AM blacklisting, disabling is based on
complete cluster resource.
Incase of partitioned cluster though, nodes applicable for AM labels only
should be considered as total nodes.
Additionally for unmanaged AM as AM blacklisting is not done as AM is not
launched by RM, it is equivalent to AM blacklisting being disabled for this
app. Hence we do not need to create a SimpleBlacklistManager object in
RMAppImpl#createNewAttempt.
We can create DisabledBlacklistManager object instead.
{code}
BlacklistManager currentAMBlacklist;
if (currentAttempt != null) {
currentAMBlacklist = currentAttempt.getAMBlacklist();
} else {
if (amBlacklistingEnabled) {
currentAMBlacklist = new SimpleBlacklistManager(
scheduler.getNumClusterNodes(), blacklistDisableThreshold);
} else {
currentAMBlacklist = new DisabledBlacklistManager();
}
}
{code}
was:
As per the current implementation for AM blacklisting, disabling is based on
complete cluster resource.
Incase of partitioned cluster though, nodes applicable for AM labels only
should be considered as total nodes.
Additionally for unmanaged AM as AM blacklisting is not done as AM is not
launched by RM, it is equivalent to blacklisting being disabled for this app.
Hence we do not need to create a SimpleBlacklistManager object in
RMAppImpl#createNewAttempt.
We can create DisabledBlacklistManager object instead.
{code}
BlacklistManager currentAMBlacklist;
if (currentAttempt != null) {
currentAMBlacklist = currentAttempt.getAMBlacklist();
} else {
if (amBlacklistingEnabled) {
currentAMBlacklist = new SimpleBlacklistManager(
scheduler.getNumClusterNodes(), blacklistDisableThreshold);
} else {
currentAMBlacklist = new DisabledBlacklistManager();
}
}
{code}
> AM blacklisting to consider node label partition
>
>
> Key: YARN-6156
> URL: https://issues.apache.org/jira/browse/YARN-6156
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch,
> YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch,
> YARN-6156.0006.patch
>
>
> As per the current implementation for AM blacklisting, disabling is based on
> complete cluster resource.
> Incase of partitioned cluster though, nodes applicable for AM labels only
> should be considered as total nodes.
> Additionally for unmanaged AM as AM blacklisting is not done as AM is not
> launched by RM, it is equivalent to AM blacklisting being disabled for this
> app. Hence we do not need to create a SimpleBlacklistManager object in
> RMAppImpl#createNewAttempt.
> We can create DisabledBlacklistManager object instead.
> {code}
> BlacklistManager currentAMBlacklist;
> if (currentAttempt != null) {
> currentAMBlacklist = currentAttempt.getAMBlacklist();
> } else {
> if (amBlacklistingEnabled) {
> currentAMBlacklist = new SimpleBlacklistManager(
> scheduler.getNumClusterNodes(), blacklistDisableThreshold);
> } else {
> currentAMBlacklist = new DisabledBlacklistManager();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6156) AM blacklisting to consider node label partition
[
https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867376#comment-15867376
]
Varun Saxena commented on YARN-6156:
For unmanaged AM as AM blacklisting is not done, it is equivalent to
blacklisting being disabled for this app. Hence we do not need to create a
SimpleBlacklistManager object in RMAppImpl#createNewAttempt. We can create
DisabledBlacklistManager object instead.
This change has also been made as part of this JIRA.
> AM blacklisting to consider node label partition
>
>
> Key: YARN-6156
> URL: https://issues.apache.org/jira/browse/YARN-6156
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch,
> YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch,
> YARN-6156.0006.patch
>
>
> As per the current implementation for AM blacklisting, disabling is based on
> complete cluster resource.
> Incase of partitioned cluster though, nodes applicable for AM labels only
> should be considered as total nodes.
> Additionally for unmanaged AM as AM blacklisting is not done, it is
> equivalent to blacklisting being disabled for this app. Hence we do not need
> to create a SimpleBlacklistManager object in RMAppImpl#createNewAttempt.
> We can create DisableBlacklistManager object instead.
> {code}
> BlacklistManager currentAMBlacklist;
> if (currentAttempt != null) {
> currentAMBlacklist = currentAttempt.getAMBlacklist();
> } else {
> if (amBlacklistingEnabled) {
> currentAMBlacklist = new SimpleBlacklistManager(
> scheduler.getNumClusterNodes(), blacklistDisableThreshold);
> } else {
> currentAMBlacklist = new DisabledBlacklistManager();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867373#comment-15867373
]
Hadoop QA commented on YARN-6163:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 11m
17s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common no findbugs output
file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/findbugsXml.xml)
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
47s{color} | {color:red}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/findbugsXml.xml)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 7s{color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 34 new + 35 unchanged -
35 fixed = 69 total (was 70) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 12 new + 315 unchanged - 3 fixed = 327 total (was 318) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
44s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 4579 new + 9158 unchanged - 0 fixed = 13737 total (was 9158) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
37s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 908 new + 1816 unchanged - 0 fixed = 2724 total (was 1816) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
46s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 4579 new + 9158 unchanged - 0 fixed = 13737 total (was 9158) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
38s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 908 new + 1816 unchanged - 0 fixed = 2724 total (was 1816) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
56s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the
[jira] [Updated] (YARN-6156) AM blacklisting to consider node label partition
[
https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-6156:
---
Summary: AM blacklisting to consider node label partition (was: AM
blacklisting consider node label partition)
> AM blacklisting to consider node label partition
>
>
> Key: YARN-6156
> URL: https://issues.apache.org/jira/browse/YARN-6156
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch,
> YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch,
> YARN-6156.0006.patch
>
>
> As per the current implementation for AM blacklisting, disabling is based on
> complete cluster resource.
> Incase of partitioned cluster though, nodes applicable for AM labels only
> should be considered as total nodes.
> Additionally for unmanaged AM as AM blacklisting is not done, it is
> equivalent to blacklisting being disabled for this app. Hence we do not need
> to create a SimpleBlacklistManager object.
> We can create DisableBlacklistManager object instead.
> {code}
> BlacklistManager currentAMBlacklist;
> if (currentAttempt != null) {
> currentAMBlacklist = currentAttempt.getAMBlacklist();
> } else {
> if (amBlacklistingEnabled) {
> currentAMBlacklist = new SimpleBlacklistManager(
> scheduler.getNumClusterNodes(), blacklistDisableThreshold);
> } else {
> currentAMBlacklist = new DisabledBlacklistManager();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6156) AM blacklisting to consider node label partition
[
https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-6156:
---
Description:
As per the current implementation for AM blacklisting, disabling is based on
complete cluster resource.
Incase of partitioned cluster though, nodes applicable for AM labels only
should be considered as total nodes.
Additionally for unmanaged AM as AM blacklisting is not done, it is equivalent
to blacklisting being disabled for this app. Hence we do not need to create a
SimpleBlacklistManager object in RMAppImpl#createNewAttempt.
We can create DisableBlacklistManager object instead.
{code}
BlacklistManager currentAMBlacklist;
if (currentAttempt != null) {
currentAMBlacklist = currentAttempt.getAMBlacklist();
} else {
if (amBlacklistingEnabled) {
currentAMBlacklist = new SimpleBlacklistManager(
scheduler.getNumClusterNodes(), blacklistDisableThreshold);
} else {
currentAMBlacklist = new DisabledBlacklistManager();
}
}
{code}
was:
As per the current implementation for AM blacklisting, disabling is based on
complete cluster resource.
Incase of partitioned cluster though, nodes applicable for AM labels only
should be considered as total nodes.
Additionally for unmanaged AM as AM blacklisting is not done, it is equivalent
to blacklisting being disabled for this app. Hence we do not need to create a
SimpleBlacklistManager object.
We can create DisableBlacklistManager object instead.
{code}
BlacklistManager currentAMBlacklist;
if (currentAttempt != null) {
currentAMBlacklist = currentAttempt.getAMBlacklist();
} else {
if (amBlacklistingEnabled) {
currentAMBlacklist = new SimpleBlacklistManager(
scheduler.getNumClusterNodes(), blacklistDisableThreshold);
} else {
currentAMBlacklist = new DisabledBlacklistManager();
}
}
{code}
> AM blacklisting to consider node label partition
>
>
> Key: YARN-6156
> URL: https://issues.apache.org/jira/browse/YARN-6156
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch,
> YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch,
> YARN-6156.0006.patch
>
>
> As per the current implementation for AM blacklisting, disabling is based on
> complete cluster resource.
> Incase of partitioned cluster though, nodes applicable for AM labels only
> should be considered as total nodes.
> Additionally for unmanaged AM as AM blacklisting is not done, it is
> equivalent to blacklisting being disabled for this app. Hence we do not need
> to create a SimpleBlacklistManager object in RMAppImpl#createNewAttempt.
> We can create DisableBlacklistManager object instead.
> {code}
> BlacklistManager currentAMBlacklist;
> if (currentAttempt != null) {
> currentAMBlacklist = currentAttempt.getAMBlacklist();
> } else {
> if (amBlacklistingEnabled) {
> currentAMBlacklist = new SimpleBlacklistManager(
> scheduler.getNumClusterNodes(), blacklistDisableThreshold);
> } else {
> currentAMBlacklist = new DisabledBlacklistManager();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6156) AM blacklisting consider node label partition
[
https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-6156:
---
Description:
As per the current implementation for AM blacklisting, disabling is based on
complete cluster resource.
Incase of partitioned cluster though, nodes applicable for AM labels only
should be considered as total nodes.
Additionally for unmanaged AM as AM blacklisting is not done, it is equivalent
to blacklisting being disabled for this app. Hence we do not need to create a
SimpleBlacklistManager object.
We can create DisableBlacklistManager object instead.
{code}
BlacklistManager currentAMBlacklist;
if (currentAttempt != null) {
currentAMBlacklist = currentAttempt.getAMBlacklist();
} else {
if (amBlacklistingEnabled) {
currentAMBlacklist = new SimpleBlacklistManager(
scheduler.getNumClusterNodes(), blacklistDisableThreshold);
} else {
currentAMBlacklist = new DisabledBlacklistManager();
}
}
{code}
was:
As per the current implementation for AM blacklisting disabling is based on
completed custer resource.
Incase of partitioned cluster, nodes applicable for AM labels only should be
considered as total nodes.
> AM blacklisting consider node label partition
> --
>
> Key: YARN-6156
> URL: https://issues.apache.org/jira/browse/YARN-6156
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch,
> YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch,
> YARN-6156.0006.patch
>
>
> As per the current implementation for AM blacklisting, disabling is based on
> complete cluster resource.
> Incase of partitioned cluster though, nodes applicable for AM labels only
> should be considered as total nodes.
> Additionally for unmanaged AM as AM blacklisting is not done, it is
> equivalent to blacklisting being disabled for this app. Hence we do not need
> to create a SimpleBlacklistManager object.
> We can create DisableBlacklistManager object instead.
> {code}
> BlacklistManager currentAMBlacklist;
> if (currentAttempt != null) {
> currentAMBlacklist = currentAttempt.getAMBlacklist();
> } else {
> if (amBlacklistingEnabled) {
> currentAMBlacklist = new SimpleBlacklistManager(
> scheduler.getNumClusterNodes(), blacklistDisableThreshold);
> } else {
> currentAMBlacklist = new DisabledBlacklistManager();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5068) Expose scheduler queue to application master
[
https://issues.apache.org/jira/browse/YARN-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867356#comment-15867356
]
Vinod Kumar Vavilapalli commented on YARN-5068:
---
It's a bit of an old JIRA but I just realized that YARN-1623 already added this
information as part of {{RegisterApplicationMasterResponse}}. Shall we revert
this change? /cc [~rohithsharma] [~vvasudev]
Also, this JIRA misses the case when an application is moved between queues. To
address that, mirroring YARN-1623, we can also send the updated queue-name as
part of {{AllocateResponse}} . The AM can see if the queue name changed between
any subsequent calls and correspondingly record the same. /cc [~djp]
> Expose scheduler queue to application master
>
>
> Key: YARN-5068
> URL: https://issues.apache.org/jira/browse/YARN-5068
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6692.patch, YARN-5068.1.patch,
> YARN-5068.2.patch, YARN-5068-branch-2.1.patch
>
>
> The AM needs to know the queue name in which it was launched.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6156) AM blacklisting consider node label partition
[ https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867355#comment-15867355 ] Varun Saxena commented on YARN-6156: Thanks [~sunilg]. Will drop a comment and update the description. > AM blacklisting consider node label partition > -- > > Key: YARN-6156 > URL: https://issues.apache.org/jira/browse/YARN-6156 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch, > YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch, > YARN-6156.0006.patch > > > As per the current implementation for AM blacklisting disabling is based on > completed custer resource. > Incase of partitioned cluster, nodes applicable for AM labels only should be > considered as total nodes. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6183) [YARN-3368] Few missing informations in Application and Application Attempt page for Basic Info table
[
https://issues.apache.org/jira/browse/YARN-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867351#comment-15867351
]
Hadoop QA commented on YARN-6183:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6183 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852750/YARN-6183.004.patch |
| Optional Tests | asflicense |
| uname | Linux 3492605f0caa 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fbc0c2b |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14956/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [YARN-3368] Few missing informations in Application and Application Attempt
> page for Basic Info table
> -
>
> Key: YARN-6183
> URL: https://issues.apache.org/jira/browse/YARN-6183
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Akhil PB
>Assignee: Akhil PB
> Attachments: YARN-6183.001.patch, YARN-6183.002.patch,
> YARN-6183.003.patch, YARN-6183.004.patch
>
>
> Fix missing information in application and app-attempt pages in new YARN-UI.
> In Basic Info table, priority and IsUnmanagedAM fields are missing for some
> applications.
> In App Master panel, change label Expr to Expression.
> In app-attempt page, logUrl and NM web UI needs to be opened in new tab.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6183) [YARN-3368] Few missing informations in Application and Application Attempt page for Basic Info table
[ https://issues.apache.org/jira/browse/YARN-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-6183: --- Attachment: YARN-6183.004.patch > [YARN-3368] Few missing informations in Application and Application Attempt > page for Basic Info table > - > > Key: YARN-6183 > URL: https://issues.apache.org/jira/browse/YARN-6183 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Akhil PB >Assignee: Akhil PB > Attachments: YARN-6183.001.patch, YARN-6183.002.patch, > YARN-6183.003.patch, YARN-6183.004.patch > > > Fix missing information in application and app-attempt pages in new YARN-UI. > In Basic Info table, priority and IsUnmanagedAM fields are missing for some > applications. > In App Master panel, change label Expr to Expression. > In app-attempt page, logUrl and NM web UI needs to be opened in new tab. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6175) Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867330#comment-15867330
]
Karthik Kambatla commented on YARN-6175:
Just skimmed through the patch. When using {{subtractFromNonNegative}}, we
should make sure we are not masking real bugs. Are all the cases here okay to
use this method?
Also, since YARN-6163 is close to be done, let us look into this once that is
committed.
> Negative vcore for resource needed to preempt
> -
>
> Key: YARN-6175
> URL: https://issues.apache.org/jira/browse/YARN-6175
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6175.001.patch
>
>
> Both old preemption code (2.8 and before) and new preemption code could have
> negative vcores while calculating resources needed to preempt.
> For old preemption, you can find following messages in RM logs:
> {code}
> Should preempt
> {code}
> The related code is in method {{resourceDeficit()}}.
> For new preemption code, there are no messages in RM logs, the related code
> is in method {{fairShareStarvation()}}.
> The negative value isn't only a display issue, but also may cause missing
> necessary preemption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6171) ConcurrentModificationException in ApplicationMasterService.allocate
[
https://issues.apache.org/jira/browse/YARN-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867323#comment-15867323
]
Karthik Kambatla commented on YARN-6171:
The latest patch looks pretty good, but undoes YARN-4690.
> ConcurrentModificationException in ApplicationMasterService.allocate
>
>
> Key: YARN-6171
> URL: https://issues.apache.org/jira/browse/YARN-6171
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6171.000.patch, YARN-6171.001.patch,
> YARN-6171.002.patch, YARN-6171.003.patch
>
>
> I have noticed an exception that closes the Application Master occasionally
> with Fair scheduler.
> {code}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.util.ConcurrentModificationException):
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
> at java.util.HashMap$KeyIterator.next(HashMap.java:956)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.allocate(FairScheduler.java:1005)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:532)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6183) [YARN-3368] Few missing informations in Application and Application Attempt page for Basic Info table
[
https://issues.apache.org/jira/browse/YARN-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867318#comment-15867318
]
Hadoop QA commented on YARN-6183:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6183 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852747/YARN-6183.003.patch |
| Optional Tests | asflicense |
| uname | Linux ab1c86076508 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fbc0c2b |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14955/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [YARN-3368] Few missing informations in Application and Application Attempt
> page for Basic Info table
> -
>
> Key: YARN-6183
> URL: https://issues.apache.org/jira/browse/YARN-6183
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Akhil PB
>Assignee: Akhil PB
> Attachments: YARN-6183.001.patch, YARN-6183.002.patch,
> YARN-6183.003.patch
>
>
> Fix missing information in application and app-attempt pages in new YARN-UI.
> In Basic Info table, priority and IsUnmanagedAM fields are missing for some
> applications.
> In App Master panel, change label Expr to Expression.
> In app-attempt page, logUrl and NM web UI needs to be opened in new tab.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6183) [YARN-3368] Few missing informations in Application and Application Attempt page for Basic Info table
[ https://issues.apache.org/jira/browse/YARN-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-6183: --- Attachment: YARN-6183.003.patch > [YARN-3368] Few missing informations in Application and Application Attempt > page for Basic Info table > - > > Key: YARN-6183 > URL: https://issues.apache.org/jira/browse/YARN-6183 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Akhil PB >Assignee: Akhil PB > Attachments: YARN-6183.001.patch, YARN-6183.002.patch, > YARN-6183.003.patch > > > Fix missing information in application and app-attempt pages in new YARN-UI. > In Basic Info table, priority and IsUnmanagedAM fields are missing for some > applications. > In App Master panel, change label Expr to Expression. > In app-attempt page, logUrl and NM web UI needs to be opened in new tab. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6038) Check other resource requests if cannot match the first one while identifying containers to preempt
[ https://issues.apache.org/jira/browse/YARN-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-6038: --- Issue Type: Bug (was: Sub-task) Parent: (was: YARN-5990) > Check other resource requests if cannot match the first one while identifying > containers to preempt > --- > > Key: YARN-6038 > URL: https://issues.apache.org/jira/browse/YARN-6038 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Yufei Gu > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6038) Check other resource requests if cannot match the first one while identifying containers to preempt
[ https://issues.apache.org/jira/browse/YARN-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla resolved YARN-6038. Resolution: Duplicate This is a duplicate of YARN-5832. > Check other resource requests if cannot match the first one while identifying > containers to preempt > --- > > Key: YARN-6038 > URL: https://issues.apache.org/jira/browse/YARN-6038 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Yufei Gu > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6183) [YARN-3368] Few missing informations in Application and Application Attempt page for Basic Info table
[ https://issues.apache.org/jira/browse/YARN-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867313#comment-15867313 ] Sunil G commented on YARN-6183: --- In cluster overview page, Donut charts are not alligned cleanly when resolution goes high. Please help to add that also here. > [YARN-3368] Few missing informations in Application and Application Attempt > page for Basic Info table > - > > Key: YARN-6183 > URL: https://issues.apache.org/jira/browse/YARN-6183 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Akhil PB >Assignee: Akhil PB > Attachments: YARN-6183.001.patch, YARN-6183.002.patch > > > Fix missing information in application and app-attempt pages in new YARN-UI. > In Basic Info table, priority and IsUnmanagedAM fields are missing for some > applications. > In App Master panel, change label Expr to Expression. > In app-attempt page, logUrl and NM web UI needs to be opened in new tab. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5832) Preemption should consider more resource-requests of a starved app if the initial set can not be satisfied
[ https://issues.apache.org/jira/browse/YARN-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-5832: --- Summary: Preemption should consider more resource-requests of a starved app if the initial set can not be satisfied (was: Preemption should consider multiple resource-requests of a starved app as needed) > Preemption should consider more resource-requests of a starved app if the > initial set can not be satisfied > -- > > Key: YARN-5832 > URL: https://issues.apache.org/jira/browse/YARN-5832 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Karthik Kambatla >Assignee: Kai Sasaki > > When identifying containers to preempt for a starved application, the new > implementation considers only the highest priority resource request. If we > can't find candidate containers for that request, we might want to consider > other requests. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6194) Cluster capacity in SchedulingPolicy is updated only on allocation file reload
[
https://issues.apache.org/jira/browse/YARN-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6194:
---
Attachment: (was: YARN-6193.000.patch)
> Cluster capacity in SchedulingPolicy is updated only on allocation file reload
> --
>
> Key: YARN-6194
> URL: https://issues.apache.org/jira/browse/YARN-6194
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Yufei Gu
>
> Some of the {{SchedulingPolicy}} methods need cluster capacity which is set
> using {{#initialize}} today. However, {{initialize()}} is called only on
> allocation reload. If nodes are added between reloads, the cluster capacity
> is not considered until the next reload.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF
[
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6193:
---
Issue Type: Sub-task (was: Bug)
Parent: YARN-5990
> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: YARN-6193.000.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this
> leads to verifying both resources instead of just the dominant resource.
> Ideally, the check should be based on policy.
> Note that current implementation of
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF
[
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6193:
---
Attachment: YARN-6193.000.patch
> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: YARN-6193.000.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this
> leads to verifying both resources instead of just the dominant resource.
> Ideally, the check should be based on policy.
> Note that current implementation of
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6193) FairScheduler might not trigger preemption when using DRF
[
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867309#comment-15867309
]
Karthik Kambatla commented on YARN-6193:
Patch that:
# Uses policy-based check instead of fitsIn to see if a container can be
preempted
# Move some of the SchedulingPolicy methods to the abstract base class instead
of sub-classes
# Augmented tests to consider DRF for both minshare and fairshare preemption
> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: YARN-6193.000.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this
> leads to verifying both resources instead of just the dominant resource.
> Ideally, the check should be based on policy.
> Note that current implementation of
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-6194) Cluster capacity in SchedulingPolicy is updated only on allocation file reload
[
https://issues.apache.org/jira/browse/YARN-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6194:
---
Comment: was deleted
(was: Patch that:
# Uses policy-based check instead of fitsIn to see if a container can be
preempted
# Move some of the SchedulingPolicy methods to the abstract base class instead
of sub-classes
# Augmented tests to consider DRF for both minshare and fairshare preemption)
> Cluster capacity in SchedulingPolicy is updated only on allocation file reload
> --
>
> Key: YARN-6194
> URL: https://issues.apache.org/jira/browse/YARN-6194
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Yufei Gu
> Attachments: YARN-6193.000.patch
>
>
> Some of the {{SchedulingPolicy}} methods need cluster capacity which is set
> using {{#initialize}} today. However, {{initialize()}} is called only on
> allocation reload. If nodes are added between reloads, the cluster capacity
> is not considered until the next reload.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6194) Cluster capacity in SchedulingPolicy is updated only on allocation file reload
[
https://issues.apache.org/jira/browse/YARN-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6194:
---
Attachment: YARN-6193.000.patch
Patch that:
# Uses policy-based check instead of fitsIn to see if a container can be
preempted
# Move some of the SchedulingPolicy methods to the abstract base class instead
of sub-classes
# Augmented tests to consider DRF for both minshare and fairshare preemption
> Cluster capacity in SchedulingPolicy is updated only on allocation file reload
> --
>
> Key: YARN-6194
> URL: https://issues.apache.org/jira/browse/YARN-6194
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Yufei Gu
> Attachments: YARN-6193.000.patch
>
>
> Some of the {{SchedulingPolicy}} methods need cluster capacity which is set
> using {{#initialize}} today. However, {{initialize()}} is called only on
> allocation reload. If nodes are added between reloads, the cluster capacity
> is not considered until the next reload.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867302#comment-15867302
]
Hadoop QA commented on YARN-6163:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 15m
26s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
46s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
31s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 31s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 53s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 106 unchanged - 212 fixed = 109 total (was 318) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 0 new + 4579 unchanged - 4579 fixed = 4579 total (was 9158) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 0 new + 908 unchanged - 908 fixed = 908 total (was 1816) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
58s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 54s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852233/yarn-6163-2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2e3be77292c6 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk
[jira] [Commented] (YARN-6194) Cluster capacity in SchedulingPolicy is updated only on allocation file reload
[
https://issues.apache.org/jira/browse/YARN-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867300#comment-15867300
]
Karthik Kambatla commented on YARN-6194:
When this is fixed, we should remove the TODO in TestFairSchedulerPreemption.
> Cluster capacity in SchedulingPolicy is updated only on allocation file reload
> --
>
> Key: YARN-6194
> URL: https://issues.apache.org/jira/browse/YARN-6194
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Yufei Gu
>
> Some of the {{SchedulingPolicy}} methods need cluster capacity which is set
> using {{#initialize}} today. However, {{initialize()}} is called only on
> allocation reload. If nodes are added between reloads, the cluster capacity
> is not considered until the next reload.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6194) Cluster capacity in SchedulingPolicy is updated only on allocation file reload
Karthik Kambatla created YARN-6194:
--
Summary: Cluster capacity in SchedulingPolicy is updated only on
allocation file reload
Key: YARN-6194
URL: https://issues.apache.org/jira/browse/YARN-6194
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 2.8.0
Reporter: Karthik Kambatla
Assignee: Yufei Gu
Some of the {{SchedulingPolicy}} methods need cluster capacity which is set
using {{#initialize}} today. However, {{initialize()}} is called only on
allocation reload. If nodes are added between reloads, the cluster capacity is
not considered until the next reload.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6148) NM node count reported to AM in Allocate Response should consider requested node label partitions.
[ https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867298#comment-15867298 ] Sunil G commented on YARN-6148: --- +1 for the current approach. > NM node count reported to AM in Allocate Response should consider requested > node label partitions. > -- > > Key: YARN-6148 > URL: https://issues.apache.org/jira/browse/YARN-6148 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6148.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6156) AM blacklisting consider node label partition
[ https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867295#comment-15867295 ] Sunil G commented on YARN-6156: --- It looks fine. But this change is not the cause of this ticket, however its fine to track it here. I would like to add some more comments for that if its ok. Thoughts? > AM blacklisting consider node label partition > -- > > Key: YARN-6156 > URL: https://issues.apache.org/jira/browse/YARN-6156 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch, > YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch, > YARN-6156.0006.patch > > > As per the current implementation for AM blacklisting disabling is based on > completed custer resource. > Incase of partitioned cluster, nodes applicable for AM labels only should be > considered as total nodes. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6148) NM node count reported to AM in Allocate Response should consider requested node label partitions.
[ https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867294#comment-15867294 ] Varun Saxena commented on YARN-6148: Yes, that's the plan. > NM node count reported to AM in Allocate Response should consider requested > node label partitions. > -- > > Key: YARN-6148 > URL: https://issues.apache.org/jira/browse/YARN-6148 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6148.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6175) Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867275#comment-15867275
]
Yufei Gu commented on YARN-6175:
Thanks [~wilfreds] for the review. [~kasha], wanna take a look?
> Negative vcore for resource needed to preempt
> -
>
> Key: YARN-6175
> URL: https://issues.apache.org/jira/browse/YARN-6175
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6175.001.patch
>
>
> Both old preemption code (2.8 and before) and new preemption code could have
> negative vcores while calculating resources needed to preempt.
> For old preemption, you can find following messages in RM logs:
> {code}
> Should preempt
> {code}
> The related code is in method {{resourceDeficit()}}.
> For new preemption code, there are no messages in RM logs, the related code
> is in method {{fairShareStarvation()}}.
> The negative value isn't only a display issue, but also may cause missing
> necessary preemption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4753) Use doxia macro to generate in-page TOC of YARN site documentation
[
https://issues.apache.org/jira/browse/YARN-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867273#comment-15867273
]
Hadoop QA commented on YARN-4753:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
53s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
42s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | YARN-4753 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852738/YARN-4753-branch-2.001.patch
|
| Optional Tests | asflicense mvnsite |
| uname | Linux 927e2e6ff831 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / 58acb67 |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14954/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Use doxia macro to generate in-page TOC of YARN site documentation
> --
>
> Key: YARN-4753
> URL: https://issues.apache.org/jira/browse/YARN-4753
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 2.7.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
> Labels: oct16-easy
> Attachments: YARN-4753.001.patch, YARN-4753.002.patch,
> YARN-4753-branch-2.001.patch
>
>
> Since maven-site-plugin 3.5 was released, we can use toc macro in Markdown.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF
[
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6193:
---
Description:
{{FSAppAttempt#canContainerBePreempted}} verifies preempting a container
doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this
leads to verifying both resources instead of just the dominant resource.
Ideally, the check should be based on policy.
Note that current implementation of
{{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken.
was:{{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} considers
all resources instead of just dominant resource. Ideally, it should use the
DominantResourceCalculator.
> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this
> leads to verifying both resources instead of just the dominant resource.
> Ideally, the check should be based on policy.
> Note that current implementation of
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF
[
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6193:
---
Summary: FairScheduler might not trigger preemption when using DRF (was:
FSAppAttempt#canContainerBePreempted considers all resources instead of just
dominant resource)
> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} considers all
> resources instead of just dominant resource. Ideally, it should use the
> DominantResourceCalculator.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6193) FSAppAttempt#canContainerBePreempted considers all resources instead of just dominant resource
[
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6193:
---
Summary: FSAppAttempt#canContainerBePreempted considers all resources
instead of just dominant resource (was:
DominantResourceFairnessPolicy#checkIfUsageOverFairShare considers all
resources instead of just dominant resource)
> FSAppAttempt#canContainerBePreempted considers all resources instead of just
> dominant resource
> --
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} considers all
> resources instead of just dominant resource. Ideally, it should use the
> DominantResourceCalculator.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6156) AM blacklisting consider node label partition
[ https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867243#comment-15867243 ] Varun Saxena commented on YARN-6156: By the way for unmanaged AM, am resource request will be null and as we are dereferencing it, the change becomes necessary. > AM blacklisting consider node label partition > -- > > Key: YARN-6156 > URL: https://issues.apache.org/jira/browse/YARN-6156 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch, > YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch, > YARN-6156.0006.patch > > > As per the current implementation for AM blacklisting disabling is based on > completed custer resource. > Incase of partitioned cluster, nodes applicable for AM labels only should be > considered as total nodes. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4753) Use doxia macro to generate in-page TOC of YARN site documentation
[ https://issues.apache.org/jira/browse/YARN-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-4753: --- Attachment: YARN-4753-branch-2.001.patch > Use doxia macro to generate in-page TOC of YARN site documentation > -- > > Key: YARN-4753 > URL: https://issues.apache.org/jira/browse/YARN-4753 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.0 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > Labels: oct16-easy > Attachments: YARN-4753.001.patch, YARN-4753.002.patch, > YARN-4753-branch-2.001.patch > > > Since maven-site-plugin 3.5 was released, we can use toc macro in Markdown. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6163) FS Preemption is a trickle for severely starved applications
[ https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-6163: --- Attachment: YARN-6163.004.patch Uploading patch v4 to match 4th commit in the PR. Not sure why the precommit wasn't picking the latest patch from PR. > FS Preemption is a trickle for severely starved applications > > > Key: YARN-6163 > URL: https://issues.apache.org/jira/browse/YARN-6163 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: YARN-6163.004.patch, yarn-6163-1.patch, yarn-6163-2.patch > > > With current logic, only one RR is considered per each instance of marking an > application starved. This marking happens only on the update call that runs > every 500ms. Due to this, an application that is severely starved takes > forever to reach fairshare based on preemptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4753) Use doxia macro to generate in-page TOC of YARN site documentation
[ https://issues.apache.org/jira/browse/YARN-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867229#comment-15867229 ] Hudson commented on YARN-4753: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11250 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11250/]) YARN-4753. Use doxia macro to generate in-page TOC of YARN site (iwasakims: rev fbc0c2bd763e3a3aad914eb9d60b05ad4ab2825f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCgroups.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WebServicesIntro.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/OpportunisticContainers.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WebApplicationProxy.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WritingYarnApplications.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRestart.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/SecureContainer.md * (edit) hadoop-project/src/site/site.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerRest.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md > Use doxia macro to generate in-page TOC of YARN site documentation > -- > > Key: YARN-4753 > URL: https://issues.apache.org/jira/browse/YARN-4753 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.0 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > Labels: oct16-easy > Attachments: YARN-4753.001.patch, YARN-4753.002.patch > > > Since maven-site-plugin 3.5 was released, we can use toc macro in Markdown. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867224#comment-15867224
]
ASF GitHub Bot commented on YARN-6163:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101201549
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestVisitedResourceRequestTracker.java
---
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at*
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import org.apache.hadoop.yarn.server.resourcemanager.MockNodes;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.List;
+
+public class TestVisitedResourceRequestTracker {
+ private final ClusterNodeTracker
+ nodeTracker = new ClusterNodeTracker<>();
+ private final ResourceRequest
+ anyRequest, rackRequest, node1Request, node2Request;
+
+ public TestVisitedResourceRequestTracker() {
+List rmNodes =
+MockNodes.newNodes(1, 2, Resources.createResource(8192, 8));
+
+FSSchedulerNode node1 = new FSSchedulerNode(rmNodes.get(0), false);
+nodeTracker.addNode(node1);
+node1Request = createRR(node1.getNodeName(), 1);
+
+FSSchedulerNode node2 = new FSSchedulerNode(rmNodes.get(1), false);
+node2Request = createRR(node2.getNodeName(), 1);
+nodeTracker.addNode(node2);
+
+anyRequest = createRR(ResourceRequest.ANY, 2);
+rackRequest = createRR(node1.getRackName(), 2);
+ }
+
+ private ResourceRequest createRR(String resourceName, int count) {
+return ResourceRequest.newInstance(
+Priority.UNDEFINED, resourceName, Resources.none(), count);
+ }
+
+ @Test
+ public void testVisitAnyRequestFirst() {
+VisitedResourceRequestTracker tracker =
+new VisitedResourceRequestTracker(nodeTracker);
+
+// Visit ANY request first
+Assert.assertTrue(tracker.visit(anyRequest));
--- End diff --
I usually insist on a message. In this case, I found it hard to come up
with a message that adds value without looking at the code. And, the comments
in the code call out the expectations clearly.
If you absolutely insist, I can add the messages.
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an
> application starved. This marking happens only on the update call that runs
> every 500ms. Due to this, an application that is severely starved takes
> forever to reach fairshare based on preemptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867217#comment-15867217
]
ASF GitHub Bot commented on YARN-6163:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101201261
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/VisitedResourceRequestTracker.java
---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Applications place {@link ResourceRequest}s at multiple levels. This is
a
+ * helper class that allows tracking if a {@link ResourceRequest} has been
+ * visited at a different locality level.
+ *
+ * This is implemented for {@link
FSAppAttempt#getStarvedResourceRequests()}.
+ * The implementation is not thread-safe.
+ */
+class VisitedResourceRequestTracker {
+ private final Map>
map =
+ new HashMap<>();
+ private final ClusterNodeTracker nodeTracker;
+
+ VisitedResourceRequestTracker(
+ ClusterNodeTracker nodeTracker) {
+this.nodeTracker = nodeTracker;
+ }
+
+ /**
+ * Check if the {@link ResourceRequest} is visited before, and track it.
+ * @param rr {@link ResourceRequest} to visit
+ * @return true if rr this is the first visit across all
+ * locality levels, false otherwise
+ */
+ boolean visit(ResourceRequest rr) {
+Priority priority = rr.getPriority();
+Resource capability = rr.getCapability();
+
+Map subMap = map.get(priority);
--- End diff --
I don't see how the Multimap simplifies anything.
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an
> application starved. This marking happens only on the update call that runs
> every 500ms. Due to this, an application that is severely starved takes
> forever to reach fairshare based on preemptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867215#comment-15867215
]
ASF GitHub Bot commented on YARN-6163:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101201228
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/VisitedResourceRequestTracker.java
---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Applications place {@link ResourceRequest}s at multiple levels. This is
a
+ * helper class that allows tracking if a {@link ResourceRequest} has been
+ * visited at a different locality level.
+ *
+ * This is implemented for {@link
FSAppAttempt#getStarvedResourceRequests()}.
+ * The implementation is not thread-safe.
+ */
+class VisitedResourceRequestTracker {
+ private final Map>
map =
+ new HashMap<>();
+ private final ClusterNodeTracker nodeTracker;
+
+ VisitedResourceRequestTracker(
+ ClusterNodeTracker nodeTracker) {
+this.nodeTracker = nodeTracker;
+ }
+
+ /**
+ * Check if the {@link ResourceRequest} is visited before, and track it.
+ * @param rr {@link ResourceRequest} to visit
+ * @return true if rr this is the first visit across all
+ * locality levels, false otherwise
+ */
+ boolean visit(ResourceRequest rr) {
+Priority priority = rr.getPriority();
+Resource capability = rr.getCapability();
+
+Map subMap = map.get(priority);
+if (subMap == null) {
+ subMap = new HashMap<>();
+ map.put(priority, subMap);
+}
+
+TrackerPerPriorityResource tracker = subMap.get(capability);
+if (tracker == null) {
+ tracker = new TrackerPerPriorityResource();
+ subMap.put(capability, tracker);
+}
+
+return tracker.visit(rr.getResourceName());
+ }
+
+ private class TrackerPerPriorityResource {
+private Set racksWithNodesVisited = new HashSet<>();
+private Set racksVisted = new HashSet<>();
+private boolean anyVisited;
+
+private boolean visitAny() {
+ if (racksVisted.isEmpty() && racksWithNodesVisited.isEmpty()) {
+anyVisited = true;
+ }
+ return anyVisited;
+}
+
+private boolean visitRack(String rackName) {
+ if (anyVisited || racksWithNodesVisited.contains(rackName)) {
+return false;
+ } else {
+racksVisted.add(rackName);
+return true;
+ }
+}
+
+private boolean visitNode(String rackName) {
+ if (anyVisited || racksVisted.contains(rackName)) {
+return false;
+ } else {
+racksWithNodesVisited.add(rackName);
+return true;
+ }
+}
+
+private boolean visit(String resourceName) {
+ if (resourceName.equals(ResourceRequest.ANY)) {
+return visitAny();
+ }
+
+ List nodes =
+ nodeTracker.getNodesByResourceName(resourceName);
+ switch (nodes.size()) {
--- End diff --
To me, this is one of those rare cases you could actually use a switch.
IAC, I changed it to if
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867211#comment-15867211
]
ASF GitHub Bot commented on YARN-6163:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101201190
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/VisitedResourceRequestTracker.java
---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Applications place {@link ResourceRequest}s at multiple levels. This is
a
+ * helper class that allows tracking if a {@link ResourceRequest} has been
+ * visited at a different locality level.
+ *
+ * This is implemented for {@link
FSAppAttempt#getStarvedResourceRequests()}.
+ * The implementation is not thread-safe.
+ */
+class VisitedResourceRequestTracker {
+ private final Map>
map =
+ new HashMap<>();
+ private final ClusterNodeTracker nodeTracker;
+
+ VisitedResourceRequestTracker(
+ ClusterNodeTracker nodeTracker) {
+this.nodeTracker = nodeTracker;
+ }
+
+ /**
+ * Check if the {@link ResourceRequest} is visited before, and track it.
+ * @param rr {@link ResourceRequest} to visit
+ * @return true if rr this is the first visit across all
+ * locality levels, false otherwise
+ */
+ boolean visit(ResourceRequest rr) {
+Priority priority = rr.getPriority();
+Resource capability = rr.getCapability();
+
+Map subMap = map.get(priority);
+if (subMap == null) {
+ subMap = new HashMap<>();
+ map.put(priority, subMap);
+}
+
+TrackerPerPriorityResource tracker = subMap.get(capability);
+if (tracker == null) {
+ tracker = new TrackerPerPriorityResource();
+ subMap.put(capability, tracker);
+}
+
+return tracker.visit(rr.getResourceName());
+ }
+
+ private class TrackerPerPriorityResource {
+private Set racksWithNodesVisited = new HashSet<>();
+private Set racksVisted = new HashSet<>();
+private boolean anyVisited;
+
+private boolean visitAny() {
+ if (racksVisted.isEmpty() && racksWithNodesVisited.isEmpty()) {
+anyVisited = true;
+ }
+ return anyVisited;
+}
+
+private boolean visitRack(String rackName) {
+ if (anyVisited || racksWithNodesVisited.contains(rackName)) {
+return false;
+ } else {
+racksVisted.add(rackName);
+return true;
+ }
+}
+
+private boolean visitNode(String rackName) {
+ if (anyVisited || racksVisted.contains(rackName)) {
+return false;
+ } else {
+racksWithNodesVisited.add(rackName);
+return true;
+ }
+}
+
+private boolean visit(String resourceName) {
+ if (resourceName.equals(ResourceRequest.ANY)) {
+return visitAny();
+ }
+
+ List nodes =
+ nodeTracker.getNodesByResourceName(resourceName);
+ switch (nodes.size()) {
+case 0:
+ // Log error
+ return false;
+case 1:
+ // Node
+
[jira] [Commented] (YARN-6175) Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867208#comment-15867208
]
Wilfred Spiegelenburg commented on YARN-6175:
-
The change overall looks good. The junit test failures are not related to the
change.
+1 (non binding)
> Negative vcore for resource needed to preempt
> -
>
> Key: YARN-6175
> URL: https://issues.apache.org/jira/browse/YARN-6175
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6175.001.patch
>
>
> Both old preemption code (2.8 and before) and new preemption code could have
> negative vcores while calculating resources needed to preempt.
> For old preemption, you can find following messages in RM logs:
> {code}
> Should preempt
> {code}
> The related code is in method {{resourceDeficit()}}.
> For new preemption code, there are no messages in RM logs, the related code
> is in method {{fairShareStarvation()}}.
> The negative value isn't only a display issue, but also may cause missing
> necessary preemption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6180) Clean unused SchedulerRequestKeys once ExecutionType updates are completed
[
https://issues.apache.org/jira/browse/YARN-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867207#comment-15867207
]
Arun Suresh commented on YARN-6180:
---
Yup, this is meant to track for container updates in general.
Anyway, I was going thru YARN-5540 again, and it looks like the changes in the
{{AppScheduleingInfo::decrementOutstanding()}} does infact clean up
schedulerKeys whose numContainers are 0.. the AM does not need to explicitly
send a 0 container ResourceRequest. Since Container Updates are just another
SchedulerKey (well the ExecutionType Updates currently... we still need to move
the increase container plumbing from the AppSchedulingInfo into the
ContainerUpdateContext).
I will verify the above with a testcase to verify the expected behavior and
either close this as not an issue or add a testcase only patch for this.
> Clean unused SchedulerRequestKeys once ExecutionType updates are completed
> --
>
> Key: YARN-6180
> URL: https://issues.apache.org/jira/browse/YARN-6180
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>
> The SchedulerRequestKeys used for ExecutionType updates, that are generated,
> tend to accumulate in the AppSchedulingInfo and over time lead to a situation
> outlined in YARN-5540.
> These keys must be removed once the container update completes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6156) AM blacklisting consider node label partition
[ https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867203#comment-15867203 ] Varun Saxena commented on YARN-6156: Thanks [~bibinchundatt]. Yeah it is not necessary to invoke simple blacklisting manager for unmanaged AM. This though existed from earlier but can be handled here itself as its one line change. [~sunilg], are you fine with the change? > AM blacklisting consider node label partition > -- > > Key: YARN-6156 > URL: https://issues.apache.org/jira/browse/YARN-6156 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch, > YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch, > YARN-6156.0006.patch > > > As per the current implementation for AM blacklisting disabling is based on > completed custer resource. > Incase of partitioned cluster, nodes applicable for AM labels only should be > considered as total nodes. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6156) AM blacklisting consider node label partition
[
https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867199#comment-15867199
]
Hadoop QA commented on YARN-6156:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 270 unchanged - 4 fixed = 270 total (was 274)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 29s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6156 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852724/YARN-6156.0006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b85f451af534 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 353a9b2 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14951/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14951/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14951/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AM blacklisting consider node label partition
>
[jira] [Commented] (YARN-4753) Use doxia macro to generate in-page TOC of YARN site documentation
[ https://issues.apache.org/jira/browse/YARN-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867191#comment-15867191 ] Masatake Iwasaki commented on YARN-4753: Thanks. Committing to trunk and backporting to branch-2. > Use doxia macro to generate in-page TOC of YARN site documentation > -- > > Key: YARN-4753 > URL: https://issues.apache.org/jira/browse/YARN-4753 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.0 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > Labels: oct16-easy > Attachments: YARN-4753.001.patch, YARN-4753.002.patch > > > Since maven-site-plugin 3.5 was released, we can use toc macro in Markdown. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867187#comment-15867187
]
ASF GitHub Bot commented on YARN-6163:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101200171
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/VisitedResourceRequestTracker.java
---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Applications place {@link ResourceRequest}s at multiple levels. This is
a
+ * helper class that allows tracking if a {@link ResourceRequest} has been
+ * visited at a different locality level.
+ *
+ * This is implemented for {@link
FSAppAttempt#getStarvedResourceRequests()}.
+ * The implementation is not thread-safe.
+ */
+class VisitedResourceRequestTracker {
+ private final Map>
map =
+ new HashMap<>();
+ private final ClusterNodeTracker nodeTracker;
+
+ VisitedResourceRequestTracker(
+ ClusterNodeTracker nodeTracker) {
+this.nodeTracker = nodeTracker;
+ }
+
+ /**
+ * Check if the {@link ResourceRequest} is visited before, and track it.
+ * @param rr {@link ResourceRequest} to visit
+ * @return true if rr this is the first visit across all
+ * locality levels, false otherwise
+ */
+ boolean visit(ResourceRequest rr) {
+Priority priority = rr.getPriority();
+Resource capability = rr.getCapability();
+
+Map subMap = map.get(priority);
+if (subMap == null) {
+ subMap = new HashMap<>();
+ map.put(priority, subMap);
+}
+
+TrackerPerPriorityResource tracker = subMap.get(capability);
+if (tracker == null) {
+ tracker = new TrackerPerPriorityResource();
+ subMap.put(capability, tracker);
+}
+
+return tracker.visit(rr.getResourceName());
+ }
+
+ private class TrackerPerPriorityResource {
+private Set racksWithNodesVisited = new HashSet<>();
+private Set racksVisted = new HashSet<>();
+private boolean anyVisited;
+
+private boolean visitAny() {
+ if (racksVisted.isEmpty() && racksWithNodesVisited.isEmpty()) {
+anyVisited = true;
+ }
+ return anyVisited;
+}
+
+private boolean visitRack(String rackName) {
+ if (anyVisited || racksWithNodesVisited.contains(rackName)) {
+return false;
+ } else {
+racksVisted.add(rackName);
+return true;
+ }
+}
+
+private boolean visitNode(String rackName) {
--- End diff --
I attempted documenting it and ended up capturing the relatively tiny logic
in English :)
Given we have documented this earlier, I see little value in documenting
again.
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee:
[jira] [Commented] (YARN-6050) AMs can't be scheduled on racks or nodes
[
https://issues.apache.org/jira/browse/YARN-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867184#comment-15867184
]
Hadoop QA commented on YARN-6050:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 26s{color}
| {color:red} root generated 1 new + 702 unchanged - 0 fixed = 703 total (was
702) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 2s{color} | {color:orange} root: The patch generated 9 new + 1918 unchanged
- 6 fixed = 1927 total (was 1924) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
31s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 1 new + 908 unchanged - 0 fixed = 909 total (was 908) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 15s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}102m
5s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
12s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}245m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6050 |
| JIRA Patch URL |
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867178#comment-15867178
]
ASF GitHub Bot commented on YARN-6163:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101199879
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
---
@@ -247,23 +247,23 @@ private Resource updateStarvedAppsFairshare(
* @param minShareStarvation minshare starvation to distribute
*/
private void updateStarvedAppsMinshare(
- TreeSet appsWithDemand, Resource minShareStarvation) {
+ TreeSet appsWithDemand, final Resource
minShareStarvation) {
--- End diff --
Fixed
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an
> application starved. This marking happens only on the update call that runs
> every 500ms. Due to this, an application that is severely starved takes
> forever to reach fairshare based on preemptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867177#comment-15867177
]
ASF GitHub Bot commented on YARN-6163:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101199875
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
---
@@ -1147,24 +1147,32 @@ private static boolean checkAndMarkRRVisited(
* starvation.
*/
List getStarvedResourceRequests() {
+// List of RRs we build in this method to return
List ret = new ArrayList<>();
+
+// Track visited RRs to avoid the same RR at multiple locality levels
Map visitedRRs= new HashMap<>();
+// Start with current starvation and track the pending amount
Resource pending = getStarvation();
for (ResourceRequest rr : appSchedulingInfo.getAllResourceRequests()) {
if (Resources.isNone(pending)) {
+// Found enough RRs to match the starvation
break;
}
+
+ // See if we have already seen this RR
if (checkAndMarkRRVisited(visitedRRs, rr)) {
continue;
}
- // Compute the number of containers of this capability that fit in
the
- // pending amount
+ // A RR can have multiple containers of a capability. We need to
+ // compute the number of containers that fit in "pending".
int ratio = (int) Math.floor(
--- End diff --
Fixed.
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an
> application starved. This marking happens only on the update call that runs
> every 500ms. Due to this, an application that is severely starved takes
> forever to reach fairshare based on preemptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867180#comment-15867180
]
ASF GitHub Bot commented on YARN-6163:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101199897
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/VisitedResourceRequestTracker.java
---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Applications place {@link ResourceRequest}s at multiple levels. This is
a
+ * helper class that allows tracking if a {@link ResourceRequest} has been
+ * visited at a different locality level.
+ *
+ * This is implemented for {@link
FSAppAttempt#getStarvedResourceRequests()}.
+ * The implementation is not thread-safe.
+ */
+class VisitedResourceRequestTracker {
+ private final Map>
map =
+ new HashMap<>();
+ private final ClusterNodeTracker nodeTracker;
+
+ VisitedResourceRequestTracker(
+ ClusterNodeTracker nodeTracker) {
+this.nodeTracker = nodeTracker;
+ }
+
+ /**
+ * Check if the {@link ResourceRequest} is visited before, and track it.
+ * @param rr {@link ResourceRequest} to visit
+ * @return true if rr this is the first visit across all
--- End diff --
Fixed.
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an
> application starved. This marking happens only on the update call that runs
> every 500ms. Due to this, an application that is severely starved takes
> forever to reach fairshare based on preemptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6148) NM node count reported to AM in Allocate Response should consider requested node label partitions.
[ https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867139#comment-15867139 ] Rohith Sharma K S commented on YARN-6148: - Basically in this JIRA lets do for YARN related changes such as adding new API of node count per partition. AM need to use this API in case of partitioned cluster. MR related changes need to be handled separately in MapReduce project. > NM node count reported to AM in Allocate Response should consider requested > node label partitions. > -- > > Key: YARN-6148 > URL: https://issues.apache.org/jira/browse/YARN-6148 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6148.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6156) AM blacklisting consider node label partition
[ https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-6156: --- Attachment: YARN-6156.0006.patch [~sunilg]/[~varun_saxena] Minor change in patch for checking unmanager AM.For unmanaged AM we can always set to Disable blacklist manager. > AM blacklisting consider node label partition > -- > > Key: YARN-6156 > URL: https://issues.apache.org/jira/browse/YARN-6156 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: YARN-6156.0001.patch, YARN-6156.0002.patch, > YARN-6156.0003.patch, YARN-6156.0004.patch, YARN-6156.0005.patch, > YARN-6156.0006.patch > > > As per the current implementation for AM blacklisting disabling is based on > completed custer resource. > Incase of partitioned cluster, nodes applicable for AM labels only should be > considered as total nodes. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6042) Fairscheduler: Dump scheduler state in log
[
https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867112#comment-15867112
]
Wilfred Spiegelenburg commented on YARN-6042:
-
Thank you for the update, we discussed it offline and the change looks good now.
I have one minor nit in the text update we should have in FSAppAttempt:
_creating_ should be _create_ in the debug text
_LOG.debug("Couldn't creating reservation for app: " + getName()_
> Fairscheduler: Dump scheduler state in log
> --
>
> Key: YARN-6042
> URL: https://issues.apache.org/jira/browse/YARN-6042
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6042.001.patch, YARN-6042.002.patch,
> YARN-6042.003.patch, YARN-6042.004.patch, YARN-6042.005.patch
>
>
> To improve the debugging of scheduler issues it would be a big improvement to
> be able to dump the scheduler state into a log on request.
> The Dump the scheduler state at a point in time would allow debugging of a
> scheduler that is not hung (deadlocked) but also not assigning containers.
> Currently we do not have a proper overview of what state the scheduler and
> the queues are in and we have to make assumptions or guess
> The scheduler and queue state needed would include (not exhaustive):
> - instantaneous and steady fair share (app / queue)
> - AM share and resources
> - weight
> - app demand
> - application run state (runnable/non runnable)
> - last time at fair/min share
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6093) Invalid AMRM token exception when using FederationRMFailoverProxyProvider at AMRMtoken renewal during a RM failover
[
https://issues.apache.org/jira/browse/YARN-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867101#comment-15867101
]
Subru Krishnan commented on YARN-6093:
--
[~botong], can you open a JIRA to track the test failure as we need to fix it.
[~jianhe], can you take a quick look at the patch before I commit as this
addresses one of your early requests, i.e. not to save the tokens in
{{FederationRMFailoverProxyProvider}}. Thanks.
> Invalid AMRM token exception when using FederationRMFailoverProxyProvider at
> AMRMtoken renewal during a RM failover
> ---
>
> Key: YARN-6093
> URL: https://issues.apache.org/jira/browse/YARN-6093
> Project: Hadoop YARN
> Issue Type: Bug
> Components: amrmproxy, federation
>Affects Versions: YARN-2915
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments:
> YARN-6093-08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch,
> YARN-6093-git08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch,
> YARN-6093.v1.patch, YARN-6093-YARN-2915.v1.patch,
> YARN-6093-YARN-2915.v2.patch, YARN-6093-YARN-2915.v3.patch,
> YARN-6093-YARN-2915.v4.patch
>
>
> AMRMProxy uses expired AMRMToken to talk to RM, leading to the "Invalid
> AMRMToken" exception. The bug is triggered when both conditions are met:
> 1. RM rolls master key and renews AMRMToken for a running AM.
> 2. Existing RPC connection between AMRMProxy and RM drops and attempt to
> reconnect via failover in FederationRMFailoverProxyProvider.
> Here's what happened:
> In DefaultRequestInterceptor.init(), we create a proxy ugi, load it with the
> initial AMRMToken issued by RM, and used it for initiating rmClient. Then we
> arrive at FederationRMFailoverProxyProvider.init(), a full copy of ugi tokens
> are saved locally, create an actual RM proxy and setup the RPC connection.
> Later when RM rolls master key and issues a new AMRMToken,
> DefaultRequestInterceptor.updateAMRMToken() updates it into the proxy ugi.
> However the new token is never used, until the existing RPC connection
> between AMRMProxy and RM drops for other reasons (say master RM crashes).
> When we try to reconnect, since the service name of the new AMRMToken is not
> yet set correctly in DefaultRequestInterceptor.updateAMRMToken(), RPC found
> no valid AMRMToken when trying to setup a new connection. We first hit a
> "Client cannot authenticate via:[TOKEN]" exception. This is expected.
> Next, FederationRMFailoverProxyProvider fails over, we reset the service
> token via ClientRMProxy.getRMAddress() and reconnect. Supposedly this would
> have worked.
> However since DefaultRequestInterceptor does not use the proxy user for later
> calls to rmClient, when performing failover in
> FederationRMFailoverProxyProvider, we are not in the proxy user. Currently
> the code solve the problem by reloading the current ugi with all tokens saved
> locally in originalTokens in method addOriginalTokens(). The problem is that
> the original AMRMToken loaded is no longer accepted by RM, and thus we keep
> hitting the "Invalid AMRMToken" exception until AM fails.
> The correct way is that rather than saving the original tokens in the proxy
> ugi, we save the original ugi itself. Every time we perform failover and
> create the new RM proxy, we use the original ugi, which is always loaded with
> the up-to-date AMRMToken.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6180) Clean unused SchedulerRequestKeys once ExecutionType updates are completed
[ https://issues.apache.org/jira/browse/YARN-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867100#comment-15867100 ] Subru Krishnan commented on YARN-6180: -- [~asuresh], IIUC this is valid for all key attributes and not just ExecutionType updates? > Clean unused SchedulerRequestKeys once ExecutionType updates are completed > -- > > Key: YARN-6180 > URL: https://issues.apache.org/jira/browse/YARN-6180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > > The SchedulerRequestKeys used for ExecutionType updates, that are generated, > tend to accumulate in the AppSchedulingInfo and over time lead to a situation > outlined in YARN-5540. > These keys must be removed once the container update completes. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6190) Bug fixes in federation polices
[ https://issues.apache.org/jira/browse/YARN-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-6190: - Issue Type: Sub-task (was: Bug) Parent: YARN-2915 > Bug fixes in federation polices > --- > > Key: YARN-6190 > URL: https://issues.apache.org/jira/browse/YARN-6190 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-6190-YARN-2915.v1.patch > > > A bug fix in LocalityMulticastAMRMProxyPolicy on policy array condition > check, along with misc cleanups. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4036) Findbugs warnings in hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867096#comment-15867096 ] Akira Ajisaka commented on YARN-4036: - Now there are no findbugs warnings in hadoop-yarn-server-common module. Closing this. > Findbugs warnings in hadoop-yarn-server-common > -- > > Key: YARN-4036 > URL: https://issues.apache.org/jira/browse/YARN-4036 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.0 >Reporter: Varun Saxena > Attachments: findbugsHtml.html > > > Refer to > https://issues.apache.org/jira/browse/YARN-3232?focusedCommentId=14679146=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14679146 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4036) Findbugs warnings in hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved YARN-4036. - Resolution: Not A Problem > Findbugs warnings in hadoop-yarn-server-common > -- > > Key: YARN-4036 > URL: https://issues.apache.org/jira/browse/YARN-4036 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.0 >Reporter: Varun Saxena > Attachments: findbugsHtml.html > > > Refer to > https://issues.apache.org/jira/browse/YARN-3232?focusedCommentId=14679146=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14679146 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6093) Invalid AMRM token exception when using FederationRMFailoverProxyProvider at AMRMtoken renewal during a RM failover
[ https://issues.apache.org/jira/browse/YARN-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867060#comment-15867060 ] Botong Huang edited comment on YARN-6093 at 2/15/17 1:23 AM: - Talked to [~curino] offline, this is a randomized test, and sometimes it fails because the threshold is too tight. This is irrelevant to the patch here. was (Author: botong): Talked to [~curino] offline, this is a randomized test, and sometimes the threshold is too tight. This is irrelevant to the patch here. > Invalid AMRM token exception when using FederationRMFailoverProxyProvider at > AMRMtoken renewal during a RM failover > --- > > Key: YARN-6093 > URL: https://issues.apache.org/jira/browse/YARN-6093 > Project: Hadoop YARN > Issue Type: Bug > Components: amrmproxy, federation >Affects Versions: YARN-2915 >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: > YARN-6093-08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch, > YARN-6093-git08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch, > YARN-6093.v1.patch, YARN-6093-YARN-2915.v1.patch, > YARN-6093-YARN-2915.v2.patch, YARN-6093-YARN-2915.v3.patch, > YARN-6093-YARN-2915.v4.patch > > > AMRMProxy uses expired AMRMToken to talk to RM, leading to the "Invalid > AMRMToken" exception. The bug is triggered when both conditions are met: > 1. RM rolls master key and renews AMRMToken for a running AM. > 2. Existing RPC connection between AMRMProxy and RM drops and attempt to > reconnect via failover in FederationRMFailoverProxyProvider. > Here's what happened: > In DefaultRequestInterceptor.init(), we create a proxy ugi, load it with the > initial AMRMToken issued by RM, and used it for initiating rmClient. Then we > arrive at FederationRMFailoverProxyProvider.init(), a full copy of ugi tokens > are saved locally, create an actual RM proxy and setup the RPC connection. > Later when RM rolls master key and issues a new AMRMToken, > DefaultRequestInterceptor.updateAMRMToken() updates it into the proxy ugi. > However the new token is never used, until the existing RPC connection > between AMRMProxy and RM drops for other reasons (say master RM crashes). > When we try to reconnect, since the service name of the new AMRMToken is not > yet set correctly in DefaultRequestInterceptor.updateAMRMToken(), RPC found > no valid AMRMToken when trying to setup a new connection. We first hit a > "Client cannot authenticate via:[TOKEN]" exception. This is expected. > Next, FederationRMFailoverProxyProvider fails over, we reset the service > token via ClientRMProxy.getRMAddress() and reconnect. Supposedly this would > have worked. > However since DefaultRequestInterceptor does not use the proxy user for later > calls to rmClient, when performing failover in > FederationRMFailoverProxyProvider, we are not in the proxy user. Currently > the code solve the problem by reloading the current ugi with all tokens saved > locally in originalTokens in method addOriginalTokens(). The problem is that > the original AMRMToken loaded is no longer accepted by RM, and thus we keep > hitting the "Invalid AMRMToken" exception until AM fails. > The correct way is that rather than saving the original tokens in the proxy > ugi, we save the original ugi itself. Every time we perform failover and > create the new RM proxy, we use the original ugi, which is always loaded with > the up-to-date AMRMToken. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6093) Invalid AMRM token exception when using FederationRMFailoverProxyProvider at AMRMtoken renewal during a RM failover
[ https://issues.apache.org/jira/browse/YARN-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867060#comment-15867060 ] Botong Huang commented on YARN-6093: Talked to [~curino] offline, this is a randomized test, and sometimes the threshold is too tight. This is irrelevant to the patch here. > Invalid AMRM token exception when using FederationRMFailoverProxyProvider at > AMRMtoken renewal during a RM failover > --- > > Key: YARN-6093 > URL: https://issues.apache.org/jira/browse/YARN-6093 > Project: Hadoop YARN > Issue Type: Bug > Components: amrmproxy, federation >Affects Versions: YARN-2915 >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: > YARN-6093-08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch, > YARN-6093-git08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch, > YARN-6093.v1.patch, YARN-6093-YARN-2915.v1.patch, > YARN-6093-YARN-2915.v2.patch, YARN-6093-YARN-2915.v3.patch, > YARN-6093-YARN-2915.v4.patch > > > AMRMProxy uses expired AMRMToken to talk to RM, leading to the "Invalid > AMRMToken" exception. The bug is triggered when both conditions are met: > 1. RM rolls master key and renews AMRMToken for a running AM. > 2. Existing RPC connection between AMRMProxy and RM drops and attempt to > reconnect via failover in FederationRMFailoverProxyProvider. > Here's what happened: > In DefaultRequestInterceptor.init(), we create a proxy ugi, load it with the > initial AMRMToken issued by RM, and used it for initiating rmClient. Then we > arrive at FederationRMFailoverProxyProvider.init(), a full copy of ugi tokens > are saved locally, create an actual RM proxy and setup the RPC connection. > Later when RM rolls master key and issues a new AMRMToken, > DefaultRequestInterceptor.updateAMRMToken() updates it into the proxy ugi. > However the new token is never used, until the existing RPC connection > between AMRMProxy and RM drops for other reasons (say master RM crashes). > When we try to reconnect, since the service name of the new AMRMToken is not > yet set correctly in DefaultRequestInterceptor.updateAMRMToken(), RPC found > no valid AMRMToken when trying to setup a new connection. We first hit a > "Client cannot authenticate via:[TOKEN]" exception. This is expected. > Next, FederationRMFailoverProxyProvider fails over, we reset the service > token via ClientRMProxy.getRMAddress() and reconnect. Supposedly this would > have worked. > However since DefaultRequestInterceptor does not use the proxy user for later > calls to rmClient, when performing failover in > FederationRMFailoverProxyProvider, we are not in the proxy user. Currently > the code solve the problem by reloading the current ugi with all tokens saved > locally in originalTokens in method addOriginalTokens(). The problem is that > the original AMRMToken loaded is no longer accepted by RM, and thus we keep > hitting the "Invalid AMRMToken" exception until AM fails. > The correct way is that rather than saving the original tokens in the proxy > ugi, we save the original ugi itself. Every time we perform failover and > create the new RM proxy, we use the original ugi, which is always loaded with > the up-to-date AMRMToken. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6093) Invalid AMRM token exception when using FederationRMFailoverProxyProvider at AMRMtoken renewal during a RM failover
[ https://issues.apache.org/jira/browse/YARN-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867042#comment-15867042 ] Botong Huang commented on YARN-6093: +[~curino]: can you please take a look at the test failure (TestWeightedRandomRouterPolicy.testClusterChosenWithRightProbability) in the last Yetus run? Thanks! > Invalid AMRM token exception when using FederationRMFailoverProxyProvider at > AMRMtoken renewal during a RM failover > --- > > Key: YARN-6093 > URL: https://issues.apache.org/jira/browse/YARN-6093 > Project: Hadoop YARN > Issue Type: Bug > Components: amrmproxy, federation >Affects Versions: YARN-2915 >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: > YARN-6093-08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch, > YARN-6093-git08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch, > YARN-6093.v1.patch, YARN-6093-YARN-2915.v1.patch, > YARN-6093-YARN-2915.v2.patch, YARN-6093-YARN-2915.v3.patch, > YARN-6093-YARN-2915.v4.patch > > > AMRMProxy uses expired AMRMToken to talk to RM, leading to the "Invalid > AMRMToken" exception. The bug is triggered when both conditions are met: > 1. RM rolls master key and renews AMRMToken for a running AM. > 2. Existing RPC connection between AMRMProxy and RM drops and attempt to > reconnect via failover in FederationRMFailoverProxyProvider. > Here's what happened: > In DefaultRequestInterceptor.init(), we create a proxy ugi, load it with the > initial AMRMToken issued by RM, and used it for initiating rmClient. Then we > arrive at FederationRMFailoverProxyProvider.init(), a full copy of ugi tokens > are saved locally, create an actual RM proxy and setup the RPC connection. > Later when RM rolls master key and issues a new AMRMToken, > DefaultRequestInterceptor.updateAMRMToken() updates it into the proxy ugi. > However the new token is never used, until the existing RPC connection > between AMRMProxy and RM drops for other reasons (say master RM crashes). > When we try to reconnect, since the service name of the new AMRMToken is not > yet set correctly in DefaultRequestInterceptor.updateAMRMToken(), RPC found > no valid AMRMToken when trying to setup a new connection. We first hit a > "Client cannot authenticate via:[TOKEN]" exception. This is expected. > Next, FederationRMFailoverProxyProvider fails over, we reset the service > token via ClientRMProxy.getRMAddress() and reconnect. Supposedly this would > have worked. > However since DefaultRequestInterceptor does not use the proxy user for later > calls to rmClient, when performing failover in > FederationRMFailoverProxyProvider, we are not in the proxy user. Currently > the code solve the problem by reloading the current ugi with all tokens saved > locally in originalTokens in method addOriginalTokens(). The problem is that > the original AMRMToken loaded is no longer accepted by RM, and thus we keep > hitting the "Invalid AMRMToken" exception until AM fails. > The correct way is that rather than saving the original tokens in the proxy > ugi, we save the original ugi itself. Every time we perform failover and > create the new RM proxy, we use the original ugi, which is always loaded with > the up-to-date AMRMToken. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6175) Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867033#comment-15867033
]
Hadoop QA commented on YARN-6175:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
36s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
23s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
39s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 113 unchanged - 3 fixed = 113 total (was 116) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
33s{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 23s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}204m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_121 Failed junit tests |
[jira] [Commented] (YARN-6093) Invalid AMRM token exception when using FederationRMFailoverProxyProvider at AMRMtoken renewal during a RM failover
[
https://issues.apache.org/jira/browse/YARN-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867030#comment-15867030
]
Hadoop QA commented on YARN-6093:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
9s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
28s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 17s{color}
| {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
26s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.federation.policies.router.TestWeightedRandomRouterPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6093 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852692/YARN-6093-YARN-2915.v4.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d555cfc3f872 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / 08dc095 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14949/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14949/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
[jira] [Commented] (YARN-6164) Expose maximum-am-resource-percent to Hadoop clients
[ https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866995#comment-15866995 ] Benson Qiu commented on YARN-6164: -- Version 003: Fix test failures > Expose maximum-am-resource-percent to Hadoop clients > > > Key: YARN-6164 > URL: https://issues.apache.org/jira/browse/YARN-6164 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Benson Qiu > Attachments: YARN-6164.001.patch, YARN-6164.002.patch, > YARN-6164.003.patch > > > To my knowledge, there is no way to programmatically obtain > `yarn.scheduler.capacity.maximum-am-resource-percent`. > I've tried looking at > [YarnClient|https://hadoop.apache.org/docs/r2.7.2/api/index.html?org/apache/hadoop/yarn/client/api/YarnClient.html], > [ResourceManager REST > APIs|https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html], > and > [YarnCommands|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html] > (via bin/yarn) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6164) Expose maximum-am-resource-percent to Hadoop clients
[ https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benson Qiu updated YARN-6164: - Attachment: YARN-6164.003.patch > Expose maximum-am-resource-percent to Hadoop clients > > > Key: YARN-6164 > URL: https://issues.apache.org/jira/browse/YARN-6164 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Benson Qiu > Attachments: YARN-6164.001.patch, YARN-6164.002.patch, > YARN-6164.003.patch > > > To my knowledge, there is no way to programmatically obtain > `yarn.scheduler.capacity.maximum-am-resource-percent`. > I've tried looking at > [YarnClient|https://hadoop.apache.org/docs/r2.7.2/api/index.html?org/apache/hadoop/yarn/client/api/YarnClient.html], > [ResourceManager REST > APIs|https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html], > and > [YarnCommands|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html] > (via bin/yarn) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6192) Move yarn-native-service API records into hadoop-yarn-slider-core module
[
https://issues.apache.org/jira/browse/YARN-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866966#comment-15866966
]
Hadoop QA commented on YARN-6192:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
56s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} yarn-native-services passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 39s{color}
| {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications
generated 5 new + 29 unchanged - 5 fixed = 34 total (was 34) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 18s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications: The patch generated
10 new + 182 unchanged - 10 fixed = 192 total (was 192) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
4s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} hadoop-yarn-slider-core in the patch passed. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services-api
generated 0 new + 0 unchanged - 123 fixed = 0 total (was 123) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-yarn-slider-core in the patch passed. {color}
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
|
| | Class org.apache.slider.api.resource.Application defines non-transient
non-serializable instance field placementPolicy In Application.java:instance
[jira] [Updated] (YARN-6050) AMs can't be scheduled on racks or nodes
[
https://issues.apache.org/jira/browse/YARN-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated YARN-6050:
Attachment: YARN-6050.007.patch
The 007 patch:
- Integrates the AM blacklisting feature by setting the number of nodes for it
to consider for the threshold to the number of "eligible" nodes instead of all
of the nodes (i.e. in the case of a strict locality, only the nodes that
qualify). For example, if you put a strict locality on the "/foo" rack, it
will only count nodes on that rack. And if you put a strict locality on the
"bar" node, it will only count that one node.
- New tests for the above
- I had to add a new method {{getNumClusterNodesByResourceName}} to the
{{YarnScheduler}} interface, similar to the existing {{getNumClusterNodes}}.
-- I wasn't sure what annotations to put, so please check this.
-- While I was here, I also moved the implementation for {{getNumClusterNodes}}
to {{AbstractYarnScheduler}} instead of in each of the other schedulers because
it was duplicate code
- I also did some more manual testing on a real cluster with various options
(different locality requirements, am blacklisting, etc)
[~leftnoteasy], please take a look at the new patch. Thanks.
> AMs can't be scheduled on racks or nodes
>
>
> Key: YARN-6050
> URL: https://issues.apache.org/jira/browse/YARN-6050
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-6050.001.patch, YARN-6050.002.patch,
> YARN-6050.003.patch, YARN-6050.004.patch, YARN-6050.005.patch,
> YARN-6050.006.patch, YARN-6050.007.patch
>
>
> Yarn itself supports rack/node aware scheduling for AMs; however, there
> currently are two problems:
> # To specify hard or soft rack/node requests, you have to specify more than
> one {{ResourceRequest}}. For example, if you want to schedule an AM only on
> "rackA", you have to create two {{ResourceRequest}}, like this:
> {code}
> ResourceRequest.newInstance(PRIORITY, ANY, CAPABILITY, NUM_CONTAINERS, false);
> ResourceRequest.newInstance(PRIORITY, "rackA", CAPABILITY, NUM_CONTAINERS,
> true);
> {code}
> The problem is that the Yarn API doesn't actually allow you to specify more
> than one {{ResourceRequest}} in the {{ApplicationSubmissionContext}}. The
> current behavior is to either build one from {{getResource}} or directly from
> {{getAMContainerResourceRequest}}, depending on if
> {{getAMContainerResourceRequest}} is null or not. We'll need to add a third
> method, say {{getAMContainerResourceRequests}}, which takes a list of
> {{ResourceRequest}} so that clients can specify the multiple resource
> requests.
> # There are some places where things are hardcoded to overwrite what the
> client specifies. These are pretty straightforward to fix.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6061) Add an UncaughtExceptionHandler for critical threads in RM
[ https://issues.apache.org/jira/browse/YARN-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866955#comment-15866955 ] Hudson commented on YARN-6061: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11249 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11249/]) YARN-6061. Addendum. Remove extraneous change. (kasha: rev 353a9b2d9165a221491395edbadf8acc3a39990b) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java > Add an UncaughtExceptionHandler for critical threads in RM > -- > > Key: YARN-6061 > URL: https://issues.apache.org/jira/browse/YARN-6061 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 2.9.0 > > Attachments: YARN-6061.001.patch, YARN-6061.002.patch, > YARN-6061.003.patch, YARN-6061.004.patch, YARN-6061.005.patch, > YARN-6061.006.patch, YARN-6061.007.patch, YARN-6061.008.patch, > YARN-6061.009.patch, YARN-6061.010.patch > > > There are several threads in fair scheduler. The thread will quit when there > is a runtime exception inside it. We should bring down the RM when that > happens. Otherwise, there may be some weird behavior in RM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866937#comment-15866937
]
ASF GitHub Bot commented on YARN-6163:
--
Github user templedf commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101171862
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/VisitedResourceRequestTracker.java
---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Applications place {@link ResourceRequest}s at multiple levels. This is
a
+ * helper class that allows tracking if a {@link ResourceRequest} has been
+ * visited at a different locality level.
+ *
+ * This is implemented for {@link
FSAppAttempt#getStarvedResourceRequests()}.
+ * The implementation is not thread-safe.
+ */
+class VisitedResourceRequestTracker {
+ private final Map>
map =
+ new HashMap<>();
+ private final ClusterNodeTracker nodeTracker;
+
+ VisitedResourceRequestTracker(
+ ClusterNodeTracker nodeTracker) {
+this.nodeTracker = nodeTracker;
+ }
+
+ /**
+ * Check if the {@link ResourceRequest} is visited before, and track it.
+ * @param rr {@link ResourceRequest} to visit
+ * @return true if rr this is the first visit across all
+ * locality levels, false otherwise
+ */
+ boolean visit(ResourceRequest rr) {
+Priority priority = rr.getPriority();
+Resource capability = rr.getCapability();
+
+Map subMap = map.get(priority);
+if (subMap == null) {
+ subMap = new HashMap<>();
+ map.put(priority, subMap);
+}
+
+TrackerPerPriorityResource tracker = subMap.get(capability);
+if (tracker == null) {
+ tracker = new TrackerPerPriorityResource();
+ subMap.put(capability, tracker);
+}
+
+return tracker.visit(rr.getResourceName());
+ }
+
+ private class TrackerPerPriorityResource {
+private Set racksWithNodesVisited = new HashSet<>();
+private Set racksVisted = new HashSet<>();
+private boolean anyVisited;
+
+private boolean visitAny() {
+ if (racksVisted.isEmpty() && racksWithNodesVisited.isEmpty()) {
+anyVisited = true;
+ }
+ return anyVisited;
+}
+
+private boolean visitRack(String rackName) {
+ if (anyVisited || racksWithNodesVisited.contains(rackName)) {
+return false;
+ } else {
+racksVisted.add(rackName);
+return true;
+ }
+}
+
+private boolean visitNode(String rackName) {
+ if (anyVisited || racksVisted.contains(rackName)) {
+return false;
+ } else {
+racksWithNodesVisited.add(rackName);
+return true;
+ }
+}
+
+private boolean visit(String resourceName) {
+ if (resourceName.equals(ResourceRequest.ANY)) {
+return visitAny();
+ }
+
+ List nodes =
+ nodeTracker.getNodesByResourceName(resourceName);
+ switch (nodes.size()) {
--- End diff --
I don't love this as a switch.
> FS Preemption is a trickle for severely starved applications
>
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866936#comment-15866936
]
ASF GitHub Bot commented on YARN-6163:
--
Github user templedf commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101172131
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/VisitedResourceRequestTracker.java
---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Applications place {@link ResourceRequest}s at multiple levels. This is
a
+ * helper class that allows tracking if a {@link ResourceRequest} has been
+ * visited at a different locality level.
+ *
+ * This is implemented for {@link
FSAppAttempt#getStarvedResourceRequests()}.
+ * The implementation is not thread-safe.
+ */
+class VisitedResourceRequestTracker {
+ private final Map>
map =
+ new HashMap<>();
+ private final ClusterNodeTracker nodeTracker;
+
+ VisitedResourceRequestTracker(
+ ClusterNodeTracker nodeTracker) {
+this.nodeTracker = nodeTracker;
+ }
+
+ /**
+ * Check if the {@link ResourceRequest} is visited before, and track it.
+ * @param rr {@link ResourceRequest} to visit
+ * @return true if rr this is the first visit across all
+ * locality levels, false otherwise
+ */
+ boolean visit(ResourceRequest rr) {
+Priority priority = rr.getPriority();
+Resource capability = rr.getCapability();
+
+Map subMap = map.get(priority);
+if (subMap == null) {
+ subMap = new HashMap<>();
+ map.put(priority, subMap);
+}
+
+TrackerPerPriorityResource tracker = subMap.get(capability);
+if (tracker == null) {
+ tracker = new TrackerPerPriorityResource();
+ subMap.put(capability, tracker);
+}
+
+return tracker.visit(rr.getResourceName());
+ }
+
+ private class TrackerPerPriorityResource {
+private Set racksWithNodesVisited = new HashSet<>();
+private Set racksVisted = new HashSet<>();
+private boolean anyVisited;
+
+private boolean visitAny() {
+ if (racksVisted.isEmpty() && racksWithNodesVisited.isEmpty()) {
+anyVisited = true;
+ }
+ return anyVisited;
+}
+
+private boolean visitRack(String rackName) {
+ if (anyVisited || racksWithNodesVisited.contains(rackName)) {
+return false;
+ } else {
+racksVisted.add(rackName);
+return true;
+ }
+}
+
+private boolean visitNode(String rackName) {
--- End diff --
Maybe add Javadoc on these guys to explain the return value? I know it's
already explained above, but doesn't hurt to be clear.
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments:
[jira] [Updated] (YARN-6093) Invalid AMRM token exception when using FederationRMFailoverProxyProvider at AMRMtoken renewal during a RM failover
[ https://issues.apache.org/jira/browse/YARN-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-6093: --- Attachment: YARN-6093-YARN-2915.v4.patch > Invalid AMRM token exception when using FederationRMFailoverProxyProvider at > AMRMtoken renewal during a RM failover > --- > > Key: YARN-6093 > URL: https://issues.apache.org/jira/browse/YARN-6093 > Project: Hadoop YARN > Issue Type: Bug > Components: amrmproxy, federation >Affects Versions: YARN-2915 >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: > YARN-6093-08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch, > YARN-6093-git08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch, > YARN-6093.v1.patch, YARN-6093-YARN-2915.v1.patch, > YARN-6093-YARN-2915.v2.patch, YARN-6093-YARN-2915.v3.patch, > YARN-6093-YARN-2915.v4.patch > > > AMRMProxy uses expired AMRMToken to talk to RM, leading to the "Invalid > AMRMToken" exception. The bug is triggered when both conditions are met: > 1. RM rolls master key and renews AMRMToken for a running AM. > 2. Existing RPC connection between AMRMProxy and RM drops and attempt to > reconnect via failover in FederationRMFailoverProxyProvider. > Here's what happened: > In DefaultRequestInterceptor.init(), we create a proxy ugi, load it with the > initial AMRMToken issued by RM, and used it for initiating rmClient. Then we > arrive at FederationRMFailoverProxyProvider.init(), a full copy of ugi tokens > are saved locally, create an actual RM proxy and setup the RPC connection. > Later when RM rolls master key and issues a new AMRMToken, > DefaultRequestInterceptor.updateAMRMToken() updates it into the proxy ugi. > However the new token is never used, until the existing RPC connection > between AMRMProxy and RM drops for other reasons (say master RM crashes). > When we try to reconnect, since the service name of the new AMRMToken is not > yet set correctly in DefaultRequestInterceptor.updateAMRMToken(), RPC found > no valid AMRMToken when trying to setup a new connection. We first hit a > "Client cannot authenticate via:[TOKEN]" exception. This is expected. > Next, FederationRMFailoverProxyProvider fails over, we reset the service > token via ClientRMProxy.getRMAddress() and reconnect. Supposedly this would > have worked. > However since DefaultRequestInterceptor does not use the proxy user for later > calls to rmClient, when performing failover in > FederationRMFailoverProxyProvider, we are not in the proxy user. Currently > the code solve the problem by reloading the current ugi with all tokens saved > locally in originalTokens in method addOriginalTokens(). The problem is that > the original AMRMToken loaded is no longer accepted by RM, and thus we keep > hitting the "Invalid AMRMToken" exception until AM fails. > The correct way is that rather than saving the original tokens in the proxy > ugi, we save the original ugi itself. Every time we perform failover and > create the new RM proxy, we use the original ugi, which is always loaded with > the up-to-date AMRMToken. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866939#comment-15866939
]
ASF GitHub Bot commented on YARN-6163:
--
Github user templedf commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101172059
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/VisitedResourceRequestTracker.java
---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Applications place {@link ResourceRequest}s at multiple levels. This is
a
+ * helper class that allows tracking if a {@link ResourceRequest} has been
+ * visited at a different locality level.
+ *
+ * This is implemented for {@link
FSAppAttempt#getStarvedResourceRequests()}.
+ * The implementation is not thread-safe.
+ */
+class VisitedResourceRequestTracker {
+ private final Map>
map =
+ new HashMap<>();
+ private final ClusterNodeTracker nodeTracker;
+
+ VisitedResourceRequestTracker(
+ ClusterNodeTracker nodeTracker) {
+this.nodeTracker = nodeTracker;
+ }
+
+ /**
+ * Check if the {@link ResourceRequest} is visited before, and track it.
+ * @param rr {@link ResourceRequest} to visit
+ * @return true if rr this is the first visit across all
--- End diff --
Extraneous "this"
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an
> application starved. This marking happens only on the update call that runs
> every 500ms. Due to this, an application that is severely starved takes
> forever to reach fairshare based on preemptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6093) Invalid AMRM token exception when using FederationRMFailoverProxyProvider at AMRMtoken renewal during a RM failover
[ https://issues.apache.org/jira/browse/YARN-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-6093: --- Attachment: (was: YARN-6093-YARN-2915.v4.patch) > Invalid AMRM token exception when using FederationRMFailoverProxyProvider at > AMRMtoken renewal during a RM failover > --- > > Key: YARN-6093 > URL: https://issues.apache.org/jira/browse/YARN-6093 > Project: Hadoop YARN > Issue Type: Bug > Components: amrmproxy, federation >Affects Versions: YARN-2915 >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: > YARN-6093-08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch, > YARN-6093-git08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch, > YARN-6093.v1.patch, YARN-6093-YARN-2915.v1.patch, > YARN-6093-YARN-2915.v2.patch, YARN-6093-YARN-2915.v3.patch, > YARN-6093-YARN-2915.v4.patch > > > AMRMProxy uses expired AMRMToken to talk to RM, leading to the "Invalid > AMRMToken" exception. The bug is triggered when both conditions are met: > 1. RM rolls master key and renews AMRMToken for a running AM. > 2. Existing RPC connection between AMRMProxy and RM drops and attempt to > reconnect via failover in FederationRMFailoverProxyProvider. > Here's what happened: > In DefaultRequestInterceptor.init(), we create a proxy ugi, load it with the > initial AMRMToken issued by RM, and used it for initiating rmClient. Then we > arrive at FederationRMFailoverProxyProvider.init(), a full copy of ugi tokens > are saved locally, create an actual RM proxy and setup the RPC connection. > Later when RM rolls master key and issues a new AMRMToken, > DefaultRequestInterceptor.updateAMRMToken() updates it into the proxy ugi. > However the new token is never used, until the existing RPC connection > between AMRMProxy and RM drops for other reasons (say master RM crashes). > When we try to reconnect, since the service name of the new AMRMToken is not > yet set correctly in DefaultRequestInterceptor.updateAMRMToken(), RPC found > no valid AMRMToken when trying to setup a new connection. We first hit a > "Client cannot authenticate via:[TOKEN]" exception. This is expected. > Next, FederationRMFailoverProxyProvider fails over, we reset the service > token via ClientRMProxy.getRMAddress() and reconnect. Supposedly this would > have worked. > However since DefaultRequestInterceptor does not use the proxy user for later > calls to rmClient, when performing failover in > FederationRMFailoverProxyProvider, we are not in the proxy user. Currently > the code solve the problem by reloading the current ugi with all tokens saved > locally in originalTokens in method addOriginalTokens(). The problem is that > the original AMRMToken loaded is no longer accepted by RM, and thus we keep > hitting the "Invalid AMRMToken" exception until AM fails. > The correct way is that rather than saving the original tokens in the proxy > ugi, we save the original ugi itself. Every time we perform failover and > create the new RM proxy, we use the original ugi, which is always loaded with > the up-to-date AMRMToken. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866938#comment-15866938
]
ASF GitHub Bot commented on YARN-6163:
--
Github user templedf commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101171746
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/VisitedResourceRequestTracker.java
---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Applications place {@link ResourceRequest}s at multiple levels. This is
a
+ * helper class that allows tracking if a {@link ResourceRequest} has been
+ * visited at a different locality level.
+ *
+ * This is implemented for {@link
FSAppAttempt#getStarvedResourceRequests()}.
+ * The implementation is not thread-safe.
+ */
+class VisitedResourceRequestTracker {
+ private final Map>
map =
+ new HashMap<>();
+ private final ClusterNodeTracker nodeTracker;
+
+ VisitedResourceRequestTracker(
+ ClusterNodeTracker nodeTracker) {
+this.nodeTracker = nodeTracker;
+ }
+
+ /**
+ * Check if the {@link ResourceRequest} is visited before, and track it.
+ * @param rr {@link ResourceRequest} to visit
+ * @return true if rr this is the first visit across all
+ * locality levels, false otherwise
+ */
+ boolean visit(ResourceRequest rr) {
+Priority priority = rr.getPriority();
+Resource capability = rr.getCapability();
+
+Map subMap = map.get(priority);
+if (subMap == null) {
+ subMap = new HashMap<>();
+ map.put(priority, subMap);
+}
+
+TrackerPerPriorityResource tracker = subMap.get(capability);
+if (tracker == null) {
+ tracker = new TrackerPerPriorityResource();
+ subMap.put(capability, tracker);
+}
+
+return tracker.visit(rr.getResourceName());
+ }
+
+ private class TrackerPerPriorityResource {
+private Set racksWithNodesVisited = new HashSet<>();
+private Set racksVisted = new HashSet<>();
+private boolean anyVisited;
+
+private boolean visitAny() {
+ if (racksVisted.isEmpty() && racksWithNodesVisited.isEmpty()) {
+anyVisited = true;
+ }
+ return anyVisited;
+}
+
+private boolean visitRack(String rackName) {
+ if (anyVisited || racksWithNodesVisited.contains(rackName)) {
+return false;
+ } else {
+racksVisted.add(rackName);
+return true;
+ }
+}
+
+private boolean visitNode(String rackName) {
+ if (anyVisited || racksVisted.contains(rackName)) {
+return false;
+ } else {
+racksWithNodesVisited.add(rackName);
+return true;
+ }
+}
+
+private boolean visit(String resourceName) {
+ if (resourceName.equals(ResourceRequest.ANY)) {
+return visitAny();
+ }
+
+ List nodes =
+ nodeTracker.getNodesByResourceName(resourceName);
+ switch (nodes.size()) {
+case 0:
+ // Log error
+ return false;
+case 1:
+ // Node
+
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866940#comment-15866940
]
ASF GitHub Bot commented on YARN-6163:
--
Github user templedf commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101171109
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/VisitedResourceRequestTracker.java
---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Applications place {@link ResourceRequest}s at multiple levels. This is
a
+ * helper class that allows tracking if a {@link ResourceRequest} has been
+ * visited at a different locality level.
+ *
+ * This is implemented for {@link
FSAppAttempt#getStarvedResourceRequests()}.
+ * The implementation is not thread-safe.
+ */
+class VisitedResourceRequestTracker {
+ private final Map>
map =
+ new HashMap<>();
+ private final ClusterNodeTracker nodeTracker;
+
+ VisitedResourceRequestTracker(
+ ClusterNodeTracker nodeTracker) {
+this.nodeTracker = nodeTracker;
+ }
+
+ /**
+ * Check if the {@link ResourceRequest} is visited before, and track it.
+ * @param rr {@link ResourceRequest} to visit
+ * @return true if rr this is the first visit across all
+ * locality levels, false otherwise
+ */
+ boolean visit(ResourceRequest rr) {
+Priority priority = rr.getPriority();
+Resource capability = rr.getCapability();
+
+Map subMap = map.get(priority);
--- End diff --
Maybe use a Guava MultiMap?
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an
> application starved. This marking happens only on the update call that runs
> every 500ms. Due to this, an application that is severely starved takes
> forever to reach fairshare based on preemptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6042) Fairscheduler: Dump scheduler state in log
[
https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866923#comment-15866923
]
Hadoop QA commented on YARN-6042:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 42m
52s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6042 |
| GITHUB PR | https://github.com/apache/hadoop/pull/193 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9df028d97374 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8e53f2b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14947/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14947/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fairscheduler: Dump scheduler state in log
> --
>
> Key: YARN-6042
> URL: https://issues.apache.org/jira/browse/YARN-6042
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6042.001.patch, YARN-6042.002.patch,
> YARN-6042.003.patch, YARN-6042.004.patch, YARN-6042.005.patch
>
>
> To improve the debugging of scheduler issues it would be
[jira] [Updated] (YARN-6192) Move yarn-native-service API records into hadoop-yarn-slider-core module
[ https://issues.apache.org/jira/browse/YARN-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-6192: -- Attachment: YARN-6192.yarn-native-services.02.patch > Move yarn-native-service API records into hadoop-yarn-slider-core module > > > Key: YARN-6192 > URL: https://issues.apache.org/jira/browse/YARN-6192 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6192-yarn-native-services.01.patch, > YARN-6192.yarn-native-services.02.patch > > > Plan to use these records in slider-core -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6191) CapacityScheduler preemption by container priority can be problematic for MapReduce
[ https://issues.apache.org/jira/browse/YARN-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866914#comment-15866914 ] Chris Douglas commented on YARN-6191: - This is related to a [discussion|http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-dev/201702.mbox/%3CCACO5Y4wVm-9_3uES+qVvi2ypzsGTvu9jbEgVfTb79unPH-E=t...@mail.gmail.com%3E] on mapreduce-dev@ on the incomplete, work-conserving preemption logic. The MR AM should react by killing reducers when it gets a preemption message (checkpointing their state, if possible). > CapacityScheduler preemption by container priority can be problematic for > MapReduce > --- > > Key: YARN-6191 > URL: https://issues.apache.org/jira/browse/YARN-6191 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: Jason Lowe > > A MapReduce job with thousands of reducers and just a couple of maps left to > go was running in a preemptable queue. Periodically other queues would get > busy and the RM would preempt some resources from the job, but it _always_ > picked the job's map tasks first because they use the lowest priority > containers. Even though the reducers had a shorter running time, most were > spared but the maps were always shot. Since the map tasks ran for a longer > time than the preemption period, the job was in a perpetual preemption loop. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6061) Add an UncaughtExceptionHandler for critical threads in RM
[ https://issues.apache.org/jira/browse/YARN-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866907#comment-15866907 ] Karthik Kambatla commented on YARN-6061: Included an extraneous change in the original commit. Just pushed an addendum to remove those changes. > Add an UncaughtExceptionHandler for critical threads in RM > -- > > Key: YARN-6061 > URL: https://issues.apache.org/jira/browse/YARN-6061 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 2.9.0 > > Attachments: YARN-6061.001.patch, YARN-6061.002.patch, > YARN-6061.003.patch, YARN-6061.004.patch, YARN-6061.005.patch, > YARN-6061.006.patch, YARN-6061.007.patch, YARN-6061.008.patch, > YARN-6061.009.patch, YARN-6061.010.patch > > > There are several threads in fair scheduler. The thread will quit when there > is a runtime exception inside it. We should bring down the RM when that > happens. Otherwise, there may be some weird behavior in RM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5280) Allow YARN containers to run with Java Security Manager
[
https://issues.apache.org/jira/browse/YARN-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866849#comment-15866849
]
Hadoop QA commented on YARN-5280:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 268 unchanged - 2 fixed = 273 total (was 270) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
58s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5280 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852664/YARN-5280.008.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 5391d5466fbc 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / aaf2713 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14942/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| Test Results |
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866847#comment-15866847
]
ASF GitHub Bot commented on YARN-6163:
--
Github user templedf commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101161807
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestVisitedResourceRequestTracker.java
---
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at*
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import org.apache.hadoop.yarn.server.resourcemanager.MockNodes;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.List;
+
+public class TestVisitedResourceRequestTracker {
+ private final ClusterNodeTracker
+ nodeTracker = new ClusterNodeTracker<>();
+ private final ResourceRequest
+ anyRequest, rackRequest, node1Request, node2Request;
+
+ public TestVisitedResourceRequestTracker() {
+List rmNodes =
+MockNodes.newNodes(1, 2, Resources.createResource(8192, 8));
+
+FSSchedulerNode node1 = new FSSchedulerNode(rmNodes.get(0), false);
+nodeTracker.addNode(node1);
+node1Request = createRR(node1.getNodeName(), 1);
+
+FSSchedulerNode node2 = new FSSchedulerNode(rmNodes.get(1), false);
+node2Request = createRR(node2.getNodeName(), 1);
+nodeTracker.addNode(node2);
+
+anyRequest = createRR(ResourceRequest.ANY, 2);
+rackRequest = createRR(node1.getRackName(), 2);
+ }
+
+ private ResourceRequest createRR(String resourceName, int count) {
+return ResourceRequest.newInstance(
+Priority.UNDEFINED, resourceName, Resources.none(), count);
+ }
+
+ @Test
+ public void testVisitAnyRequestFirst() {
+VisitedResourceRequestTracker tracker =
+new VisitedResourceRequestTracker(nodeTracker);
+
+// Visit ANY request first
+Assert.assertTrue(tracker.visit(anyRequest));
--- End diff --
assertTrue() without a message is evil
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an
> application starved. This marking happens only on the update call that runs
> every 500ms. Due to this, an application that is severely starved takes
> forever to reach fairshare based on preemptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866848#comment-15866848
]
ASF GitHub Bot commented on YARN-6163:
--
Github user templedf commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101161043
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/VisitedResourceRequestTracker.java
---
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Applications place {@link ResourceRequest}s at multiple levels. This is
a
+ * helper class that allows tracking if a {@link ResourceRequest} has been
+ * visited at a different locality level.
+ *
+ * This is implemented for {@link
FSAppAttempt#getStarvedResourceRequests()}.
+ * The implementation is not thread-safe.
--- End diff --
I love you.
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an
> application starved. This marking happens only on the update call that runs
> every 500ms. Due to this, an application that is severely starved takes
> forever to reach fairshare based on preemptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866841#comment-15866841
]
ASF GitHub Bot commented on YARN-6163:
--
Github user templedf commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101159725
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
---
@@ -1147,24 +1147,32 @@ private static boolean checkAndMarkRRVisited(
* starvation.
*/
List getStarvedResourceRequests() {
+// List of RRs we build in this method to return
List ret = new ArrayList<>();
+
+// Track visited RRs to avoid the same RR at multiple locality levels
Map visitedRRs= new HashMap<>();
+// Start with current starvation and track the pending amount
Resource pending = getStarvation();
for (ResourceRequest rr : appSchedulingInfo.getAllResourceRequests()) {
if (Resources.isNone(pending)) {
+// Found enough RRs to match the starvation
break;
}
+
+ // See if we have already seen this RR
if (checkAndMarkRRVisited(visitedRRs, rr)) {
continue;
}
- // Compute the number of containers of this capability that fit in
the
- // pending amount
+ // A RR can have multiple containers of a capability. We need to
+ // compute the number of containers that fit in "pending".
int ratio = (int) Math.floor(
--- End diff --
Given that ratio is the number of containers that fit in "pending," ratio
is probably a bad name. That was a good chunk of my initial confusion.
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an
> application starved. This marking happens only on the update call that runs
> every 500ms. Due to this, an application that is severely starved takes
> forever to reach fairshare based on preemptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866842#comment-15866842
]
ASF GitHub Bot commented on YARN-6163:
--
Github user templedf commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/192#discussion_r101160719
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
---
@@ -247,23 +247,23 @@ private Resource updateStarvedAppsFairshare(
* @param minShareStarvation minshare starvation to distribute
*/
private void updateStarvedAppsMinshare(
- TreeSet appsWithDemand, Resource minShareStarvation) {
+ TreeSet appsWithDemand, final Resource
minShareStarvation) {
--- End diff --
I would make both or neither final
> FS Preemption is a trickle for severely starved applications
>
>
> Key: YARN-6163
> URL: https://issues.apache.org/jira/browse/YARN-6163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-6163-1.patch, yarn-6163-2.patch
>
>
> With current logic, only one RR is considered per each instance of marking an
> application starved. This marking happens only on the update call that runs
> every 500ms. Due to this, an application that is severely starved takes
> forever to reach fairshare based on preemptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6190) Bug fixes in federation polices
[
https://issues.apache.org/jira/browse/YARN-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866837#comment-15866837
]
Hadoop QA commented on YARN-6190:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
10s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
5s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 34s{color}
| {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
|
| | Inconsistent synchronization of
org.apache.hadoop.yarn.server.federation.policies.amrmproxy.LocalityMulticastAMRMProxyPolicy.headroom;
locked 85% of time Unsynchronized access at
LocalityMulticastAMRMProxyPolicy.java:85% of time Unsynchronized access at
LocalityMulticastAMRMProxyPolicy.java:[line 463] |
| | Inconsistent synchronization of
org.apache.hadoop.yarn.server.federation.policies.amrmproxy.LocalityMulticastAMRMProxyPolicy.weights;
locked 66% of time Unsynchronized access at
LocalityMulticastAMRMProxyPolicy.java:66% of time Unsynchronized access at
LocalityMulticastAMRMProxyPolicy.java:[line 448] |
| Failed junit tests |
hadoop.yarn.server.federation.policies.router.TestWeightedRandomRouterPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6190 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852670/YARN-6190-YARN-2915.v1.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d81b3d166353 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / 08dc095 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| findbugs
[jira] [Commented] (YARN-6171) ConcurrentModificationException in ApplicationMasterService.allocate
[
https://issues.apache.org/jira/browse/YARN-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866830#comment-15866830
]
Hadoop QA commented on YARN-6171:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 39 unchanged - 1 fixed = 39 total (was 40) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 20s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6171 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852663/YARN-6171.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 325ad9e616b3 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / aaf2713 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14941/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14941/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14941/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Updated] (YARN-6042) Fairscheduler: Dump scheduler state in log
[ https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6042: --- Attachment: YARN-6042.005.patch Patch v5 is a rebase. The upstream changes too fast. > Fairscheduler: Dump scheduler state in log > -- > > Key: YARN-6042 > URL: https://issues.apache.org/jira/browse/YARN-6042 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6042.001.patch, YARN-6042.002.patch, > YARN-6042.003.patch, YARN-6042.004.patch, YARN-6042.005.patch > > > To improve the debugging of scheduler issues it would be a big improvement to > be able to dump the scheduler state into a log on request. > The Dump the scheduler state at a point in time would allow debugging of a > scheduler that is not hung (deadlocked) but also not assigning containers. > Currently we do not have a proper overview of what state the scheduler and > the queues are in and we have to make assumptions or guess > The scheduler and queue state needed would include (not exhaustive): > - instantaneous and steady fair share (app / queue) > - AM share and resources > - weight > - app demand > - application run state (runnable/non runnable) > - last time at fair/min share -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6190) Bug fixes in federation polices
[ https://issues.apache.org/jira/browse/YARN-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866793#comment-15866793 ] Carlo Curino edited comment on YARN-6190 at 2/14/17 10:04 PM: -- Hi [~botong], Thanks for the patch, it looks good in general. I believe that: # The synchronized are probably redundant (the invoker is single threaded, and we guarded headroom, but making it ConcurrentHashMap... the only thing to double check is whether/if the scan of headroom is compatible with a put). # Don't fix java 8 / java 7 issues like ArralyList<> in this, worth doing only if/when we backport with java7 only branches # I think you are correct on using ResourceRequest instead of instance when invoking static methods (cleaner) # I think the null check for weights is also a correct fix. Please add a test to verify that we cover this. was (Author: curino): Hi [~botong], Thanks for the patch, it looks good in general. I believe that: * The synchronized are probably redundant (the invoker is single threaded, and we guarded headroom, but making it ConcurrentHashMap... the only thing to double check is whether/if the scan of headroom is compatible with a put). * Don't fix java 8 / java 7 issues like ArralyList<> in this, worth doing only if/when we backport with java7 only branches * I think you are correct on using ResourceRequest instead of instance when invoking static methods (cleaner) * I think the null check for weights is also a correct fix. Please add a test to verify that we cover this. > Bug fixes in federation polices > --- > > Key: YARN-6190 > URL: https://issues.apache.org/jira/browse/YARN-6190 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-6190-YARN-2915.v1.patch > > > A bug fix in LocalityMulticastAMRMProxyPolicy on policy array condition > check, along with misc cleanups. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6190) Bug fixes in federation polices
[ https://issues.apache.org/jira/browse/YARN-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866793#comment-15866793 ] Carlo Curino edited comment on YARN-6190 at 2/14/17 10:03 PM: -- Hi [~botong], Thanks for the patch, it looks good in general. I believe that: * The synchronized are probably redundant (the invoker is single threaded, and we guarded headroom, but making it ConcurrentHashMap... the only thing to double check is whether/if the scan of headroom is compatible with a put). * Don't fix java 8 / java 7 issues like ArralyList<> in this, worth doing only if/when we backport with java7 only branches * I think you are correct on using ResourceRequest instead of instance when invoking static methods (cleaner) * I think the null check for weights is also a correct fix. Please add a test to verify that we cover this. was (Author: curino): Hi [~botong], Thanks for the patch, it looks good in general. I believe that: 1) The synchronized are probably redundant (the invoker is single threaded, and we guarded headroom, but making it ConcurrentHashMap... the only thing to double check is whether/if the scan of headroom is compatible with a put). 2) Don't fix java 8 / java 7 issues like ArralyList<> in this, worth doing only if/when we backport with java7 only branches 3) I think you are correct on using ResourceRequest instead of instance when invoking static methods (cleaner) 4) I think the null check for weights is also a correct fix. Please add a test to verify that we cover this. > Bug fixes in federation polices > --- > > Key: YARN-6190 > URL: https://issues.apache.org/jira/browse/YARN-6190 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-6190-YARN-2915.v1.patch > > > A bug fix in LocalityMulticastAMRMProxyPolicy on policy array condition > check, along with misc cleanups. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6190) Bug fixes in federation polices
[ https://issues.apache.org/jira/browse/YARN-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866793#comment-15866793 ] Carlo Curino commented on YARN-6190: Hi [~botong], Thanks for the patch, it looks good in general. I believe that: 1) The synchronized are probably redundant (the invoker is single threaded, and we guarded headroom, but making it ConcurrentHashMap... the only thing to double check is whether/if the scan of headroom is compatible with a put). 2) Don't fix java 8 / java 7 issues like ArralyList<> in this, worth doing only if/when we backport with java7 only branches 3) I think you are correct on using ResourceRequest instead of instance when invoking static methods (cleaner) 4) I think the null check for weights is also a correct fix. Please add a test to verify that we cover this. > Bug fixes in federation polices > --- > > Key: YARN-6190 > URL: https://issues.apache.org/jira/browse/YARN-6190 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-6190-YARN-2915.v1.patch > > > A bug fix in LocalityMulticastAMRMProxyPolicy on policy array condition > check, along with misc cleanups. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6193) DominantResourceFairnessPolicy#checkIfUsageOverFairShare considers all resources instead of just dominant resource
Karthik Kambatla created YARN-6193:
--
Summary: DominantResourceFairnessPolicy#checkIfUsageOverFairShare
considers all resources instead of just dominant resource
Key: YARN-6193
URL: https://issues.apache.org/jira/browse/YARN-6193
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.8.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
{{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} considers all
resources instead of just dominant resource. Ideally, it should use the
DominantResourceCalculator.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6061) Add an UncaughtExceptionHandler for critical threads in RM
[ https://issues.apache.org/jira/browse/YARN-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866776#comment-15866776 ] Hudson commented on YARN-6061: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11248 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11248/]) YARN-6061. Add an UncaughtExceptionHandler for critical threads in RM. (kasha: rev 652679aa8ad6f9e61b8ed8e2b04b3e0332025e94) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMCriticalThreadUncaughtExceptionHandler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMFatalEventType.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java > Add an UncaughtExceptionHandler for critical threads in RM > -- > > Key: YARN-6061 > URL: https://issues.apache.org/jira/browse/YARN-6061 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 2.9.0 > > Attachments: YARN-6061.001.patch, YARN-6061.002.patch, > YARN-6061.003.patch, YARN-6061.004.patch, YARN-6061.005.patch, > YARN-6061.006.patch, YARN-6061.007.patch, YARN-6061.008.patch, > YARN-6061.009.patch, YARN-6061.010.patch > > > There are several threads in fair scheduler. The thread will quit when there > is a runtime exception inside it. We should bring down the RM when that > happens. Otherwise, there may be some weird behavior in RM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6061) Add an UncaughtExceptionHandler for critical threads in RM
[ https://issues.apache.org/jira/browse/YARN-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866773#comment-15866773 ] Yufei Gu commented on YARN-6061: Yeah! Thanks [~kasha] for review, advice and commit. > Add an UncaughtExceptionHandler for critical threads in RM > -- > > Key: YARN-6061 > URL: https://issues.apache.org/jira/browse/YARN-6061 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 2.9.0 > > Attachments: YARN-6061.001.patch, YARN-6061.002.patch, > YARN-6061.003.patch, YARN-6061.004.patch, YARN-6061.005.patch, > YARN-6061.006.patch, YARN-6061.007.patch, YARN-6061.008.patch, > YARN-6061.009.patch, YARN-6061.010.patch > > > There are several threads in fair scheduler. The thread will quit when there > is a runtime exception inside it. We should bring down the RM when that > happens. Otherwise, there may be some weird behavior in RM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6192) Move yarn-native-service API records into hadoop-yarn-slider-core module
[
https://issues.apache.org/jira/browse/YARN-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866772#comment-15866772
]
Hadoop QA commented on YARN-6192:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
1s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-6192 does not apply to yarn-native-services. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6192 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852672/YARN-6192-yarn-native-services.01.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14945/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Move yarn-native-service API records into hadoop-yarn-slider-core module
>
>
> Key: YARN-6192
> URL: https://issues.apache.org/jira/browse/YARN-6192
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-6192-yarn-native-services.01.patch
>
>
> Plan to use these records in slider-core
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
