[jira] [Commented] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575924#comment-14575924
]
Hadoop QA commented on YARN-3655:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 2s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 23s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 4s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 50m 17s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 86m 55s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738190/YARN-3655.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 71de367 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8208/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8208/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8208/console |
This message was automatically generated.
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch, YARN-3655.004.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3778) Fix Yarn resourcemanger CLI usage
Brahma Reddy Battula created YARN-3778:
--
Summary: Fix Yarn resourcemanger CLI usage
Key: YARN-3778
URL: https://issues.apache.org/jira/browse/YARN-3778
Project: Hadoop YARN
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
The usage message from code does not match with the one documented.
1. java ResourceManager should be yarn resourcemanager
{code}
private static void printUsage(PrintStream out) {
out.println(Usage: java ResourceManager [-format-state-store]);
out.println(
+ [-remove-application-from-state-store appId] + \n);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated YARN-3655:
Attachment: (was: YARN-3655.004.patch)
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch, YARN-3655.004.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated YARN-3655:
Attachment: YARN-3655.004.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch, YARN-3655.004.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3778) Fix Yarn resourcemanger CLI usage
[
https://issues.apache.org/jira/browse/YARN-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated YARN-3778:
---
Attachment: YARN-3778.patch
Attaching the patch..Kindly review..
Fix Yarn resourcemanger CLI usage
-
Key: YARN-3778
URL: https://issues.apache.org/jira/browse/YARN-3778
Project: Hadoop YARN
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
Attachments: YARN-3778.patch
The usage message from code does not match with the one documented.
1. java ResourceManager should be yarn resourcemanager
{code}
private static void printUsage(PrintStream out) {
out.println(Usage: java ResourceManager [-format-state-store]);
out.println(
+ [-remove-application-from-state-store appId] + \n);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3259) FairScheduler: Trigger fairShare updates on node events
[ https://issues.apache.org/jira/browse/YARN-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575678#comment-14575678 ] Hudson commented on YARN-3259: -- FAILURE: Integrated in Hadoop-Yarn-trunk #950 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/950/]) YARN-3259. FairScheduler: Trigger fairShare updates on node events. (Anubhav Dhoot via kasha) (kasha: rev 75885852cc19dd6de12e62498b112d5d70ce87f4) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestSchedulingUpdate.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSOpDurations.java FairScheduler: Trigger fairShare updates on node events --- Key: YARN-3259 URL: https://issues.apache.org/jira/browse/YARN-3259 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Fix For: 2.8.0 Attachments: YARN-3259.001.patch, YARN-3259.002.patch, YARN-3259.003.patch Instead of waiting for update interval unconditionally, we can trigger early updates on important events - for eg node join and leave. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1462) AHS API and other AHS changes to handle tags for completed MR jobs
[ https://issues.apache.org/jira/browse/YARN-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575677#comment-14575677 ] Hudson commented on YARN-1462: -- FAILURE: Integrated in Hadoop-Yarn-trunk #950 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/950/]) YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. Contributed by Xuan Gong (xgong: rev 3e000a919fede85230fcfb06309a1f1d5e0c479c) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/ApplicationCreatedEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java AHS API and other AHS changes to handle tags for completed MR jobs -- Key: YARN-1462 URL: https://issues.apache.org/jira/browse/YARN-1462 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Xuan Gong Fix For: 2.8.0 Attachments: YARN-1462-branch-2.7-1.2.patch, YARN-1462-branch-2.7-1.patch, YARN-1462.1.patch, YARN-1462.2.patch, YARN-1462.3.patch, YARN-1462.4.patch AHS related work for tags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3706) Generalize native HBase writer for additional tables
[ https://issues.apache.org/jira/browse/YARN-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated YARN-3706: --- Attachment: YARN-3726-YARN-2928.007.patch YARN-3726-YARN-2928.007.patch : Renamed ColumnImpl to ColumnHelper to clarify relationship to *ColumnT instances of interface Columnt TimelineWriterUtils.encode needed two things per separator, the original and the URLencoded version. I like the idea of not having to URLEncode the separators over and over again, so I made separators a separate class with their own behavior. That ends up working out much cleaner now. No more URLEncoding and Decoding on the fly over and over. Byte representation is calculated once and reused. Encoding and decoding logic moved to Separator class, which is nicer than stashing too many things in the helper class. Also found out that URLEncoding does actually HTML encoding and encodes space as plus, which is an issue when the values contain a plus. Chose different encoding that does not seem likely to be used in source, so even URL encoded strings can be used as identifiers now. Patch is ready for review again. Generalize native HBase writer for additional tables Key: YARN-3706 URL: https://issues.apache.org/jira/browse/YARN-3706 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Joep Rottinghuis Assignee: Joep Rottinghuis Priority: Minor Attachments: YARN-3706-YARN-2928.001.patch, YARN-3726-YARN-2928.002.patch, YARN-3726-YARN-2928.003.patch, YARN-3726-YARN-2928.004.patch, YARN-3726-YARN-2928.005.patch, YARN-3726-YARN-2928.006.patch, YARN-3726-YARN-2928.007.patch When reviewing YARN-3411 we noticed that we could change the class hierarchy a little in order to accommodate additional tables easily. In order to get ready for benchmark testing we left the original layout in place, as performance would not be impacted by the code hierarchy. Here is a separate jira to address the hierarchy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3259) FairScheduler: Trigger fairShare updates on node events
[ https://issues.apache.org/jira/browse/YARN-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575669#comment-14575669 ] Hudson commented on YARN-3259: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #220 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/220/]) YARN-3259. FairScheduler: Trigger fairShare updates on node events. (Anubhav Dhoot via kasha) (kasha: rev 75885852cc19dd6de12e62498b112d5d70ce87f4) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestSchedulingUpdate.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSOpDurations.java FairScheduler: Trigger fairShare updates on node events --- Key: YARN-3259 URL: https://issues.apache.org/jira/browse/YARN-3259 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Fix For: 2.8.0 Attachments: YARN-3259.001.patch, YARN-3259.002.patch, YARN-3259.003.patch Instead of waiting for update interval unconditionally, we can trigger early updates on important events - for eg node join and leave. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1462) AHS API and other AHS changes to handle tags for completed MR jobs
[ https://issues.apache.org/jira/browse/YARN-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575668#comment-14575668 ] Hudson commented on YARN-1462: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #220 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/220/]) YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. Contributed by Xuan Gong (xgong: rev 3e000a919fede85230fcfb06309a1f1d5e0c479c) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/ApplicationCreatedEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java AHS API and other AHS changes to handle tags for completed MR jobs -- Key: YARN-1462 URL: https://issues.apache.org/jira/browse/YARN-1462 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Xuan Gong Fix For: 2.8.0 Attachments: YARN-1462-branch-2.7-1.2.patch, YARN-1462-branch-2.7-1.patch, YARN-1462.1.patch, YARN-1462.2.patch, YARN-1462.3.patch, YARN-1462.4.patch AHS related work for tags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3259) FairScheduler: Trigger fairShare updates on node events
[ https://issues.apache.org/jira/browse/YARN-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575741#comment-14575741 ] Hudson commented on YARN-3259: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2148 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2148/]) YARN-3259. FairScheduler: Trigger fairShare updates on node events. (Anubhav Dhoot via kasha) (kasha: rev 75885852cc19dd6de12e62498b112d5d70ce87f4) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSOpDurations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestSchedulingUpdate.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java FairScheduler: Trigger fairShare updates on node events --- Key: YARN-3259 URL: https://issues.apache.org/jira/browse/YARN-3259 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Fix For: 2.8.0 Attachments: YARN-3259.001.patch, YARN-3259.002.patch, YARN-3259.003.patch Instead of waiting for update interval unconditionally, we can trigger early updates on important events - for eg node join and leave. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1462) AHS API and other AHS changes to handle tags for completed MR jobs
[ https://issues.apache.org/jira/browse/YARN-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575740#comment-14575740 ] Hudson commented on YARN-1462: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2148 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2148/]) YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. Contributed by Xuan Gong (xgong: rev 3e000a919fede85230fcfb06309a1f1d5e0c479c) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/ApplicationCreatedEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java AHS API and other AHS changes to handle tags for completed MR jobs -- Key: YARN-1462 URL: https://issues.apache.org/jira/browse/YARN-1462 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Xuan Gong Fix For: 2.8.0 Attachments: YARN-1462-branch-2.7-1.2.patch, YARN-1462-branch-2.7-1.patch, YARN-1462.1.patch, YARN-1462.2.patch, YARN-1462.3.patch, YARN-1462.4.patch AHS related work for tags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3259) FairScheduler: Trigger fairShare updates on node events
[ https://issues.apache.org/jira/browse/YARN-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575778#comment-14575778 ] Hudson commented on YARN-3259: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2166 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2166/]) YARN-3259. FairScheduler: Trigger fairShare updates on node events. (Anubhav Dhoot via kasha) (kasha: rev 75885852cc19dd6de12e62498b112d5d70ce87f4) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSOpDurations.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestSchedulingUpdate.java FairScheduler: Trigger fairShare updates on node events --- Key: YARN-3259 URL: https://issues.apache.org/jira/browse/YARN-3259 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Fix For: 2.8.0 Attachments: YARN-3259.001.patch, YARN-3259.002.patch, YARN-3259.003.patch Instead of waiting for update interval unconditionally, we can trigger early updates on important events - for eg node join and leave. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1462) AHS API and other AHS changes to handle tags for completed MR jobs
[ https://issues.apache.org/jira/browse/YARN-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575777#comment-14575777 ] Hudson commented on YARN-1462: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2166 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2166/]) YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. Contributed by Xuan Gong (xgong: rev 3e000a919fede85230fcfb06309a1f1d5e0c479c) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/ApplicationCreatedEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java AHS API and other AHS changes to handle tags for completed MR jobs -- Key: YARN-1462 URL: https://issues.apache.org/jira/browse/YARN-1462 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Xuan Gong Fix For: 2.8.0 Attachments: YARN-1462-branch-2.7-1.2.patch, YARN-1462-branch-2.7-1.patch, YARN-1462.1.patch, YARN-1462.2.patch, YARN-1462.3.patch, YARN-1462.4.patch AHS related work for tags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1462) AHS API and other AHS changes to handle tags for completed MR jobs
[ https://issues.apache.org/jira/browse/YARN-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575748#comment-14575748 ] Hudson commented on YARN-1462: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #209 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/209/]) YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. Contributed by Xuan Gong (xgong: rev 3e000a919fede85230fcfb06309a1f1d5e0c479c) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/ApplicationCreatedEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java AHS API and other AHS changes to handle tags for completed MR jobs -- Key: YARN-1462 URL: https://issues.apache.org/jira/browse/YARN-1462 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Xuan Gong Fix For: 2.8.0 Attachments: YARN-1462-branch-2.7-1.2.patch, YARN-1462-branch-2.7-1.patch, YARN-1462.1.patch, YARN-1462.2.patch, YARN-1462.3.patch, YARN-1462.4.patch AHS related work for tags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3259) FairScheduler: Trigger fairShare updates on node events
[ https://issues.apache.org/jira/browse/YARN-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575749#comment-14575749 ] Hudson commented on YARN-3259: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #209 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/209/]) YARN-3259. FairScheduler: Trigger fairShare updates on node events. (Anubhav Dhoot via kasha) (kasha: rev 75885852cc19dd6de12e62498b112d5d70ce87f4) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSOpDurations.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestSchedulingUpdate.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java FairScheduler: Trigger fairShare updates on node events --- Key: YARN-3259 URL: https://issues.apache.org/jira/browse/YARN-3259 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Fix For: 2.8.0 Attachments: YARN-3259.001.patch, YARN-3259.002.patch, YARN-3259.003.patch Instead of waiting for update interval unconditionally, we can trigger early updates on important events - for eg node join and leave. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3259) FairScheduler: Trigger fairShare updates on node events
[ https://issues.apache.org/jira/browse/YARN-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575767#comment-14575767 ] Hudson commented on YARN-3259: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #218 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/218/]) YARN-3259. FairScheduler: Trigger fairShare updates on node events. (Anubhav Dhoot via kasha) (kasha: rev 75885852cc19dd6de12e62498b112d5d70ce87f4) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSOpDurations.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestSchedulingUpdate.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java FairScheduler: Trigger fairShare updates on node events --- Key: YARN-3259 URL: https://issues.apache.org/jira/browse/YARN-3259 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Fix For: 2.8.0 Attachments: YARN-3259.001.patch, YARN-3259.002.patch, YARN-3259.003.patch Instead of waiting for update interval unconditionally, we can trigger early updates on important events - for eg node join and leave. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1462) AHS API and other AHS changes to handle tags for completed MR jobs
[ https://issues.apache.org/jira/browse/YARN-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575766#comment-14575766 ] Hudson commented on YARN-1462: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #218 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/218/]) YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. Contributed by Xuan Gong (xgong: rev 3e000a919fede85230fcfb06309a1f1d5e0c479c) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/ApplicationCreatedEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md * hadoop-yarn-project/CHANGES.txt AHS API and other AHS changes to handle tags for completed MR jobs -- Key: YARN-1462 URL: https://issues.apache.org/jira/browse/YARN-1462 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Xuan Gong Fix For: 2.8.0 Attachments: YARN-1462-branch-2.7-1.2.patch, YARN-1462-branch-2.7-1.patch, YARN-1462.1.patch, YARN-1462.2.patch, YARN-1462.3.patch, YARN-1462.4.patch AHS related work for tags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3706) Generalize native HBase writer for additional tables
[
https://issues.apache.org/jira/browse/YARN-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575830#comment-14575830
]
Hadoop QA commented on YARN-3706:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 58s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 42s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 49s | The applied patch generated 7
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 16s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 11s | The patch has 3 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 40s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 0m 52s | The patch appears to introduce 5
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 1m 15s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 38m 49s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-timelineservice |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738152/YARN-3726-YARN-2928.007.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / d88f30b |
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/8203/artifact/patchprocess/diffJavadocWarnings.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8203/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8203/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-timelineservice.html
|
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8203/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8203/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8203/console |
This message was automatically generated.
Generalize native HBase writer for additional tables
Key: YARN-3706
URL: https://issues.apache.org/jira/browse/YARN-3706
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Minor
Attachments: YARN-3706-YARN-2928.001.patch,
YARN-3726-YARN-2928.002.patch, YARN-3726-YARN-2928.003.patch,
YARN-3726-YARN-2928.004.patch, YARN-3726-YARN-2928.005.patch,
YARN-3726-YARN-2928.006.patch, YARN-3726-YARN-2928.007.patch
When reviewing YARN-3411 we noticed that we could change the class hierarchy
a little in order to accommodate additional tables easily.
In order to get ready for benchmark testing we left the original layout in
place, as performance would not be impacted by the code hierarchy.
Here is a separate jira to address the hierarchy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575992#comment-14575992
]
zhihai xu commented on YARN-3655:
-
Updated the patch to fix the checkstyle issue. The latest patch
YARN-3655.004.patch passed the Jenkins test.
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch, YARN-3655.004.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated YARN-3655:
Attachment: (was: YARN-3655.004.patch)
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated YARN-3655:
Attachment: YARN-3655.004.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch, YARN-3655.004.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575983#comment-14575983
]
Hadoop QA commented on YARN-3655:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 57s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 45s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 50m 13s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 88m 11s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738203/YARN-3655.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 71de367 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8209/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8209/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8209/console |
This message was automatically generated.
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch, YARN-3655.004.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3748) Cleanup Findbugs volatile warnings
[ https://issues.apache.org/jira/browse/YARN-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576108#comment-14576108 ] Sean Busbey commented on YARN-3748: --- +1 v5 looks good. Cleanup Findbugs volatile warnings -- Key: YARN-3748 URL: https://issues.apache.org/jira/browse/YARN-3748 Project: Hadoop YARN Issue Type: Bug Reporter: Gabor Liptak Priority: Minor Attachments: YARN-3748.1.patch, YARN-3748.2.patch, YARN-3748.3.patch, YARN-3748.5.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3748) Cleanup Findbugs volatile warnings
[
https://issues.apache.org/jira/browse/YARN-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575847#comment-14575847
]
Hadoop QA commented on YARN-3748:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 48s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 56s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 47s | The applied patch generated 1
new checkstyle issues (total was 227, now 226). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 28s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 50m 42s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 89m 50s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738108/YARN-3748.5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e46cb80 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8205/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8205/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8205/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8205/console |
This message was automatically generated.
Cleanup Findbugs volatile warnings
--
Key: YARN-3748
URL: https://issues.apache.org/jira/browse/YARN-3748
Project: Hadoop YARN
Issue Type: Bug
Reporter: Gabor Liptak
Priority: Minor
Attachments: YARN-3748.1.patch, YARN-3748.2.patch, YARN-3748.3.patch,
YARN-3748.5.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3748) Cleanup Findbugs volatile warnings
[ https://issues.apache.org/jira/browse/YARN-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575863#comment-14575863 ] Gabor Liptak commented on YARN-3748: [~busbey] The tests did pass this time. Cleanup Findbugs volatile warnings -- Key: YARN-3748 URL: https://issues.apache.org/jira/browse/YARN-3748 Project: Hadoop YARN Issue Type: Bug Reporter: Gabor Liptak Priority: Minor Attachments: YARN-3748.1.patch, YARN-3748.2.patch, YARN-3748.3.patch, YARN-3748.5.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated YARN-3655:
Attachment: YARN-3655.004.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch, YARN-3655.004.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2716) Refactor ZKRMStateStore retry code with Apache Curator
[
https://issues.apache.org/jira/browse/YARN-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575848#comment-14575848
]
Hadoop QA commented on YARN-2716:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 54s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 7m 49s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 56s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 24s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 6s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 1s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 30s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 49m 18s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 91m 36s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738095/yarn-2716-3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e46cb80 |
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8204/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8204/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8204/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8204/console |
This message was automatically generated.
Refactor ZKRMStateStore retry code with Apache Curator
--
Key: YARN-2716
URL: https://issues.apache.org/jira/browse/YARN-2716
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Karthik Kambatla
Attachments: yarn-2716-1.patch, yarn-2716-2.patch, yarn-2716-3.patch,
yarn-2716-prelim.patch, yarn-2716-prelim.patch, yarn-2716-super-prelim.patch
Per suggestion by [~kasha] in YARN-2131, it's nice to use curator to
simplify the retry logic in ZKRMStateStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3778) Fix Yarn resourcemanger CLI usage
[
https://issues.apache.org/jira/browse/YARN-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575846#comment-14575846
]
Hadoop QA commented on YARN-3778:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 55s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 44s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 24s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 50m 15s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 88m 5s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738148/YARN-3778.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b3ffa87 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8201/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8201/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8201/console |
This message was automatically generated.
Fix Yarn resourcemanger CLI usage
-
Key: YARN-3778
URL: https://issues.apache.org/jira/browse/YARN-3778
Project: Hadoop YARN
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
Attachments: YARN-3778.patch
The usage message from code does not match with the one documented.
1. java ResourceManager should be yarn resourcemanager
{code}
private static void printUsage(PrintStream out) {
out.println(Usage: java ResourceManager [-format-state-store]);
out.println(
+ [-remove-application-from-state-store appId] + \n);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575845#comment-14575845
]
Hadoop QA commented on YARN-3655:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 51s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 46s | The applied patch generated 1
new checkstyle issues (total was 122, now 119). |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 24s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 50m 12s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 87m 55s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738145/YARN-3655.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b3ffa87 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8202/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8202/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8202/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8202/console |
This message was automatically generated.
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch, YARN-3655.004.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3706) Generalize native HBase writer for additional tables
[ https://issues.apache.org/jira/browse/YARN-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated YARN-3706: --- Attachment: YARN-3726-YARN-2928.008.patch YARN-3726-YARN-2928.008.patch : Fixed spaces, pulled in branch changes (no conflicts). FindBugs 3.0.1 is showing no warnings in Eclipse and I can't seem to spot the 5 new ones this patch is supposed to have introduced in the FindBugs warnings report above, it might be that FindBugs setup is already broken on this branch in Jenkins? I'll submit a new patch and see what happens. Generalize native HBase writer for additional tables Key: YARN-3706 URL: https://issues.apache.org/jira/browse/YARN-3706 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Joep Rottinghuis Assignee: Joep Rottinghuis Priority: Minor Attachments: YARN-3706-YARN-2928.001.patch, YARN-3726-YARN-2928.002.patch, YARN-3726-YARN-2928.003.patch, YARN-3726-YARN-2928.004.patch, YARN-3726-YARN-2928.005.patch, YARN-3726-YARN-2928.006.patch, YARN-3726-YARN-2928.007.patch, YARN-3726-YARN-2928.008.patch When reviewing YARN-3411 we noticed that we could change the class hierarchy a little in order to accommodate additional tables easily. In order to get ready for benchmark testing we left the original layout in place, as performance would not be impacted by the code hierarchy. Here is a separate jira to address the hierarchy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3706) Generalize native HBase writer for additional tables
[ https://issues.apache.org/jira/browse/YARN-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575858#comment-14575858 ] Joep Rottinghuis commented on YARN-3706: I'll fix the JavaDoc warnings later today. Generalize native HBase writer for additional tables Key: YARN-3706 URL: https://issues.apache.org/jira/browse/YARN-3706 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Joep Rottinghuis Assignee: Joep Rottinghuis Priority: Minor Attachments: YARN-3706-YARN-2928.001.patch, YARN-3726-YARN-2928.002.patch, YARN-3726-YARN-2928.003.patch, YARN-3726-YARN-2928.004.patch, YARN-3726-YARN-2928.005.patch, YARN-3726-YARN-2928.006.patch, YARN-3726-YARN-2928.007.patch, YARN-3726-YARN-2928.008.patch When reviewing YARN-3411 we noticed that we could change the class hierarchy a little in order to accommodate additional tables easily. In order to get ready for benchmark testing we left the original layout in place, as performance would not be impacted by the code hierarchy. Here is a separate jira to address the hierarchy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-574) PrivateLocalizer does not support parallel resource download via ContainerLocalizer
[
https://issues.apache.org/jira/browse/YARN-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575869#comment-14575869
]
Hadoop QA commented on YARN-574:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 21s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 11s | The applied patch generated 1
new checkstyle issues (total was 213, now 213). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 46s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 22s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 6m 3s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 46m 38s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738110/YARN-574.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 71de367 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8206/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8206/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8206/console |
This message was automatically generated.
PrivateLocalizer does not support parallel resource download via
ContainerLocalizer
---
Key: YARN-574
URL: https://issues.apache.org/jira/browse/YARN-574
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.6.0, 2.8.0, 2.7.1
Reporter: Omkar Vinit Joshi
Assignee: Omkar Vinit Joshi
Attachments: YARN-574.1.patch, YARN-574.2.patch
At present private resources will be downloaded in parallel only if multiple
containers request the same resource. However otherwise it will be serial.
The protocol between PrivateLocalizer and ContainerLocalizer supports
multiple downloads however it is not used and only one resource is sent for
downloading at a time.
I think we can increase / assure parallelism (even for single container
requesting resource) for private/application resources by making multiple
downloads per ContainerLocalizer.
Total Parallelism before
= number of threads allotted for PublicLocalizer [public resource] + number
of containers[private and application resource]
Total Parallelism after
= number of threads allotted for PublicLocalizer [public resource] + number
of containers * max downloads per container [private and application resource]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3706) Generalize native HBase writer for additional tables
[
https://issues.apache.org/jira/browse/YARN-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575873#comment-14575873
]
Hadoop QA commented on YARN-3706:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 17s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 45s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 43s | The applied patch generated 7
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 14s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 12s | The patch has 3 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 40s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 0m 51s | The patch appears to introduce 5
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 1m 16s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 38m 3s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-timelineservice |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738186/YARN-3726-YARN-2928.008.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / d88f30b |
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/8207/artifact/patchprocess/diffJavadocWarnings.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8207/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8207/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-timelineservice.html
|
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8207/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8207/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8207/console |
This message was automatically generated.
Generalize native HBase writer for additional tables
Key: YARN-3706
URL: https://issues.apache.org/jira/browse/YARN-3706
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Minor
Attachments: YARN-3706-YARN-2928.001.patch,
YARN-3726-YARN-2928.002.patch, YARN-3726-YARN-2928.003.patch,
YARN-3726-YARN-2928.004.patch, YARN-3726-YARN-2928.005.patch,
YARN-3726-YARN-2928.006.patch, YARN-3726-YARN-2928.007.patch,
YARN-3726-YARN-2928.008.patch
When reviewing YARN-3411 we noticed that we could change the class hierarchy
a little in order to accommodate additional tables easily.
In order to get ready for benchmark testing we left the original layout in
place, as performance would not be impacted by the code hierarchy.
Here is a separate jira to address the hierarchy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
