[jira] [Commented] (YARN-3543) ApplicationReport should be able to tell whether the Application is AM managed or not.
[
https://issues.apache.org/jira/browse/YARN-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558049#comment-14558049
]
Hadoop QA commented on YARN-3543:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 14 new or modified test files. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 3m 10s | The applied patch generated 1
new checkstyle issues (total was 14, now 14). |
| {color:green}+1{color} | whitespace | 0m 11s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 59s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 109m 8s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| {color:green}+1{color} | yarn tests | 0m 29s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 7m 3s | Tests passed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 2m 2s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 3m 14s | Tests passed in
hadoop-yarn-server-applicationhistoryservice. |
| {color:green}+1{color} | yarn tests | 0m 29s | Tests passed in
hadoop-yarn-server-common. |
| {color:green}+1{color} | yarn tests | 50m 18s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 218m 21s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735120/0004-YARN-3543.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 446d515 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8070/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8070/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8070/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8070/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8070/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-applicationhistoryservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8070/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
|
| hadoop-yarn-server-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8070/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8070/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8070/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8070/console |
This message was automatically generated.
ApplicationReport should be able to tell whether the Application is AM
managed or not.
---
Key: YARN-3543
URL: https://issues.apache.org/jira/browse/YARN-3543
Project: Hadoop YARN
Issue Type: Improvement
Components: api
Affects Versions: 2.6.0
Reporter: Spandan Dutta
Assignee: Rohith
Attachments: 0001-YARN-3543.patch, 0001-YARN-3543.patch,
0002-YARN-3543.patch, 0002-YARN-3543.patch, 0003-YARN-3543.patch,
0004-YARN-3543.patch, 0004-YARN-3543.patch, 0004-YARN-3543.patch,
YARN-3543-AH.PNG, YARN-3543-RM.PNG
Currently we can know whether the application submitted by the user is AM
managed from the applicationSubmissionContext. This can be only done at the
time when the user submits the job. We
[jira] [Commented] (YARN-3543) ApplicationReport should be able to tell whether the Application is AM managed or not.
[ https://issues.apache.org/jira/browse/YARN-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558050#comment-14558050 ] Rohith commented on YARN-3543: -- [~vinodkv] Kindly review the updated patch.. ApplicationReport should be able to tell whether the Application is AM managed or not. --- Key: YARN-3543 URL: https://issues.apache.org/jira/browse/YARN-3543 Project: Hadoop YARN Issue Type: Improvement Components: api Affects Versions: 2.6.0 Reporter: Spandan Dutta Assignee: Rohith Attachments: 0001-YARN-3543.patch, 0001-YARN-3543.patch, 0002-YARN-3543.patch, 0002-YARN-3543.patch, 0003-YARN-3543.patch, 0004-YARN-3543.patch, 0004-YARN-3543.patch, 0004-YARN-3543.patch, YARN-3543-AH.PNG, YARN-3543-RM.PNG Currently we can know whether the application submitted by the user is AM managed from the applicationSubmissionContext. This can be only done at the time when the user submits the job. We should have access to this info from the ApplicationReport as well so that we can check whether an app is AM managed or not anytime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3686) CapacityScheduler should trim default_node_label_expression
[ https://issues.apache.org/jira/browse/YARN-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-3686: -- Attachment: 0004-YARN-3686.patch Thank you [~leftnoteasy] for pointing out the same. Uploading patch after adding new test. CapacityScheduler should trim default_node_label_expression --- Key: YARN-3686 URL: https://issues.apache.org/jira/browse/YARN-3686 Project: Hadoop YARN Issue Type: Sub-task Components: api, client, resourcemanager Reporter: Wangda Tan Assignee: Sunil G Priority: Critical Attachments: 0001-YARN-3686.patch, 0002-YARN-3686.patch, 0003-YARN-3686.patch, 0004-YARN-3686.patch We should trim default_node_label_expression for queue before using it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3709) RM Web UI AM link shown before MRAppMaster launch
Bibin A Chundatt created YARN-3709: -- Summary: RM Web UI AM link shown before MRAppMaster launch Key: YARN-3709 URL: https://issues.apache.org/jira/browse/YARN-3709 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.0 Environment: Suse 11 Sp3 Reporter: Bibin A Chundatt Priority: Minor Steps to reproduce === 1.Configure HA setup with 2 NM 2.AM allocated memory 1024 MB in CS 3.Submit 5 pi jobs in parallel 4.2 AM runs in parallel *Expected :* Only for running Applications Tracking Url/AM link should be shown *Actual:* For all 5 application *ApplicationMaster* link shown For application unassigned with AM Tracking URL should be shown as *UNASSIGNED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3709) RM Web UI AM link shown before MRAppMaster launch
[ https://issues.apache.org/jira/browse/YARN-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3709: --- Attachment: ApplicationMasterLink.png RM Web UI AM link shown before MRAppMaster launch - Key: YARN-3709 URL: https://issues.apache.org/jira/browse/YARN-3709 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.0 Environment: Suse 11 Sp3 Reporter: Bibin A Chundatt Priority: Minor Attachments: ApplicationMasterLink.png Steps to reproduce === 1.Configure HA setup with 2 NM 2.AM allocated memory 1024 MB in CS 3.Submit 5 pi jobs in parallel 4.2 AM runs in parallel *Expected :* Only for running Applications Tracking Url/AM link should be shown *Actual:* For all 5 application *ApplicationMaster* link shown For application unassigned with AM Tracking URL should be shown as *UNASSIGNED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3686) CapacityScheduler should trim default_node_label_expression
[
https://issues.apache.org/jira/browse/YARN-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558286#comment-14558286
]
Hadoop QA commented on YARN-3686:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 42s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 32s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 36s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 4 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 1m 57s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 50m 13s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 90m 40s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735160/0004-YARN-3686.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ada233b |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8071/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8071/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8071/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8071/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8071/console |
This message was automatically generated.
CapacityScheduler should trim default_node_label_expression
---
Key: YARN-3686
URL: https://issues.apache.org/jira/browse/YARN-3686
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, client, resourcemanager
Reporter: Wangda Tan
Assignee: Sunil G
Priority: Critical
Attachments: 0001-YARN-3686.patch, 0002-YARN-3686.patch,
0003-YARN-3686.patch, 0004-YARN-3686.patch
We should trim default_node_label_expression for queue before using it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated YARN-3655:
Attachment: YARN-3655.003.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2238) filtering on UI sticks even if I move away from the page
[ https://issues.apache.org/jira/browse/YARN-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558459#comment-14558459 ] Naganarasimha G R commented on YARN-2238: - +1 lgtm (non-binding), Tested the patch with trunk, but would it be also good to add the filter params some where in the UI to indicate whats the current filter ? filtering on UI sticks even if I move away from the page Key: YARN-2238 URL: https://issues.apache.org/jira/browse/YARN-2238 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.4.0 Reporter: Sangjin Lee Assignee: Jian He Labels: usability Attachments: YARN-2238.patch, YARN-2238.png, filtered.png The main data table in many web pages (RM, AM, etc.) seems to show an unexpected filtering behavior. If I filter the table by typing something in the key or value field (or I suspect any search field), the data table gets filtered. The example I used is the job configuration page for a MR job. That is expected. However, when I move away from that page and visit any other web page of the same type (e.g. a job configuration page), the page is rendered with the filtering! That is unexpected. What's even stranger is that it does not render the filtering term. As a result, I have a page that's mysteriously filtered but doesn't tell me what it's filtering on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3581) Deprecate -directlyAccessNodeLabelStore in RMAdminCLI
[
https://issues.apache.org/jira/browse/YARN-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558458#comment-14558458
]
Hadoop QA commented on YARN-3581:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 29s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 28s | The applied patch generated 2
new checkstyle issues (total was 40, now 42). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 38s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 43s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 6m 53s | Tests passed in
hadoop-yarn-client. |
| | | 42m 15s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735195/YARN-3581.20150525-1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ada233b |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8074/artifact/patchprocess/diffcheckstylehadoop-yarn-client.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8074/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8074/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8074/console |
This message was automatically generated.
Deprecate -directlyAccessNodeLabelStore in RMAdminCLI
-
Key: YARN-3581
URL: https://issues.apache.org/jira/browse/YARN-3581
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, client, resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-3581.20150525-1.patch
In 2.6.0, we added an option called -directlyAccessNodeLabelStore to make
RM can start with label-configured queue settings. After YARN-2918, we don't
need this option any more, admin can configure queue setting, start RM and
configure node label via RMAdminCLI without any error.
In addition, this option is very restrictive, first it needs to run on the
same node where RM is running if admin configured to store labels in local
disk.
Second, when admin run the option when RM is running, multiple process write
to a same file can happen, this could make node label store becomes invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3710) FairScheduler: Should allocate more containers for assign-multiple after assignReservedContainer turns the reservation into an allocation.
zhihai xu created YARN-3710: --- Summary: FairScheduler: Should allocate more containers for assign-multiple after assignReservedContainer turns the reservation into an allocation. Key: YARN-3710 URL: https://issues.apache.org/jira/browse/YARN-3710 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: zhihai xu Assignee: zhihai xu FairScheduler: Should allocate more containers for assign-multiple after assignReservedContainer turns the reservation into an allocation. Currently FairScheduler#attemptScheduling does not assign more containers for assign-multiple after assignReservedContainer turns the reservation into an allocation successfully. We should try to assign more containers on the same node if assignMultiple is enabled after assignReservedContainer turns the reservation into an allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558454#comment-14558454
]
Hadoop QA commented on YARN-3655:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 37s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 31s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 46s | The applied patch generated 1
new checkstyle issues (total was 123, now 120). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 15s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 50m 10s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 86m 31s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735189/YARN-3655.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ada233b |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8072/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8072/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8072/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8072/console |
This message was automatically generated.
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
[
https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558495#comment-14558495
]
Chris Nauroth commented on YARN-3626:
-
Hi Craig. This looks good to me. I have just one minor nitpick. I think the
logic in {{ContainerLaunch}} for setting {{preferLocalizedJars}} could be
simplified to this:
{code}
boolean preferLocalizedJars = Boolean.valueOf(classpathPrependDistCache);
{code}
{{Boolean#valueOf}} is null-safe.
Thanks!
On Windows localized resources are not moved to the front of the classpath
when they should be
--
Key: YARN-3626
URL: https://issues.apache.org/jira/browse/YARN-3626
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Environment: Windows
Reporter: Craig Welch
Assignee: Craig Welch
Fix For: 2.7.1
Attachments: YARN-3626.0.patch, YARN-3626.11.patch,
YARN-3626.14.patch, YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch
In response to the mapreduce.job.user.classpath.first setting the classpath
is ordered differently so that localized resources will appear before system
classpath resources when tasks execute. On Windows this does not work
because the localized resources are not linked into their final location when
the classpath jar is created. To compensate for that localized jar resources
are added directly to the classpath generated for the jar rather than being
discovered from the localized directories. Unfortunately, they are always
appended to the classpath, and so are never preferred over system resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558428#comment-14558428
]
zhihai xu commented on YARN-3655:
-
Hi [~kasha], thanks for the review.
bq. 1. okToUnreserve
fixed in the new patch YARN-3655.003.patch
bq. 2. Add an if (isValidReservation) check in FSAppAttempt#reserve so all the
reservation logic stays in one place?
IMHO, It is not good to add if (isValidReservation) check in
FSAppAttempt#reserve because all the conditions checked in isValidReservation
are already checked before we call FSAppAttempt#reserve, it will be duplicate
code which will affect the performance.
bq. 3.In {{FSAppAttempt#assignContainer(node, request, nodeType, reserved)}}...
fixed in the new patch YARN-3655.003.patch, In order to remove fitsInMaxShare
check, I merged {{fitsInMaxShare}} check into {{hasContainerForNode}}, which
also make the code cleaner.
bq. 4. While adding this check in FSAppAttempt#assignContainer(node) might work
in practice, it somehow feels out of place. Also, assignReservedContainer could
also lead to a reservation?
It looks like assignReservedContainer won't lead to a
reservation({{FSAppAttempt#reserve}}), assignReservedContainer won't call
{{FSAppAttempt#reserve}} because {{FSAppAttempt#reserve}} will only be called
when the node Available Resource is smaller than the requested/reserved
resource. assignReservedContainer will only call assignContainer when the node
Available Resource is no less than the reserved resource. So only
{{FSAppAttempt#assignContainer(node)}} can lead to a reservation when the node
Available Resource is smaller than the requested resource.
bq. 5. Instead of calling okToUnreserve/!isValidReservation in
FairScheduler#attemptScheduling...
fixed in the new patch YARN-3655.003.patch
bq. 6. Looks like assign-multiple is broken with reserved-containers. The
while-loop for assign-multiple should look at both reserved and un-reserved
containers assigned. Can we file a follow-up JIRA to fix this?
I suppose you mean assign-multiple is broken after assignReservedContainer
turns the reservation into an allocation.
Yes, I created YARN-3710 to fix this issue.
bq. Oh, and I found it hard to understand the test
fixed in the new patch YARN-3655.003.patch, please review it.
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558526#comment-14558526
]
Hadoop QA commented on YARN-2369:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 37s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 33s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 29s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 16s | The applied patch generated 2
new checkstyle issues (total was 176, now 177). |
| {color:red}-1{color} | checkstyle | 2m 49s | The applied patch generated
11 new checkstyle issues (total was 509, now 520). |
| {color:red}-1{color} | checkstyle | 3m 15s | The applied patch generated
11 new checkstyle issues (total was 7, now 18). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 5 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 15s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 22s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 45s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 36s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| | | 70m 19s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735207/YARN-2369-4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ada233b |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/diffcheckstylehadoop-common.txt
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/diffcheckstylehadoop-yarn-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/console |
This message was automatically generated.
Environment variable handling assumes values should be appended
---
Key: YARN-2369
URL: https://issues.apache.org/jira/browse/YARN-2369
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jason Lowe
Assignee: Dustin Cote
Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch,
YARN-2369-4.patch
When processing environment variables for a container context the code
assumes that the value should be appended to any pre-existing value in the
environment. This may be desired behavior for handling path-like environment
variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
non-intuitive and harmful way to handle any variable that does not have
path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3581) Deprecate -directlyAccessNodeLabelStore in RMAdminCLI
[ https://issues.apache.org/jira/browse/YARN-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-3581: Attachment: (was: YARN-3581.20150525-1.patch) Deprecate -directlyAccessNodeLabelStore in RMAdminCLI - Key: YARN-3581 URL: https://issues.apache.org/jira/browse/YARN-3581 Project: Hadoop YARN Issue Type: Sub-task Components: api, client, resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-3581.20150525-1.patch In 2.6.0, we added an option called -directlyAccessNodeLabelStore to make RM can start with label-configured queue settings. After YARN-2918, we don't need this option any more, admin can configure queue setting, start RM and configure node label via RMAdminCLI without any error. In addition, this option is very restrictive, first it needs to run on the same node where RM is running if admin configured to store labels in local disk. Second, when admin run the option when RM is running, multiple process write to a same file can happen, this could make node label store becomes invalid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3581) Deprecate -directlyAccessNodeLabelStore in RMAdminCLI
[ https://issues.apache.org/jira/browse/YARN-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-3581: Attachment: YARN-3581.20150525-1.patch missed to add changes for ClusterCLI... Deprecate -directlyAccessNodeLabelStore in RMAdminCLI - Key: YARN-3581 URL: https://issues.apache.org/jira/browse/YARN-3581 Project: Hadoop YARN Issue Type: Sub-task Components: api, client, resourcemanager Reporter: Wangda Tan Assignee: Naganarasimha G R Attachments: YARN-3581.20150525-1.patch, YARN-3581.20150525-1.patch In 2.6.0, we added an option called -directlyAccessNodeLabelStore to make RM can start with label-configured queue settings. After YARN-2918, we don't need this option any more, admin can configure queue setting, start RM and configure node label via RMAdminCLI without any error. In addition, this option is very restrictive, first it needs to run on the same node where RM is running if admin configured to store labels in local disk. Second, when admin run the option when RM is running, multiple process write to a same file can happen, this could make node label store becomes invalid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Cote updated YARN-2369: -- Attachment: YARN-2369-4.patch Checkstyle fixes, first try Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3581) Deprecate -directlyAccessNodeLabelStore in RMAdminCLI
[
https://issues.apache.org/jira/browse/YARN-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-3581:
Attachment: YARN-3581.20150525-1.patch
Hi [~Wangda],
I have corrected the issues mentioned earlier and also added one more fix:
{{[ ..] square brackets are used when the parameter is optional, hence
converted them to .. as arguments are required in case of labels
modification.}}
Please inform if you require a separate patch for 7.1 branch (1,3,4 the one
mentioned above).
Deprecate -directlyAccessNodeLabelStore in RMAdminCLI
-
Key: YARN-3581
URL: https://issues.apache.org/jira/browse/YARN-3581
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, client, resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-3581.20150525-1.patch
In 2.6.0, we added an option called -directlyAccessNodeLabelStore to make
RM can start with label-configured queue settings. After YARN-2918, we don't
need this option any more, admin can configure queue setting, start RM and
configure node label via RMAdminCLI without any error.
In addition, this option is very restrictive, first it needs to run on the
same node where RM is running if admin configured to store labels in local
disk.
Second, when admin run the option when RM is running, multiple process write
to a same file can happen, this could make node label store becomes invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3581) Deprecate -directlyAccessNodeLabelStore in RMAdminCLI
[
https://issues.apache.org/jira/browse/YARN-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558435#comment-14558435
]
Hadoop QA commented on YARN-3581:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 36s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 30s | The applied patch generated 1
new checkstyle issues (total was 37, now 38). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 44s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 6m 58s | Tests passed in
hadoop-yarn-client. |
| | | 42m 26s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735192/YARN-3581.20150525-1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ada233b |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8073/artifact/patchprocess/diffcheckstylehadoop-yarn-client.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8073/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8073/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8073/console |
This message was automatically generated.
Deprecate -directlyAccessNodeLabelStore in RMAdminCLI
-
Key: YARN-3581
URL: https://issues.apache.org/jira/browse/YARN-3581
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, client, resourcemanager
Reporter: Wangda Tan
Assignee: Naganarasimha G R
Attachments: YARN-3581.20150525-1.patch
In 2.6.0, we added an option called -directlyAccessNodeLabelStore to make
RM can start with label-configured queue settings. After YARN-2918, we don't
need this option any more, admin can configure queue setting, start RM and
configure node label via RMAdminCLI without any error.
In addition, this option is very restrictive, first it needs to run on the
same node where RM is running if admin configured to store labels in local
disk.
Second, when admin run the option when RM is running, multiple process write
to a same file can happen, this could make node label store becomes invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558578#comment-14558578
]
Hadoop QA commented on YARN-2369:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 30s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 11s | The applied patch generated 2
new checkstyle issues (total was 176, now 176). |
| {color:red}-1{color} | checkstyle | 2m 43s | The applied patch generated 8
new checkstyle issues (total was 509, now 517). |
| {color:red}-1{color} | checkstyle | 3m 10s | The applied patch generated 2
new checkstyle issues (total was 7, now 9). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 7 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 16s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 19s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 35s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 1m 58s | Tests passed in
hadoop-yarn-common. |
| | | 70m 13s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735218/YARN-2369-5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ada233b |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/diffcheckstylehadoop-common.txt
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/diffcheckstylehadoop-yarn-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/console |
This message was automatically generated.
Environment variable handling assumes values should be appended
---
Key: YARN-2369
URL: https://issues.apache.org/jira/browse/YARN-2369
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jason Lowe
Assignee: Dustin Cote
Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch,
YARN-2369-4.patch, YARN-2369-5.patch
When processing environment variables for a container context the code
assumes that the value should be appended to any pre-existing value in the
environment. This may be desired behavior for handling path-like environment
variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
non-intuitive and harmful way to handle any variable that does not have
path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558618#comment-14558618
]
Hadoop QA commented on YARN-3655:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 30s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 54s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 16s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 50m 14s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 86m 38s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735229/YARN-3655.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ada233b |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8077/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8077/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8077/console |
This message was automatically generated.
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated YARN-3655:
Attachment: YARN-3655.003.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch, YARN-3655.003.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Cote updated YARN-2369: -- Attachment: YARN-2369-5.patch Missed a couple of style items. Submitting again. Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, YARN-2369-5.patch When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3655) FairScheduler: potential livelock due to maxAMShare limitation and container reservation
[
https://issues.apache.org/jira/browse/YARN-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated YARN-3655:
Attachment: (was: YARN-3655.003.patch)
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation
-
Key: YARN-3655
URL: https://issues.apache.org/jira/browse/YARN-3655
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: YARN-3655.000.patch, YARN-3655.001.patch,
YARN-3655.002.patch
FairScheduler: potential livelock due to maxAMShare limitation and container
reservation.
If a node is reserved by an application, all the other applications don't
have any chance to assign a new container on this node, unless the
application which reserves the node assigns a new container on this node or
releases the reserved container on this node.
The problem is if an application tries to call assignReservedContainer and
fail to get a new container due to maxAMShare limitation, it will block all
other applications to use the nodes it reserves. If all other running
applications can't release their AM containers due to being blocked by these
reserved containers. A livelock situation can happen.
The following is the code at FSAppAttempt#assignContainer which can cause
this potential livelock.
{code}
// Check the AM resource usage for the leaf queue
if (!isAmRunning() !getUnmanagedAM()) {
ListResourceRequest ask = appSchedulingInfo.getAllResourceRequests();
if (ask.isEmpty() || !getQueue().canRunAppAM(
ask.get(0).getCapability())) {
if (LOG.isDebugEnabled()) {
LOG.debug(Skipping allocation because maxAMShare limit would +
be exceeded);
}
return Resources.none();
}
}
{code}
To fix this issue, we can unreserve the node if we can't allocate the AM
container on the node due to Max AM share limitation and the node is reserved
by the application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3712) ContainersLauncher: handle event CLEANUP_CONTAINER asynchronously
[ https://issues.apache.org/jira/browse/YARN-3712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Gong updated YARN-3712: --- Attachment: YARN-3712.01.patch ContainersLauncher: handle event CLEANUP_CONTAINER asynchronously - Key: YARN-3712 URL: https://issues.apache.org/jira/browse/YARN-3712 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Jun Gong Assignee: Jun Gong Attachments: YARN-3712.01.patch It will save some time by handling event CLEANUP_CONTAINER asynchronously. This improvement will be useful for cases that cleaning up container cost a little long time(e.g. for our case: we are running Docker container on NM, it will take above 1 seconds to clean up one docker container. ) and many containers to clean up(e.g. NM need clean up all running containers when NM shutdown). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3712) ContainersLauncher: handle event CLEANUP_CONTAINER asynchronously
Jun Gong created YARN-3712: -- Summary: ContainersLauncher: handle event CLEANUP_CONTAINER asynchronously Key: YARN-3712 URL: https://issues.apache.org/jira/browse/YARN-3712 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Jun Gong Assignee: Jun Gong It will save some time by handling event CLEANUP_CONTAINER asynchronously. This improvement will be useful for cases that cleaning up container cost a little long time(e.g. for our case: we are running Docker container on NM, it will take above 1 seconds to clean up one docker container. ) and many containers to clean up(e.g. NM need clean up all running containers when NM shutdown). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2238) filtering on UI sticks even if I move away from the page
[ https://issues.apache.org/jira/browse/YARN-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558708#comment-14558708 ] Xuan Gong commented on YARN-2238: - Committed into trunk/branch-2/branch-2.7. Thanks, Jian filtering on UI sticks even if I move away from the page Key: YARN-2238 URL: https://issues.apache.org/jira/browse/YARN-2238 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.4.0 Reporter: Sangjin Lee Assignee: Jian He Labels: usability Fix For: 2.7.1 Attachments: YARN-2238.patch, YARN-2238.png, filtered.png The main data table in many web pages (RM, AM, etc.) seems to show an unexpected filtering behavior. If I filter the table by typing something in the key or value field (or I suspect any search field), the data table gets filtered. The example I used is the job configuration page for a MR job. That is expected. However, when I move away from that page and visit any other web page of the same type (e.g. a job configuration page), the page is rendered with the filtering! That is unexpected. What's even stranger is that it does not render the filtering term. As a result, I have a page that's mysteriously filtered but doesn't tell me what it's filtering on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2238) filtering on UI sticks even if I move away from the page
[ https://issues.apache.org/jira/browse/YARN-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558711#comment-14558711 ] Hudson commented on YARN-2238: -- FAILURE: Integrated in Hadoop-trunk-Commit #7899 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7899/]) YARN-2238. Filtering on UI sticks even if I move away from the page. (xgong: rev 39077dba2e877420e7470df253f6154f6ecc64ec) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java filtering on UI sticks even if I move away from the page Key: YARN-2238 URL: https://issues.apache.org/jira/browse/YARN-2238 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.4.0 Reporter: Sangjin Lee Assignee: Jian He Labels: usability Fix For: 2.7.1 Attachments: YARN-2238.patch, YARN-2238.png, filtered.png The main data table in many web pages (RM, AM, etc.) seems to show an unexpected filtering behavior. If I filter the table by typing something in the key or value field (or I suspect any search field), the data table gets filtered. The example I used is the job configuration page for a MR job. That is expected. However, when I move away from that page and visit any other web page of the same type (e.g. a job configuration page), the page is rendered with the filtering! That is unexpected. What's even stranger is that it does not render the filtering term. As a result, I have a page that's mysteriously filtered but doesn't tell me what it's filtering on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3712) ContainersLauncher: handle event CLEANUP_CONTAINER asynchronously
[
https://issues.apache.org/jira/browse/YARN-3712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558710#comment-14558710
]
Hadoop QA commented on YARN-3712:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 37s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 33s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 39s | The applied patch generated 2
new checkstyle issues (total was 32, now 28). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 2s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 6m 4s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 42m 0s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735253/YARN-3712.01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ada233b |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8078/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8078/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8078/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8078/console |
This message was automatically generated.
ContainersLauncher: handle event CLEANUP_CONTAINER asynchronously
-
Key: YARN-3712
URL: https://issues.apache.org/jira/browse/YARN-3712
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Reporter: Jun Gong
Assignee: Jun Gong
Attachments: YARN-3712.01.patch
It will save some time by handling event CLEANUP_CONTAINER asynchronously.
This improvement will be useful for cases that cleaning up container cost a
little long time(e.g. for our case: we are running Docker container on NM, it
will take above 1 seconds to clean up one docker container. ) and many
containers to clean up(e.g. NM need clean up all running containers when NM
shutdown).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3711) Documentation of ResourceManager HA should explain about webapp address configuration
Masatake Iwasaki created YARN-3711:
--
Summary: Documentation of ResourceManager HA should explain about
webapp address configuration
Key: YARN-3711
URL: https://issues.apache.org/jira/browse/YARN-3711
Project: Hadoop YARN
Issue Type: Improvement
Components: documentation
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor
Proper URL of AM Web UI could not be got without setting
{{yarn.resourcemanager.webapp.address._node-id_}} and/or
{{yarn.resourcemanager.webapp.https.address._node-id_}} if RM-HA is enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2238) filtering on UI sticks even if I move away from the page
[ https://issues.apache.org/jira/browse/YARN-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558705#comment-14558705 ] Xuan Gong commented on YARN-2238: - Thanks for the validation. bq. but would it be also good to add the filter params some where in the UI to indicate whats the current filter ? We could do that separately if we have requirement in future. +1, check this in. filtering on UI sticks even if I move away from the page Key: YARN-2238 URL: https://issues.apache.org/jira/browse/YARN-2238 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.4.0 Reporter: Sangjin Lee Assignee: Jian He Labels: usability Attachments: YARN-2238.patch, YARN-2238.png, filtered.png The main data table in many web pages (RM, AM, etc.) seems to show an unexpected filtering behavior. If I filter the table by typing something in the key or value field (or I suspect any search field), the data table gets filtered. The example I used is the job configuration page for a MR job. That is expected. However, when I move away from that page and visit any other web page of the same type (e.g. a job configuration page), the page is rendered with the filtering! That is unexpected. What's even stranger is that it does not render the filtering term. As a result, I have a page that's mysteriously filtered but doesn't tell me what it's filtering on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
