[jira] [Comment Edited] (YARN-5597) YARN Federation improvements
[ https://issues.apache.org/jira/browse/YARN-5597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577824#comment-16577824 ] Bibin A Chundatt edited comment on YARN-5597 at 8/13/18 4:50 AM: - Thank you [~subru] Could you clarify query related to State Store usage IIUC we have option of using Zookeeper as federation store and zookeeper as Resourcemanager State Store. In deployment are we suggesting to use single zookeeper instance for all the RM subclusters (as RMStateStore) and also for FederationStateStore ? While evaluating the store implementation of statestore + kerberos for zk connection if find following limitation Currently in zookeeper 3.4.6 doesn't support multiple secure zk client with different configuration on same jvm instance. ZOOKEEPER-2139 is included in 3.5.2 only. was (Author: bibinchundatt): [~subru] IIUC we have option of using Zookeeper as federation store and zookeeper as Resourcemanager State Store. In deployment are we suggesting to use single zookeeper instance for all the RM subclusters (as RMStateStore) and also for FederationStateStore ? While evaluating the store implementation of statestore + kerberos for zk connection if find following limitation Currently in zookeeper 3.4.6 doesn't support multiple secure zk client with different configuration on same jvm instance. ZOOKEEPER-2139 is included in 3.5.2 only. > YARN Federation improvements > > > Key: YARN-5597 > URL: https://issues.apache.org/jira/browse/YARN-5597 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Subru Krishnan >Assignee: Subru Krishnan >Priority: Major > > This umbrella JIRA tracks set of improvements over the YARN Federation MVP > (YARN-2915) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5597) YARN Federation improvements
[ https://issues.apache.org/jira/browse/YARN-5597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577824#comment-16577824 ] Bibin A Chundatt commented on YARN-5597: [~subru] IIUC we have option of using Zookeeper as federation store and zookeeper as Resourcemanager State Store. In deployment are we suggesting to use single zookeeper instance for all the RM subclusters (as RMStateStore) and also for FederationStateStore ? While evaluating the store implementation of statestore + kerberos for zk connection if find following limitation Currently in zookeeper 3.4.6 doesn't support multiple secure zk client with different configuration on same jvm instance. ZOOKEEPER-2139 is included in 3.5.2 only. > YARN Federation improvements > > > Key: YARN-5597 > URL: https://issues.apache.org/jira/browse/YARN-5597 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Subru Krishnan >Assignee: Subru Krishnan >Priority: Major > > This umbrella JIRA tracks set of improvements over the YARN Federation MVP > (YARN-2915) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7494) Add muti node lookup support for better placement
[ https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577817#comment-16577817 ] Sunil Govindan commented on YARN-7494: -- Thanks [~cheersyang] for quick review. Yes, I modified manager class to have a wrapper to do this extra check. I tried to make the scheduler lookup path as lean as possible. Pls help to review. > Add muti node lookup support for better placement > - > > Key: YARN-7494 > URL: https://issues.apache.org/jira/browse/YARN-7494 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7494.001.patch, YARN-7494.002.patch, > YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.005.patch, > YARN-7494.006.patch, YARN-7494.007.patch, YARN-7494.008.patch, > YARN-7494.009.patch, YARN-7494.010.patch, YARN-7494.11.patch, > YARN-7494.12.patch, YARN-7494.13.patch, YARN-7494.14.patch, > YARN-7494.15.patch, YARN-7494.16.patch, YARN-7494.v0.patch, > YARN-7494.v1.patch, multi-node-designProposal.png > > > Instead of single node, for effectiveness we can consider a multi node lookup > based on partition to start with. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7494) Add muti node lookup support for better placement
[ https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-7494: - Attachment: YARN-7494.16.patch > Add muti node lookup support for better placement > - > > Key: YARN-7494 > URL: https://issues.apache.org/jira/browse/YARN-7494 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7494.001.patch, YARN-7494.002.patch, > YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.005.patch, > YARN-7494.006.patch, YARN-7494.007.patch, YARN-7494.008.patch, > YARN-7494.009.patch, YARN-7494.010.patch, YARN-7494.11.patch, > YARN-7494.12.patch, YARN-7494.13.patch, YARN-7494.14.patch, > YARN-7494.15.patch, YARN-7494.16.patch, YARN-7494.v0.patch, > YARN-7494.v1.patch, multi-node-designProposal.png > > > Instead of single node, for effectiveness we can consider a multi node lookup > based on partition to start with. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8559) Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint
[ https://issues.apache.org/jira/browse/YARN-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577797#comment-16577797 ] Weiwei Yang commented on YARN-8559: --- Thank you [~jhung] for taking care of those branches, as now all commits are pushed, lets close this ticket. Thanks all. > Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint > > > Key: YARN-8559 > URL: https://issues.apache.org/jira/browse/YARN-8559 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Anna Savarin >Assignee: Weiwei Yang >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: YARN-8559-branch-2.001.patch, > YARN-8559-branch-3.0.001.patch, YARN-8559.001.patch, YARN-8559.002.patch, > YARN-8559.003.patch, YARN-8559.004.patch > > > All Hadoop services provide a set of common endpoints (/stacks, /logLevel, > /metrics, /jmx, /conf). In the case of the Resource Manager, part of the > configuration comes from the scheduler being used. Currently, these > configuration key/values are not exposed through the /conf endpoint, thereby > revealing an incomplete configuration picture. > Make an improvement and expose the scheduling configuration info through the > RM's /conf endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7494) Add muti node lookup support for better placement
[
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1652#comment-1652
]
Weiwei Yang commented on YARN-7494:
---
Thanks [~sunilg] for the updates, please see my followup comments below:
ResourceUsageMultiNodeLookupPolicy.java:
{code:java}
Iterator getPreferredNodeIterator(Collection nodes, String partition,
boolean isAutoSortEnabled);
{code}
I understand you have added this to make background-sort completely controlled
by the sorting interval argument, but it complex the API. I think we can add a
wrapper in {{MultiNodeSortingManager#getPreferredNodeIterator}}, and let app
placement allocator call this instead. This way, we can remove {{boolean
isAutoSortEnabled}} argument because the manager is able to tell if the sorting
thread is running or not (or simply checking the sorting interval value). This
also gives the benefit that we don't need to expose the \{{sorter}} class to
app allocator.
Minor things to fix:
# ResourceUsageMultiNodeLookupPolicy.java: replace line 23 import star
# TestCapacityScheduler.java: replace line 33 import start
# TestCapacitySchedulerNodeLabelUpdate.java: remove unused imports at line 25
and 49
# ResourceUsageMultiNodeLookupPolicy.java: replace line 23 import start
# ApplicationSchedulingConfig.java: line 20, 24 unnecessary changes
# FiCaSchedulerApp.java: line 174-187: can we put this in a private method e.g
{{updateNodesSortingPolicyIfNecessary()}}?
Other looks good to me! Please let me know if this makes sense.
Thanks!
> Add muti node lookup support for better placement
> -
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.002.patch,
> YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.005.patch,
> YARN-7494.006.patch, YARN-7494.007.patch, YARN-7494.008.patch,
> YARN-7494.009.patch, YARN-7494.010.patch, YARN-7494.11.patch,
> YARN-7494.12.patch, YARN-7494.13.patch, YARN-7494.14.patch,
> YARN-7494.15.patch, YARN-7494.v0.patch, YARN-7494.v1.patch,
> multi-node-designProposal.png
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup
> based on partition to start with.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8621) Add REST API tests for Resource Types fields for the apps/ endpoint
[
https://issues.apache.org/jira/browse/YARN-8621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577731#comment-16577731
]
genericqa commented on YARN-8621:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 40s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8621 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935297/YARN-8621.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 76e235e484c0 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3ac07b7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/21577/artifact/out/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/21577/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21577/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-8621) Add REST API tests for Resource Types fields for the apps/ endpoint
[ https://issues.apache.org/jira/browse/YARN-8621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577710#comment-16577710 ] Szilard Nemeth commented on YARN-8621: -- By checking the latest code: AppAttempt has no field in the response that would refer to resource requests, see: https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/dao/AppAttemptInfo.java Uploaded patch for testing the apps/ endpoint only. I would create a separate jira that would implement adding a new field to AppAttempt, as it's a whole different story than this one. [~templedf]: Do you agree creating a follow-up jira that would add the resource request field to the App Attempt and add testcases at the same time? Is it required to have those fields in the response at all? [~leftnoteasy]: You might be also interested in the question above as you implemented YARN-7817 earlier this year. Also, code review would be appreciated. Thanks! > Add REST API tests for Resource Types fields for the apps/ endpoint > -- > > Key: YARN-8621 > URL: https://issues.apache.org/jira/browse/YARN-8621 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8621.001.patch > > > This is a complement for YARN-7451 that already added unit tests for the apps > and scheduler endpoints. > The following API endpoints should be tested as well: > /ws/v1/cluster/apps/ > -/ws/v1/cluster/apps//appattempts- > -/ws/v1/cluster/apps//appattempts/- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8621) Add REST API tests for Resource Types fields for apps/
[ https://issues.apache.org/jira/browse/YARN-8621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8621: - Summary: Add REST API tests for Resource Types fields for apps/ (was: Add REST API tests for Resource Types fields) > Add REST API tests for Resource Types fields for apps/ > - > > Key: YARN-8621 > URL: https://issues.apache.org/jira/browse/YARN-8621 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8621.001.patch > > > This is a complement for YARN-7451 that already added unit tests for the apps > and scheduler endpoints. > The following API endpoints should be tested as well: > /ws/v1/cluster/apps/ > -/ws/v1/cluster/apps//appattempts- > -/ws/v1/cluster/apps//appattempts/- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8621) Add REST API tests for Resource Types fields for the apps/ endpoint
[ https://issues.apache.org/jira/browse/YARN-8621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8621: - Summary: Add REST API tests for Resource Types fields for the apps/ endpoint (was: Add REST API tests for Resource Types fields for apps/) > Add REST API tests for Resource Types fields for the apps/ endpoint > -- > > Key: YARN-8621 > URL: https://issues.apache.org/jira/browse/YARN-8621 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8621.001.patch > > > This is a complement for YARN-7451 that already added unit tests for the apps > and scheduler endpoints. > The following API endpoints should be tested as well: > /ws/v1/cluster/apps/ > -/ws/v1/cluster/apps//appattempts- > -/ws/v1/cluster/apps//appattempts/- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8621) Add REST API tests for Resource Types fields
[ https://issues.apache.org/jira/browse/YARN-8621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8621: - Attachment: YARN-8621.001.patch > Add REST API tests for Resource Types fields > > > Key: YARN-8621 > URL: https://issues.apache.org/jira/browse/YARN-8621 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8621.001.patch > > > This is a complement for YARN-7451 that already added unit tests for the apps > and scheduler endpoints. > The following API endpoints should be tested as well: > /ws/v1/cluster/apps/ > -/ws/v1/cluster/apps//appattempts- > -/ws/v1/cluster/apps//appattempts/- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8621) Add REST API tests for Resource Types fields
[ https://issues.apache.org/jira/browse/YARN-8621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8621: - Description: This is a complement for YARN-7451 that already added unit tests for the apps and scheduler endpoints. The following API endpoints should be tested as well: /ws/v1/cluster/apps/ -/ws/v1/cluster/apps//appattempts- -/ws/v1/cluster/apps//appattempts/- was: This is a complement for YARN-7451 that already added unit tests for the apps and scheduler endpoints. The following API endpoints should be tested as well: /ws/v1/cluster/apps/ -/ws/v1/cluster/apps//appattempts- /ws/v1/cluster/apps//appattempts/ > Add REST API tests for Resource Types fields > > > Key: YARN-8621 > URL: https://issues.apache.org/jira/browse/YARN-8621 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > > This is a complement for YARN-7451 that already added unit tests for the apps > and scheduler endpoints. > The following API endpoints should be tested as well: > /ws/v1/cluster/apps/ > -/ws/v1/cluster/apps//appattempts- > -/ws/v1/cluster/apps//appattempts/- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8621) Add REST API tests for Resource Types fields
[ https://issues.apache.org/jira/browse/YARN-8621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8621: - Description: This is a complement for YARN-7451 that already added unit tests for the apps and scheduler endpoints. The following API endpoints should be tested as well: /ws/v1/cluster/apps/ -/ws/v1/cluster/apps//appattempts- /ws/v1/cluster/apps//appattempts/ was: This is a complement for YARN-7451 that already added unit tests for the apps and scheduler endpoints. The following API endpoints should be tested as well: /ws/v1/cluster/apps/ -/ws/v1/cluster/apps//appattempts /ws/v1/cluster/apps//appattempts/ - > Add REST API tests for Resource Types fields > > > Key: YARN-8621 > URL: https://issues.apache.org/jira/browse/YARN-8621 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > > This is a complement for YARN-7451 that already added unit tests for the apps > and scheduler endpoints. > The following API endpoints should be tested as well: > /ws/v1/cluster/apps/ > -/ws/v1/cluster/apps//appattempts- > /ws/v1/cluster/apps//appattempts/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8621) Add REST API tests for Resource Types fields
[ https://issues.apache.org/jira/browse/YARN-8621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8621: - Description: This is a complement for YARN-7451 that already added unit tests for the apps and scheduler endpoints. The following API endpoints should be tested as well: /ws/v1/cluster/apps/ -/ws/v1/cluster/apps//appattempts /ws/v1/cluster/apps//appattempts/ - was: This is a complement for YARN-7451 that already added unit tests for the apps and scheduler endpoints. The following API endpoints should be tested as well: /ws/v1/cluster/apps/ /ws/v1/cluster/apps//appattempts /ws/v1/cluster/apps//appattempts/ > Add REST API tests for Resource Types fields > > > Key: YARN-8621 > URL: https://issues.apache.org/jira/browse/YARN-8621 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > > This is a complement for YARN-7451 that already added unit tests for the apps > and scheduler endpoints. > The following API endpoints should be tested as well: > /ws/v1/cluster/apps/ > -/ws/v1/cluster/apps//appattempts > /ws/v1/cluster/apps//appattempts/ - -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7494) Add muti node lookup support for better placement
[
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577698#comment-16577698
]
genericqa commented on YARN-7494:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 9s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-7494 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935293/YARN-7494.15.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 8f6f0567f9f3 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3ac07b7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/21576/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21576/testReport/ |
| Max. process+thread count | 922 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Assigned] (YARN-8099) Add InterfaceAudience for Node label request and response
[ https://issues.apache.org/jira/browse/YARN-8099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi reassigned YARN-8099: --- Assignee: Abhishek Modi > Add InterfaceAudience for Node label request and response > - > > Key: YARN-8099 > URL: https://issues.apache.org/jira/browse/YARN-8099 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Abhishek Modi >Priority: Minor > > GetNodesToLabelsRequest > GetNodesToLabelsResponse > GetLabelsToNodesRequest > GetLabelsToNodesResponse > Class level above are not made public this need to be marked as public -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7494) Add muti node lookup support for better placement
[ https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577651#comment-16577651 ] Sunil Govindan commented on YARN-7494: -- Updating v15 patch after correcting default class name. cc [~cheersyang] > Add muti node lookup support for better placement > - > > Key: YARN-7494 > URL: https://issues.apache.org/jira/browse/YARN-7494 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7494.001.patch, YARN-7494.002.patch, > YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.005.patch, > YARN-7494.006.patch, YARN-7494.007.patch, YARN-7494.008.patch, > YARN-7494.009.patch, YARN-7494.010.patch, YARN-7494.11.patch, > YARN-7494.12.patch, YARN-7494.13.patch, YARN-7494.14.patch, > YARN-7494.15.patch, YARN-7494.v0.patch, YARN-7494.v1.patch, > multi-node-designProposal.png > > > Instead of single node, for effectiveness we can consider a multi node lookup > based on partition to start with. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7494) Add muti node lookup support for better placement
[ https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-7494: - Attachment: YARN-7494.15.patch > Add muti node lookup support for better placement > - > > Key: YARN-7494 > URL: https://issues.apache.org/jira/browse/YARN-7494 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7494.001.patch, YARN-7494.002.patch, > YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.005.patch, > YARN-7494.006.patch, YARN-7494.007.patch, YARN-7494.008.patch, > YARN-7494.009.patch, YARN-7494.010.patch, YARN-7494.11.patch, > YARN-7494.12.patch, YARN-7494.13.patch, YARN-7494.14.patch, > YARN-7494.15.patch, YARN-7494.v0.patch, YARN-7494.v1.patch, > multi-node-designProposal.png > > > Instead of single node, for effectiveness we can consider a multi node lookup > based on partition to start with. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577644#comment-16577644 ] Naganarasimha G R commented on YARN-7863: - Thanks for the patch [~sunilg] and apologies for the delayed review, as i was not acquainted with Placement Constraint part of the code, i required some time in understanding the placement constraints and then in relation how you have modified it. But still could not figure out few of the stuff * Not completely clear sure how the parsing happens for the *And* and *Or* cases which was mentioned by [~cheersyang] in his earlier comment "_-placement_spec OR(IN,python=true:NOTIN,java=1.8):IN,os!=centos:NOTIN,env=prod,dev"_ Just tried to replace the same in your test case but seems like it fails at PlacementConstraintParser.parseNodeAttributeExpression ln no 669. Whether its expected to fail or it should assume it as *And* and proceed? * In PlacementConstraintParser.NodeConstraintParser.parse() PlacementTargets target in ln no 399 seems to include keys also in val. Was not sure why we need to store in this way, would it not be bettert only values here * Can we place both constraints related to attributes and allocation tags together in a single placement spec for distributed shell? As I understand the Placement spec API supports to take multiple type of placement constraints for a single resource/allocation request. We too should support it right ? * NodeConstraintParser.parse seems to assume that attributeNameSpace has "rm.yarn.io/" but IIRC we concluded that user need to specify this explicitly right else exception is thrown right ? some minor comments * TestPlacementConstraintParser ln no 453 : testParseNodeAttributeSpec has attributeName1, attributeName2; which is unused * ResourceManager ln no 652-654 createNodeAttributesManager should either return NodeAttributesManagerImpl or take parameter as rmContext instead of type casting * Seems like build seems to have find bugs and test case failures. Can you please have a look at it ? > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
