[jira] [Commented] (YARN-7172) ResourceCalculator.fitsIn() should not take a cluster resource parameter

2017-09-17 Thread Sen Zhao (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169597#comment-16169597
 ] 

Sen Zhao commented on YARN-7172:


Thank you, [~leftnoteasy] and [~templedf]

> ResourceCalculator.fitsIn() should not take a cluster resource parameter
> 
>
> Key: YARN-7172
> URL: https://issues.apache.org/jira/browse/YARN-7172
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Sen Zhao
>  Labels: newbie
> Fix For: 3.1.0
>
> Attachments: YARN-7172.001.patch, YARN-7172.002.patch, 
> YARN-7172.003.patch, YARN-7172.004.patch, YARN-7172.005.patch, 
> YARN-7172.006.patch
>
>
> There are numerous calls to {{ClusterNodeTracker.getClusterResource()}} 
> (which involves a lock) to get a value to pass as the cluster resource 
> parameter to {{Resources.fitsIn()}}, but the parameter is (quite reasonably) 
> ignored.  We should remove the parameter.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7172) ResourceCalculator.fitsIn() should not take a cluster resource parameter

2017-09-17 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169578#comment-16169578
 ] 

Hudson commented on YARN-7172:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12894 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/12894/])
YARN-7172. ResourceCalculator.fitsIn() should not take a cluster (wangda: rev 
e81596d06d226f1cfa44b2390ce3095ed4dee621)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceCalculator.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DefaultResourceCalculator.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/CapacitySchedulerPreemptionUtils.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/QueuePriorityContainerCandidateSelector.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResourceCalculator.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ReservedContainerCandidatesSelector.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java


> ResourceCalculator.fitsIn() should not take a cluster resource parameter
> 
>
> Key: YARN-7172
> URL: https://issues.apache.org/jira/browse/YARN-7172
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Sen Zhao
>  Labels: newbie
> Fix For: 3.1.0
>
> Attachments: YARN-7172.001.patch, YARN-7172.002.patch, 
> YARN-7172.003.patch, YARN-7172.004.patch, YARN-7172.005.patch, 
> YARN-7172.006.patch
>
>
> There are numerous calls to {{ClusterNodeTracker.getClusterResource()}} 
> (which involves a lock) to get a value to pass as the cluster resource 
> parameter to {{Resources.fitsIn()}}, but the parameter is (quite reasonably) 
> ignored.  We should remove the parameter.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups

2017-09-17 Thread Wangda Tan (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169573#comment-16169573
 ] 

Wangda Tan commented on YARN-6620:
--

[~tangzhankun], 

Sorry to cause confusing, the attached patch still need some cleanups and 
additional code works, which will take another 2-3 days to finish. Will update 
the patch once it in a good state. 

Regarding to ur comments:
bq. 1. Current GPUResourceAllocator is not got from LocalResourceAllocators but 
created in GpuResourceHandlerImpl directly. Is this intended?
Yes it is intended, ideally each plugin should maintain its own 
allocator/handler, etc. I plan to remove LocalResourceAllocators. 

bq. 2. The GpuResourceHandler get container's requested GPU from an environment 
key "REQUESTED_GPU_NUM". So in fact, there's no need to define the allowed GPU 
resource in "node-resouce.xml"
Ah, this code is done before YARN-3926 merge, so I will update to use resource 
profile in the next uploaded patch.

bq. For instance, if different vendors' GPU cards are installed in the cluster, 
how can a user distinguish them? thru node attributes?
Good point, I think we should use node attribute to distinguish them. I think 
this might be unavoidable: different DL workload needs different driver 
versions / GPU architectures, and different frameworks like OpenCL/CUDA, we 
need node attribute anyway.

> [YARN-6223] NM Java side code changes to support isolate GPU devices by using 
> CGroups
> -
>
> Key: YARN-6620
> URL: https://issues.apache.org/jira/browse/YARN-6620
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6620.001.patch, YARN-6620.002.patch, 
> YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, 
> YARN-6620.006-WIP.patch
>
>
> This JIRA plan to add support of:
> 1) GPU configuration for NodeManagers
> 2) Isolation in CGroups. (Java side).
> 3) NM restart and recovery allocated GPU devices



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups

2017-09-17 Thread Zhankun Tang (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169566#comment-16169566
 ] 

Zhankun Tang commented on YARN-6620:


[~wangda], Thanks for the patch!
Now we have defined the resource plugin framework in NM which manages various 
resources that implemented ResourcePlugin and ResourceHandler interface easily. 
It also provide a good way for resource plugin to update NodeStatus which 
provide possibility for FPGA IP update thru node labels/attributes.

Two questions/comments from me:
1. Current GPUResourceAllocator is not got from LocalResourceAllocators but 
created in GpuResourceHandlerImpl directly. Is this intended?
2. The GpuResourceHandler get container's requested GPU from an environment key 
"REQUESTED_GPU_NUM". So in fact, there's no need to define the allowed GPU 
resource in "node-resouce.xml". This is not an issue at present, but may brings 
a potential limitation that the end-user cannot declare different type of GPU 
vendor resource. For instance, if different vendors' GPU cards are installed in 
the cluster, how can a user distinguish them? thru node attributes?

> [YARN-6223] NM Java side code changes to support isolate GPU devices by using 
> CGroups
> -
>
> Key: YARN-6620
> URL: https://issues.apache.org/jira/browse/YARN-6620
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6620.001.patch, YARN-6620.002.patch, 
> YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, 
> YARN-6620.006-WIP.patch
>
>
> This JIRA plan to add support of:
> 1) GPU configuration for NodeManagers
> 2) Isolation in CGroups. (Java side).
> 3) NM restart and recovery allocated GPU devices



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7172) ResourceCalculator.fitsIn() should not take a cluster resource parameter

2017-09-17 Thread Sen Zhao (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169538#comment-16169538
 ] 

Sen Zhao commented on YARN-7172:


HI, [~templedf]. Could you help me to review the latest patch? If there are 
other problems, I will fix it

> ResourceCalculator.fitsIn() should not take a cluster resource parameter
> 
>
> Key: YARN-7172
> URL: https://issues.apache.org/jira/browse/YARN-7172
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Sen Zhao
>  Labels: newbie
> Attachments: YARN-7172.001.patch, YARN-7172.002.patch, 
> YARN-7172.003.patch, YARN-7172.004.patch, YARN-7172.005.patch, 
> YARN-7172.006.patch
>
>
> There are numerous calls to {{ClusterNodeTracker.getClusterResource()}} 
> (which involves a lock) to get a value to pass as the cluster resource 
> parameter to {{Resources.fitsIn()}}, but the parameter is (quite reasonably) 
> ignored.  We should remove the parameter.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6182) [YARN-3368] Fix alignment issues and missing information in queue pages

2017-09-17 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169366#comment-16169366
 ] 

Hadoop QA commented on YARN-6182:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  5m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
49s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  6m 41s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6182 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887560/YARN-6182.004.patch |
| Optional Tests |  asflicense  |
| uname | Linux 074428277d04 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8d7cc22 |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/17495/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> [YARN-3368] Fix alignment issues and missing information in queue pages
> ---
>
> Key: YARN-6182
> URL: https://issues.apache.org/jira/browse/YARN-6182
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
> Attachments: YARN-6182.001.patch, YARN-6182.002.patch, 
> YARN-6182.003.patch, YARN-6182.004.patch
>
>
> This patch fixes following issues:
> In Queues page:
> # Queue Capacities: Absolute Max Capacity should be aligned better.
> # Queue Information: State is coming empty
> # Queues tab becomes inactive while hovering on the queue.
> # Fixes the capacity decimal places to two places.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-6182) [YARN-3368] Fix alignment issues and missing information in queue pages

2017-09-17 Thread Akhil PB (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169364#comment-16169364
 ] 

Akhil PB edited comment on YARN-6182 at 9/17/17 5:39 PM:
-

v4 patch: rebased on latest trunk, fixes following
# Queue Capacities: Absolute Max Capacity should be aligned better.
# Queue Information: State is coming empty
# Queues tab becomes inactive while hovering on the queue.
# Fixes the capacity decimal places to two places.
# jshint issues

cc [~sunilg]


was (Author: akhilpb):
v4 patch: rebased on latest trunk, fixes following
# Queue Capacities: Absolute Max Capacity should be aligned better.
# Queue Information: State is coming empty
# Queues tab becomes inactive while hovering on the queue.
# Fixes the capacity decimal places to two places.
# jshint issues

> [YARN-3368] Fix alignment issues and missing information in queue pages
> ---
>
> Key: YARN-6182
> URL: https://issues.apache.org/jira/browse/YARN-6182
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
> Attachments: YARN-6182.001.patch, YARN-6182.002.patch, 
> YARN-6182.003.patch, YARN-6182.004.patch
>
>
> This patch fixes following issues:
> In Queues page:
> # Queue Capacities: Absolute Max Capacity should be aligned better.
> # Queue Information: State is coming empty
> # Queues tab becomes inactive while hovering on the queue.
> # Fixes the capacity decimal places to two places.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6182) [YARN-3368] Fix alignment issues and missing information in queue pages

2017-09-17 Thread Akhil PB (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akhil PB updated YARN-6182:
---
Attachment: YARN-6182.004.patch

v4 patch: rebased on latest trunk, fixes following
# Queue Capacities: Absolute Max Capacity should be aligned better.
# Queue Information: State is coming empty
# Queues tab becomes inactive while hovering on the queue.
# Fixes the capacity decimal places to two places.
# jshint issues

> [YARN-3368] Fix alignment issues and missing information in queue pages
> ---
>
> Key: YARN-6182
> URL: https://issues.apache.org/jira/browse/YARN-6182
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
> Attachments: YARN-6182.001.patch, YARN-6182.002.patch, 
> YARN-6182.003.patch, YARN-6182.004.patch
>
>
> This patch fixes following issues:
> In Queues page:
> # Queue Capacities: Absolute Max Capacity should be aligned better.
> # Queue Information: State is coming empty
> # Queues tab becomes inactive while hovering on the queue.
> # Fixes the capacity decimal places to two places.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6626) Embed REST API service into RM

2017-09-17 Thread Eric Yang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated YARN-6626:

Attachment: YARN-6626.yarn-native-services.001.patch

First draft for bundle api service REST API in RM.  This JIRA depends on 
pom.xml file modification in YARN-7193.

The current approach is to bundle the REST API code as a jar file, and allow 
hadoop-yarn-ui project to bundle the REST API jar file, and configure web.xml 
to load REST API classes.

The drawback of this approach is the URI is prefixed with /ui2 instead of 
/ws/v1/.  /ws/v1/ is already taken by resource manager's own REST API.  We 
probably want to label this as /ws/v2/ or /app/v1/ as prefix of REST API to 
avoid confusion between resource manager API vs application REST API.

The alternative method is to move the code from ApiServer.java into 
RMWebApp.java, which makes that file bloated, and can't be separated as another 
standalone server.

> Embed REST API service into RM
> --
>
> Key: YARN-6626
> URL: https://issues.apache.org/jira/browse/YARN-6626
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
> Fix For: yarn-native-services
>
> Attachments: YARN-6626.yarn-native-services.001.patch
>
>
> As of now the deployment model of the Native Services REST API service is 
> standalone. There are several cross-cutting solutions that can be inherited 
> for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API 
> service if it is embedded into the RM process. In fact we can expose the REST 
> API via the same port as RM UI (8088 default). The URI path 
> /services/v1/applications will distinguish the REST API calls from other RM 
> APIs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7209) [YARN-3368] CSS backgound missing for breadcrumbs in new YARN-UI

2017-09-17 Thread Akhil PB (JIRA) 
Akhil PB created YARN-7209:
--

 Summary: [YARN-3368] CSS backgound missing for breadcrumbs in new 
YARN-UI
 Key: YARN-7209
 URL: https://issues.apache.org/jira/browse/YARN-7209
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Akhil PB
Assignee: Akhil PB


After latest styles changes in YANN-UI, CSS background for breadcrumbs is 
missing which was there initially.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org