date:20161205


 [ 
https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangshilong updated YARN-5969:
---
Description: 
in FairShareComparator class, the performance of function getResourceUsage()  
is very poor. It will be executed above 100,000,000 times per second.
In our scene, It  takes 20 seconds per minute.  
A simple solution is to reduce call counts  of the function.

  was:
in FairShareComparator.java, the performance of function getResourceUsage()  is 
very poor. It will be executed above 100,000,000 times per second.
In our scene, It  takes 20 seconds per minute.  
A simple solution is to reduce call counts  of the function.


> FairShareComparator getResourceUsage poor performance
> -
>
> Key: YARN-5969
> URL: https://issues.apache.org/jira/browse/YARN-5969
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: zhangshilong
> Attachments: 20161206.patch
>
>
> in FairShareComparator class, the performance of function getResourceUsage()  
> is very poor. It will be executed above 100,000,000 times per second.
> In our scene, It  takes 20 seconds per minute.  
> A simple solution is to reduce call counts  of the function.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5292) Support for PAUSED container state for OPPORTUNISTIC containers

2016-12-05 Thread Subru Krishnan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724410#comment-15724410
 ] 

Subru Krishnan commented on YARN-5292:
--

Thanks [~hrsharma]/[~asuresh] for working on this. I am on vacation so don't 
have the opportunity to follow this closely but I did raise a few concerns 
([above|https://issues.apache.org/jira/browse/YARN-5292?focusedCommentId=15685939]).
  [~hrsharma] replied to one of them based on which I have updated the JIRA 
title but the rest do *not* seem to be addressed. Am I missing something?

> Support for PAUSED container state for OPPORTUNISTIC containers
> ---
>
> Key: YARN-5292
> URL: https://issues.apache.org/jira/browse/YARN-5292
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Hitesh Sharma
>Assignee: Hitesh Sharma
> Attachments: YARN-5292.001.patch, YARN-5292.002.patch, 
> YARN-5292.003.patch, YARN-5292.004.patch, YARN-5292.005.patch, yarn-5292.pdf
>
>
> YARN-2877 introduced OPPORTUNISTIC containers, and YARN-5216 proposes to add 
> capability to customize how OPPORTUNISTIC containers get preempted.
> In this JIRA we propose introducing a PAUSED container state.
> When a running container gets preempted, it enters the PAUSED state, where it 
> remains until resources get freed up on the node then the preempted container 
> can resume to the running state.
>  
> One scenario where this capability is useful is work preservation. How 
> preemption is done, and whether the container supports it, is implementation 
> specific.
> For instance, if the container is a virtual machine, then preempt would pause 
> the VM and resume would restore it back to the running state.
> If the container doesn't support preemption, then preempt would default to 
> killing the container. 
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5292) Support for PAUSED container state for OPPORTUNISTIC containers

2016-12-05 Thread Subru Krishnan (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subru Krishnan updated YARN-5292:
-
Summary: Support for PAUSED container state for OPPORTUNISTIC containers  
(was: Support for PAUSED container state)

> Support for PAUSED container state for OPPORTUNISTIC containers
> ---
>
> Key: YARN-5292
> URL: https://issues.apache.org/jira/browse/YARN-5292
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Hitesh Sharma
>Assignee: Hitesh Sharma
> Attachments: YARN-5292.001.patch, YARN-5292.002.patch, 
> YARN-5292.003.patch, YARN-5292.004.patch, YARN-5292.005.patch, yarn-5292.pdf
>
>
> YARN-2877 introduced OPPORTUNISTIC containers, and YARN-5216 proposes to add 
> capability to customize how OPPORTUNISTIC containers get preempted.
> In this JIRA we propose introducing a PAUSED container state.
> When a running container gets preempted, it enters the PAUSED state, where it 
> remains until resources get freed up on the node then the preempted container 
> can resume to the running state.
>  
> One scenario where this capability is useful is work preservation. How 
> preemption is done, and whether the container supports it, is implementation 
> specific.
> For instance, if the container is a virtual machine, then preempt would pause 
> the VM and resume would restore it back to the running state.
> If the container doesn't support preemption, then preempt would default to 
> killing the container. 
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5969) FairShareComparator getResourceUsage poor performance


 [ 
https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangshilong updated YARN-5969:
---
Description: 
in FairShareComparator.java, the performance of function getResourceUsage()  is 
very poor. It will be executed above 100,000,000 times per second.
In our scene, It  takes 20 seconds per minute.  
A simple solution is to reduce call counts  of the function.

  was:
in FairShareComparator.java, the performance of function getResourceUsage()  is 
very pool. It will be executed above 100,000,000 times per second.
In our scene, It  takes 20 seconds per minute.  
A simple solution is to reduce call counts  of the function.


> FairShareComparator getResourceUsage poor performance
> -
>
> Key: YARN-5969
> URL: https://issues.apache.org/jira/browse/YARN-5969
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: zhangshilong
> Attachments: 20161206.patch
>
>
> in FairShareComparator.java, the performance of function getResourceUsage()  
> is very poor. It will be executed above 100,000,000 times per second.
> In our scene, It  takes 20 seconds per minute.  
> A simple solution is to reduce call counts  of the function.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5969) FairShareComparator getResourceUsage poor performance


 [ 
https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangshilong updated YARN-5969:
---
Summary: FairShareComparator getResourceUsage poor performance  (was: 
FairShareComparator getResourceUsage pool performance)

> FairShareComparator getResourceUsage poor performance
> -
>
> Key: YARN-5969
> URL: https://issues.apache.org/jira/browse/YARN-5969
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: zhangshilong
> Attachments: 20161206.patch
>
>
> in FairShareComparator.java, the performance of function getResourceUsage()  
> is very pool. It will be executed above 100,000,000 times per second.
> In our scene, It  takes 20 seconds per minute.  
> A simple solution is to reduce call counts  of the function.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5734) OrgQueue for easy CapacityScheduler queue configuration management


[ 
https://issues.apache.org/jira/browse/YARN-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724356#comment-15724356
 ] 

Jian He commented on YARN-5734:
---

[~mshen], [~jhung], [~zhz], very useful feature! thanks for the contribution, 
Some questions I had about the design:
- Does add/remove also support a full qualified queue name, not just a 
hierachical structure ? I think supporting a single full qualified queue name 
would be handy, especially for CLI add/remove
- IIUC, the xml-file will still be used for initialization on startup, even if 
the API-based approach is enabled ? Then, if the RM gets restarted, will the RM 
honor the xml file or the config store for initialization ? I feel both 
scenarios may be possible:
-- If it is a crash-and-restart, probably we should honor the config 
store.
-- If RM is going through a rolling upgrade. User may need to provide a 
new queue structure for initialization, then, the xml file will conflict with 
what's in config store.
- Is the implementation that the caller will block until the update is 
completed - both in store and memory ? 
- IIUC, the EmbededDerbyDatabase is suitable for single RM only. Do you run RM 
HA in your cluster? Also, I guess Derby does not support fencing ? If so, we 
could potentially have two RMs writing together in a split-brain situation and 
cause data inconsistency. Therefore, I think ZKRMStateStore might be a better 
store option by default, especially because of RM HA. 
- Regarding PluggableConfigurationPolicy for authorization, has the 
implementation considered using YarnAuthorizationProvider ? 
YarnAuthorizationProvider is a interface which can be implemented by other 
authorization plugin(Apache Ranger). Ranger has a nice web portal where it can 
define arbitrary authorization policies such as restricting certain user/groups 
from doing certain operations. It would be useful if it did, as Ranger plugin 
just needs to implement the necessary interface and get the config 
authorization for free.

> OrgQueue for easy CapacityScheduler queue configuration management
> --
>
> Key: YARN-5734
> URL: https://issues.apache.org/jira/browse/YARN-5734
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Min Shen
>Assignee: Min Shen
> Attachments: OrgQueue_API-Based_Config_Management_v1.pdf, 
> OrgQueue_Design_v0.pdf
>
>
> The current xml based configuration mechanism in CapacityScheduler makes it 
> very inconvenient to apply any changes to the queue configurations. We saw 2 
> main drawbacks in the file based configuration mechanism:
> # This makes it very inconvenient to automate queue configuration updates. 
> For example, in our cluster setup, we leverage the queue mapping feature from 
> YARN-2411 to route users to their dedicated organization queues. It could be 
> extremely cumbersome to keep updating the config file to manage the very 
> dynamic mapping between users to organizations.
> # Even a user has the admin permission on one specific queue, that user is 
> unable to make any queue configuration changes to resize the subqueues, 
> changing queue ACLs, or creating new queues. All these operations need to be 
> performed in a centralized manner by the cluster administrators.
> With these current limitations, we realized the need of a more flexible 
> configuration mechanism that allows queue configurations to be stored and 
> managed more dynamically. We developed the feature internally at LinkedIn 
> which introduces the concept of MutableConfigurationProvider. What it 
> essentially does is to provide a set of configuration mutation APIs that 
> allows queue configurations to be updated externally with a set of REST APIs. 
> When performing the queue configuration changes, the queue ACLs will be 
> honored, which means only queue administrators can make configuration changes 
> to a given queue. MutableConfigurationProvider is implemented as a pluggable 
> interface, and we have one implementation of this interface which is based on 
> Derby embedded database.
> This feature has been deployed at LinkedIn's Hadoop cluster for a year now, 
> and have gone through several iterations of gathering feedbacks from users 
> and improving accordingly. With this feature, cluster administrators are able 
> to automate lots of thequeue configuration management tasks, such as setting 
> the queue capacities to adjust cluster resources between queues based on 
> established resource consumption patterns, or managing updating the user to 
> queue mappings. We have attached our design documentation with this ticket 
> and would like to receive feedbacks from the community regarding how to best 
> integrate it with the latest version of YARN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-5969) FairShareComparator getResourceUsage pool performance


 [ 
https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangshilong updated YARN-5969:
---
Attachment: 20161206.patch

> FairShareComparator getResourceUsage pool performance
> -
>
> Key: YARN-5969
> URL: https://issues.apache.org/jira/browse/YARN-5969
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: zhangshilong
> Attachments: 20161206.patch
>
>
> in FairShareComparator.java, the performance of function getResourceUsage()  
> is very pool. It will be executed above 100,000,000 times per second.
> In our scene, It  takes 20 seconds per minute.  
> A simple solution is to reduce call counts  of the function.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5969) FairShareComparator getResourceUsage pool performance

zhangshilong created YARN-5969:
--

 Summary: FairShareComparator getResourceUsage pool performance
 Key: YARN-5969
 URL: https://issues.apache.org/jira/browse/YARN-5969
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: fairscheduler
Affects Versions: 2.7.1
Reporter: zhangshilong


in FairShareComparator.java, the performance of function getResourceUsage()  is 
very pool. It will be executed above 100,000,000 times per second.
In our scene, It  takes 20 seconds per minute.  
A simple solution is to reduce call counts  of the function.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5967) Fix slider core module findbugs warnings


[ 
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724210#comment-15724210
 ] 

Hadoop QA commented on YARN-5967:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
26s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
16s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
52s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
 in yarn-native-services has 268 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
27s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services 
failed. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
23s{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
18s{color} | {color:green} hadoop-yarn-slider-core in the patch passed. {color} 
|
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 20s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5967 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841876/YARN-5967-yarn-native-services.01.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 9b50268b92fc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | yarn-native-services / 69283ee |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs | 
https://builds.apache.org/job/PreCommit-YARN-Build/14194/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-slider_hadoop-yarn-slider-core-warnings.html
 |
| javadoc | 
https://builds.apache.org/job/PreCommit-YARN-Build/14194/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-slider_hadoop-yarn-slider-core.txt
 |
| whitespace |

[jira] [Updated] (YARN-5967) Fix slider core module findbugs warnings


 [ 
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5967:
--
Attachment: YARN-5967-yarn-native-services.01.patch

> Fix slider core module findbugs warnings 
> -
>
> Key: YARN-5967
> URL: https://issues.apache.org/jira/browse/YARN-5967
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5967-yarn-native-services.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5967) Fix slider core module findbugs warnings


 [ 
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5967:
--
Attachment: (was: YARN-5967-yarn-native-services-01.patch)

> Fix slider core module findbugs warnings 
> -
>
> Key: YARN-5967
> URL: https://issues.apache.org/jira/browse/YARN-5967
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5967-yarn-native-services.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3409) Add constraint node labels

2016-12-05 Thread Naganarasimha G R (JIRA)

[
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724142#comment-15724142
]

Naganarasimha G R commented on YARN-3409:
-

[~leftnoteasy]

bq. The problem is the predefined type cannot solve the problem like r+version,
or version in format like (x.y.z-build). And customized type sounds too heavy
for the solution.
bq. This is not a perfect solution, but I think we cannot handle it well in the
predefined-type too, for example, how to avoid two versions with different
format, one is 1.8.3 and another one is 1.8.
well its not one version type solving for all versions, what i tried to capture
was representation of one such version comparison. Version format is not going
to be so dynamic for a software that we add or remove dynamically some sections
to it. so if for a given software they want to parse string at the end or the
number of longs separated by a "." can be easily custom coded into a class.

bq. Long array: Any version could be converted to long array.
the above example {{x.y.z-build}} which you gave doesnt fit in long array right
? suppose we some how come with a pattern to match these stuff with a internal
structure, how many such patterns we need to support and how to verify patterns
against other patterns ?

bq. And I found all the long constraints in design doc should be addressed by
resource profile.
LONG types in the example are
{code}
NUM_OF_DISKS :=> run container on nodes which has more local disks. Spark kind
of tasks doing a lot of data processing would like to have more disks to
process the data parallelly,
NUM_OF_INTF :=> run container on nodes which has dual networks (generally
required for AM’s to be run in dual plane so that its accessible for clients),
MAX_MEM :=> run container on nodes which has more memory (so that there will be
chances for resizing the container even though the current allocation is Less ,
suitable for spark )
{code}
all these scenarios captured are of non consumeable resources types, hence
cannot be addressed as another resource type (so as to solve through resource
profiles).

bq. But if we really need it, we can treat it as a long\[1\], or have a new
internal type long.
how to capture internal type as long/ string or any other custom type ? if its
inherent logic and if it doesnt match what user intended then none of the
resource requests matches or lead to undesired allocations.

bq. Other type: I don't really think we need any other type, for example,
double sounds not necessary, and types like DATE/BOOLEAN can be covered by
string.
Agree that Boolean can be acheived with String itself and for DATE might not be
immediately required but consider for date different formats of parsing to be
supported. And also consider the scenario where in we want to schedule on the
node where load average for past (1/5/15) mins is less than a particular value ?
All these kinds non consumables resources are supported in one way or the other
in other schedulers

bq. For example, we can have a internal field in RMNodeLabal to save its long
array (or any other required internal type beyond string).
Based on regex identifying internal type is risky we may match wrongly than
what user intends to (if we do it based on some order), this would lead to
allocations happening in a undesired manner. Also how/who provides what is the
internal type/regex again there would be some kind of input which needs to be
taken for a constraint right ? If its predefined type then as mentioned it
would be done in some order which might lead to undesired allocations.

bq. For the type compatibility issue, if both of them have different internal
type (like long array v long array), we can downgrade to string comparison.
This is not a perfect solution, but I think we cannot handle it well in the
predefined-type too.
If we downgrade to string comarison for long and decimal values then obviously
there will be cases where in compare operations (like greater than , less than)
will fail.
And in my predefined type we do not allow constraint expression if it doesnt
match the constraint type and for the example (1.8 and 1.8.3), it needs to be
handled as custom version type rather a generic version type.

bq. And customized type sounds too heavy for the solution, I don't really think
we should support such complex case.
Well is it heavy in terms of implementation or usage ? As we keep string as the
default type in usage front its pretty simple and implementation and
maintainence front i dont see too much of a problem other than changes which is
required on the existing labels.
Most of the back end implementation would hardly require any huge change as it
needs to store another kind and we can make use most of the commonNodeLabels
Manager and store.
IMHO I would like it to be flexible enough for future scenarios too hence i
would still prefer for

[jira] [Commented] (YARN-5554) MoveApplicationAcrossQueues does not check user permission on the target queue


[ 
https://issues.apache.org/jira/browse/YARN-5554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724046#comment-15724046
 ] 

Hadoop QA commented on YARN-5554:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 0 new + 77 unchanged - 3 fixed = 77 total (was 80) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 11s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 50s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5554 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841865/YARN-5554.11.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 0b2be41230d2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / dcedb72 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/14192/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14192/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14192/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> MoveApplicationAcrossQueues does not check user permission on the target queue
>

[jira] [Commented] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler


[ 
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723998#comment-15723998
 ] 

Hadoop QA commented on YARN-4390:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 
47s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 13 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
21s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
28s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
32s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
43s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
19s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
15s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
24s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 27s{color} 
| {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_111
 with JDK v1.8.0_111 generated 1 new + 26 unchanged - 11 fixed = 27 total (was 
37) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 28s{color} 
| {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121
 with JDK v1.7.0_121 generated 1 new + 26 unchanged - 11 fixed = 27 total (was 
37) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 22s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 30 new + 526 unchanged - 17 fixed = 556 total (was 543) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
17s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_111
 with JDK v1.8.0_111 generated 1 new + 975 unchanged - 0 fixed = 976 total (was 
975) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 20s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK 
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 21s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_111 Failed junit tests | 
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
|   |

[jira] [Commented] (YARN-5921) Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState

2016-12-05 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723985#comment-15723985
 ] 

Hudson commented on YARN-5921:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10942 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10942/])
YARN-5921. Incorrect synchronization in (naganarasimha_gr: rev 
f3b8ff54ab08545d7093bf8861b44ec9912e8dc3)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java


> Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState
> --
>
> Key: YARN-5921
> URL: https://issues.apache.org/jira/browse/YARN-5921
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-5921.01.patch, YARN-5921.02.patch
>
>
> Code in RMContextImpl is as under:
> {code:title=RMContextImpl.java|borderStyle=solid}
>   void setHAServiceState(HAServiceState haServiceState) {
> synchronized (haServiceState) {
>   this.haServiceState = haServiceState;
> }
>   }
>   public HAServiceState getHAServiceState() {
> synchronized (haServiceState) {
>   return haServiceState;
> }
>   }
> {code}
> As can be seen above, in setHAServiceState, we are synchronizing on the 
> passed haServiceState instead of haServiceState in RMContextImpl which will 
> not lead to desired effect. This does not seem to be intentional.
> We can use a RW lock or synchronize on some object here. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5967) Fix slider core module findbugs warnings


[ 
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723983#comment-15723983
 ] 

Hadoop QA commented on YARN-5967:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  7s{color} 
| {color:red} YARN-5967 does not apply to trunk. Rebase required? Wrong Branch? 
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5967 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841868/YARN-5967-yarn-native-services-01.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14193/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Fix slider core module findbugs warnings 
> -
>
> Key: YARN-5967
> URL: https://issues.apache.org/jira/browse/YARN-5967
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5967-yarn-native-services-01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5968) Fix slider core module javadocs

Jian He created YARN-5968:
-

 Summary: Fix slider core module javadocs
 Key: YARN-5968
 URL: https://issues.apache.org/jira/browse/YARN-5968
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5967) Fix slider core module findbugs warnings


 [ 
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5967:
--
Attachment: YARN-5967-yarn-native-services-01.patch

a dummy patch to trigger jenkins

> Fix slider core module findbugs warnings 
> -
>
> Key: YARN-5967
> URL: https://issues.apache.org/jira/browse/YARN-5967
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5967-yarn-native-services-01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5967) Fix slider core module findbugs warnings

Jian He created YARN-5967:
-

 Summary: Fix slider core module findbugs warnings 
 Key: YARN-5967
 URL: https://issues.apache.org/jira/browse/YARN-5967
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4330) MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages.

2016-12-05 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723951#comment-15723951
 ] 

Naganarasimha G R commented on YARN-4330:
-

Hi [~varun_saxena] & [~steve_l],
Is it ok to fix this issue in 2.9 and trunk ? shall i resolve this issue or 
still you guys have plans for 2.8 ? 
IMO its not a must for 2.8 but feel its good to have. Please share your 
thoughts 

> MiniYARNCluster is showing multiple  Failed to instantiate default resource 
> calculator warning messages.
> 
>
> Key: YARN-4330
> URL: https://issues.apache.org/jira/browse/YARN-4330
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test, yarn
>Affects Versions: 2.8.0
> Environment: OSX, JUnit
>Reporter: Steve Loughran
>Assignee: Varun Saxena
>Priority: Blocker
>  Labels: oct16-hard
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-4330.002.patch, YARN-4330.003.patch, 
> YARN-4330.004.patch, YARN-4330.01.patch
>
>
> Whenever I try to start a MiniYARNCluster on Branch-2 (commit #0b61cca), I 
> see multiple stack traces warning me that a resource calculator plugin could 
> not be created
> {code}
> (ResourceCalculatorPlugin.java:getResourceCalculatorPlugin(184)) - 
> java.lang.UnsupportedOperationException: Could not determine OS: Failed to 
> instantiate default resource calculator.
> java.lang.UnsupportedOperationException: Could not determine OS
> {code}
> This is a minicluster. It doesn't need resource calculation. It certainly 
> doesn't need test logs being cluttered with even more stack traces which will 
> only generate false alarms about tests failing. 
> There needs to be a way to turn this off, and the minicluster should have it 
> that way by default.
> Being ruthless and marking as a blocker, because its a fairly major 
> regression for anyone testing with the minicluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3409) Add constraint node labels


[ 
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723947#comment-15723947
 ] 

Wangda Tan commented on YARN-3409:
--

[~Naganarasimha],

bq. ... this is one such example we had faced earlier, but based on scenario it 
can vary, but if we have a custom type and define how to compare it would be 
better to adapt anything in future. ...
The problem is the predefined type cannot solve the problem like r+version, or 
version in format like (x.y.z-build). And customized type sounds too heavy for 
the solution, I don't really think we should support such complex case.

To me we should only support common constraint use cases. In my mind, we don't 
need explicit type (I will explain why), and two internal type will be enough:
1) String:
2) Long array: Any version could be converted to long array. And I found all 
the long constraints in design doc should be addressed by resource profile. But 
if we really need it, we can treat it as a long\[1\], or have a new internal 
type long.
3) Other type: I don't really think we need any other type, for example, double 
sounds not necessary, and types like DATE/BOOLEAN can be covered by string.

The reason you mentioned to type is valid but we can still get good performance 
without predefine the type. For example, we can have a internal field in 
RMNodeLabal to save its long array (or any other required internal type beyond 
string). This could be done automatically based on regex matching when we add 
the constraint. And we can do the same thing on node constraint expr. Once they 
are done, if the expr is >=, and both of constraint pattern and node constraint 
have pre-saved long array, we can just compare the arrays.

For the type compatibility issue, if both of them have different internal type 
(like long array v long array), we can downgrade to string comparison. This is 
not a perfect solution, but I think we cannot handle it well in the 
predefined-type too, for example, how to avoid two versions with different 
format, one is 1.8.3 and another one is 1.8.

bq. further it would be better to have for constraint Name too, which will be 
helpful for Boolean type.
I think we should have constraint name, a constraint should have name and 
value, the type is arguable per my above comment. 

Thoughts?


> Add constraint node labels
> --
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, capacityscheduler, client
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
> Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, 
> YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to 
> determinate how resources of a special set of nodes could be shared by a 
> group of entities (like teams, departments, etc.). Partitions of a cluster 
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has 
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% 
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of 
> node’s hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 
> 2.20 && JDK.version >= 8u20 && x86_64).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5756) Add state-machine implementation for queues


[ 
https://issues.apache.org/jira/browse/YARN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723920#comment-15723920
 ] 

Hadoop QA commented on YARN-5756:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 57s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 2 new + 609 unchanged - 3 fixed = 611 total (was 612) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
32s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 42m 44s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 39s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5756 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841852/YARN-5756.5.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 2b3a1e8e885f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / dcedb72 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/14191/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/14191/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results |

[jira] [Updated] (YARN-5554) MoveApplicationAcrossQueues does not check user permission on the target queue

2016-12-05 Thread Wilfred Spiegelenburg (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wilfred Spiegelenburg updated YARN-5554:

Attachment: YARN-5554.11.patch

removing unused imports to fix checkstyle warnings

> MoveApplicationAcrossQueues does not check user permission on the target queue
> --
>
> Key: YARN-5554
> URL: https://issues.apache.org/jira/browse/YARN-5554
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.2
>Reporter: Haibo Chen
>Assignee: Wilfred Spiegelenburg
>  Labels: oct16-medium
> Attachments: YARN-5554.10.patch, YARN-5554.11.patch, 
> YARN-5554.2.patch, YARN-5554.3.patch, YARN-5554.4.patch, YARN-5554.5.patch, 
> YARN-5554.6.patch, YARN-5554.7.patch, YARN-5554.8.patch, YARN-5554.9.patch
>
>
> moveApplicationAcrossQueues operation currently does not check user 
> permission on the target queue. This incorrectly allows one user to move 
> his/her own applications to a queue that the user has no access to



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5525) Make log aggregation service class configurable


[ 
https://issues.apache.org/jira/browse/YARN-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723872#comment-15723872
 ] 

Jian He commented on YARN-5525:
---

yeah, we should probably have both log aggregator and logFetcher pluggable so 
that this is a useful feature from end to end. It's ok to get one part in 
first, but we should have an end-to-end design.. 

> Make log aggregation service class configurable
> ---
>
> Key: YARN-5525
> URL: https://issues.apache.org/jira/browse/YARN-5525
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation
>Reporter: Giovanni Matteo Fumarola
>Assignee: Botong Huang
>Priority: Minor
>  Labels: oct16-medium
> Attachments: YARN-5525.v1.patch, YARN-5525.v2.patch, 
> YARN-5525.v3.patch, YARN-5525.v4.patch, YARN-5525.v5.patch
>
>
> Make the log aggregation class configurable and extensible, so that 
> alternative log aggregation behaviors like app specific log aggregation 
> directory, log aggregation format can be implemented and plugged in.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3409) Add constraint node labels

2016-12-05 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723856#comment-15723856
 ] 

Naganarasimha G R commented on YARN-3409:
-

Thanks [~wangda] for the quick feedback,
bq. I still think constraint is one special kind of label, we should reuse 
existing APIs: 
IMHO It depends on how we conclude on whether we require *constraint type*, 
lets further discuss on it and then come back to this. and further i cross 
verified, my bad we support replace nodelabel mappings and adding cluster node 
labels through REST.

bq. Do we really need define type of the constraint? 
In the example which i had given software version x.y.z it always doesn't have 
the numerical digits to be padded for example {{r2 and r19}}, so ascii 
comparison might fail, this is one such example we had faced earlier, but based 
on scenario it can vary, but if we have a custom type and define how to compare 
it would be better to adapt anything in future.

bq. I think we can do late-binding it. 
May be value for the node's constraint is given as {{Long}} and in the 
expression we give it as {{Double}} or vice versa, then evaluation will lead to 
run time failures right ? or for each evaluation we need to check and define 
what is the better match or how to compare two different kinds of values. Late 
binding would be *too much costly/expensive* when each request has a constraint 
expression. As we need to almost validate for such scenarios for each node 
against the constraint expression of a resource request in each schedule cycle. 
In the approach which i have specified in the patch its almost static 
comparison of long to long, or double to double, or type to type, as the type 
is predetermined type for the Node's value and the RHS value of the expression.

bq. sometimes we need compare it (like version > 2.3.4.5 and version < 2.3.4.9) 
and sometimes we need do string much (like version ~ 2.3.4.*)
bq. Maybe we can support "like" operator as well, it will compare string with 
given regex.
Agree i too was thinking about supporting regex for the the String Constraint 
type. further it would be better to have for constraint Name too, which will be 
helpful for Boolean type. 

bq. We only need support one set of syntax (for example, equal is "==", but 
"eq" will not be a valid operation name)
Yeah this is not a basic requirement, it would have been required if we 
directly use it this expression in REST api's but we usually use as part of 
ResourceRequest object, but just thought in case we want to support dynamic 
Constraint expression modification for a RR in future.

I would like to get more feedback on the constraint type from others too, hope 
to get early feedback
cc [~kkaranasos] [~devaraj.k] [~cchen317] [~grey]  and others in the 
subscriber's list.




> Add constraint node labels
> --
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, capacityscheduler, client
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
> Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, 
> YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to 
> determinate how resources of a special set of nodes could be shared by a 
> group of entities (like teams, departments, etc.). Partitions of a cluster 
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has 
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% 
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of 
> node’s hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 
> 2.20 && JDK.version >= 8u20 && x86_64).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5910) Support for multi-cluster delegation tokens


[ 
https://issues.apache.org/jira/browse/YARN-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723829#comment-15723829
 ] 

Jian He commented on YARN-5910:
---

bq. ideally, the token should be self-sufficient to discover the renewer 
address.
After digging the code more for this approach, even in non-HA mode, conf is 
also required for things like retry settings, also the principal name is 
required for secure setting.  Basically the Token has to selectively carry all 
the necessary conf for connecting to the renewer in HA, non-HA, secure 
scenarios.  How to maintain such an unknown list is a non-trivial task in the 
first place. I'd prefer the passing via appSubmissionContext approach now. 

> Support for multi-cluster delegation tokens
> ---
>
> Key: YARN-5910
> URL: https://issues.apache.org/jira/browse/YARN-5910
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: security
>Reporter: Clay B.
>Priority: Minor
>
> As an administrator running many secure (kerberized) clusters, some which 
> have peer clusters managed by other teams, I am looking for a way to run jobs 
> which may require services running on other clusters. Particular cases where 
> this rears itself are running something as core as a distcp between two 
> kerberized clusters (e.g. {{hadoop --config /home/user292/conf/ distcp 
> hdfs://LOCALCLUSTER/user/user292/test.out 
> hdfs://REMOTECLUSTER/user/user292/test.out.result}}).
> Thanks to YARN-3021, once can run for a while but if the delegation token for 
> the remote cluster needs renewal the job will fail[1]. One can pre-configure 
> their {{hdfs-site.xml}} loaded by the YARN RM to know of all possible HDFSes 
> available but that requires coordination that is not always feasible, 
> especially as a cluster's peers grow into the tens of clusters or across 
> management teams. Ideally, one could have core systems configured this way 
> but jobs could also specify their own handling of tokens and management when 
> needed?
> [1]: Example stack trace when the RM is unaware of a remote service:
> 
> {code}
> 2016-03-23 14:59:50,528 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
>  application_1458441356031_3317 found existing hdfs token Kind: 
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:REMOTECLUSTER, Ident: 
> (HDFS_DELEGATION_TOKEN token
>  10927 for user292)
> 2016-03-23 14:59:50,557 WARN 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
>  Unable to add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, 
> Service: ha-hdfs:REMOTECLUSTER, Ident: (HDFS_DELEGATION_TOKEN token 10927 for 
> user292)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:427)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:78)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:781)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:762)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.IOException: Unable to map logical nameservice URI 
> 'hdfs://REMOTECLUSTER' to a NameNode. Local configuration does not have a 
> failover proxy provider configured.
> at org.apache.hadoop.hdfs.DFSClient$Renewer.getNNProxy(DFSClient.java:1164)
> at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:1128)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:516)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:513)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:511)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:425)
> ... 6 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (YARN-5768) Integrate remaining app lifetime using feature implemented in YARN-4206


 [ 
https://issues.apache.org/jira/browse/YARN-5768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He reassigned YARN-5768:
-

Assignee: Jian He

> Integrate remaining app lifetime using feature implemented in YARN-4206
> ---
>
> Key: YARN-5768
> URL: https://issues.apache.org/jira/browse/YARN-5768
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Jian He
> Fix For: yarn-native-services
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5756) Add state-machine implementation for queues

2016-12-05 Thread Xuan Gong (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-5756:

Attachment: YARN-5756.5.patch

Fix the test-case failures

> Add state-machine implementation for queues
> ---
>
> Key: YARN-5756
> URL: https://issues.apache.org/jira/browse/YARN-5756
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-5756.1.patch, YARN-5756.2.patch, YARN-5756.3.patch, 
> YARN-5756.4.patch, YARN-5756.5.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5184) Fix up incompatible changes introduced on ContainerStatus and NodeReport

2016-12-05 Thread Junping Du (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723717#comment-15723717
 ] 

Junping Du commented on YARN-5184:
--

+1 on patch for branch-2 (and trunk) and branch-2.8.
Will commit it tomorrow if no other objections.

> Fix up incompatible changes introduced on ContainerStatus and NodeReport
> 
>
> Key: YARN-5184
> URL: https://issues.apache.org/jira/browse/YARN-5184
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: api
>Affects Versions: 2.8.0, 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Sangjin Lee
>Priority: Blocker
> Attachments: YARN-5184-branch-2.8.01.patch, YARN-5184.01.patch
>
>
> YARN-2882 and YARN-5430 broke compatibility by adding abstract methods to 
> ContainerStatus. Since ContainerStatus is a Public-Stable class, adding 
> abstract methods to this class breaks any extensions. 
> To fix this, we should add default implementations to these new methods and 
> not leave them as abstract. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5739) Provide timeline reader API to list available timeline entity types for one application


[ 
https://issues.apache.org/jira/browse/YARN-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723699#comment-15723699
 ] 

Hadoop QA commented on YARN-5739:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
27s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
6s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
2s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
37s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
13s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
33s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m  
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 36s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 
29 unchanged - 1 fixed = 30 total (was 30) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
56s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
15s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in 
the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 24s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5739 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841829/YARN-5739-YARN-5355.007.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 29a372d1a6c1 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-5355 / f734977 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/14189/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14189/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice

[jira] [Commented] (YARN-5739) Provide timeline reader API to list available timeline entity types for one application


[ 
https://issues.apache.org/jira/browse/YARN-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723688#comment-15723688
 ] 

Sangjin Lee commented on YARN-5739:
---

The latest patch LGTM. [~varun_saxena], let me know what you think.

> Provide timeline reader API to list available timeline entity types for one 
> application
> ---
>
> Key: YARN-5739
> URL: https://issues.apache.org/jira/browse/YARN-5739
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Li Lu
>Assignee: Li Lu
> Attachments: YARN-5739-YARN-5355.001.patch, 
> YARN-5739-YARN-5355.002.patch, YARN-5739-YARN-5355.003.patch, 
> YARN-5739-YARN-5355.004.patch, YARN-5739-YARN-5355.005.patch, 
> YARN-5739-YARN-5355.006.patch, YARN-5739-YARN-5355.007.patch
>
>
> Right now we only show a part of available timeline entity data in the new 
> YARN UI. However, some data (especially library specific data) are not 
> possible to be queried out by the web UI. It will be appealing for the UI to 
> provide an "entity browser" for each YARN application. Actually, simply 
> dumping out available timeline entities (with proper pagination, of course) 
> would be pretty helpful for UI users. 
> On timeline side, we're not far away from this goal. Right now I believe the 
> only thing missing is to list all available entity types within one 
> application. The challenge here is that we're not storing this data for each 
> application, but given this kind of call is relatively rare (compare to 
> writes and updates) we can perform some scanning during the read time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5292) Support for PAUSED container state


[ 
https://issues.apache.org/jira/browse/YARN-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723652#comment-15723652
 ] 

Arun Suresh commented on YARN-5292:
---

Thanks for the update [~hrsharma].

I am +1 for the latest patch.
Would like to hear [~sidharta-s]'s and [~vvasudev]'s thoughts on integrating 
this functionality with the {{LinuxContainerExecutor}} and/or possibly the 
associated Runtimes since docker does support Container pausing which 
internally uses cgroups freezer module, which means it should be exposable via 
the default runtime as well.

Will commit this tomorrow, if its ok with everyone else tracking this.


> Support for PAUSED container state
> --
>
> Key: YARN-5292
> URL: https://issues.apache.org/jira/browse/YARN-5292
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Hitesh Sharma
>Assignee: Hitesh Sharma
> Attachments: YARN-5292.001.patch, YARN-5292.002.patch, 
> YARN-5292.003.patch, YARN-5292.004.patch, YARN-5292.005.patch, yarn-5292.pdf
>
>
> YARN-2877 introduced OPPORTUNISTIC containers, and YARN-5216 proposes to add 
> capability to customize how OPPORTUNISTIC containers get preempted.
> In this JIRA we propose introducing a PAUSED container state.
> When a running container gets preempted, it enters the PAUSED state, where it 
> remains until resources get freed up on the node then the preempted container 
> can resume to the running state.
>  
> One scenario where this capability is useful is work preservation. How 
> preemption is done, and whether the container supports it, is implementation 
> specific.
> For instance, if the container is a virtual machine, then preempt would pause 
> the VM and resume would restore it back to the running state.
> If the container doesn't support preemption, then preempt would default to 
> killing the container. 
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5756) Add state-machine implementation for queues

2016-12-05 Thread Li Lu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723639#comment-15723639
 ] 

Li Lu commented on YARN-5756:
-

Thanks [~xgong]. Looks fine but I'm not extremely familiar with 
queue/schedulers. Maybe [~wangda] or [~jianhe] can take a look at it? 

> Add state-machine implementation for queues
> ---
>
> Key: YARN-5756
> URL: https://issues.apache.org/jira/browse/YARN-5756
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-5756.1.patch, YARN-5756.2.patch, YARN-5756.3.patch, 
> YARN-5756.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5559) Analyse 2.8.0/3.0.0 jdiff reports and fix any issues


[ 
https://issues.apache.org/jira/browse/YARN-5559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723625#comment-15723625
 ] 

Wangda Tan commented on YARN-5559:
--

Really apologize for my delayed response, thanks a lot to [~ajisakaa] for 
taking care of this. And thanks [~jianhe]/[~djp] for reviewing the patch.

> Analyse 2.8.0/3.0.0 jdiff reports and fix any issues
> 
>
> Key: YARN-5559
> URL: https://issues.apache.org/jira/browse/YARN-5559
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Wangda Tan
>Assignee: Akira Ajisaka
>Priority: Blocker
>  Labels: oct16-easy
> Fix For: 2.8.0, 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5559.1.patch, YARN-5559.2.patch, YARN-5559.3.patch, 
> YARN-5559.4.patch, YARN-5559.5.patch, YARN-5559.6.patch, YARN-5559.7.patch, 
> YARN-5559.8.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler

2016-12-05 Thread Eric Payne (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated YARN-4390:
-
Attachment: YARN-4390.branch-2.8.001.patch

> Do surgical preemption based on reserved container in CapacityScheduler
> ---
>
> Key: YARN-4390
> URL: https://issues.apache.org/jira/browse/YARN-4390
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1
>Reporter: Eric Payne
>Assignee: Wangda Tan
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: QueueNotHittingMax.jpg, YARN-4390-design.1.pdf, 
> YARN-4390-test-results.pdf, YARN-4390.1.patch, YARN-4390.2.patch, 
> YARN-4390.3.branch-2.patch, YARN-4390.3.patch, YARN-4390.4.patch, 
> YARN-4390.5.patch, YARN-4390.6.patch, YARN-4390.7.patch, YARN-4390.8.patch, 
> YARN-4390.branch-2.8.001.patch
>
>
> There are multiple reasons why preemption could unnecessarily preempt 
> containers. One is that an app could be requesting a large container (say 
> 8-GB), and the preemption monitor could conceivably preempt multiple 
> containers (say 8, 1-GB containers) in order to fill the large container 
> request. These smaller containers would then be rejected by the requesting AM 
> and potentially given right back to the preempted app.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Reopened] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler

2016-12-05 Thread Eric Payne (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne reopened YARN-4390:
--

The backport to 2.8 was a little tricky. I would like to reopen this JIRA, 
attach the 2.8 patch, and move the JIRA to Patch Available to trigger a 2.8 
build.

> Do surgical preemption based on reserved container in CapacityScheduler
> ---
>
> Key: YARN-4390
> URL: https://issues.apache.org/jira/browse/YARN-4390
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1
>Reporter: Eric Payne
>Assignee: Wangda Tan
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: QueueNotHittingMax.jpg, YARN-4390-design.1.pdf, 
> YARN-4390-test-results.pdf, YARN-4390.1.patch, YARN-4390.2.patch, 
> YARN-4390.3.branch-2.patch, YARN-4390.3.patch, YARN-4390.4.patch, 
> YARN-4390.5.patch, YARN-4390.6.patch, YARN-4390.7.patch, YARN-4390.8.patch
>
>
> There are multiple reasons why preemption could unnecessarily preempt 
> containers. One is that an app could be requesting a large container (say 
> 8-GB), and the preemption monitor could conceivably preempt multiple 
> containers (say 8, 1-GB containers) in order to fill the large container 
> request. These smaller containers would then be rejected by the requesting AM 
> and potentially given right back to the preempted app.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5739) Provide timeline reader API to list available timeline entity types for one application

2016-12-05 Thread Li Lu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Li Lu updated YARN-5739:

Attachment: YARN-5739-YARN-5355.007.patch

Thanks [~varun_saxena] for the comments. Addressed all of them. 

> Provide timeline reader API to list available timeline entity types for one 
> application
> ---
>
> Key: YARN-5739
> URL: https://issues.apache.org/jira/browse/YARN-5739
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Li Lu
>Assignee: Li Lu
> Attachments: YARN-5739-YARN-5355.001.patch, 
> YARN-5739-YARN-5355.002.patch, YARN-5739-YARN-5355.003.patch, 
> YARN-5739-YARN-5355.004.patch, YARN-5739-YARN-5355.005.patch, 
> YARN-5739-YARN-5355.006.patch, YARN-5739-YARN-5355.007.patch
>
>
> Right now we only show a part of available timeline entity data in the new 
> YARN UI. However, some data (especially library specific data) are not 
> possible to be queried out by the web UI. It will be appealing for the UI to 
> provide an "entity browser" for each YARN application. Actually, simply 
> dumping out available timeline entities (with proper pagination, of course) 
> would be pretty helpful for UI users. 
> On timeline side, we're not far away from this goal. Right now I believe the 
> only thing missing is to list all available entity types within one 
> application. The challenge here is that we're not storing this data for each 
> application, but given this kind of call is relatively rare (compare to 
> writes and updates) we can perform some scanning during the read time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5931) Document timeout interfaces CLI and REST APIs


[ 
https://issues.apache.org/jira/browse/YARN-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723542#comment-15723542
 ] 

Jian He commented on YARN-5931:
---

- " Update timeout of an application from NOW. ApplicationId can be passed 
using 'appId' option. Timeout value is in seconds." 
May be "Update timeout of an application from the time of request in seconds.  
ApplicationId can be specified using 'appId' option" ?

> Document timeout interfaces CLI and REST APIs
> -
>
> Key: YARN-5931
> URL: https://issues.apache.org/jira/browse/YARN-5931
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: ResourceManagerRest.html, YARN-5931.0.patch, 
> YarnCommands.html
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5647) [Security] Collector and reader side changes for loading auth filters and principals


[ 
https://issues.apache.org/jira/browse/YARN-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723523#comment-15723523
 ] 

Sangjin Lee commented on YARN-5647:
---

[~varun_saxena], thanks for the patch!

Just curious, why was the change for TestTimelineReaderWebServicesHBaseStorage 
necessary (replacing {{setupOptions}} with {{addFilters}})?

I assume that you're still working on the unit tests for this? Also, can you 
please fix the checkstyle issues? Seems straightforward enough.

> [Security] Collector and reader side changes for loading auth filters and 
> principals
> 
>
> Key: YARN-5647
> URL: https://issues.apache.org/jira/browse/YARN-5647
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: oct16-hard
> Attachments: YARN-5647-YARN-5355.wip.002.patch, 
> YARN-5647-YARN-5355.wip.003.patch, YARN-5647-YARN-5355.wip.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler


[ 
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723497#comment-15723497
 ] 

Wangda Tan commented on YARN-4390:
--

Thanks, once you have the patch ready, I can help with review.

> Do surgical preemption based on reserved container in CapacityScheduler
> ---
>
> Key: YARN-4390
> URL: https://issues.apache.org/jira/browse/YARN-4390
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1
>Reporter: Eric Payne
>Assignee: Wangda Tan
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: QueueNotHittingMax.jpg, YARN-4390-design.1.pdf, 
> YARN-4390-test-results.pdf, YARN-4390.1.patch, YARN-4390.2.patch, 
> YARN-4390.3.branch-2.patch, YARN-4390.3.patch, YARN-4390.4.patch, 
> YARN-4390.5.patch, YARN-4390.6.patch, YARN-4390.7.patch, YARN-4390.8.patch
>
>
> There are multiple reasons why preemption could unnecessarily preempt 
> containers. One is that an app could be requesting a large container (say 
> 8-GB), and the preemption monitor could conceivably preempt multiple 
> containers (say 8, 1-GB containers) in order to fill the large container 
> request. These smaller containers would then be rejected by the requesting AM 
> and potentially given right back to the preempted app.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5694) ZKRMStateStore can prevent the transition to standby in branch-2.7 if the ZK node is unreachable

2016-12-05 Thread Daniel Templeton (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723483#comment-15723483
 ] 

Daniel Templeton commented on YARN-5694:


Thanks for the reviews and commit, [~jianhe]!

> ZKRMStateStore can prevent the transition to standby in branch-2.7 if the ZK 
> node is unreachable
> 
>
> Key: YARN-5694
> URL: https://issues.apache.org/jira/browse/YARN-5694
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.3
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>  Labels: oct16-medium
> Fix For: 2.6.5, 2.7.4
>
> Attachments: YARN-5694.001.patch, YARN-5694.002.patch, 
> YARN-5694.003.patch, YARN-5694.004.patch, YARN-5694.004.patch, 
> YARN-5694.005.patch, YARN-5694.006.patch, YARN-5694.007.patch, 
> YARN-5694.008.patch, YARN-5694.branch-2.6.001.patch, 
> YARN-5694.branch-2.6.002.patch, YARN-5694.branch-2.7.001.patch, 
> YARN-5694.branch-2.7.002.patch, YARN-5694.branch-2.7.004.patch, 
> YARN-5694.branch-2.7.005.patch
>
>
> {{ZKRMStateStore.doStoreMultiWithRetries()}} holds the lock while trying to 
> talk to ZK.  If the connection fails, it will retry while still holding the 
> lock.  The retries are intended to be strictly time limited, but in the case 
> that the ZK node is unreachable, the time limit fails, resulting in the 
> thread holding the lock for over an hour.  Transitioning the RM to standby 
> requires that same lock, so in exactly the case that the RM should be 
> transitioning to standby, the {{VerifyActiveStatusThread}} blocks it from 
> happening.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5965) Revisit ApplicationReport #getApplicationTimeouts


[ 
https://issues.apache.org/jira/browse/YARN-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723482#comment-15723482
 ] 

Jian He commented on YARN-5965:
---

yep, sounds good to me.

> Revisit ApplicationReport #getApplicationTimeouts
> -
>
> Key: YARN-5965
> URL: https://issues.apache.org/jira/browse/YARN-5965
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Jian He
>Assignee: Rohith Sharma K S
>
> Currently it returns a list of ApplicationTimeout objects,  to get a 
> particular timeout, the caller code needs to iterate the list and compare the 
> timeoutType to get the corresponding value. Is a map data structure easier 
> for use code? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler

2016-12-05 Thread Eric Payne (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723438#comment-15723438
 ] 

Eric Payne commented on YARN-4390:
--

Thanks [~leftnoteasy]. I think I have it figured out. This JIRA depends on both 
YARN-4865 and YARN-4934. I have backported those to 2.8 and will now work on 
this JIRA.

> Do surgical preemption based on reserved container in CapacityScheduler
> ---
>
> Key: YARN-4390
> URL: https://issues.apache.org/jira/browse/YARN-4390
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1
>Reporter: Eric Payne
>Assignee: Wangda Tan
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: QueueNotHittingMax.jpg, YARN-4390-design.1.pdf, 
> YARN-4390-test-results.pdf, YARN-4390.1.patch, YARN-4390.2.patch, 
> YARN-4390.3.branch-2.patch, YARN-4390.3.patch, YARN-4390.4.patch, 
> YARN-4390.5.patch, YARN-4390.6.patch, YARN-4390.7.patch, YARN-4390.8.patch
>
>
> There are multiple reasons why preemption could unnecessarily preempt 
> containers. One is that an app could be requesting a large container (say 
> 8-GB), and the preemption monitor could conceivably preempt multiple 
> containers (say 8, 1-GB containers) in order to fill the large container 
> request. These smaller containers would then be rejected by the requesting AM 
> and potentially given right back to the preempted app.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5756) Add state-machine implementation for queues


[ 
https://issues.apache.org/jira/browse/YARN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723427#comment-15723427
 ] 

Hadoop QA commented on YARN-5756:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 45s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 9 new + 236 unchanged - 2 fixed = 245 total (was 238) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
31s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 46s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
34s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 17s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
|   | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue |
|   | 
hadoop.yarn.server.resourcemanager.reservation.TestCapacitySchedulerPlanFollower
 |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
 |
|   | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5756 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841798/YARN-5756.4.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 47f665eaf0b2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8c46808 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |

[jira] [Commented] (YARN-5184) Fix up incompatible changes introduced on ContainerStatus and NodeReport


[ 
https://issues.apache.org/jira/browse/YARN-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723363#comment-15723363
 ] 

Hadoop QA commented on YARN-5184:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m  
3s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
16s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
34s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
27s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
23s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
27s{color} | {color:green} hadoop-yarn-api in the patch passed with JDK 
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 12s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:5af2af1 |
| JIRA Issue | YARN-5184 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841806/YARN-5184-branch-2.8.01.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux f6668fc1c8ef 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2.8 / 802a1fb |
| Default Java | 1.7.0_121 |
| Multi-JDK versions |

[jira] [Commented] (YARN-5184) Fix up incompatible changes introduced on ContainerStatus and NodeReport


[ 
https://issues.apache.org/jira/browse/YARN-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723243#comment-15723243
 ] 

Sangjin Lee commented on YARN-5184:
---

Renamed the patches. The trunk patch should apply cleanly to branch-2 also 
(2.9).

> Fix up incompatible changes introduced on ContainerStatus and NodeReport
> 
>
> Key: YARN-5184
> URL: https://issues.apache.org/jira/browse/YARN-5184
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: api
>Affects Versions: 2.8.0, 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Sangjin Lee
>Priority: Blocker
> Attachments: YARN-5184-branch-2.8.01.patch, YARN-5184.01.patch
>
>
> YARN-2882 and YARN-5430 broke compatibility by adding abstract methods to 
> ContainerStatus. Since ContainerStatus is a Public-Stable class, adding 
> abstract methods to this class breaks any extensions. 
> To fix this, we should add default implementations to these new methods and 
> not leave them as abstract. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5085) Add support for change of container ExecutionType


 [ 
https://issues.apache.org/jira/browse/YARN-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5085:
--
Attachment: YARN-5085.001.patch

Attaching a combined patch for all the subpatches to kick off Jenkins.
The Sub tasks can be used for focused reviews.

> Add support for change of container ExecutionType
> -
>
> Key: YARN-5085
> URL: https://issues.apache.org/jira/browse/YARN-5085
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5085.001.patch
>
>
> YARN-2882 introduced the concept of {{ExecutionType}} for containers and it 
> also introduced the concept of OPPORTUNISTIC ExecutionType.
> YARN-4335 introduced changes to the ResourceRequest so that AMs may request 
> that the Container allocated against the ResourceRequest is of a particular 
> {{ExecutionType}}.
> This JIRA proposes to provide support for the AM to change the ExecutionType 
> of a previously requested Container.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5184) Fix up incompatible changes introduced on ContainerStatus and NodeReport


 [ 
https://issues.apache.org/jira/browse/YARN-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-5184:
--
Attachment: YARN-5184.01.patch

> Fix up incompatible changes introduced on ContainerStatus and NodeReport
> 
>
> Key: YARN-5184
> URL: https://issues.apache.org/jira/browse/YARN-5184
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: api
>Affects Versions: 2.8.0, 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Sangjin Lee
>Priority: Blocker
> Attachments: YARN-5184-branch-2.8.01.patch, YARN-5184.01.patch
>
>
> YARN-2882 and YARN-5430 broke compatibility by adding abstract methods to 
> ContainerStatus. Since ContainerStatus is a Public-Stable class, adding 
> abstract methods to this class breaks any extensions. 
> To fix this, we should add default implementations to these new methods and 
> not leave them as abstract. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5184) Fix up incompatible changes introduced on ContainerStatus and NodeReport


 [ 
https://issues.apache.org/jira/browse/YARN-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-5184:
--
Attachment: YARN-5184-branch-2.8.01.patch

> Fix up incompatible changes introduced on ContainerStatus and NodeReport
> 
>
> Key: YARN-5184
> URL: https://issues.apache.org/jira/browse/YARN-5184
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: api
>Affects Versions: 2.8.0, 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Sangjin Lee
>Priority: Blocker
> Attachments: YARN-5184-branch-2.8.01.patch, YARN-5184.01.patch
>
>
> YARN-2882 and YARN-5430 broke compatibility by adding abstract methods to 
> ContainerStatus. Since ContainerStatus is a Public-Stable class, adding 
> abstract methods to this class breaks any extensions. 
> To fix this, we should add default implementations to these new methods and 
> not leave them as abstract. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5184) Fix up incompatible changes introduced on ContainerStatus and NodeReport


 [ 
https://issues.apache.org/jira/browse/YARN-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-5184:
--
Attachment: (was: YARN-5184-branch-2.8.poc.patch)

> Fix up incompatible changes introduced on ContainerStatus and NodeReport
> 
>
> Key: YARN-5184
> URL: https://issues.apache.org/jira/browse/YARN-5184
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: api
>Affects Versions: 2.8.0, 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Sangjin Lee
>Priority: Blocker
>
> YARN-2882 and YARN-5430 broke compatibility by adding abstract methods to 
> ContainerStatus. Since ContainerStatus is a Public-Stable class, adding 
> abstract methods to this class breaks any extensions. 
> To fix this, we should add default implementations to these new methods and 
> not leave them as abstract. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5959) RM changes to support change of container ExecutionType


[ 
https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723235#comment-15723235
 ] 

Hadoop QA commented on YARN-5959:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  7s{color} 
| {color:red} YARN-5959 does not apply to trunk. Rebase required? Wrong Branch? 
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5959 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841803/YARN-5959.wip.002.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14187/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> RM changes to support change of container ExecutionType
> ---
>
> Key: YARN-5959
> URL: https://issues.apache.org/jira/browse/YARN-5959
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5959.combined.001.patch, YARN-5959.wip.002.patch, 
> YARN-5959.wip.patch
>
>
> RM side changes to allow an AM to ask for change of ExecutionType.
> Currently, there are two cases:
> # *Promotion* : OPPORTUNISTIC to GUARANTEED.
> # *Demotion* : GUARANTEED to OPPORTUNISTIC.
> This is similar in YARN-1197 which allows for change in Container resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5184) Fix up incompatible changes introduced on ContainerStatus and NodeReport


 [ 
https://issues.apache.org/jira/browse/YARN-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-5184:
--
Attachment: (was: YARN-5184-branch-2.poc.patch)

> Fix up incompatible changes introduced on ContainerStatus and NodeReport
> 
>
> Key: YARN-5184
> URL: https://issues.apache.org/jira/browse/YARN-5184
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: api
>Affects Versions: 2.8.0, 2.9.0
>Reporter: Karthik Kambatla
>Assignee: Sangjin Lee
>Priority: Blocker
>
> YARN-2882 and YARN-5430 broke compatibility by adding abstract methods to 
> ContainerStatus. Since ContainerStatus is a Public-Stable class, adding 
> abstract methods to this class breaks any extensions. 
> To fix this, we should add default implementations to these new methods and 
> not leave them as abstract. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5966) AMRMClient changes to support ExecutionType update


 [ 
https://issues.apache.org/jira/browse/YARN-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5966:
--
Attachment: YARN-5966.wip.001.patch

Attaching initial patch

> AMRMClient changes to support ExecutionType update
> --
>
> Key: YARN-5966
> URL: https://issues.apache.org/jira/browse/YARN-5966
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5966.wip.001.patch
>
>
> {{AMRMClient}} changes to support change of container ExecutionType



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5966) AMRMClient changes to support ExecutionType update


 [ 
https://issues.apache.org/jira/browse/YARN-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5966:
--
Description: {{AMRMClient}} changes to support change of container 
ExecutionType

> AMRMClient changes to support ExecutionType update
> --
>
> Key: YARN-5966
> URL: https://issues.apache.org/jira/browse/YARN-5966
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>
> {{AMRMClient}} changes to support change of container ExecutionType



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-5966) AMRMClient changes to support ExecutionType update


 [ 
https://issues.apache.org/jira/browse/YARN-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh reassigned YARN-5966:
-

Assignee: Arun Suresh

> AMRMClient changes to support ExecutionType update
> --
>
> Key: YARN-5966
> URL: https://issues.apache.org/jira/browse/YARN-5966
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>
> {{AMRMClient}} changes to support change of container ExecutionType



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType


 [ 
https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5959:
--
Attachment: YARN-5959.wip.002.patch

Moved out the AMRMClient changes to YARN-5966 and added more tests.

> RM changes to support change of container ExecutionType
> ---
>
> Key: YARN-5959
> URL: https://issues.apache.org/jira/browse/YARN-5959
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5959.combined.001.patch, YARN-5959.wip.002.patch, 
> YARN-5959.wip.patch
>
>
> RM side changes to allow an AM to ask for change of ExecutionType.
> Currently, there are two cases:
> # *Promotion* : OPPORTUNISTIC to GUARANTEED.
> # *Demotion* : GUARANTEED to OPPORTUNISTIC.
> This is similar in YARN-1197 which allows for change in Container resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5966) AMRMClient changes to support ExecutionType update

Arun Suresh created YARN-5966:
-

 Summary: AMRMClient changes to support ExecutionType update
 Key: YARN-5966
 URL: https://issues.apache.org/jira/browse/YARN-5966
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Arun Suresh






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5559) Analyse 2.8.0/3.0.0 jdiff reports and fix any issues

2016-12-05 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723210#comment-15723210
 ] 

Hudson commented on YARN-5559:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10939 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10939/])
YARN-5559. Analyse 2.8.0/3.0.0 jdiff reports and fix any issues. (jianhe: rev 
43ebff2e354142bddcb42755766a965ae8a503a6)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodeLabelsResponsePBImpl.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/InvalidStateTransitionException.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterNodeLabelsResponse.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/InvalidStateTransitonException.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java


> Analyse 2.8.0/3.0.0 jdiff reports and fix any issues
> 
>
> Key: YARN-5559
> URL: https://issues.apache.org/jira/browse/YARN-5559
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Wangda Tan
>Assignee: Akira Ajisaka
>Priority: Blocker
>  Labels: oct16-easy
> Fix For: 2.8.0, 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5559.1.patch, YARN-5559.2.patch, YARN-5559.3.patch, 
> YARN-5559.4.patch, YARN-5559.5.patch, YARN-5559.6.patch, YARN-5559.7.patch, 
> YARN-5559.8.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5756) Add state-machine implementation for queues

2016-12-05 Thread Xuan Gong (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-5756:

Attachment: YARN-5756.4.patch

> Add state-machine implementation for queues
> ---
>
> Key: YARN-5756
> URL: https://issues.apache.org/jira/browse/YARN-5756
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-5756.1.patch, YARN-5756.2.patch, YARN-5756.3.patch, 
> YARN-5756.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5849) Automatically create YARN control group for pre-mounted cgroups


[ 
https://issues.apache.org/jira/browse/YARN-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723124#comment-15723124
 ] 

Hadoop QA commented on YARN-5849:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
11s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 26 unchanged - 23 fixed = 26 total (was 49) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
33s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
 generated 0 new + 230 unchanged - 5 fixed = 230 total (was 235) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
26s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 
40s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
14s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m

[jira] [Commented] (YARN-3409) Add constraint node labels


[ 
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723113#comment-15723113
 ] 

Wangda Tan commented on YARN-3409:
--

[~varun_saxena], [~Naganarasimha],

Thanks for your feedbacks, however I still think constraint is one special kind 
of label, we should reuse existing APIs: they share a lots of common fields and 
internal implementations. For example we already have fully support of label 
modifications in REST API, I don't think adding a full set of node constraint 
API is a good idea.

We can still make REST API / CLI backward compatible.

And thanks [~Naganarasimha] for the POC patch, I look a very brief look at it. 
Some comments:
1) Do we really need define type of the constraint? 
I think we can do late-binding it. For example, software version could be: 
x.y.z, it is not double and not long, sometimes we need compare it (like 
version > 2.3.4.5 and version < 2.3.4.9) and sometimes we need do string much 
(like version ~ 2.3.4.*)
Another benefit is, with late-binding, we don't need to add node constraint 
upfront. Node constraint will be added/removed to the cluster when we change 
constraint on node, but they should not conflict with known partition name. 
With this, we can avoid type conflicting as well. In your existing design, if 
constraint-A is defined as long, but on node1 admin set a double value, it will 
report error. Instead, if we store everything as string, and evaluate it 
according to constraint expression, it should be more easier to be used.

2) For operations
- We only need support one set of syntax (for example, equal is "==", but "eq" 
will not be a valid operation name)
- Maybe we can support "like" operator as well, it will compare string with 
given regex. 


> Add constraint node labels
> --
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, capacityscheduler, client
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
> Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, 
> YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to 
> determinate how resources of a special set of nodes could be shared by a 
> group of entities (like teams, departments, etc.). Partitions of a cluster 
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has 
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% 
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of 
> node’s hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 
> 2.20 && JDK.version >= 8u20 && x86_64).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5903) Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl beforeclass setup method

2016-12-05 Thread Haibo Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated YARN-5903:
-
Attachment: YARN-5903.03.patch

Upload another patch to make it consistent with YARN-5901 as [~varun_saxena] 
suggested.

> Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl 
> beforeclass setup method
> 
>
> Key: YARN-5903
> URL: https://issues.apache.org/jira/browse/YARN-5903
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-5903.02.patch, YARN-5903.03.patch, 
> yarn5903.001.patch
>
>
> This is essentially the same race condition as in YARN-5901, that is, 
> resourcemanager.getServiceState() == STATE.STARTED does not guarantee 
> resource manager is fully started.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5849) Automatically create YARN control group for pre-mounted cgroups

2016-12-05 Thread Miklos Szegedi (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Miklos Szegedi updated YARN-5849:
-
Attachment: YARN-5849.007.patch

Indeed. Thank you, [~templedf]. I added the javadocs

> Automatically create YARN control group for pre-mounted cgroups
> ---
>
> Key: YARN-5849
> URL: https://issues.apache.org/jira/browse/YARN-5849
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
> Attachments: YARN-5849.000.patch, YARN-5849.001.patch, 
> YARN-5849.002.patch, YARN-5849.003.patch, YARN-5849.004.patch, 
> YARN-5849.005.patch, YARN-5849.006.patch, YARN-5849.007.patch
>
>
> Yarn can be launched with linux-container-executor.cgroups.mount set to 
> false. It will search for the cgroup mount paths set up by the administrator 
> parsing the /etc/mtab file. You can also specify 
> resource.percentage-physical-cpu-limit to limit the CPU resources assigned to 
> containers.
> linux-container-executor.cgroups.hierarchy is the root of the settings of all 
> YARN containers. If this is specified but not created YARN will fail at 
> startup:
> Caused by: java.io.FileNotFoundException: 
> /cgroups/cpu/hadoop-yarn/cpu.cfs_period_us (Permission denied)
> org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler.updateCgroup(CgroupsLCEResourcesHandler.java:263)
> This JIRA is about automatically creating YARN control group in the case 
> above. It reduces the cost of administration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5888) [YARN-3368] Improve unit tests for YARN UI

2016-12-05 Thread Sreenath Somarajapuram (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722215#comment-15722215
 ] 

Sreenath Somarajapuram commented on YARN-5888:
--

- Why is 'config/configs.env' required for test?
- Tabs are being used instead of space.
- I don't see any major test cases/conditions getting added.
- What is the "Test fields" in models supposed to test? - Model is just another 
object and set & get work irrespective of the field names defined in them.

Probably this ticket can be used for making UTs run as expected. And further, 
test cases for each class can be added under an umbrella in individual tickets.

> [YARN-3368] Improve unit tests for YARN UI
> --
>
> Key: YARN-5888
> URL: https://issues.apache.org/jira/browse/YARN-5888
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
>Priority: Minor
> Attachments: YARN-5888.001.patch
>
>
> - Add missing test cases in new YARN UI
> - Fix test cases errors in new YARN UI 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5924) Resource Manager fails to load state with InvalidProtocolBufferException

2016-12-05 Thread Oleksii Dymytrov (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksii Dymytrov updated YARN-5924:
---
Assignee: (was: Oleksii Dymytrov)

> Resource Manager fails to load state with InvalidProtocolBufferException
> 
>
> Key: YARN-5924
> URL: https://issues.apache.org/jira/browse/YARN-5924
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Oleksii Dymytrov
>
> InvalidProtocolBufferException is thrown during recovering of the 
> application's state if application's data has invalid format (or is broken) 
> under FSRMStateRoot/RMAppRoot/application_1477986176766_0134/ directory in 
> HDFS:
> {noformat}
> com.google.protobuf.InvalidProtocolBufferException: Protocol message 
> end-group tag did not match expected tag.
>   at 
> com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
>   at 
> com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
>   at 
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:143)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
>   at 
> org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$ApplicationStateDataProto.parseFrom(YarnServerResourceManagerRecoveryProtos.java:1028)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore$RMAppStateFileProcessor.processChildNode(FileSystemRMStateStore.java:966)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.processDirectoriesOfFiles(FileSystemRMStateStore.java:317)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.loadRMAppState(FileSystemRMStateStore.java:281)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.loadState(FileSystemRMStateStore.java:232)
> {noformat}
> The solution can be to catch "InvalidProtocolBufferException", show warning 
> and remove application's folder that contains invalid data to prevent RM 
> restart failure. 
> Additionally, I've added catch for other exceptions that can appear during 
> recovering of the specific application, to avoid RM failure even if the only 
> one application's state can't be loaded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5866) [YARN-3368] Fix few issues reported by jshint in new YARN UI

2016-12-05 Thread Sreenath Somarajapuram (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722122#comment-15722122
 ] 

Sreenath Somarajapuram commented on YARN-5866:
--

Looks mostly good to me. One suggestion is that it would be better to comment 
the function arguments instead of removing them. This would maintain code 
readability.
{code}
urlForFindRecord(id /*, modelName, snapshot*/) {
{code}

> [YARN-3368] Fix few issues reported by jshint in new YARN UI
> 
>
> Key: YARN-5866
> URL: https://issues.apache.org/jira/browse/YARN-5866
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
> Attachments: YARN-5866.001.patch, YARN-5866.002.patch
>
>
> There are few minor issues reported by jshint (javascript lint tool).
> This jira is to track and fix those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5931) Document timeout interfaces CLI and REST APIs

2016-12-05 Thread Rohith Sharma K S (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721841#comment-15721841
 ] 

Rohith Sharma K S commented on YARN-5931:
-

Thanks Sunil for the review. I will update the patch fixing review comments. 

> Document timeout interfaces CLI and REST APIs
> -
>
> Key: YARN-5931
> URL: https://issues.apache.org/jira/browse/YARN-5931
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: ResourceManagerRest.html, YARN-5931.0.patch, 
> YarnCommands.html
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5932) Retrospect moveApplicationToQueue in align with YARN-5611

2016-12-05 Thread Rohith Sharma K S (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721684#comment-15721684
 ] 

Rohith Sharma K S commented on YARN-5932:
-

+1 LGTM,  I will commit it later of the day if no more objections 

> Retrospect moveApplicationToQueue in align with YARN-5611
> -
>
> Key: YARN-5932
> URL: https://issues.apache.org/jira/browse/YARN-5932
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler, resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5932.0001.patch, YARN-5932.0002.patch, 
> YARN-5932.0003.patch, YARN-5932.v0.patch, YARN-5932.v1.patch
>
>
> All dynamic api's of an application's state change could follow a general 
> design approach. Currently priority and app timeouts are following this 
> approach all corner cases.
> *Steps*
> - Do a pre-validate check to ensure that changes are fine.
> - Update this information to state-store
> - Perform real move operation and update in-memory data structures.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5956) Refactor ClientRMService

2016-12-05 Thread Sunil G (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721663#comment-15721663
 ] 

Sunil G commented on YARN-5956:
---

[~lewuathe]

few comments.

1. 

{code} 
 if (application.isAppInCompletedStates()) {  
  RMAuditLogger.logSuccess(callerUGI.getShortUserName(),
  AuditConstants.FAIL_ATTEMPT_REQUEST, "ClientRMService"
  applicationId);
  return response;
  } 
{code}

For {{failApplicationAttempt}}, this is not correct. We want to send normal 
response only for those apps whihc are in final states. But if app is in 
NEW_SAVING etc, we must throw exception. That is missed now.

2. {{updateApplicationTimeouts}} has some unwanted code. Pls remove it.
{code}
1634if (application.isAppInCompletedStates()) {
1635
1636}
{code}



> Refactor ClientRMService
> 
>
> Key: YARN-5956
> URL: https://issues.apache.org/jira/browse/YARN-5956
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: YARN-5956.01.patch, YARN-5956.02.patch
>
>
> Some refactoring can be done in {{ClientRMService}}.
> - Remove redundant variable declaration
> - Fill in missing javadocs
> - Proper variable access modifier
> - Fix some typos in method name and exception messages



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5955) Use threadpool or multiple thread to recover app

2016-12-05 Thread zhangyubiao (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721355#comment-15721355
 ] 

zhangyubiao edited comment on YARN-5955 at 12/5/16 8:53 AM:


[~templedf], is it zookeeper is the best way to store the app state? is it can 
improve good enough performance to the large cluster to recovery apps?
 as we [~tangshangwen] [~zhengchenyu] see in  
http://www.slideshare.net/arinto/next-generation-hadoop-high-availability-for-yarn.
 it seem like that zookeeper is not the best way to store the app state.
[~Naganarasimha]  [~varun_saxena]  [~rohithsharma]


was (Author: piaoyu zhang):
[~templedf], is it zookeeper is the best way to store the app state? is it can 
improve performance to the large cluster to recovery apps?
 as we [~tangshangwen] [~zhengchenyu] see in  
http://www.slideshare.net/arinto/next-generation-hadoop-high-availability-for-yarn.
 it seem like that zookeeper is not the best way to store the app state.
[~Naganarasimha]  [~varun_saxena]  [~rohithsharma]

> Use threadpool or multiple thread to recover app
> 
>
> Key: YARN-5955
> URL: https://issues.apache.org/jira/browse/YARN-5955
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: Zhaofei Meng
>Assignee: Ajith S
> Fix For: 2.7.1
>
>
> current app recovery is one by one,use thead pool can make recovery faster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5955) Use threadpool or multiple thread to recover app

2016-12-05 Thread zhangyubiao (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721355#comment-15721355
 ] 

zhangyubiao edited comment on YARN-5955 at 12/5/16 8:52 AM:


[~templedf], is it zookeeper is the best way to store the app state? is it can 
improve performance to the large cluster to recovery apps?
 as we [~tangshangwen] [~zhengchenyu] see in  
http://www.slideshare.net/arinto/next-generation-hadoop-high-availability-for-yarn.
 it seem like that zookeeper is not the best way to store the app state.
[~Naganarasimha]  [~varun_saxena]  [~rohithsharma]


was (Author: piaoyu zhang):
[~templedf]，is it zookeeper is the best way to store the app state? is it can 
improve performance to the large cluster to recovery apps?
 as we [~tangshangwen] [~zhengchenyu] see in  
http://www.slideshare.net/arinto/next-generation-hadoop-high-availability-for-yarn.
 it seem like that zookeeper is not the best way to store the app state.
[~Naganarasimha]  [~varun_saxena]  [~rohithsharma]

> Use threadpool or multiple thread to recover app
> 
>
> Key: YARN-5955
> URL: https://issues.apache.org/jira/browse/YARN-5955
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: Zhaofei Meng
>Assignee: Ajith S
> Fix For: 2.7.1
>
>
> current app recovery is one by one,use thead pool can make recovery faster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5965) Revisit ApplicationReport #getApplicationTimeouts

2016-12-05 Thread Sunil G (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721642#comment-15721642
 ] 

Sunil G commented on YARN-5965:
---

Hi.

Currently we use map in REST end and for cli. So I think we can keep the same 
syntax here. Something like below.

{{public abstract Map 
getApplicationTimeouts();}}
Thoughts?

> Revisit ApplicationReport #getApplicationTimeouts
> -
>
> Key: YARN-5965
> URL: https://issues.apache.org/jira/browse/YARN-5965
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Jian He
>Assignee: Rohith Sharma K S
>
> Currently it returns a list of ApplicationTimeout objects,  to get a 
> particular timeout, the caller code needs to iterate the list and compare the 
> timeoutType to get the corresponding value. Is a map data structure easier 
> for use code? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-5965) Revisit ApplicationReport #getApplicationTimeouts

2016-12-05 Thread Rohith Sharma K S (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S reassigned YARN-5965:
---

Assignee: Rohith Sharma K S

> Revisit ApplicationReport #getApplicationTimeouts
> -
>
> Key: YARN-5965
> URL: https://issues.apache.org/jira/browse/YARN-5965
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Jian He
>Assignee: Rohith Sharma K S
>
> Currently it returns a list of ApplicationTimeout objects,  to get a 
> particular timeout, the caller code needs to iterate the list and compare the 
> timeoutType to get the corresponding value. Is a map data structure easier 
> for use code? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5965) Revisit ApplicationReport #getApplicationTimeouts


[ 
https://issues.apache.org/jira/browse/YARN-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721574#comment-15721574
 ] 

Jian He commented on YARN-5965:
---

[~sunilg], [~rohithsharma] your opinion ?  
I was working on YARN-5768,  don't have strong opinion on this, just want to 
hear your thoughts whether it's worth to change it or not. 

> Revisit ApplicationReport #getApplicationTimeouts
> -
>
> Key: YARN-5965
> URL: https://issues.apache.org/jira/browse/YARN-5965
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Jian He
>
> Currently it returns a list of ApplicationTimeout objects,  to get a 
> particular timeout, the caller code needs to iterate the list and compare the 
> timeoutType to get the corresponding value. Is a map data structure easier 
> for use code? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5965) Revisit ApplicationReport #getApplicationTimeouts