[jira] [Commented] (YARN-5611) Provide an API to update lifetime of an application.

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635311#comment-15635311
 ] 

Rohith Sharma K S commented on YARN-5611:
-

bq. 1. ApplicationUpdateTimeoutMapProto -> ApplicationUpdateTimeoutProto. May 
not need to have "map" in the name of proto
It is naming convention I maintained with other map protos. Do not want to 
change its naming convention. 

bq. There is a any chance that NEW_SAVING could fail and app could no longer be 
in next state. We could consider apps which are from ACCEPTED state, correct? 
Any specific reason to consider NEW_SAVING?
Update timeout can also possible in NEW_SAVING state. This is because  
state-store update is done from dispatcher queue which ensures that 
updateTimeout event will go always behind new_saving event. So, should not be 
problem updating in new_saving. In case, new_saving fails, then App it self is 
failed. 

bq. In validateISO8601AndConvertToLocalTimeEpoch, if (expireTime < 
currentTimeMillis) helps to validate timeout's of past. But still there could 
be a delta time from current time to next timeout check thread. Could that also 
to be validated here? Or such entries will be discarded by the monitor thread?
When dealing with milliseconds, there will be delta changes. It should be fine. 
Actually, as user I would expect to update timeout at least with 5minutes gap. 

bq. In general thought and thinking out loud, Map 
}} why we need a map here. Could we have {{List and each 
ApplicationTimeouts could have its type and real timeout value in long. I am 
not seeing much of benefit from map here as app timeout types could no be more 
than 3/4 atmost.
Good question. There are couple of reason for considering MAP instead of List.
# It is defensive code style, where RM do not want to expect any duplicate 
applicationTimeout types. If we go for List, user is allowed to input duplicate 
timeout types so that RM server need to take decision which one to consider and 
it is debatable topic. Advantage of is MAP is  user send single timeout types. 
It is more clarity for user as well as RM. 
# Populating MAP is much easier in RM. In List, need to look up for all the 
entry correspond to timeoutType. Let say, in new_saving, timeout update is for 
LIFETIME. It is easier to look up and get value in map. But in list, on every 
state need to iterate for all the values and find does LIFETIME exist or not. 




> Provide an API to update lifetime of an application.
> 
>
> Key: YARN-5611
> URL: https://issues.apache.org/jira/browse/YARN-5611
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: oct16-hard
> Attachments: 0001-YARN-5611.patch, 0002-YARN-5611.patch, 
> 0003-YARN-5611.patch, YARN-5611.0004.patch, YARN-5611.0005.patch, 
> YARN-5611.v0.patch
>
>
> YARN-4205 monitors an Lifetime of an applications is monitored if required. 
> Add an client api to update lifetime of an application. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4498) Application level node labels stats to be available in REST

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635283#comment-15635283
 ] 

Naganarasimha G R commented on YARN-4498:
-

Thanks [~bibinchundatt], i feel latest approach is fine and i feel i can commit 
the addendum patches to the specific branch, [~rohithsharma] any other points 
on this ?

> Application level node labels stats to be available in REST
> ---
>
> Key: YARN-4498
> URL: https://issues.apache.org/jira/browse/YARN-4498
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, client, resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>  Labels: oct16-medium
> Fix For: 2.8.0, 2.9.0, 3.0.0-alpha2
>
> Attachments: 0001-YARN-4498.patch, YARN-4498.0002.patch, 
> YARN-4498.0003.patch, YARN-4498.0004.patch, YARN-4498.addendum.001.patch, 
> YARN-4498.branch-2.8.0001.patch, YARN-4498.branch-2.8.addendum.001.patch, 
> apps.xml
>
>
> Currently nodelabel stats per application is not available through REST like 
> currently used labels by all live containers, total stats of containers per 
> label for app etc..
> CLI and web UI scenarios will be handled separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4498) Application level node labels stats to be available in REST

[Bibin A Chundatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635188#comment-15635188
 ] 

Bibin A Chundatt commented on YARN-4498:


As per the currently implementation at all cases branch and trunk will be in 
sync for api. Hoping in current patch no changes are required

> Application level node labels stats to be available in REST
> ---
>
> Key: YARN-4498
> URL: https://issues.apache.org/jira/browse/YARN-4498
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, client, resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>  Labels: oct16-medium
> Fix For: 2.8.0, 2.9.0, 3.0.0-alpha2
>
> Attachments: 0001-YARN-4498.patch, YARN-4498.0002.patch, 
> YARN-4498.0003.patch, YARN-4498.0004.patch, YARN-4498.addendum.001.patch, 
> YARN-4498.branch-2.8.0001.patch, YARN-4498.branch-2.8.addendum.001.patch, 
> apps.xml
>
>
> Currently nodelabel stats per application is not available through REST like 
> currently used labels by all live containers, total stats of containers per 
> label for app etc..
> CLI and web UI scenarios will be handled separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4849) [YARN-3368] cleanup code base, integrate web UI related build to mvn, and fix licenses.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635183#comment-15635183
 ] 

Hadoop QA commented on YARN-4849:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  3m 
59s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  3m 
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
14s{color} | {color:green} YARN-3368 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m  
7s{color} | {color:green} YARN-3368 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green} YARN-3368 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
25s{color} | {color:green} YARN-3368 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} YARN-3368 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m  
7s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
35s{color} | {color:green} hadoop-yarn-ui in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
36s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 49s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:183a5e9 |
| JIRA Issue | YARN-4849 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12837034/YARN-4849-YARN-3368.addendum.5.patch
 |
| Optional Tests |  asflicense  mvnsite  compile  javac  javadoc  mvninstall  
unit  xml  |
| uname | Linux 254310b92374 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-3368 / 6b20b8c |
| Default Java | 1.8.0_101 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/13779/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: 
hadoop-yarn-project/hadoop-yarn |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/13779/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> [YARN-3368] cleanup code base, integrate web UI related build to mvn, and fix 
> licenses.
> ---
>
> Key: YARN-4849
> URL: https://issues.apache.org/jira/browse/YARN-4849
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>

[jira] [Commented] (YARN-4734) Merge branch:YARN-3368 to trunk

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635143#comment-15635143
 ] 

Hadoop QA commented on YARN-4734:
-

(!) A patch to the testing environment has been detected. 
Re-executing against the patched versions to perform further tests. 
The console is at 
https://builds.apache.org/job/PreCommit-YARN-Build/13780/console in case of 
problems.


> Merge branch:YARN-3368 to trunk
> ---
>
> Key: YARN-4734
> URL: https://issues.apache.org/jira/browse/YARN-4734
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-4734.1.patch, YARN-4734.10-NOT_READY.patch, 
> YARN-4734.11-NOT_READY.patch, YARN-4734.12-NOT_READY.patch, 
> YARN-4734.13.patch, YARN-4734.14.patch, YARN-4734.15.patch, 
> YARN-4734.16.patch, YARN-4734.17-rebased.patch, YARN-4734.17.patch, 
> YARN-4734.18.patch, YARN-4734.19.patch, YARN-4734.2.patch, 
> YARN-4734.20.patch, YARN-4734.3.patch, YARN-4734.4.patch, YARN-4734.5.patch, 
> YARN-4734.6.patch, YARN-4734.7.patch, YARN-4734.8.patch, 
> YARN-4734.9-NOT_READY.patch
>
>
> YARN-2928 branch is planned to merge back to trunk shortly, it depends on 
> changes of YARN-3368. This JIRA is to track the merging task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4734) Merge branch:YARN-3368 to trunk

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-4734:
-
Attachment: YARN-4734.20.patch

According to people's feedbacks, attached ver.20 patch, which included two 
minor updates:
1) When profile (-Pyarn-ui) is not specified, use pom as packaging format 
(instead of war) to make sure not dev files packaged to the release tar ball. 
2) Added a note to documentation to emphasis that Hadoop need to be built by 
-Pyarn-ui to try to new UI.

> Merge branch:YARN-3368 to trunk
> ---
>
> Key: YARN-4734
> URL: https://issues.apache.org/jira/browse/YARN-4734
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-4734.1.patch, YARN-4734.10-NOT_READY.patch, 
> YARN-4734.11-NOT_READY.patch, YARN-4734.12-NOT_READY.patch, 
> YARN-4734.13.patch, YARN-4734.14.patch, YARN-4734.15.patch, 
> YARN-4734.16.patch, YARN-4734.17-rebased.patch, YARN-4734.17.patch, 
> YARN-4734.18.patch, YARN-4734.19.patch, YARN-4734.2.patch, 
> YARN-4734.20.patch, YARN-4734.3.patch, YARN-4734.4.patch, YARN-4734.5.patch, 
> YARN-4734.6.patch, YARN-4734.7.patch, YARN-4734.8.patch, 
> YARN-4734.9-NOT_READY.patch
>
>
> YARN-2928 branch is planned to merge back to trunk shortly, it depends on 
> changes of YARN-3368. This JIRA is to track the merging task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4849) [YARN-3368] cleanup code base, integrate web UI related build to mvn, and fix licenses.

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635124#comment-15635124
 ] 

Wangda Tan commented on YARN-4849:
--

[~sunilg] plz review.

> [YARN-3368] cleanup code base, integrate web UI related build to mvn, and fix 
> licenses.
> ---
>
> Key: YARN-4849
> URL: https://issues.apache.org/jira/browse/YARN-4849
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: YARN-3368
>
> Attachments: YARN-4849-YARN-3368.1.patch, 
> YARN-4849-YARN-3368.2.patch, YARN-4849-YARN-3368.3.patch, 
> YARN-4849-YARN-3368.4.patch, YARN-4849-YARN-3368.5.patch, 
> YARN-4849-YARN-3368.6.patch, YARN-4849-YARN-3368.7.patch, 
> YARN-4849-YARN-3368.8.patch, YARN-4849-YARN-3368.addendum.1.patch, 
> YARN-4849-YARN-3368.addendum.2.patch, YARN-4849-YARN-3368.addendum.3.patch, 
> YARN-4849-YARN-3368.addendum.4.patch, YARN-4849-YARN-3368.addendum.5.patch, 
> YARN-4849-YARN-3368.doc-fix-08172016.1.patch, 
> YARN-4849-YARN-3368.doc-fix-08232016.1.patch, 
> YARN-4849-YARN-3368.javadoc-fix-09082016.1.patch, 
> YARN-4849-YARN-3368.javadoc-fix-09082016.2.patch, 
> YARN-4849-YARN-3368.javadoc-fix-09082016.3.patch, 
> YARN-4849-YARN-3368.license-fix-08172016.1.patch, 
> YARN-4849-YARN-3368.license-fix-08232016.1.patch, 
> YARN-4849-YARN-3368.rat-fix-08302016.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4849) [YARN-3368] cleanup code base, integrate web UI related build to mvn, and fix licenses.

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-4849:
-
Attachment: YARN-4849-YARN-3368.addendum.5.patch

Attached addendum.5 patch, which contains two changes:
1) Use pom as default packing for UI project when -Pyarn-ui is not specified.
2) Added a note to doc which mentioned Hadoop should be built by -Pyarn-ui in 
order to run the new UI.

> [YARN-3368] cleanup code base, integrate web UI related build to mvn, and fix 
> licenses.
> ---
>
> Key: YARN-4849
> URL: https://issues.apache.org/jira/browse/YARN-4849
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: YARN-3368
>
> Attachments: YARN-4849-YARN-3368.1.patch, 
> YARN-4849-YARN-3368.2.patch, YARN-4849-YARN-3368.3.patch, 
> YARN-4849-YARN-3368.4.patch, YARN-4849-YARN-3368.5.patch, 
> YARN-4849-YARN-3368.6.patch, YARN-4849-YARN-3368.7.patch, 
> YARN-4849-YARN-3368.8.patch, YARN-4849-YARN-3368.addendum.1.patch, 
> YARN-4849-YARN-3368.addendum.2.patch, YARN-4849-YARN-3368.addendum.3.patch, 
> YARN-4849-YARN-3368.addendum.4.patch, YARN-4849-YARN-3368.addendum.5.patch, 
> YARN-4849-YARN-3368.doc-fix-08172016.1.patch, 
> YARN-4849-YARN-3368.doc-fix-08232016.1.patch, 
> YARN-4849-YARN-3368.javadoc-fix-09082016.1.patch, 
> YARN-4849-YARN-3368.javadoc-fix-09082016.2.patch, 
> YARN-4849-YARN-3368.javadoc-fix-09082016.3.patch, 
> YARN-4849-YARN-3368.license-fix-08172016.1.patch, 
> YARN-4849-YARN-3368.license-fix-08232016.1.patch, 
> YARN-4849-YARN-3368.rat-fix-08302016.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Reopened] (YARN-4849) [YARN-3368] cleanup code base, integrate web UI related build to mvn, and fix licenses.

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan reopened YARN-4849:
--

> [YARN-3368] cleanup code base, integrate web UI related build to mvn, and fix 
> licenses.
> ---
>
> Key: YARN-4849
> URL: https://issues.apache.org/jira/browse/YARN-4849
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: YARN-3368
>
> Attachments: YARN-4849-YARN-3368.1.patch, 
> YARN-4849-YARN-3368.2.patch, YARN-4849-YARN-3368.3.patch, 
> YARN-4849-YARN-3368.4.patch, YARN-4849-YARN-3368.5.patch, 
> YARN-4849-YARN-3368.6.patch, YARN-4849-YARN-3368.7.patch, 
> YARN-4849-YARN-3368.8.patch, YARN-4849-YARN-3368.addendum.1.patch, 
> YARN-4849-YARN-3368.addendum.2.patch, YARN-4849-YARN-3368.addendum.3.patch, 
> YARN-4849-YARN-3368.addendum.4.patch, 
> YARN-4849-YARN-3368.doc-fix-08172016.1.patch, 
> YARN-4849-YARN-3368.doc-fix-08232016.1.patch, 
> YARN-4849-YARN-3368.javadoc-fix-09082016.1.patch, 
> YARN-4849-YARN-3368.javadoc-fix-09082016.2.patch, 
> YARN-4849-YARN-3368.javadoc-fix-09082016.3.patch, 
> YARN-4849-YARN-3368.license-fix-08172016.1.patch, 
> YARN-4849-YARN-3368.license-fix-08232016.1.patch, 
> YARN-4849-YARN-3368.rat-fix-08302016.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5585) [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635056#comment-15635056
 ] 

Rohith Sharma K S commented on YARN-5585:
-

bq. What would we emit as part of the UID if the entity id prefix was not set? 
An empty string ("")?
Default value is zero. So, default value zero will be returned as part of UID

bq. I am assuming that we will need to add a new method (or methods) that 
explicitly supports pagination and connects that to the 2nd branch of logic you 
added in GenericEntityReader.getResults(), right?
I am thinking not to add any new methods. Perhaps I try to fit in a existing 
code flow. 

> [Atsv2] Reader side changes for entity prefix and support for pagination via 
> additional filters
> ---
>
> Key: YARN-5585
> URL: https://issues.apache.org/jira/browse/YARN-5585
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Critical
>  Labels: oct16-hard
> Attachments: 0001-YARN-5585.patch, YARN-5585-YARN-5355.0001.patch, 
> YARN-5585-workaround.patch, YARN-5585.v0.patch
>
>
> TimelineReader REST API's provides lot of filters to retrieve the 
> applications. Along with those, it would be good to add new filter i.e fromId 
> so that entities can be retrieved after the fromId. 
> Current Behavior : Default limit is set to 100. If there are 1000 entities 
> then REST call gives first/last 100 entities. How to retrieve next set of 100 
> entities i.e 101 to 200 OR 900 to 801?
> Example : If applications are stored database, app-1 app-2 ... app-10.
> *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is 
> no way to achieve this. 
> So proposal is to have fromId in the filter like 
> *getApps?limit=5&=app-5* which gives list of apps from app-6 to 
> app-10. 
> Since ATS is targeting large number of entities storage, it is very common 
> use case to get next set of entities using fromId rather than querying all 
> the entites. This is very useful for pagination in web UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634966#comment-15634966
 ] 

Joep Rottinghuis commented on YARN-4061:


Opened HBASE-17018 and attached a pdf of my doc there.

> [Fault tolerance] Fault tolerant writer for timeline v2
> ---
>
> Key: YARN-4061
> URL: https://issues.apache.org/jira/browse/YARN-4061
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Li Lu
>Assignee: Joep Rottinghuis
>  Labels: YARN-5355
> Attachments: FaulttolerantwriterforTimelinev2.pdf
>
>
> We need to build a timeline writer that can be resistant to backend storage 
> down time and timeline collector failures. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634927#comment-15634927
 ] 

Hadoop QA commented on YARN-5600:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  2m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 44s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 4 new + 374 unchanged - 6 fixed = 378 total (was 380) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
25s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
19s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 
17s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5600 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12837006/YARN-5600.003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  cc  |
| uname | Linux b451a2178d1b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 5cad93d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/13778/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
 |
|  Test Results | 

[jira] [Commented] (YARN-5808) Add gc log options to the yarn daemon script when starting services-api

[Gour Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634925#comment-15634925
 ] 

Gour Saha commented on YARN-5808:
-

Patch 002 LGTM.

> Add gc log options to the yarn daemon script when starting services-api
> ---
>
> Key: YARN-5808
> URL: https://issues.apache.org/jira/browse/YARN-5808
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Billie Rinaldi
> Fix For: yarn-native-services
>
> Attachments: YARN-5808-yarn-native-services.001.patch, 
> YARN-5808-yarn-native-services.002.patch
>
>
> We need to add the gc log options as below when starting services-api using 
> the yarn-daemon.sh script -
> {code}
> -XX:+PrintGC -Xloggc:$YARN_LOG_DIR/services-api-gc.log -XX:+PrintGCDetails 
> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2497) Changes for fair scheduler to support allocate resource respect labels

[Tao Jie (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634894#comment-15634894
 ] 

Tao Jie commented on YARN-2497:
---

Any updates for this jira? It might be very useful to me.

> Changes for fair scheduler to support allocate resource respect labels
> --
>
> Key: YARN-2497
> URL: https://issues.apache.org/jira/browse/YARN-2497
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5823) Update NMTokens in case of requests with only opportunistic containers

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634883#comment-15634883
 ] 

Hadoop QA commented on YARN-5823:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: 
The patch generated 0 new + 13 unchanged - 1 fixed = 13 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
27s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 
56s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 35m 
57s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 20s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5823 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12837000/YARN-5823.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux cbe578009b89 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 5cad93d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/13776/testReport/ |
| modules | C: 

[jira] [Assigned] (YARN-5782) Slider version output includes variables buildNumber and hadoop.version

[Gour Saha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gour Saha reassigned YARN-5782:
---

Assignee: Gour Saha

> Slider version output includes variables buildNumber and hadoop.version
> ---
>
> Key: YARN-5782
> URL: https://issues.apache.org/jira/browse/YARN-5782
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Gour Saha
>
> The hadoop.version variable could be switched to project.version, and perhaps 
> buildNumber could be removed since Hadoop does not use the buildnumber plugin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5782) Slider version output includes variables buildNumber and hadoop.version

[Gour Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634869#comment-15634869
 ] 

Gour Saha commented on YARN-5782:
-

Currently this is what it dumps -
{code}
{
  "slider_version": 
"{org.apache.hadoop.yarn.services.hadoop.build.info=${hadoop.version}, 
org.apache.hadoop.yarn.services.hadoop.deployed.info=(HEAD detached at 7e711ef) 
@baf52eab8ab3c6c75b33bb8e21dc38, 
org.apache.hadoop.yarn.services.application.build.info=Apache Hadoop YARN 
Slider Core-2.7.1.2.4.2.1-108 Built against commit# ${buildNumber} on Java 
1.7.0_21 by jenkins}",
  "hadoop_version": "Hadoop runtime version (HEAD detached at 7e711ef) with 
source checksum baf52eab8ab3c6c75b33bb8e21dc38 and build date 2016-11-02T23:28Z"
}
{code}

> Slider version output includes variables buildNumber and hadoop.version
> ---
>
> Key: YARN-5782
> URL: https://issues.apache.org/jira/browse/YARN-5782
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Billie Rinaldi
>
> The hadoop.version variable could be switched to project.version, and perhaps 
> buildNumber could be removed since Hadoop does not use the buildnumber plugin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5828) Native services client errors out when config formats are uppercase

[Gour Saha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gour Saha updated YARN-5828:

Fix Version/s: yarn-native-services

> Native services client errors out when config formats are uppercase
> ---
>
> Key: YARN-5828
> URL: https://issues.apache.org/jira/browse/YARN-5828
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Fix For: yarn-native-services
>
> Attachments: YARN-5828-yarn-native-services.001.patch
>
>
> The services API is using uppercase config file formats now, which makes our 
> client error out with "org.apache.slider.core.exceptions.BadConfigException: 
> Config format null doesn't exist." We can make this error more informative 
> and make the config formats not be case sensitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5835) NoClassDefFoundError when reading ATS v1.5

[Zhiyuan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhiyuan Yang updated YARN-5835:
---
Description: 
LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in 
$HADOOP_HOME/share/hadoop/tools/lib but is never added in timeline server's 
classpath. Reading data from LevelDBCacheTimelineStore will get 
NoClassDefFoundError. Here's the stacktrace:
{code:java}
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
at 
org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
at 
org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
at 
org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
at 
org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
at 
org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresFromCacheIds(EntityGroupFSTimelineStore.java:853)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresForRead(EntityGroupFSTimelineStore.java:906)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getEntities(EntityGroupFSTimelineStore.java:959)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doGetEntities(TimelineDataManager.java:169)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.getEntities(TimelineDataManager.java:139)
at 
org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEntities(TimelineWebServices.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}

A temporary fix would be adding $HADOOP_HOME/share/hadoop/tools/lib/* into 
HADOOP_CLASSPATH manually.

  was:
LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in 
/hadoop/shared/hadoop/tools but is never added in timeline server's classpath. 
Reading data from LevelDBCacheTimelineStore will get NoClassDefFoundError. 
Here's the stacktrace:
{code:java}
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
at 
org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
at 
org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
at 
org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
at 
org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
at 
org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
at 

[jira] [Updated] (YARN-5835) NoClassDefFoundError when reading ATS v1.5

[Zhiyuan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhiyuan Yang updated YARN-5835:
---
Description: 
LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in 
/hadoop/shared/hadoop/tools but is never added in timeline server's classpath. 
Reading data from LevelDBCacheTimelineStore will get NoClassDefFoundError. 
Here's the stacktrace:
{code:java}
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
at 
org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
at 
org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
at 
org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
at 
org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
at 
org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresFromCacheIds(EntityGroupFSTimelineStore.java:853)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresForRead(EntityGroupFSTimelineStore.java:906)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getEntities(EntityGroupFSTimelineStore.java:959)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doGetEntities(TimelineDataManager.java:169)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.getEntities(TimelineDataManager.java:139)
at 
org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEntities(TimelineWebServices.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}

A temporary fix would be adding /hadoop/shared/hadoop/tools into 
HADOOP_CLASSPATH manually.

  was:
LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in 
/hadoop/shared/hadoop/tools but is never added in timeline server's classpath. 
Reading data from LevelDBCacheTimelineStore will get NoClassDefFoundError. 
Here's the stacktrace:
{code:java}
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
at 
org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
at 
org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
at 
org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
at 
org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
at 
org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
at 

[jira] [Commented] (YARN-5835) NoClassDefFoundError when reading ATS v1.5

[Zhiyuan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634846#comment-15634846
 ] 

Zhiyuan Yang commented on YARN-5835:


Thanks [~gtCarrera9] for helping debug on this! 
[~hitesh], [~Sreenath], have you ever gotten this issue? This occurs when I run 
Tez UI on hadoop branch-2.8. 

> NoClassDefFoundError when reading ATS v1.5
> --
>
> Key: YARN-5835
> URL: https://issues.apache.org/jira/browse/YARN-5835
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Zhiyuan Yang
>
> LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in 
> /hadoop/shared/hadoop/tools but is never added in timeline server's 
> classpath. Reading data from LevelDBCacheTimelineStore will get 
> NoClassDefFoundError. Here's the stacktrace:
> {code:java}
> java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
>   at 
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
>   at 
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
>   at 
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
>   at 
> org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
>   at 
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
>   at 
> org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
>   at 
> org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
>   at 
> org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
>   at 
> org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
>   at 
> org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
>   at 
> org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
>   at 
> org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
>   at 
> org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresFromCacheIds(EntityGroupFSTimelineStore.java:853)
>   at 
> org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresForRead(EntityGroupFSTimelineStore.java:906)
>   at 
> org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getEntities(EntityGroupFSTimelineStore.java:959)
>   at 
> org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doGetEntities(TimelineDataManager.java:169)
>   at 
> org.apache.hadoop.yarn.server.timeline.TimelineDataManager.getEntities(TimelineDataManager.java:139)
>   at 
> org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEntities(TimelineWebServices.java:119)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5828) Native services client errors out when config formats are uppercase

[Gour Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634840#comment-15634840
 ] 

Gour Saha commented on YARN-5828:
-

[~billie.rinaldi] the patch looks good. I tested it with an app with configs 
and it starts up fine. +1 for commit.

> Native services client errors out when config formats are uppercase
> ---
>
> Key: YARN-5828
> URL: https://issues.apache.org/jira/browse/YARN-5828
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-5828-yarn-native-services.001.patch
>
>
> The services API is using uppercase config file formats now, which makes our 
> client error out with "org.apache.slider.core.exceptions.BadConfigException: 
> Config format null doesn't exist." We can make this error more informative 
> and make the config formats not be case sensitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5835) NoClassDefFoundError when reading ATS v1.5

[Zhiyuan Yang (JIRA)]

Zhiyuan Yang created YARN-5835:
--

 Summary: NoClassDefFoundError when reading ATS v1.5
 Key: YARN-5835
 URL: https://issues.apache.org/jira/browse/YARN-5835
 Project: Hadoop YARN
  Issue Type: Task
Reporter: Zhiyuan Yang


LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in 
/hadoop/shared/hadoop/tools but is never added in timeline server's classpath. 
Reading data from LevelDBCacheTimelineStore will get NoClassDefFoundError. 
Here's the stacktrace:
{code:java}
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
at 
org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
at 
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
at 
org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
at 
org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
at 
org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
at 
org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresFromCacheIds(EntityGroupFSTimelineStore.java:853)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresForRead(EntityGroupFSTimelineStore.java:906)
at 
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getEntities(EntityGroupFSTimelineStore.java:959)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doGetEntities(TimelineDataManager.java:169)
at 
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.getEntities(TimelineDataManager.java:139)
at 
org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEntities(TimelineWebServices.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5834) TestNodeStatusUpdater.testNMRMConnectionConf compares nodemanager wait time to the incorrect value

[Miklos Szegedi (JIRA)]

Miklos Szegedi created YARN-5834:


 Summary: TestNodeStatusUpdater.testNMRMConnectionConf compares 
nodemanager wait time to the incorrect value
 Key: YARN-5834
 URL: https://issues.apache.org/jira/browse/YARN-5834
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Miklos Szegedi
Priority: Minor


The function is TestNodeStatusUpdater#testNMRMConnectionConf()
I believe the connectionWaitMs references below were meant to be 
nmRmConnectionWaitMs.
{code}
conf.setLong(YarnConfiguration.NM_RESOURCEMANAGER_CONNECT_MAX_WAIT_MS,
nmRmConnectionWaitMs);
conf.setLong(YarnConfiguration.RESOURCEMANAGER_CONNECT_MAX_WAIT_MS,
connectionWaitMs);
...
  long t = System.currentTimeMillis();
  long duration = t - waitStartTime;
  boolean waitTimeValid = (duration >= nmRmConnectionWaitMs) &&
  (duration < (*connectionWaitMs* + delta));

  if(!waitTimeValid) {
// throw exception if NM doesn't retry long enough
throw new Exception("NM should have tried re-connecting to RM during " +
  "period of at least " + *connectionWaitMs* + " ms, but " +
  "stopped retrying within " + (*connectionWaitMs* + delta) +
  " ms: " + e, e);
  }
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5783) Verify applications are identified starved

[Karthik Kambatla (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634830#comment-15634830
 ] 

Karthik Kambatla commented on YARN-5783:


bq. In SchedulerApplicationAttempt.hashCode(), what is the value of adding the 
prime * positive?
Honestly, not sure. This [stackoverflow 
link|http://stackoverflow.com/questions/113511/best-implementation-for-hashcode-method]
 suggested it, and all auto-generated protobuf files seem to adopt that 
approach too. 

Addressing the other two comments in the next patch. 

> Verify applications are identified starved
> --
>
> Key: YARN-5783
> URL: https://issues.apache.org/jira/browse/YARN-5783
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>  Labels: oct16-medium
> Attachments: yarn-5783.YARN-4752.1.patch, 
> yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch, 
> yarn-5783.YARN-4752.4.patch, yarn-5783.YARN-4752.5.patch
>
>
> JIRA to track unit tests to verify the identification of starved 
> applications. An application should be marked starved only when:
> # Cluster allocation is over the configured threshold for preemption.
> # Preemption is enabled for a queue and any of the following:
> ## The queue is under its minshare for longer than minsharePreemptionTimeout
> ## One of the queue’s applications is under its fairshare for longer than 
> fairsharePreemptionTimeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis

[Miklos Szegedi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Miklos Szegedi updated YARN-5600:
-
Attachment: YARN-5600.003.patch

Updated code based on comments.

> Add a parameter to ContainerLaunchContext to emulate 
> yarn.nodemanager.delete.debug-delay-sec on a per-application basis
> ---
>
> Key: YARN-5600
> URL: https://issues.apache.org/jira/browse/YARN-5600
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Miklos Szegedi
>  Labels: oct16-medium
> Attachments: YARN-5600.000.patch, YARN-5600.001.patch, 
> YARN-5600.002.patch, YARN-5600.003.patch
>
>
> To make debugging application launch failures simpler, I'd like to add a 
> parameter to the CLC to allow an application owner to request delayed 
> deletion of the application's launch artifacts.
> This JIRA solves largely the same problem as YARN-5599, but for cases where 
> ATS is not in use, e.g. branch-2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis

[Miklos Szegedi (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634788#comment-15634788
 ] 

Miklos Szegedi commented on YARN-5600:
--

Thank you for the review [~templedf]!

It seems to me that you're doing extra work to keep the delete time as a Date, 
not to mention adding potential time zone concerns. Millis since the epoch may 
be simpler.
  I do not necessarily agree with using a long of milliseconds here. The value 
is a time stamp and Date is the natural representation. I just use the standard 
Date functions.
  Date does not bring in timezone concerns, it is implemented with the same 
long inside. It is supposed to be timezone independent. A custom implementation 
may add review overhead.
  I could try MonotonicClock, but it uses System.nanoTime(). It is not safe to 
use System.nanoTime() across threads and getDelayedDeletionTime() may be called 
from multiple threads.

The DeletionService.scheduleFileDeletionTask() methods can and probably should 
be private.
   Unfortunately ResourceLocalizationService#cleanUpFilesPerUserDir uses it, so 
I have to keep it public.
   I agree that this needs to be refactored. Feel free to file a JIRA.

In your tests, instead of sleeping and asserting, sleep for short periods in a 
loop to minimize the test time.
  testDelayedDeletionContainerOnly(), testDelayedDeletion(): I need to wait as 
long as the specified retention time but I added the suggested pattern once 
that has elapsed.
  testDelayedKeep(): the condition is that nothing happens for X amount of 
time, so the wait is expected. I am happy to accept suggestions on the wait 
time, here.
I think 200ms should be enough to catch any pending deletes.

> Add a parameter to ContainerLaunchContext to emulate 
> yarn.nodemanager.delete.debug-delay-sec on a per-application basis
> ---
>
> Key: YARN-5600
> URL: https://issues.apache.org/jira/browse/YARN-5600
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Miklos Szegedi
>  Labels: oct16-medium
> Attachments: YARN-5600.000.patch, YARN-5600.001.patch, 
> YARN-5600.002.patch
>
>
> To make debugging application launch failures simpler, I'd like to add a 
> parameter to the CLC to allow an application owner to request delayed 
> deletion of the application's launch artifacts.
> This JIRA solves largely the same problem as YARN-5599, but for cases where 
> ATS is not in use, e.g. branch-2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis

[Miklos Szegedi (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634787#comment-15634787
 ] 

Miklos Szegedi commented on YARN-5600:
--

Thank you for the review [~templedf]!

It seems to me that you're doing extra work to keep the delete time as a Date, 
not to mention adding potential time zone concerns. Millis since the epoch may 
be simpler.
  I do not necessarily agree with using a long of milliseconds here. The value 
is a time stamp and Date is the natural representation. I just use the standard 
Date functions.
  Date does not bring in timezone concerns, it is implemented with the same 
long inside. It is supposed to be timezone independent. A custom implementation 
may add review overhead.
  I could try MonotonicClock, but it uses System.nanoTime(). It is not safe to 
use System.nanoTime() across threads and getDelayedDeletionTime() may be called 
from multiple threads.

The DeletionService.scheduleFileDeletionTask() methods can and probably should 
be private.
   Unfortunately ResourceLocalizationService#cleanUpFilesPerUserDir uses it, so 
I have to keep it public.
   I agree that this needs to be refactored. Feel free to file a JIRA.

In your tests, instead of sleeping and asserting, sleep for short periods in a 
loop to minimize the test time.
  testDelayedDeletionContainerOnly(), testDelayedDeletion(): I need to wait as 
long as the specified retention time but I added the suggested pattern once 
that has elapsed.
  testDelayedKeep(): the condition is that nothing happens for X amount of 
time, so the wait is expected. I am happy to accept suggestions on the wait 
time, here.
I think 200ms should be enough to catch any pending deletes.

> Add a parameter to ContainerLaunchContext to emulate 
> yarn.nodemanager.delete.debug-delay-sec on a per-application basis
> ---
>
> Key: YARN-5600
> URL: https://issues.apache.org/jira/browse/YARN-5600
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Miklos Szegedi
>  Labels: oct16-medium
> Attachments: YARN-5600.000.patch, YARN-5600.001.patch, 
> YARN-5600.002.patch
>
>
> To make debugging application launch failures simpler, I'd like to add a 
> parameter to the CLC to allow an application owner to request delayed 
> deletion of the application's launch artifacts.
> This JIRA solves largely the same problem as YARN-5599, but for cases where 
> ATS is not in use, e.g. branch-2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2995) Enhance UI to show cluster resource utilization of various container types

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634782#comment-15634782
 ] 

Hadoop QA commented on YARN-2995:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
41s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
44s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 38s{color} | {color:orange} root: The patch generated 1 new + 428 unchanged 
- 4 fixed = 429 total (was 432) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
22s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
 generated 1 new + 235 unchanged - 0 fixed = 236 total (was 235) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
32s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 57s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 36m 18s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 
25s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 27s{color} 
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 19s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
 |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-2995 |
| JIRA Patch URL | 

[jira] [Commented] (YARN-5833) Change default port for AMRMProxy

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634778#comment-15634778
 ] 

Hadoop QA commented on YARN-5833:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
23s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 27s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5833 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12837002/YARN-5883.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 6f1ce2fa83d2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 5cad93d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/13777/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/13777/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Change default port for AMRMProxy
> -
>
> Key: YARN-5833
> URL: https://issues.apache.org/jira/browse/YARN-5833
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5883.001.patch
>
>
> The default port for the AMRMProxy coincides with the one for the Collector 
> Service (port 8048). Will 

[jira] [Commented] (YARN-5377) TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in trunk

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634767#comment-15634767
 ] 

Hadoop QA commented on YARN-5377:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
 The patch generated 0 new + 32 unchanged - 2 fixed = 32 total (was 34) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 
29s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 48s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5377 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12836999/YARN-5377.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 23cf1854165d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 5cad93d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/13775/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/13775/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in 
> trunk
> --
>
> Key: YARN-5377
> URL: https://issues.apache.org/jira/browse/YARN-5377
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>

[jira] [Updated] (YARN-5833) Change default port for AMRMProxy

[Konstantinos Karanasos (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-5833:
-
Attachment: YARN-5883.001.patch

Attaching patch.

> Change default port for AMRMProxy
> -
>
> Key: YARN-5833
> URL: https://issues.apache.org/jira/browse/YARN-5833
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5883.001.patch
>
>
> The default port for the AMRMProxy coincides with the one for the Collector 
> Service (port 8048). Will use a different port for the AMRMProxy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5783) Verify applications are identified starved

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634725#comment-15634725
 ] 

Hadoop QA commented on YARN-5783:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
51s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
46s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
2s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 22s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 5 new + 85 unchanged - 0 fixed = 90 total (was 85) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 57s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 41s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5783 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12836728/yarn-5783.YARN-4752.5.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux c700ec1bcc86 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-4752 / b425ca2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/13774/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/13774/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/13774/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output | 

[jira] [Commented] (YARN-5585) [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634721#comment-15634721
 ] 

Sangjin Lee commented on YARN-5585:
---

Thanks [~rohithsharma] for the latest patch!

In addition to Varun's comments above, I have a few comments.

(TimelineReaderContext.java)
- Do we need to maintain 2 constructors? Now that the entity id prefix is a key 
part of this, we should probably drop the old constructor. I presume you could 
update this as we update the tests?

(TimelineReaderWebServices.java)
- I am assuming that we will need to add a new method (or methods) that 
explicitly supports pagination and connects that to the 2nd branch of logic you 
added in {{GenericEntityReader.getResults()}}, right?

(TimelineUIDConverter.java)
- What would we emit as part of the UID if the entity id prefix was not set? An 
empty string ("")?

(EntityRowKeyPrefix.java)
- we need to add javadoc to the new method

(GenericEntityReader.java)
- l.85: we should remove {{sortedKeys}} from this constructor too, right?

> [Atsv2] Reader side changes for entity prefix and support for pagination via 
> additional filters
> ---
>
> Key: YARN-5585
> URL: https://issues.apache.org/jira/browse/YARN-5585
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Critical
>  Labels: oct16-hard
> Attachments: 0001-YARN-5585.patch, YARN-5585-YARN-5355.0001.patch, 
> YARN-5585-workaround.patch, YARN-5585.v0.patch
>
>
> TimelineReader REST API's provides lot of filters to retrieve the 
> applications. Along with those, it would be good to add new filter i.e fromId 
> so that entities can be retrieved after the fromId. 
> Current Behavior : Default limit is set to 100. If there are 1000 entities 
> then REST call gives first/last 100 entities. How to retrieve next set of 100 
> entities i.e 101 to 200 OR 900 to 801?
> Example : If applications are stored database, app-1 app-2 ... app-10.
> *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is 
> no way to achieve this. 
> So proposal is to have fromId in the filter like 
> *getApps?limit=5&=app-5* which gives list of apps from app-6 to 
> app-10. 
> Since ATS is targeting large number of entities storage, it is very common 
> use case to get next set of entities using fromId rather than querying all 
> the entites. This is very useful for pagination in web UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5823) Update NMTokens in case of requests with only opportunistic containers

[Konstantinos Karanasos (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-5823:
-
Attachment: YARN-5823.002.patch

Attaching new patch -- fixing findbug and checkstyle issues.

> Update NMTokens in case of requests with only opportunistic containers
> --
>
> Key: YARN-5823
> URL: https://issues.apache.org/jira/browse/YARN-5823
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>Priority: Blocker
> Attachments: YARN-5823.001.patch, YARN-5823.002.patch
>
>
> At the moment, when an {{AllocateRequest}} contains only opportunistic 
> {{ResourceRequests}}, the updated NMTokens are not properly added to the 
> {{AllocateResponse}}.
> In such a case the AM does not get back the needed NMTokens that are required 
> to start the opportunistic containers at the respective nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5377) TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in trunk

[Konstantinos Karanasos (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634701#comment-15634701
 ] 

Konstantinos Karanasos edited comment on YARN-5377 at 11/3/16 11:59 PM:


The problem with the test is that the container was moving fast from the DONE 
to the CONTAINER_CLEANUP_AFTER_KILL state, and the DONE state was not observed 
by the {{waitForNMContainerState}} method of the {{BaseContainerManagerTest}}.

I added a new {{waitForNMContainerState}} method that takes as input a list of 
final container states, instead of a single one like before. When any of the 
states of this list is reached, the {{waitForNMContainerState}} exits 
successfully.


was (Author: kkaranasos):
The problem with the test is that the container was moving fast from the DONE 
to the CONTAINER_CLEANUP_AFTER_KILL, and the DONE state was not observed by the 
{{waitForNMContainerState}} method of the {{BaseContainerManagerTest}}.

I added a new {{waitForNMContainerState}} method that takes as input a list of 
final container states, instead of a single one like before. When any of the 
states of this list is reached, the {{waitForNMContainerState}} exits 
successfully.

> TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in 
> trunk
> --
>
> Key: YARN-5377
> URL: https://issues.apache.org/jira/browse/YARN-5377
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5377.001.patch
>
>
> Test case fails jenkin build 
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/12228/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt]
> {noformat}
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 134.586 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
> testKillMultipleOpportunisticContainers(org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager)
>   Time elapsed: 32.134 sec  <<< FAILURE!
> java.lang.AssertionError: ContainerState is not correct (timedout) 
> expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.BaseContainerManagerTest.waitForNMContainerState(BaseContainerManagerTest.java:363)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager.testKillMultipleOpportunisticContainers(TestQueuingContainerManager.java:470)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5377) TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in trunk

[Konstantinos Karanasos (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-5377:
-
Attachment: YARN-5377.001.patch

The problem with the test is that the container was moving fast from the DONE 
to the CONTAINER_CLEANUP_AFTER_KILL, and the DONE state was not observed by the 
{{waitForNMContainerState}} method of the {{BaseContainerManagerTest}}.

I added a new {{waitForNMContainerState}} method that takes as input a list of 
final container states, instead of a single one like before. When any of the 
states of this list is reached, the {{waitForNMContainerState}} exits 
successfully.

> TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in 
> trunk
> --
>
> Key: YARN-5377
> URL: https://issues.apache.org/jira/browse/YARN-5377
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5377.001.patch
>
>
> Test case fails jenkin build 
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/12228/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt]
> {noformat}
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 134.586 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
> testKillMultipleOpportunisticContainers(org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager)
>   Time elapsed: 32.134 sec  <<< FAILURE!
> java.lang.AssertionError: ContainerState is not correct (timedout) 
> expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.BaseContainerManagerTest.waitForNMContainerState(BaseContainerManagerTest.java:363)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager.testKillMultipleOpportunisticContainers(TestQueuingContainerManager.java:470)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5833) Change default port for AMRMProxy

[Konstantinos Karanasos (JIRA)]

Konstantinos Karanasos created YARN-5833:


 Summary: Change default port for AMRMProxy
 Key: YARN-5833
 URL: https://issues.apache.org/jira/browse/YARN-5833
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Konstantinos Karanasos


The default port for the AMRMProxy coincides with the one for the Collector 
Service (port 8048). Will use a different port for the AMRMProxy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-5833) Change default port for AMRMProxy

[Konstantinos Karanasos (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos reassigned YARN-5833:


Assignee: Konstantinos Karanasos

> Change default port for AMRMProxy
> -
>
> Key: YARN-5833
> URL: https://issues.apache.org/jira/browse/YARN-5833
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>
> The default port for the AMRMProxy coincides with the one for the Collector 
> Service (port 8048). Will use a different port for the AMRMProxy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5258) Document Use of Docker with LinuxContainerExecutor

[Sidharta Seethana (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634608#comment-15634608
 ] 

Sidharta Seethana edited comment on YARN-5258 at 11/3/16 11:10 PM:
---

Feedback : 

{code}
Docker combines an easy-to-use interface to Linux containers with 
easy-to-construct image files for those containers. In short, Docker launches 
very light weight virtual machines.
{code}

IMO, this is not an accurate characterization and we should drop it. This blog 
post explains this is more detail : 
https://blog.docker.com/2016/03/containers-are-not-vms/ . It might also be a 
good idea to link to http://docs.docker.com/ . 

{code}
The Linux Container Executor (LCE) allows the YARN NodeManager to launch YARN 
containers into Docker containers. 
{code}

I believe a brief description of ‘container runtimes’ would be warranted in 
this section - LCE currently supports two - the default/‘process’ based runtime 
and the docker runtime. It is possible to choose between these on a per 
container basis. Alternatively, additional information could be added in a 
follow up patch(es).

{code}
The Docker suuport in the LCE is still evolving.
{code}

minor typo. suuport -> support

{code}
To track progress, follow JIRA-3611, 
{code}

It might be better to say YARN-3611 - with a link. 

{code}
sudo docker pull images/hadoop-docker:latest
{code}

IMO, this should be a working example. That said, I am not aware of any popular 
vendor-neutral images that would be a good candidate. This has been one of the 
barriers to creating good documentation for this functionality. Should we 
consider hosting ‘official’ apache hadoop images on docker hub ? Thoughts ? 

{code}
The following properties should be set in yarn-site.xml:
{code}

Some of the properties described here don’t have values that are inline with 
yarn-default.xml . Specifically, 
{{yarn.nodemanager.runtime.linux.docker.allowed-container-networks}} and 
{{yarn.nodemanager.runtime.linux.docker.privileged-containers.acl}} . There is 
also a setting that isn’t mentioned here : 
{{yarn.nodemanager.runtime.linux.docker.default-container-network}} . I think a 
separate section on networking is warranted - I’ll submit a follow up patch 
with additional documentation. 

{code}
feature.docker.enabled
{code}

In the context of this functionality, this is not ‘optional’ - this must be set 
of 1. 


{code}
In order to work with YARN, there are two requirements for Docker images.
{code}

There are additional limitations - again, these could be added in subsequent 
updates to the documentation. An important limitation that comes to mind is 
that because YARN always overrides the command the container is launched with, 
images with an {{ENTRYPOINT}} directive will not work. Application frameworks 
may impose their additional requirements. For example, using slider with Docker 
and YARN (currently) requires that all images have python installed in them (in 
order to run the slider agent). 

{code}
First, the Docker container will be explicitly launched with the application 
owner as the container user. If the application owner is not a valid user (by 
UID) in the Docker image, the application will fail.
{code}

By UID? This is not clear - it might be useful to provide an example here. One 
example I can think of here - the UID of ‘nobody’ is different in CentOS vs 
Ubuntu - so running an Ubuntu container on CentOS as user ‘nobody’ is likely to 
cause failures. 


{code}
In order to run an application in a Docker container, set the following 
environment variables in the application's environment:
{code}

 It might be worth pointing out that while this is not ideal, it does allow for 
some existing applications that can inject environment variables to run in 
docker containers without modifications e.g spark and map reduce. 

{code}
Example: Spark
{code}

As mentioned earlier : I think this should be an actual working example and we 
should consider exploring what it would take to make that possible. 

I think this is a great start, thanks again [~templedf] for taking this on. 




was (Author: sidharta-s):
Feedback : 

{code}
Docker combines an easy-to-use interface to Linux containers with 
easy-to-construct image files for those containers. In short, Docker launches 
very light weight virtual machines.
{code}

IMO, this is not an accurate characterization and we should drop it. This blog 
post explains this is more detail : 
https://blog.docker.com/2016/03/containers-are-not-vms/ . It might also be a 
good idea to link to http://docs.docker.com/ . 

{code}
The Linux Container Executor (LCE) allows the YARN NodeManager to launch YARN 
containers into Docker containers. 
{code}

I believe a brief description of ‘container runtimes’ would be warranted in 
this section - LCE currently supports two - the default/‘process’ based runtime 
and the docker runtime. It is 

[jira] [Commented] (YARN-5258) Document Use of Docker with LinuxContainerExecutor

[Sidharta Seethana (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634608#comment-15634608
 ] 

Sidharta Seethana commented on YARN-5258:
-

Feedback : 

{code}
Docker combines an easy-to-use interface to Linux containers with 
easy-to-construct image files for those containers. In short, Docker launches 
very light weight virtual machines.
{code}

IMO, this is not an accurate characterization and we should drop it. This blog 
post explains this is more detail : 
https://blog.docker.com/2016/03/containers-are-not-vms/ . It might also be a 
good idea to link to http://docs.docker.com/ . 

{code}
The Linux Container Executor (LCE) allows the YARN NodeManager to launch YARN 
containers into Docker containers. 
{code}

I believe a brief description of ‘container runtimes’ would be warranted in 
this section - LCE currently supports two - the default/‘process’ based runtime 
and the docker runtime. It is possible to choose between these on a per 
container basis. Alternatively, additional information could be added in a 
follow up patch(es).

{code}
The Docker suuport in the LCE is still evolving.
{code}

minor typo. suuport -> support

{code}
To track progress, follow JIRA-3611, 
{code}

It might be better to say YARN-3611 - with a link. 

{code}
sudo docker pull images/hadoop-docker:latest
{code}

IMO, this should be a working example. That said, I am not aware of any popular 
vendor-neutral images that would be a good candidate. This has been one of the 
barriers to creating good documentation for this functionality. Should we 
consider hosting ‘official’ apache hadoop images on docker hub ? Thoughts ? 

{code}
The following properties should be set in yarn-site.xml:
{code}

Some of the properties described here don’t have values that are inline with 
yarn-default.xml . Specifically, 
yarn.nodemanager.runtime.linux.docker.allowed-container-networks and 
yarn.nodemanager.runtime.linux.docker.privileged-containers.acl . There is also 
a setting that isn’t mentioned here : 
yarn.nodemanager.runtime.linux.docker.default-container-network . I think a 
separate section on networking is warranted - I’ll submit a follow up patch 
with additional documentation. 

{code}
feature.docker.enabled
{code}

In the context of this functionality, this is not ‘optional’ - this must be set 
of 1. 


{code}
In order to work with YARN, there are two requirements for Docker images.
{code}

There are additional limitations - again, these could be added in subsequent 
updates to the documentation. An important limitation that comes to mind is 
that because YARN always overrides the command the container is launched with, 
images with an {{ENTRYPOINT}} directive will not work. Application frameworks 
may impose their additional requirements. For example, using slider with Docker 
and YARN (currently) requires that all images have python installed in them (in 
order to run the slider agent). 

{code}
First, the Docker container will be explicitly launched with the application 
owner as the container user. If the application owner is not a valid user (by 
UID) in the Docker image, the application will fail.
{code}

By UID? This is not clear - it might be useful to provide an example here. One 
example I can think of here - the UID of ‘nobody’ is different in CentOS vs 
Ubuntu - so running an Ubuntu container on CentOS as user ‘nobody’ is likely to 
cause failures. 


{code}
In order to run an application in a Docker container, set the following 
environment variables in the application's environment:
{code}

 It might be worth pointing out that while this is not ideal, it does allow for 
some existing applications that can inject environment variables to run in 
docker containers without modifications e.g spark and map reduce. 

{code}
Example: Spark
{code}

As mentioned earlier : I think this should be an actual working example and we 
should consider exploring what it would take to make that possible. 

I think this is a great start, thanks again [~templedf] for taking this on. 



> Document Use of Docker with LinuxContainerExecutor
> --
>
> Key: YARN-5258
> URL: https://issues.apache.org/jira/browse/YARN-5258
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>  Labels: oct16-easy
> Attachments: YARN-5258.001.patch, YARN-5258.002.patch
>
>
> There aren't currently any docs that explain how to configure Docker and all 
> of its various options aside from reading all of the JIRAs.  We need to 
> document the configuration, use, and troubleshooting, along with helpful 
> examples.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (YARN-4355) NPE while processing localizer heartbeat

[Jonathan Hung (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634576#comment-15634576
 ] 

Jonathan Hung edited comment on YARN-4355 at 11/3/16 10:54 PM:
---

The initial patch looks good to me.

[~Naganarasimha], [~templedf], I rebased and uploaded a patch based on 
[~varun_saxena]'s patch (conflict was due to YARN-1942) and addressed the 
comments, can you please take a look? Thanks!

Also it seems if this race condition is hit, the localization service on that 
node will die which seems like a pretty serious issue? Let me know if I 
understood correctly.


was (Author: jhung):
Hi [~Naganarasimha], [~templedf], I rebased and uploaded a patch based on 
[~varun_saxena]'s patch (conflict was due to YARN-1942) and addressed the 
comments, can you please take a look? Thanks!

Also it seems if this race condition is hit, the localization service on that 
node will die which seems like a pretty serious issue? Let me know if I 
understood correctly.

> NPE while processing localizer heartbeat
> 
>
> Key: YARN-4355
> URL: https://issues.apache.org/jira/browse/YARN-4355
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.2
>Reporter: Jason Lowe
>Assignee: Varun Saxena
> Attachments: YARN-4355.01.patch, YARN-4355.02.patch
>
>
> While analyzing YARN-4354 I noticed a nodemanager was getting NPEs while 
> processing a private localizer heartbeat.  I think there's a race where we 
> can cleanup resources for an application and therefore remove the app local 
> resource tracker just as we are trying to handle the localizer heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4355) NPE while processing localizer heartbeat

[Jonathan Hung (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634576#comment-15634576
 ] 

Jonathan Hung commented on YARN-4355:
-

Hi [~Naganarasimha], [~templedf], I rebased and uploaded a patch based on 
[~varun_saxena]'s patch (conflict was due to YARN-1942) and addressed the 
comments, can you please take a look? Thanks!

Also it seems if this race condition is hit, the localization service on that 
node will die which seems like a pretty serious issue? Let me know if I 
understood correctly.

> NPE while processing localizer heartbeat
> 
>
> Key: YARN-4355
> URL: https://issues.apache.org/jira/browse/YARN-4355
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.2
>Reporter: Jason Lowe
>Assignee: Varun Saxena
> Attachments: YARN-4355.01.patch, YARN-4355.02.patch
>
>
> While analyzing YARN-4354 I noticed a nodemanager was getting NPEs while 
> processing a private localizer heartbeat.  I think there's a race where we 
> can cleanup resources for an application and therefore remove the app local 
> resource tracker just as we are trying to handle the localizer heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4355) NPE while processing localizer heartbeat

[Jonathan Hung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hung updated YARN-4355:

Attachment: YARN-4355.02.patch

> NPE while processing localizer heartbeat
> 
>
> Key: YARN-4355
> URL: https://issues.apache.org/jira/browse/YARN-4355
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.2
>Reporter: Jason Lowe
>Assignee: Varun Saxena
> Attachments: YARN-4355.01.patch, YARN-4355.02.patch
>
>
> While analyzing YARN-4354 I noticed a nodemanager was getting NPEs while 
> processing a private localizer heartbeat.  I think there's a race where we 
> can cleanup resources for an application and therefore remove the app local 
> resource tracker just as we are trying to handle the localizer heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4355) NPE while processing localizer heartbeat

[Jonathan Hung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hung updated YARN-4355:

Attachment: (was: YARN-4355.02.patch)

> NPE while processing localizer heartbeat
> 
>
> Key: YARN-4355
> URL: https://issues.apache.org/jira/browse/YARN-4355
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.2
>Reporter: Jason Lowe
>Assignee: Varun Saxena
> Attachments: YARN-4355.01.patch
>
>
> While analyzing YARN-4354 I noticed a nodemanager was getting NPEs while 
> processing a private localizer heartbeat.  I think there's a race where we 
> can cleanup resources for an application and therefore remove the app local 
> resource tracker just as we are trying to handle the localizer heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4355) NPE while processing localizer heartbeat

[Jonathan Hung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hung updated YARN-4355:

Attachment: YARN-4355.02.patch

> NPE while processing localizer heartbeat
> 
>
> Key: YARN-4355
> URL: https://issues.apache.org/jira/browse/YARN-4355
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.2
>Reporter: Jason Lowe
>Assignee: Varun Saxena
> Attachments: YARN-4355.01.patch, YARN-4355.02.patch
>
>
> While analyzing YARN-4354 I noticed a nodemanager was getting NPEs while 
> processing a private localizer heartbeat.  I think there's a race where we 
> can cleanup resources for an application and therefore remove the app local 
> resource tracker just as we are trying to handle the localizer heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5828) Native services client errors out when config formats are uppercase

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634546#comment-15634546
 ] 

Hadoop QA commented on YARN-5828:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
48s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
29s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
22s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
32s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  
4s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
 in yarn-native-services has 314 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
30s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services 
failed. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
8s{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
 generated 0 new + 313 unchanged - 1 fixed = 313 total (was 314) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
25s{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
25s{color} | {color:green} hadoop-yarn-slider-core in the patch passed. {color} 
|
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
20s{color} | {color:red} The patch generated 11 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 50s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5828 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12836980/YARN-5828-yarn-native-services.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 29e2acb04a50 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | yarn-native-services / 2ec9e9d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs | 
https://builds.apache.org/job/PreCommit-YARN-Build/13772/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-slider_hadoop-yarn-slider-core-warnings.html
 |
| javadoc | 
https://builds.apache.org/job/PreCommit-YARN-Build/13772/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-slider_hadoop-yarn-slider-core.txt
 |
| javadoc | 

[jira] [Created] (YARN-5832) Preemption should consider multiple resource-requests of a starved app as needed

[Karthik Kambatla (JIRA)]

Karthik Kambatla created YARN-5832:
--

 Summary: Preemption should consider multiple resource-requests of 
a starved app as needed
 Key: YARN-5832
 URL: https://issues.apache.org/jira/browse/YARN-5832
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: fairscheduler
Reporter: Karthik Kambatla


When identifying containers to preempt for a starved application, the new 
implementation considers only the highest priority resource request. If we 
can't find candidate containers for that request, we might want to consider 
other requests. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2995) Enhance UI to show cluster resource utilization of various container types

[Konstantinos Karanasos (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634527#comment-15634527
 ] 

Konstantinos Karanasos commented on YARN-2995:
--

Regarding the remaining issues:
* The checkstyle issue is about a method that takes more than 7 parameters in 
one of the test classes. That was already the case for that method; I just 
added some more parameters.
* The javadoc issue, as I already explained in a comment above, is related to 
Java 8 complaining about using '_' as identifiers. This is used in multiple 
places in the Web UI classes, and should be treated in a separate JIRA.
* The unit test issue regarding {{TestQueuingContainerManager}} is unrelated to 
the present JIRA and is tracked in YARN-5377.
* There is a build issue in sls when running the corresponding tests, but it 
might be a Jenkins issue, since it builds fine for me locally. Just kicked-off 
Jenkins again to see if the problem persists.

> Enhance UI to show cluster resource utilization of various container types
> --
>
> Key: YARN-2995
> URL: https://issues.apache.org/jira/browse/YARN-2995
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Sriram Rao
>Assignee: Konstantinos Karanasos
>Priority: Blocker
> Attachments: YARN-2995.001.patch, YARN-2995.002.patch, 
> YARN-2995.003.patch, YARN-2995.004.patch, all-nodes.png, all-nodes.png, 
> opp-container.png
>
>
> This JIRA proposes to extend the Resource manager UI to show how cluster 
> resources are being used to run *guaranteed start* and *queueable* 
> containers.  For example, a graph that shows over time, the fraction of  
> running containers that are *guaranteed start* and the fraction of running 
> containers that are *queueable*. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5783) Verify applications are identified starved

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634515#comment-15634515
 ] 

Daniel Templeton commented on YARN-5783:


Comments:

* In {{SchedulerApplicationAttempt.hashCode()}}, what is the value of adding 
the prime \* positive?
* In {{FSStarvedApps.take()}} javadoc you don't need the hyphen in the @throws 
message.
* In {{TestFSAppsStarvation}} test methods, should {code}
setupStarvedCluster();
submitAppsToEachLeafQueue();
sendNodeUpdateEvents();

// Sleep to hit the preemption timeouts
Thread.sleep(10);

// Scheduler update to populate starved apps
scheduler.update();

// Wait for apps to be processed by MockPreemptionThread
Thread.yield();{code} be put into a common method to be called from all the 
test methods?  And maybe call that from {{testPreemptionDisabled()}} as well so 
that you give it a fair chance to prove that it really is disabled?

> Verify applications are identified starved
> --
>
> Key: YARN-5783
> URL: https://issues.apache.org/jira/browse/YARN-5783
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>  Labels: oct16-medium
> Attachments: yarn-5783.YARN-4752.1.patch, 
> yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch, 
> yarn-5783.YARN-4752.4.patch, yarn-5783.YARN-4752.5.patch
>
>
> JIRA to track unit tests to verify the identification of starved 
> applications. An application should be marked starved only when:
> # Cluster allocation is over the configured threshold for preemption.
> # Preemption is enabled for a queue and any of the following:
> ## The queue is under its minshare for longer than minsharePreemptionTimeout
> ## One of the queue’s applications is under its fairshare for longer than 
> fairsharePreemptionTimeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5377) TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in trunk

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634513#comment-15634513
 ] 

Arun Suresh commented on YARN-5377:
---

Assigning this to [~kkaranasos], since he the original author of the testcase 
and he has a patch available for this.

> TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in 
> trunk
> --
>
> Key: YARN-5377
> URL: https://issues.apache.org/jira/browse/YARN-5377
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Konstantinos Karanasos
>
> Test case fails jenkin build 
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/12228/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt]
> {noformat}
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 134.586 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
> testKillMultipleOpportunisticContainers(org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager)
>   Time elapsed: 32.134 sec  <<< FAILURE!
> java.lang.AssertionError: ContainerState is not correct (timedout) 
> expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.BaseContainerManagerTest.waitForNMContainerState(BaseContainerManagerTest.java:363)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager.testKillMultipleOpportunisticContainers(TestQueuingContainerManager.java:470)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5377) TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in trunk

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634513#comment-15634513
 ] 

Arun Suresh edited comment on YARN-5377 at 11/3/16 10:26 PM:
-

Assigning this to [~kkaranasos], since he's the original author of the testcase 
and he has a patch available for this.


was (Author: asuresh):
Assigning this to [~kkaranasos], since he the original author of the testcase 
and he has a patch available for this.

> TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in 
> trunk
> --
>
> Key: YARN-5377
> URL: https://issues.apache.org/jira/browse/YARN-5377
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Konstantinos Karanasos
>
> Test case fails jenkin build 
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/12228/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt]
> {noformat}
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 134.586 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
> testKillMultipleOpportunisticContainers(org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager)
>   Time elapsed: 32.134 sec  <<< FAILURE!
> java.lang.AssertionError: ContainerState is not correct (timedout) 
> expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.BaseContainerManagerTest.waitForNMContainerState(BaseContainerManagerTest.java:363)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager.testKillMultipleOpportunisticContainers(TestQueuingContainerManager.java:470)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5377) TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in trunk

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5377:
--
Assignee: Konstantinos Karanasos

> TestQueuingContainerManager.testKillMultipleOpportunisticContainers fails in 
> trunk
> --
>
> Key: YARN-5377
> URL: https://issues.apache.org/jira/browse/YARN-5377
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Konstantinos Karanasos
>
> Test case fails jenkin build 
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/12228/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt]
> {noformat}
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 134.586 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
> testKillMultipleOpportunisticContainers(org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager)
>   Time elapsed: 32.134 sec  <<< FAILURE!
> java.lang.AssertionError: ContainerState is not correct (timedout) 
> expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.BaseContainerManagerTest.waitForNMContainerState(BaseContainerManagerTest.java:363)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager.testKillMultipleOpportunisticContainers(TestQueuingContainerManager.java:470)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5831) Propagate allowPreemptionFrom flag all the way down to the app

[Karthik Kambatla (JIRA)]

Karthik Kambatla created YARN-5831:
--

 Summary: Propagate allowPreemptionFrom flag all the way down to 
the app
 Key: YARN-5831
 URL: https://issues.apache.org/jira/browse/YARN-5831
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: fairscheduler
Reporter: Karthik Kambatla


FairScheduler allows disallowing preemption from a queue. When checking if 
preemption for an application is allowed, the new preemption code recurses all 
the way to the root queue to check this flag. 

Propagating this information all the way to the app will be more efficient. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5821) Drop left-over preemption-related code and clean up method visibilities in the Schedulable hierarchy

[Karthik Kambatla (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634467#comment-15634467
 ] 

Karthik Kambatla commented on YARN-5821:


Checking this in..

> Drop left-over preemption-related code and clean up method visibilities in 
> the Schedulable hierarchy
> 
>
> Key: YARN-5821
> URL: https://issues.apache.org/jira/browse/YARN-5821
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-5821.YARN-4752.1.patch, yarn-5821.YARN-4752.1.patch
>
>
> There is some code left-over from old preemption. We need to drop that.
> Also, looks like the visibilities in the {{Schedulable}} hierarchy need to be 
> revisited. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5258) Document Use of Docker with LinuxContainerExecutor

[Sidharta Seethana (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634430#comment-15634430
 ] 

Sidharta Seethana commented on YARN-5258:
-

[~templedf], I was able to apply the patch to branch-2. I'll do a review and 
get back to you soon. 

> Document Use of Docker with LinuxContainerExecutor
> --
>
> Key: YARN-5258
> URL: https://issues.apache.org/jira/browse/YARN-5258
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>  Labels: oct16-easy
> Attachments: YARN-5258.001.patch, YARN-5258.002.patch
>
>
> There aren't currently any docs that explain how to configure Docker and all 
> of its various options aside from reading all of the JIRAs.  We need to 
> document the configuration, use, and troubleshooting, along with helpful 
> examples.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5828) Native services client errors out when config formats are uppercase

[Billie Rinaldi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billie Rinaldi updated YARN-5828:
-
Attachment: YARN-5828-yarn-native-services.001.patch

> Native services client errors out when config formats are uppercase
> ---
>
> Key: YARN-5828
> URL: https://issues.apache.org/jira/browse/YARN-5828
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-5828-yarn-native-services.001.patch
>
>
> The services API is using uppercase config file formats now, which makes our 
> client error out with "org.apache.slider.core.exceptions.BadConfigException: 
> Config format null doesn't exist." We can make this error more informative 
> and make the config formats not be case sensitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5258) Document Use of Docker with LinuxContainerExecutor

[Sidharta Seethana (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634373#comment-15634373
 ] 

Sidharta Seethana commented on YARN-5258:
-

[~templedf] thank you for this patch. I had some trouble applying this patch to 
trunk - was this generated against branch-2? 

> Document Use of Docker with LinuxContainerExecutor
> --
>
> Key: YARN-5258
> URL: https://issues.apache.org/jira/browse/YARN-5258
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>  Labels: oct16-easy
> Attachments: YARN-5258.001.patch, YARN-5258.002.patch
>
>
> There aren't currently any docs that explain how to configure Docker and all 
> of its various options aside from reading all of the JIRAs.  We need to 
> document the configuration, use, and troubleshooting, along with helpful 
> examples.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5830) Avoid preempting AM containers

[Karthik Kambatla (JIRA)]

Karthik Kambatla created YARN-5830:
--

 Summary: Avoid preempting AM containers
 Key: YARN-5830
 URL: https://issues.apache.org/jira/browse/YARN-5830
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: fairscheduler
Reporter: Karthik Kambatla


While considering containers for preemption, avoid AM containers unless they 
are the only container for the app. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-3359) Recover collector list in RM failed over

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634344#comment-15634344
 ] 

Varun Saxena edited comment on YARN-3359 at 11/3/16 9:31 PM:
-

Thanks [~gtCarrera9] for the patch. Code largely seems fine and consistent with 
the approach adopted.

Coming to the approach, it seems on resync we will create a list of all the 
known collectors but this is not sent on register request to RM but on 
subsequent HB to RM. I guess this is done because if we send it on registration 
this would mean protocol changes and would mean additional changes on RM side 
too.
Also, we are sending all collectors known to this NM and not only the NMs' 
launched on it. Can't we keep some additional info for known collectors 
indicating if each one of those collectors is a collector launched on this NM. 
And hence NM reports only these collectors to RM on resync. Otherwise many NMs' 
may pretty much report similar set of collector infos in first NM HB on 
reconnection. This can be potentially optimized ? May not cause much impact 
though considering we only set collector info in RMAppImpl. Thoughts  ?

In NodeManager#reregisterCollectors, in the piece of code below, just to be 
safe should we check for null after get from map ? Looking at the code it is 
highly unlikely to have a NPE here as application is only removed from map when 
aggregation finishes. But better to be safe considering 
ConcurrentHashMap#entrySet is weakly consistent ? 
{code}
  if (!ApplicationState.FINISHED.equals(context.getApplications()
  .get(entry.getKey()).getApplicationState())) {
{code}

Nit. In method NodeManager#resyncWithRM, line 
{{context.getKnownCollectors().clear();}} has been commented out. It can be 
removed.


was (Author: varun_saxena):
Thanks [~gtCarrera9] for the patch. Code largely seems fine and consistent with 
the approach adopted.

Coming to the approach, it seems on resync we will create a list of all the 
known collectors but this is not sent on register request to RM but on 
subsequent HB to RM. I guess this is done because if we send it on registration 
this would mean protocol changes and would mean additional changes on RM side 
too.
Also, we are sending all collectors known to this NM and not only the NMs' 
launched on it. Can't we keep some additional info for known collectors 
indicating if each one of those collectors is a collector launched on this NM. 
And hence NM reports only these collectors to RM on resync. Otherwise all NMs' 
may pretty much report the same collector info in first NM HB on reconnection. 
This can be potentially optimized ? May not cause much impact though 
considering we only set collector info in RMAppImpl. Thoughts  ?

In NodeManager#reregisterCollectors, in the piece of code below, just to be 
safe should we check for null after get from map ? Looking at the code it is 
highly unlikely to have a NPE here as application is only removed from map when 
aggregation finishes. But better to be safe considering 
ConcurrentHashMap#entrySet is weakly consistent ? 
{code}
  if (!ApplicationState.FINISHED.equals(context.getApplications()
  .get(entry.getKey()).getApplicationState())) {
{code}

Nit. In method NodeManager#resyncWithRM, line 
{{context.getKnownCollectors().clear();}} has been commented out. It can be 
removed.

> Recover collector list in RM failed over
> 
>
> Key: YARN-3359
> URL: https://issues.apache.org/jira/browse/YARN-3359
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Junping Du
>Assignee: Li Lu
>  Labels: YARN-5355, oct16-medium
> Attachments: YARN-3359-YARN-5355.001.patch, 
> YARN-3359-YARN-5355.002.patch, YARN-3359-YARN-5638.patch
>
>
> Per discussion in YARN-3039, split the recover work from RMStateStore in a 
> separated JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5821) Drop left-over preemption-related code and clean up method visibilities in the Schedulable hierarchy

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634359#comment-15634359
 ] 

Daniel Templeton commented on YARN-5821:


Looks fine to me. +1

> Drop left-over preemption-related code and clean up method visibilities in 
> the Schedulable hierarchy
> 
>
> Key: YARN-5821
> URL: https://issues.apache.org/jira/browse/YARN-5821
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-5821.YARN-4752.1.patch, yarn-5821.YARN-4752.1.patch
>
>
> There is some code left-over from old preemption. We need to drop that.
> Also, looks like the visibilities in the {{Schedulable}} hierarchy need to be 
> revisited. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3359) Recover collector list in RM failed over

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634344#comment-15634344
 ] 

Varun Saxena commented on YARN-3359:


Thanks [~gtCarrera9] for the patch. Code largely seems fine and consistent with 
the approach adopted.

Coming to the approach, it seems on resync we will create a list of all the 
known collectors but this is not sent on register request to RM but on 
subsequent HB to RM. I guess this is done because if we send it on registration 
this would mean protocol changes and would mean additional changes on RM side 
too.
Also, we are sending all collectors known to this NM and not only the NMs' 
launched on it. Can't we keep some additional info for known collectors 
indicating if each one of those collectors is a collector launched on this NM. 
And hence NM reports only these collectors to RM on resync. Otherwise all NMs' 
may pretty much report the same collector info in first NM HB on reconnection. 
This can be potentially optimized ? May not cause much impact though 
considering we only set collector info in RMAppImpl. Thoughts  ?

In NodeManager#reregisterCollectors, in the piece of code below, just to be 
safe should we check for null after get from map ? Looking at the code it is 
highly unlikely to have a NPE here as application is only removed from map when 
aggregation finishes. But better to be safe considering 
ConcurrentHashMap#entrySet is weakly consistent ? 
{code}
  if (!ApplicationState.FINISHED.equals(context.getApplications()
  .get(entry.getKey()).getApplicationState())) {
{code}

Nit. In method NodeManager#resyncWithRM, line 
{{context.getKnownCollectors().clear();}} has been commented out. It can be 
removed.

> Recover collector list in RM failed over
> 
>
> Key: YARN-3359
> URL: https://issues.apache.org/jira/browse/YARN-3359
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Junping Du
>Assignee: Li Lu
>  Labels: YARN-5355, oct16-medium
> Attachments: YARN-3359-YARN-5355.001.patch, 
> YARN-3359-YARN-5355.002.patch, YARN-3359-YARN-5638.patch
>
>
> Per discussion in YARN-3039, split the recover work from RMStateStore in a 
> separated JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3735) Retain JRE Fatal error logs upon container failure

[JESSE CHEN (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634224#comment-15634224
 ] 

JESSE CHEN commented on YARN-3735:
--

Is there a workaround even?

> Retain JRE Fatal error logs upon container failure
> --
>
> Key: YARN-3735
> URL: https://issues.apache.org/jira/browse/YARN-3735
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Reporter: Srikanth Sundarrajan
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption

[Karthik Kambatla (JIRA)]

Karthik Kambatla created YARN-5829:
--

 Summary: FS preemption should reserve a node before considering 
containers on it for preemption
 Key: YARN-5829
 URL: https://issues.apache.org/jira/browse/YARN-5829
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: fairscheduler
Reporter: Karthik Kambatla


FS preemption evaluates nodes for preemption, and subsequently preempts 
identified containers. If this node is not reserved for a specific application, 
any other application could be allocated resources on this node. 

Reserving the node for the starved application before preempting containers 
would help avoid this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634089#comment-15634089
 ] 

Hadoop QA commented on YARN-5356:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
9s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
27s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
21s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. 
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m 
24s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red}  3m 24s{color} | 
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  3m 24s{color} 
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 43s{color} | {color:orange} root: The patch generated 4 new + 161 unchanged 
- 3 fixed = 165 total (was 164) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
29s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
23s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. 
{color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
25s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
22s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 28s{color} 
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
35s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 24s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 16s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
1s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 41s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| 

[jira] [Commented] (YARN-5587) Add support for resource profiles

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634036#comment-15634036
 ] 

Hadoop QA commented on YARN-5587:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 8 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
50s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
16s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
25s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
 9s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
59s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
38s{color} | {color:green} YARN-3926 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  2m 13s{color} 
| {color:red} hadoop-yarn-project_hadoop-yarn generated 3 new + 38 unchanged - 
1 fixed = 41 total (was 39) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 48s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 37 new + 898 unchanged - 8 fixed = 935 total (was 906) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
36s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
25s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
19s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 14s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 40m 
16s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  7m 52s{color} 
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 15s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
|  |  Class 
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl$ProfileCapabilityComparator
 defines non-transient non-serializable instance field resourceProfilesMap  In 
AMRMClientImpl.java:instance field resourceProfilesMap  In AMRMClientImpl.java |
| Failed junit tests | hadoop.yarn.server.nodemanager.TestDirectoryCollection |
|   | 

[jira] [Commented] (YARN-5611) Provide an API to update lifetime of an application.

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633911#comment-15633911
 ] 

Sunil G commented on YARN-5611:
---

Thanks [~rohithsharma] for the patch.

Few comments

1. ApplicationUpdateTimeoutMapProto -> ApplicationUpdateTimeoutProto. May not 
need to have "map" in the name of proto.
2. Few more comment about the map "applicationTimeouts" in 
{{updateApplicationTimeouts}} of ClientRMService
3. I think we need to create an event internally here for 
{{RPCUtil.getRemoteException}}.
{code}
if (applicationTimeouts.isEmpty()) {
  String message =
   "At least one ApplicationTimeoutType should be configured for 
updating timeouts.";
  RMAuditLogger.logFailure(callerUGI.getShortUserName(),
  AuditConstants.UPDATE_APP_TIMEOUTS, "UNKNOWN", 
"ClientRMService",
  message, applicationId);
  throw RPCUtil.getRemoteException(message);
   }
{code}
4. There is a any chance that NEW_SAVING could fail and app could no longer be 
in next state. We could consider apps which are from ACCEPTED state, correct? 
Any specific reason to consider NEW_SAVING?
{code}
if (RMAppState.NEW.equals(application.getState())
|| COMPLETED_APP_STATES.contains(application.getState())) {
{code}
5. In {{validateISO8601AndConvertToLocalTimeEpoch}}, instead of using {{long 
currentTimeMillis = System.currentTimeMillis();}}, its better to use Clock.
6. In {{validateISO8601AndConvertToLocalTimeEpoch}}, {{if (expireTime < 
currentTimeMillis)}} helps to validate timeout's of past. But still there could 
be a delta time from current time to next timeout check thread. Could that also 
to be validated here? Or such entries will be discarded by the monitor thread?
7. In general thought and thinking out loud, {{Map }} why we need a map here. Could we have {{List}} 
and each *ApplicationTimeouts* could have its type and real timeout value in 
long. I am not seeing much of benefit from map here as app timeout types could 
no be more than 3/4 atmost.



> Provide an API to update lifetime of an application.
> 
>
> Key: YARN-5611
> URL: https://issues.apache.org/jira/browse/YARN-5611
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: oct16-hard
> Attachments: 0001-YARN-5611.patch, 0002-YARN-5611.patch, 
> 0003-YARN-5611.patch, YARN-5611.0004.patch, YARN-5611.0005.patch, 
> YARN-5611.v0.patch
>
>
> YARN-4205 monitors an Lifetime of an applications is monitored if required. 
> Add an client api to update lifetime of an application. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2009) CapacityScheduler: Add intra-queue preemption for app priority support

[Eric Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633851#comment-15633851
 ] 

Eric Payne commented on YARN-2009:
--

Good point. There's no hurry, so we should wait for a few days to see if that 
discussion gets resolved.

> CapacityScheduler: Add intra-queue preemption for app priority support
> --
>
> Key: YARN-2009
> URL: https://issues.apache.org/jira/browse/YARN-2009
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler
>Reporter: Devaraj K
>Assignee: Sunil G
>  Labels: oct16-medium
> Fix For: 2.9.0
>
> Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch, 
> YARN-2009.0003.patch, YARN-2009.0004.patch, YARN-2009.0005.patch, 
> YARN-2009.0006.patch, YARN-2009.0007.patch, YARN-2009.0008.patch, 
> YARN-2009.0009.patch, YARN-2009.0010.patch, YARN-2009.0011.patch, 
> YARN-2009.0012.patch, YARN-2009.0013.patch, YARN-2009.0014.patch, 
> YARN-2009.0015.patch, YARN-2009.0016.patch
>
>
> While preempting containers based on the queue ideal assignment, we may need 
> to consider preempting the low priority application containers first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5828) Native services client errors out when config formats are uppercase

[Billie Rinaldi (JIRA)]

Billie Rinaldi created YARN-5828:


 Summary: Native services client errors out when config formats are 
uppercase
 Key: YARN-5828
 URL: https://issues.apache.org/jira/browse/YARN-5828
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi


The services API is using uppercase config file formats now, which makes our 
client error out with "org.apache.slider.core.exceptions.BadConfigException: 
Config format null doesn't exist." We can make this error more informative and 
make the config formats not be case sensitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5552) Add Builder methods for common yarn API records

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633772#comment-15633772
 ] 

Wangda Tan commented on YARN-5552:
--

Patch looks good, will wait till next Monday before committing. Any other 
suggestions? [~asuresh] / [~kasha].

> Add Builder methods for common yarn API records
> ---
>
> Key: YARN-5552
> URL: https://issues.apache.org/jira/browse/YARN-5552
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Arun Suresh
>Assignee: Tao Jie
> Attachments: YARN-5552.000.patch, YARN-5552.001.patch, 
> YARN-5552.002.patch, YARN-5552.003.patch, YARN-5552.004.patch, 
> YARN-5552.005.patch, YARN-5552.006.patch, YARN-5552.007.patch, 
> YARN-5552.008.patch, YARN-5552.009.patch
>
>
> Currently yarn API records such as ResourceRequest, AllocateRequest/Respone 
> as well as AMRMClient.ContainerRequest have multiple constructors / 
> newInstance methods. This makes it very difficult to add new fields to these 
> records.
> It would probably be better if we had Builder classes for many of these 
> records, which would make evolution of these records a bit easier.
> (suggested by [~kasha])



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5587) Add support for resource profiles

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633746#comment-15633746
 ] 

Wangda Tan commented on YARN-5587:
--

Thanks for update, [~vvasudev].

I will let [~asuresh] finish reviews of the AMRMClient related implementations, 
the approach and other part of the patch looks fine.

> Add support for resource profiles
> -
>
> Key: YARN-5587
> URL: https://issues.apache.org/jira/browse/YARN-5587
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>  Labels: oct16-hard
> Attachments: YARN-5587-YARN-3926.001.patch, 
> YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, 
> YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, 
> YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, 
> YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch
>
>
> Add support for resource profiles on the RM side to allow users to use 
> shorthands to specify resource requirements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5820) yarn node CLI help should be clearer

[Grant Sohn (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633743#comment-15633743
 ] 

Grant Sohn commented on YARN-5820:
--

There is extra whitespace, otherwise it's ok.

{noformat}
node [-list [-states | -showDetails | -all] | -status ]
{noformat}

should be:
{noformat}
node [-list [-states |-showDetails|-all] |-status ]
{noformat}

> yarn node CLI help should be clearer
> 
>
> Key: YARN-5820
> URL: https://issues.apache.org/jira/browse/YARN-5820
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.6.0
>Reporter: Grant Sohn
>Assignee: Ajith S
>Priority: Trivial
> Attachments: YARN-5820.01.patch
>
>
> Current message is:
> {noformat}
> usage: node
>  -all   Works with -list to list all nodes.
>  -list  List all running nodes. Supports optional use of
> -states to filter nodes based on node state, all -all
> to list all nodes.
>  -statesWorks with -list to filter nodes based on input
> comma-separated list of node states.
>  -statusPrints the status report of the node.
> {noformat}
> It should be either this:
> {noformat}
> usage: yarn node [-list [-states |-all] | -status ]
>  -all   Works with -list to list all nodes.
>  -list  List all running nodes. Supports optional use of
> -states to filter nodes based on node state, all -all
> to list all nodes.
>  -statesWorks with -list to filter nodes based on input
> comma-separated list of node states.
>  -statusPrints the status report of the node.
> {noformat}
> or that.
> {noformat}
> usage: yarn node -list [-states |-all] 
>yarn node -status 
>  -all   Works with -list to list all nodes.
>  -list  List all running nodes. Supports optional use of
> -states to filter nodes based on node state, all -all
> to list all nodes.
>  -statesWorks with -list to filter nodes based on input
> comma-separated list of node states.
>  -statusPrints the status report of the node.
> {noformat}
> The latter is the least ambiguous.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2009) CapacityScheduler: Add intra-queue preemption for app priority support

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633729#comment-15633729
 ] 

Wangda Tan commented on YARN-2009:
--

[~eepayne], I'm OK with backport all required patches including 
YARN-4108/YARN-4822 and even YARN-4390.

However, there's an ongoing discussion on community about is it better to 
re-branch branch-2.8 from branch-2, with that, you don't need all these 
backports, will it be an option to you? +[~jlowe].

Thanks,
Wangda

> CapacityScheduler: Add intra-queue preemption for app priority support
> --
>
> Key: YARN-2009
> URL: https://issues.apache.org/jira/browse/YARN-2009
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler
>Reporter: Devaraj K
>Assignee: Sunil G
>  Labels: oct16-medium
> Fix For: 2.9.0
>
> Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch, 
> YARN-2009.0003.patch, YARN-2009.0004.patch, YARN-2009.0005.patch, 
> YARN-2009.0006.patch, YARN-2009.0007.patch, YARN-2009.0008.patch, 
> YARN-2009.0009.patch, YARN-2009.0010.patch, YARN-2009.0011.patch, 
> YARN-2009.0012.patch, YARN-2009.0013.patch, YARN-2009.0014.patch, 
> YARN-2009.0015.patch, YARN-2009.0016.patch
>
>
> While preempting containers based on the queue ideal assignment, we may need 
> to consider preempting the low priority application containers first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

[Inigo Goiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Inigo Goiri updated YARN-5356:
--
Attachment: YARN-5356.009.patch

> NodeManager should communicate physical resource capability to ResourceManager
> --
>
> Key: YARN-5356
> URL: https://issues.apache.org/jira/browse/YARN-5356
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Nathan Roberts
>Assignee: Inigo Goiri
>  Labels: oct16-medium
> Attachments: YARN-5356.000.patch, YARN-5356.001.patch, 
> YARN-5356.002.patch, YARN-5356.002.patch, YARN-5356.003.patch, 
> YARN-5356.004.patch, YARN-5356.005.patch, YARN-5356.006.patch, 
> YARN-5356.007.patch, YARN-5356.008.patch, YARN-5356.009.patch
>
>
> Currently ResourceUtilization contains absolute quantities of resource used 
> (e.g. 4096MB memory used). It would be good if the NM also communicated the 
> actual physical resource capabilities of the node so that the RM can use this 
> data to schedule more effectively (overcommit, etc)
> Currently the only available information is the Resource the node registered 
> with (or later updated using updateNodeResource). However, these aren't 
> really sufficient to get a good view of how utilized a resource is. For 
> example, if a node reports 400% CPU utilization, does that mean it's 
> completely full, or barely utilized? Today there is no reliable way to figure 
> this out.
> [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you 
> have thoughts/opinions on this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5587) Add support for resource profiles

[Varun Vasudev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-5587:

Attachment: YARN-5587-YARN-3926.009.patch

Uploaded a new patch to fix the failing test cases and the javadoc issues. 
[~asuresh], [~leftnoteasy] - can you you please review? Thanks!

> Add support for resource profiles
> -
>
> Key: YARN-5587
> URL: https://issues.apache.org/jira/browse/YARN-5587
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>  Labels: oct16-hard
> Attachments: YARN-5587-YARN-3926.001.patch, 
> YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, 
> YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, 
> YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, 
> YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch
>
>
> Add support for resource profiles on the RM side to allow users to use 
> shorthands to specify resource requirements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3359) Recover collector list in RM failed over

[Li Lu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633666#comment-15633666
 ] 

Li Lu commented on YARN-3359:
-

I cannot reproduce those two test failures and they appears to be unrelated. 
Maybe we'd like to put this in before the rebase? 

> Recover collector list in RM failed over
> 
>
> Key: YARN-3359
> URL: https://issues.apache.org/jira/browse/YARN-3359
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Junping Du
>Assignee: Li Lu
>  Labels: YARN-5355, oct16-medium
> Attachments: YARN-3359-YARN-5355.001.patch, 
> YARN-3359-YARN-5355.002.patch, YARN-3359-YARN-5638.patch
>
>
> Per discussion in YARN-3039, split the recover work from RMStateStore in a 
> separated JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5827) DockerLinuxContainer can't limit the cpu usage

[Vinod Kumar Vavilapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633663#comment-15633663
 ] 

Vinod Kumar Vavilapalli commented on YARN-5827:
---

Cgroup support is already added via YARN-4553. Can this be closed as dup?

> DockerLinuxContainer can't limit the cpu usage
> --
>
> Key: YARN-5827
> URL: https://issues.apache.org/jira/browse/YARN-5827
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.8.0
> Environment: CentOS7.1
>Reporter: zhengchenyu
>   Original Estimate: 240h
>  Remaining Estimate: 240h
>
> When I use DockerLinuxContainer, I found that the function of limiting cpu 
> usage doesn't make effect. Because DockerLinuxContainer regard the exitCode 
> of  "docker inpsect" as pid. But When I got this pid, the command of 
> "launch.sh" has started, then we can't add the child process or threads into 
> the cgroup, so the restrictions of cpu usage doesn't make effect。



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5827) DockerLinuxContainer can't limit the cpu usage

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated YARN-5827:
--
Fix Version/s: (was: 2.8.0)

> DockerLinuxContainer can't limit the cpu usage
> --
>
> Key: YARN-5827
> URL: https://issues.apache.org/jira/browse/YARN-5827
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.8.0
> Environment: CentOS7.1
>Reporter: zhengchenyu
>   Original Estimate: 240h
>  Remaining Estimate: 240h
>
> When I use DockerLinuxContainer, I found that the function of limiting cpu 
> usage doesn't make effect. Because DockerLinuxContainer regard the exitCode 
> of  "docker inpsect" as pid. But When I got this pid, the command of 
> "launch.sh" has started, then we can't add the child process or threads into 
> the cgroup, so the restrictions of cpu usage doesn't make effect。



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2009) CapacityScheduler: Add intra-queue preemption for app priority support

[Vinod Kumar Vavilapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633601#comment-15633601
 ] 

Vinod Kumar Vavilapalli commented on YARN-2009:
---

Great persistent effort here, kudos [~sunilg], [~eepayne] and [~leftnoteasy]!

> CapacityScheduler: Add intra-queue preemption for app priority support
> --
>
> Key: YARN-2009
> URL: https://issues.apache.org/jira/browse/YARN-2009
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler
>Reporter: Devaraj K
>Assignee: Sunil G
>  Labels: oct16-medium
> Fix For: 2.9.0
>
> Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch, 
> YARN-2009.0003.patch, YARN-2009.0004.patch, YARN-2009.0005.patch, 
> YARN-2009.0006.patch, YARN-2009.0007.patch, YARN-2009.0008.patch, 
> YARN-2009.0009.patch, YARN-2009.0010.patch, YARN-2009.0011.patch, 
> YARN-2009.0012.patch, YARN-2009.0013.patch, YARN-2009.0014.patch, 
> YARN-2009.0015.patch, YARN-2009.0016.patch
>
>
> While preempting containers based on the queue ideal assignment, we may need 
> to consider preempting the low priority application containers first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5814) Add druid as storage backend in YARN Timeline Service

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633581#comment-15633581
 ] 

Junping Du commented on YARN-5814:
--

Thanks [~sjlee0] for positive feedback. Current ATS v2 weekly call doesn't work 
for him in Beijing (mid-night), I can help to corodinate some calls for 
initiative efforts and design as I know guys from both side (I saw their demo 
in my recent visit to Beijing). 

>  Add druid as storage backend in YARN Timeline Service
> --
>
> Key: YARN-5814
> URL: https://issues.apache.org/jira/browse/YARN-5814
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Affects Versions: 3.0.0-alpha2
>Reporter: Bingxue Qiu
>
> h3. Introduction
> I propose to add druid as storage backend in YARN Timeline Service.
> We run more than 6000 applications and generate 450 million metrics daily in 
> Alibaba Clusters with thousands of nodes. We need to collect and store 
> meta/events/metrics data, online analyze the utilization reports of various 
> dimensions and display the trends of allocation/usage resources for cluster 
> by joining and aggregating data. It helps us to manage and optimize the 
> cluster by tracking resource utilization.
> To achieve our goal we have changed to use druid as the storage instead of 
> HBase and have achieved sub-second OLAP performance in our production 
> environment for few months. 
> h3. Analysis
> Currently YARN Timeline Service only supports aggregating metrics at a) flow 
> level by FlowRunCoprocessor and b) application level metrics aggregating by 
> AppLevelTimelineCollector, offline (time-based periodic) aggregation for 
> flows/users/queues for reporting and analysis is planned but not yet 
> implemented. YARN Timeline Service chooses Apache HBase as the primary 
> storage backend. As we all know that HBase doesn't fit for OLAP.
>  For arbitrary exploration of data,such as online analyze the utilization 
> reports of various dimensions(Queue,Flow,Users,Application,CPU,Memory) by 
> joining and aggregating data, Druid's custom column format enables ad-hoc 
> queries without pre-computation. The format also enables fast scans on 
> columns, which is important for good aggregation performance.
> To achieve our goal that support to online analyze the utilization reports of 
> various dimensions, display the variation trends of allocation/usage 
> resources for cluster, and arbitrary exploration of data, we propose to add 
> druid storage and implement DruidWriter /DruidReader in YARN Timeline Service.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633586#comment-15633586
 ] 

Hadoop QA commented on YARN-5356:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
48s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
16s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
20s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. 
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m 
20s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red}  3m 20s{color} | 
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  3m 20s{color} 
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 29s{color} | {color:orange} root: The patch generated 4 new + 161 unchanged 
- 3 fixed = 165 total (was 164) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
20s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. 
{color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
20s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. 
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 17s{color} 
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
31s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 22s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 41m 
26s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
0s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 28s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem 

[jira] [Commented] (YARN-2009) CapacityScheduler: Add intra-queue preemption for app priority support

[Eric Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633578#comment-15633578
 ] 

Eric Payne commented on YARN-2009:
--

Hi [~leftnoteasy] and [~sunilg]. I would like this feature to be backported to 
branch-2.8. In order for this to happen, we need to backport the prerequisites, 
YARN-4108 and YARN-4822. I have branch-2.8 patches for both of them. Can I 
backport these?

Would you like me to post the patches on the respective JIRAs or should I just 
backport them directly? There were a few differences, but nothing major.

> CapacityScheduler: Add intra-queue preemption for app priority support
> --
>
> Key: YARN-2009
> URL: https://issues.apache.org/jira/browse/YARN-2009
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler
>Reporter: Devaraj K
>Assignee: Sunil G
>  Labels: oct16-medium
> Fix For: 2.9.0
>
> Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch, 
> YARN-2009.0003.patch, YARN-2009.0004.patch, YARN-2009.0005.patch, 
> YARN-2009.0006.patch, YARN-2009.0007.patch, YARN-2009.0008.patch, 
> YARN-2009.0009.patch, YARN-2009.0010.patch, YARN-2009.0011.patch, 
> YARN-2009.0012.patch, YARN-2009.0013.patch, YARN-2009.0014.patch, 
> YARN-2009.0015.patch, YARN-2009.0016.patch
>
>
> While preempting containers based on the queue ideal assignment, we may need 
> to consider preempting the low priority application containers first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5802) updateApplicationPriority api in scheduler should ensure to re-insert app to correct ordering policy

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-5802:
--
Summary: updateApplicationPriority api in scheduler should ensure to 
re-insert app to correct ordering policy  (was: Application priority updates 
add pending apps to running ordering policy)

> updateApplicationPriority api in scheduler should ensure to re-insert app to 
> correct ordering policy
> 
>
> Key: YARN-5802
> URL: https://issues.apache.org/jira/browse/YARN-5802
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-5802.0001.patch, YARN-5802.0002.patch, 
> YARN-5802.0003.patch, YARN-5802.0004.patch, YARN-5802.0005.patch, 
> YARN-5802.0006.patch
>
>
> {{LeafQueue#updateApplicationPriority}}
> {code}
>  getOrderingPolicy().removeSchedulableEntity(attempt);
>   // Update new priority in SchedulerApplication
>   attempt.setPriority(newAppPriority);
>   getOrderingPolicy().addSchedulableEntity(attempt);
> {code}
> We should add again to ordering policy only when  attempt available in first 
> case.Else during application attempt removal will try to iterate on killed 
> application still available in pending Ordering policy.Which can cause RM to 
> crash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5822) Log ContainerRuntime initialization error in LinuxContainerExecutor

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633530#comment-15633530
 ] 

Hudson commented on YARN-5822:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10763 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10763/])
YARN-5822. Log ContainerRuntime initialization error in (vvasudev: rev 
9ee0e3172ed20b951b458cbbdc4799675f2a2a51)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java


> Log ContainerRuntime initialization error in LinuxContainerExecutor 
> 
>
> Key: YARN-5822
> URL: https://issues.apache.org/jira/browse/YARN-5822
> Project: Hadoop YARN
>  Issue Type: Task
>  Components: nodemanager
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Trivial
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5822.001.patch
>
>
> LinuxContainerExecutor does not log information corresponding to 
> ContainerRuntime initialization failure. This makes it hard to identify the 
> root cause for Nodemanager start failure. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5265) Make HBase configuration for the timeline service configurable

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633517#comment-15633517
 ] 

Sangjin Lee commented on YARN-5265:
---

The latest patch LGTM. I don't see the test failures from the timelineservice 
modules. I'll commit it shortly.

> Make HBase configuration for the timeline service configurable
> --
>
> Key: YARN-5265
> URL: https://issues.apache.org/jira/browse/YARN-5265
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Joep Rottinghuis
>Assignee: Joep Rottinghuis
>  Labels: YARN-5355, oct16-medium
> Attachments: ATS v2 cluster deployment v1.png, 
> YARN-5265-YARN-2928.01.patch, YARN-5265-YARN-2928.02.patch, 
> YARN-5265-YARN-2928.03.patch, YARN-5265-YARN-2928.04.patch, 
> YARN-5265-YARN-2928.05.patch, YARN-5265-YARN-5355.06.patch, 
> YARN-5265-YARN-5355.07.patch, YARN-5265-YARN-5355.08.patch, 
> YARN-5265-YARN-5355.09.patch, YARN-5265-YARN-5355.10.patch, 
> YARN-5265-YARN-5355.11.patch
>
>
> Currently we create "default" HBase configurations, this works as long as the 
> user places the appropriate configuration on the classpath.
> This works fine for a standalone Hadoop cluster.
> However, if a user wants to monitor an HBase cluster and has a separate ATS 
> HBase cluster, then it can become tricky to create the right classpath for 
> the nodemanagers and still have tasks have their separate configs.
> It will be much easier to add a yarn configuration to let cluster admins 
> configure which HBase to connect to to write ATS metrics to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5265) Make HBase configuration for the timeline service configurable

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633477#comment-15633477
 ] 

Hadoop QA commented on YARN-5265:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  3m 
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
33s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
18s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
38s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
50s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
30s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
31s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
31s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
52s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m  
8s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5265 |
| JIRA Patch URL | 

[jira] [Commented] (YARN-5822) Log ContainerRuntime initialization error in LinuxContainerExecutor

[Varun Vasudev (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633443#comment-15633443
 ] 

Varun Vasudev commented on YARN-5822:
-

+1 - pretty straight forward patch.

> Log ContainerRuntime initialization error in LinuxContainerExecutor 
> 
>
> Key: YARN-5822
> URL: https://issues.apache.org/jira/browse/YARN-5822
> Project: Hadoop YARN
>  Issue Type: Task
>  Components: nodemanager
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Trivial
> Attachments: YARN-5822.001.patch
>
>
> LinuxContainerExecutor does not log information corresponding to 
> ContainerRuntime initialization failure. This makes it hard to identify the 
> root cause for Nodemanager start failure. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5814) Add druid as storage backend in YARN Timeline Service

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633432#comment-15633432
 ] 

Sangjin Lee commented on YARN-5814:
---

Thanks [~BINGXUE QIU] for your proposal!

I am certainly +1 on the proposal. It would give people more choice in terms of 
what they want to use as their storage backend. The HBase backend is chosen 
primarily for its scalability, and the basic data is more about OLTP than OLAP. 
The OLAP side of things would be implemented via Phoenix offline aggregation.

We tried our best to be pluggable when it comes to swapping storage backends, 
but no doubt there are places where the HBase details leak. We would need to 
address them together. As Junping mentioned, I also look forward to a more 
detailed proposal.

The group that's working closely on the timeline service v2. has weekly status 
calls, but the timing might not work out well for you. Please do reach out to 
me via email to see how we can work more closely.

>  Add druid as storage backend in YARN Timeline Service
> --
>
> Key: YARN-5814
> URL: https://issues.apache.org/jira/browse/YARN-5814
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Affects Versions: 3.0.0-alpha2
>Reporter: Bingxue Qiu
>
> h3. Introduction
> I propose to add druid as storage backend in YARN Timeline Service.
> We run more than 6000 applications and generate 450 million metrics daily in 
> Alibaba Clusters with thousands of nodes. We need to collect and store 
> meta/events/metrics data, online analyze the utilization reports of various 
> dimensions and display the trends of allocation/usage resources for cluster 
> by joining and aggregating data. It helps us to manage and optimize the 
> cluster by tracking resource utilization.
> To achieve our goal we have changed to use druid as the storage instead of 
> HBase and have achieved sub-second OLAP performance in our production 
> environment for few months. 
> h3. Analysis
> Currently YARN Timeline Service only supports aggregating metrics at a) flow 
> level by FlowRunCoprocessor and b) application level metrics aggregating by 
> AppLevelTimelineCollector, offline (time-based periodic) aggregation for 
> flows/users/queues for reporting and analysis is planned but not yet 
> implemented. YARN Timeline Service chooses Apache HBase as the primary 
> storage backend. As we all know that HBase doesn't fit for OLAP.
>  For arbitrary exploration of data,such as online analyze the utilization 
> reports of various dimensions(Queue,Flow,Users,Application,CPU,Memory) by 
> joining and aggregating data, Druid's custom column format enables ad-hoc 
> queries without pre-computation. The format also enables fast scans on 
> columns, which is important for good aggregation performance.
> To achieve our goal that support to online analyze the utilization reports of 
> various dimensions, display the variation trends of allocation/usage 
> resources for cluster, and arbitrary exploration of data, we propose to add 
> druid storage and implement DruidWriter /DruidReader in YARN Timeline Service.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5814) Add druid as storage backend in YARN Timeline Service

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633315#comment-15633315
 ] 

Junping Du commented on YARN-5814:
--

Thanks [~BINGXUE QIU] for reporting this issue. 
I think this use case and implementation from Alibaba could benefit our 
community for several reasons:
1. It will show case our ATS v2 design and implementation are flexible to 
different storage backend due to different use cases. It can be NoSQL (HBase), 
filesystem (HDFS, guys from NTT seems to work on it) and of course, some OLAP 
implementations.
2. Our current backend implementation on HBase is lacking of ad-hoc query on 
timeline info. In our previous assumption for accessing these timeline info in 
limited ways - like getting runtime or offline aggregation info from UI, it 
won't be a problem. However, if we would like to support the case of 
interactive queries for timeline info on a large and busy cluster, HBase may 
not be the best fit. I believe there could be other YARN users than Alibaba to 
have similar requirements if we are thinking analysis of yarn application info 
is really a big data problem, and the proposed effort can expand our ATS v2 
scenario.
I think we should consider to merge this proposal to our ATS v2 ongoing effort 
(may be under YARN-5355?) if [~BINGXUE QIU] can work out a more concrete design.
ATS v2 folks ([~sjlee0], [~vinodkv], [~gtCarrera9], [~vrushalic], 
[~jrottinghuis], [~varun_saxena], [~Naganarasimha] and [~rohithsharma]), what 
do you guys think?


>  Add druid as storage backend in YARN Timeline Service
> --
>
> Key: YARN-5814
> URL: https://issues.apache.org/jira/browse/YARN-5814
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Affects Versions: 3.0.0-alpha2
>Reporter: Bingxue Qiu
>
> h3. Introduction
> I propose to add druid as storage backend in YARN Timeline Service.
> We run more than 6000 applications and generate 450 million metrics daily in 
> Alibaba Clusters with thousands of nodes. We need to collect and store 
> meta/events/metrics data, online analyze the utilization reports of various 
> dimensions and display the trends of allocation/usage resources for cluster 
> by joining and aggregating data. It helps us to manage and optimize the 
> cluster by tracking resource utilization.
> To achieve our goal we have changed to use druid as the storage instead of 
> HBase and have achieved sub-second OLAP performance in our production 
> environment for few months. 
> h3. Analysis
> Currently YARN Timeline Service only supports aggregating metrics at a) flow 
> level by FlowRunCoprocessor and b) application level metrics aggregating by 
> AppLevelTimelineCollector, offline (time-based periodic) aggregation for 
> flows/users/queues for reporting and analysis is planned but not yet 
> implemented. YARN Timeline Service chooses Apache HBase as the primary 
> storage backend. As we all know that HBase doesn't fit for OLAP.
>  For arbitrary exploration of data,such as online analyze the utilization 
> reports of various dimensions(Queue,Flow,Users,Application,CPU,Memory) by 
> joining and aggregating data, Druid's custom column format enables ad-hoc 
> queries without pre-computation. The format also enables fast scans on 
> columns, which is important for good aggregation performance.
> To achieve our goal that support to online analyze the utilization reports of 
> various dimensions, display the variation trends of allocation/usage 
> resources for cluster, and arbitrary exploration of data, we propose to add 
> druid storage and implement DruidWriter /DruidReader in YARN Timeline Service.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5765) LinuxContainerExecutor creates appcache and its subdirectories with wrong group owner.

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated YARN-5765:
-
Description: 
LinuxContainerExecutor creates usercache/\{userId\}/appcache/\{appId\} with 
wrong group owner, causing Log aggregation and ShuffleHandler to fail because 
node manager process does not have permission to read the files under the 
directory.
This can be easily reproduced by enabling LCE and submitting a MR example job 
as a user that does not belong to the same group that NM process belongs to. 

  was:
LinuxContainerExecutor creates usercache/\{userId\}/appcache/\{appId\} with 
wrong group owner, causing Log aggregation and ShuffleHandler to fail because 
node manager process does not have permission to read the files under the 
directory.
This can be easily reproduced by enabling LCE and submitting a MR example job. 


> LinuxContainerExecutor creates appcache and its subdirectories with wrong 
> group owner.
> --
>
> Key: YARN-5765
> URL: https://issues.apache.org/jira/browse/YARN-5765
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Naganarasimha G R
>Priority: Blocker
>
> LinuxContainerExecutor creates usercache/\{userId\}/appcache/\{appId\} with 
> wrong group owner, causing Log aggregation and ShuffleHandler to fail because 
> node manager process does not have permission to read the files under the 
> directory.
> This can be easily reproduced by enabling LCE and submitting a MR example job 
> as a user that does not belong to the same group that NM process belongs to. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5265) Make HBase configuration for the timeline service configurable

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633271#comment-15633271
 ] 

Joep Rottinghuis commented on YARN-5265:


I'm still looking at local test failures that are probably unrelated:
{noformat}
java.lang.NoSuchMethodError: 
org.apache.hadoop.http.HttpServer2.getWebAppContext()Lorg/mortbay/jetty/webapp/WebAppContext;
{noformat}

> Make HBase configuration for the timeline service configurable
> --
>
> Key: YARN-5265
> URL: https://issues.apache.org/jira/browse/YARN-5265
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Joep Rottinghuis
>Assignee: Joep Rottinghuis
>  Labels: YARN-5355, oct16-medium
> Attachments: ATS v2 cluster deployment v1.png, 
> YARN-5265-YARN-2928.01.patch, YARN-5265-YARN-2928.02.patch, 
> YARN-5265-YARN-2928.03.patch, YARN-5265-YARN-2928.04.patch, 
> YARN-5265-YARN-2928.05.patch, YARN-5265-YARN-5355.06.patch, 
> YARN-5265-YARN-5355.07.patch, YARN-5265-YARN-5355.08.patch, 
> YARN-5265-YARN-5355.09.patch, YARN-5265-YARN-5355.10.patch, 
> YARN-5265-YARN-5355.11.patch
>
>
> Currently we create "default" HBase configurations, this works as long as the 
> user places the appropriate configuration on the classpath.
> This works fine for a standalone Hadoop cluster.
> However, if a user wants to monitor an HBase cluster and has a separate ATS 
> HBase cluster, then it can become tricky to create the right classpath for 
> the nodemanagers and still have tasks have their separate configs.
> It will be much easier to add a yarn configuration to let cluster admins 
> configure which HBase to connect to to write ATS metrics to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5265) Make HBase configuration for the timeline service configurable

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated YARN-5265:
---
Attachment: YARN-5265-YARN-5355.11.patch

Thanks [~sjlee0] for the suggestions. Uploading YARN-5265-YARN-5355.11.patch to 
address them.

> Make HBase configuration for the timeline service configurable
> --
>
> Key: YARN-5265
> URL: https://issues.apache.org/jira/browse/YARN-5265
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Joep Rottinghuis
>Assignee: Joep Rottinghuis
>  Labels: YARN-5355, oct16-medium
> Attachments: ATS v2 cluster deployment v1.png, 
> YARN-5265-YARN-2928.01.patch, YARN-5265-YARN-2928.02.patch, 
> YARN-5265-YARN-2928.03.patch, YARN-5265-YARN-2928.04.patch, 
> YARN-5265-YARN-2928.05.patch, YARN-5265-YARN-5355.06.patch, 
> YARN-5265-YARN-5355.07.patch, YARN-5265-YARN-5355.08.patch, 
> YARN-5265-YARN-5355.09.patch, YARN-5265-YARN-5355.10.patch, 
> YARN-5265-YARN-5355.11.patch
>
>
> Currently we create "default" HBase configurations, this works as long as the 
> user places the appropriate configuration on the classpath.
> This works fine for a standalone Hadoop cluster.
> However, if a user wants to monitor an HBase cluster and has a separate ATS 
> HBase cluster, then it can become tricky to create the right classpath for 
> the nodemanagers and still have tasks have their separate configs.
> It will be much easier to add a yarn configuration to let cluster admins 
> configure which HBase to connect to to write ATS metrics to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

[Inigo Goiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Inigo Goiri updated YARN-5356:
--
Attachment: YARN-5356.008.patch

> NodeManager should communicate physical resource capability to ResourceManager
> --
>
> Key: YARN-5356
> URL: https://issues.apache.org/jira/browse/YARN-5356
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Nathan Roberts
>Assignee: Inigo Goiri
>  Labels: oct16-medium
> Attachments: YARN-5356.000.patch, YARN-5356.001.patch, 
> YARN-5356.002.patch, YARN-5356.002.patch, YARN-5356.003.patch, 
> YARN-5356.004.patch, YARN-5356.005.patch, YARN-5356.006.patch, 
> YARN-5356.007.patch, YARN-5356.008.patch
>
>
> Currently ResourceUtilization contains absolute quantities of resource used 
> (e.g. 4096MB memory used). It would be good if the NM also communicated the 
> actual physical resource capabilities of the node so that the RM can use this 
> data to schedule more effectively (overcommit, etc)
> Currently the only available information is the Resource the node registered 
> with (or later updated using updateNodeResource). However, these aren't 
> really sufficient to get a good view of how utilized a resource is. For 
> example, if a node reports 400% CPU utilization, does that mean it's 
> completely full, or barely utilized? Today there is no reliable way to figure 
> this out.
> [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you 
> have thoughts/opinions on this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

[Inigo Goiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Inigo Goiri updated YARN-5356:
--
Attachment: (was: YARN-5356.008.patch)

> NodeManager should communicate physical resource capability to ResourceManager
> --
>
> Key: YARN-5356
> URL: https://issues.apache.org/jira/browse/YARN-5356
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Nathan Roberts
>Assignee: Inigo Goiri
>  Labels: oct16-medium
> Attachments: YARN-5356.000.patch, YARN-5356.001.patch, 
> YARN-5356.002.patch, YARN-5356.002.patch, YARN-5356.003.patch, 
> YARN-5356.004.patch, YARN-5356.005.patch, YARN-5356.006.patch, 
> YARN-5356.007.patch, YARN-5356.008.patch
>
>
> Currently ResourceUtilization contains absolute quantities of resource used 
> (e.g. 4096MB memory used). It would be good if the NM also communicated the 
> actual physical resource capabilities of the node so that the RM can use this 
> data to schedule more effectively (overcommit, etc)
> Currently the only available information is the Resource the node registered 
> with (or later updated using updateNodeResource). However, these aren't 
> really sufficient to get a good view of how utilized a resource is. For 
> example, if a node reports 400% CPU utilization, does that mean it's 
> completely full, or barely utilized? Today there is no reliable way to figure 
> this out.
> [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you 
> have thoughts/opinions on this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5585) [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633179#comment-15633179
 ] 

Varun Saxena commented on YARN-5585:


Thanks [~rohithsharma] for the patch. Will touch upon the areas which have been 
completed.

# In GenericEntityReader#getResults we need to have a PageFilter based on 
TimelineEntityFilters#limit.
# You commented about the GenericEntityReader#getResults method. I agree that 
we can have {{Long}} in EntityRowKey to get the prefix which can then be used 
to set stop row. Current code has a problem as you mentioned. We should have 
now either 2 filters fromPrefix and fromId or a single filter which carries 
both separated by some separator. Right ?
# Should we move calculateTheClosestNextRowKeyForPrefix to TimelineStorageUtils 
?
# Javadoc in TimelineReader should be changed. It currently says entities would 
be sorted by created time which is no longer true.
{code}
   * @return A set of TimelineEntity instances of the given entity
   *type in the given context scope which matches the given predicates
   *ordered by created time, descending. Each entity will only contain the
   *metadata(id, type and created time) plus the given fields to retrieve.
{code}
# Regarding below piece of code in GenericEntityReader#getResults. I think to 
differentiate between pagination and non pagination mode we should merely use 
the new filters we add instead checking the context. These filter(s) should go 
into TimelineEntityFilters (semantically speaking) instead of going in 
TimelineReaderContext i.e. we should not be filling entity ID and prefix in 
context based on fromId + fromPrefix filter.
{code}
if (context.getEntityId() == null) {
 ...
} else { // pagination mode, will scan from given entityPrefix!enitityId
  ...
}
{code}

> [Atsv2] Reader side changes for entity prefix and support for pagination via 
> additional filters
> ---
>
> Key: YARN-5585
> URL: https://issues.apache.org/jira/browse/YARN-5585
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Critical
>  Labels: oct16-hard
> Attachments: 0001-YARN-5585.patch, YARN-5585-YARN-5355.0001.patch, 
> YARN-5585-workaround.patch, YARN-5585.v0.patch
>
>
> TimelineReader REST API's provides lot of filters to retrieve the 
> applications. Along with those, it would be good to add new filter i.e fromId 
> so that entities can be retrieved after the fromId. 
> Current Behavior : Default limit is set to 100. If there are 1000 entities 
> then REST call gives first/last 100 entities. How to retrieve next set of 100 
> entities i.e 101 to 200 OR 900 to 801?
> Example : If applications are stored database, app-1 app-2 ... app-10.
> *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is 
> no way to achieve this. 
> So proposal is to have fromId in the filter like 
> *getApps?limit=5&=app-5* which gives list of apps from app-6 to 
> app-10. 
> Since ATS is targeting large number of entities storage, it is very common 
> use case to get next set of entities using fromId rather than querying all 
> the entites. This is very useful for pagination in web UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

[Inigo Goiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Inigo Goiri updated YARN-5356:
--
Attachment: YARN-5356.008.patch

Setting ResourceCalculatorPlugin consistently for the NM and removing 
read/write lock.

> NodeManager should communicate physical resource capability to ResourceManager
> --
>
> Key: YARN-5356
> URL: https://issues.apache.org/jira/browse/YARN-5356
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Nathan Roberts
>Assignee: Inigo Goiri
>  Labels: oct16-medium
> Attachments: YARN-5356.000.patch, YARN-5356.001.patch, 
> YARN-5356.002.patch, YARN-5356.002.patch, YARN-5356.003.patch, 
> YARN-5356.004.patch, YARN-5356.005.patch, YARN-5356.006.patch, 
> YARN-5356.007.patch, YARN-5356.008.patch
>
>
> Currently ResourceUtilization contains absolute quantities of resource used 
> (e.g. 4096MB memory used). It would be good if the NM also communicated the 
> actual physical resource capabilities of the node so that the RM can use this 
> data to schedule more effectively (overcommit, etc)
> Currently the only available information is the Resource the node registered 
> with (or later updated using updateNodeResource). However, these aren't 
> really sufficient to get a good view of how utilized a resource is. For 
> example, if a node reports 400% CPU utilization, does that mean it's 
> completely full, or barely utilized? Today there is no reliable way to figure 
> this out.
> [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you 
> have thoughts/opinions on this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4852) Resource Manager Ran Out of Memory

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633063#comment-15633063
 ] 

Rohith Sharma K S commented on YARN-4852:
-

[~sharadag] [~slukog] As per analysis in earlier discussions, YARN-4862 was 
identified as reason for OOM. 
Should this JIRA can be resolved since YARN-4862 patch got committed? Does 
YARN-4862 need to backported to Hadoop-2.6?

> Resource Manager Ran Out of Memory
> --
>
> Key: YARN-4852
> URL: https://issues.apache.org/jira/browse/YARN-4852
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Gokul
> Attachments: threadDump.log
>
>
> Resource Manager went out of memory (max heap size: 8 GB, CMS GC) and shut 
> down itself. 
> Heap dump analysis reveals that 1200 instances of RMNodeImpl class hold 86% 
> of memory. When digging  deeper, there are around 0.5 million objects of 
> UpdatedContainerInfo (nodeUpdateQueue inside RMNodeImpl). This in turn 
> contains around 1.7 million objects of YarnProtos$ContainerIdProto, 
> ContainerStatusProto, ApplicationAttemptIdProto, ApplicationIdProto each of 
> which retain around 1 GB heap.
> Back to Back Full GC kept on happening. GC wasn't able to recover any heap 
> and went OOM. JVM dumped the heap before quitting. We analyzed the heap. 
> RM's usual heap usage is around 4 GB but it suddenly spiked to 8 GB in 20 
> mins time and went OOM.
> There are no spike in job submissions, container numbers at the time of issue 
> occurrence. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

[Inigo Goiri (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632983#comment-15632983
 ] 

Inigo Goiri commented on YARN-5356:
---

Thanks [~jlowe] for the review.

Regarding the {{ResourceCalculatorPlugin}}, I'm very tempted to create a method 
{{ResourceCalculatorPlugin#getNMResourceCalculatorPlugin}} to encapsulate the 
code for RCP creation in {{ContainersMonitorImpl}} and make it easier to 
instantiate.

Regarding the locks, the fewer locks the better, as it's just an assignment, 
volatile should be more than enough.

> NodeManager should communicate physical resource capability to ResourceManager
> --
>
> Key: YARN-5356
> URL: https://issues.apache.org/jira/browse/YARN-5356
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Nathan Roberts
>Assignee: Inigo Goiri
>  Labels: oct16-medium
> Attachments: YARN-5356.000.patch, YARN-5356.001.patch, 
> YARN-5356.002.patch, YARN-5356.002.patch, YARN-5356.003.patch, 
> YARN-5356.004.patch, YARN-5356.005.patch, YARN-5356.006.patch, 
> YARN-5356.007.patch
>
>
> Currently ResourceUtilization contains absolute quantities of resource used 
> (e.g. 4096MB memory used). It would be good if the NM also communicated the 
> actual physical resource capabilities of the node so that the RM can use this 
> data to schedule more effectively (overcommit, etc)
> Currently the only available information is the Resource the node registered 
> with (or later updated using updateNodeResource). However, these aren't 
> really sufficient to get a good view of how utilized a resource is. For 
> example, if a node reports 400% CPU utilization, does that mean it's 
> completely full, or barely utilized? Today there is no reliable way to figure 
> this out.
> [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you 
> have thoughts/opinions on this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Resolved] (YARN-5279) Potential Container leak in NM in preemption flow

[Rohith Sharma K S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-5279.
-
  Resolution: Implemented
Target Version/s:   (was: 2.9.0)

As part of YARN-4862, this scenario also has been take care. Closing as 
Implemented. 

> Potential Container leak in NM in preemption flow
> -
>
> Key: YARN-5279
> URL: https://issues.apache.org/jira/browse/YARN-5279
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5279.patch
>
>
> In discussion YARN-4862 
> [comment|https://issues.apache.org/jira/browse/YARN-4862?focusedCommentId=15341538=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15341538],
>  it is observed that there could be a container leak in NodeManager whenever 
> container is preempted from RM
> Basically if NM receives same containerId details in  {{containersToCleanUp}} 
> and {{containersToBeRemovedFromNM}} in the same heartbeat  then container 
> will never-ever removed in NMContext. Rather NM kills the container of 
> containersToCleanup and send back status again to RM. But RM blindly reject 
> the status since RMContainer is already removed and it is null.
> I think whenever RMContainer is null, RMNode should be informed to send 
> {{containersToBeRemovedFromNM}} so that NM will remove from its context.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4862) Handle duplicate completed containers in RMNodeImpl

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632952#comment-15632952
 ] 

Rohith Sharma K S commented on YARN-4862:
-

Thanks [~jlowe] for the reviewing and committing patch :-)

> Handle duplicate completed containers in RMNodeImpl
> ---
>
> Key: YARN-4862
> URL: https://issues.apache.org/jira/browse/YARN-4862
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: 0001-YARN-4862.patch, 0002-YARN-4862.patch, 
> 0003-YARN-4862.patch, YARN-4862-004.patch, YARN-4862-005.patch, 
> YARN-4862-006.patch
>
>
> As per 
> [comment|https://issues.apache.org/jira/browse/YARN-4852?focusedCommentId=15209689=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15209689]
>  from [~sharadag], there should be safe guard for duplicated container status 
> in RMNodeImpl before creating UpdatedContainerInfo. 
> Or else in heavily loaded cluster where event processing is gradually slow, 
> if any duplicated container are sent to RM(may be bug in NM also), there is 
> significant impact that RMNodImpl always create UpdatedContainerInfo for 
> duplicated containers. This result in increase in the heap memory and causes 
> problem like YARN-4852.
> This is an optimization for issue kind YARN-4852



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org