[jira] [Updated] (YARN-6384) Add configuratin to set max cpu usage when strict-resource-usage is false with cgroups

[dengkai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dengkai updated YARN-6384:
--
Attachment: YARN-6384-4.patch

> Add configuratin to set max cpu usage when strict-resource-usage is false 
> with cgroups
> --
>
> Key: YARN-6384
> URL: https://issues.apache.org/jira/browse/YARN-6384
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: dengkai
> Attachments: YARN-6384-0.patch, YARN-6384-1.patch, YARN-6384-2.patch, 
> YARN-6384-3.patch, YARN-6384-4.patch
>
>
> When using cgroups on yarn, if 
> yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage is 
> false, user may get very more cpu time than expected based on the vcores. 
> There should be a upper limit even resource-usage is not strict, just like a 
> percentage which user can get more than promised by vcores. I think it's 
> important in a shared cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7276) Federation Router Web Service fixes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214174#comment-16214174
 ] 

Hadoop QA commented on YARN-7276:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  9m 
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m  9s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 56s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
35s{color} | {color:green} hadoop-yarn-server-router in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 51s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:ca8ddc6 |
| JIRA Issue | YARN-7276 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12893440/YARN-7276.006.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 64fd721b2f20 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 248d9b6 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/18077/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/18077/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: 

[jira] [Updated] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated YARN-4511:
-
Attachment: YARN-4511-YARN-1011.07.patch

Patch update to address comments + added a unit test for SchedulerNode

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7276) Federation Router Web Service fixes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214158#comment-16214158
 ] 

Hadoop QA commented on YARN-7276:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m  
7s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
23s{color} | {color:red} hadoop-yarn-server-router in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 7s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-yarn-server-router in trunk failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red}  0m 
32s{color} | {color:red} branch has errors when building and testing our client 
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
12s{color} | {color:red} hadoop-yarn-server-router in trunk failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
12s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
12s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 12s{color} 
| {color:red} hadoop-yarn-server-router in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
13s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red}  0m  
7s{color} | {color:red} patch has errors when building and testing our client 
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
11s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 11s{color} 
| {color:red} hadoop-yarn-server-router in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  3m 54s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:17213a0 |
| JIRA Issue | YARN-7276 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12893440/YARN-7276.006.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux d9cac05cf5fd 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 248d9b6 |
| Default Java | 1.7.0_151 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-YARN-Build/18076/artifact/patchprocess/branch-mvninstall-root.txt
 |
| compile | 
https://builds.apache.org/job/PreCommit-YARN-Build/18076/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
 |
| mvnsite | 
https://builds.apache.org/job/PreCommit-YARN-Build/18076/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
 |
| findbugs | 

[jira] [Updated] (YARN-7276) Federation Router Web Service fixes

[JIRA]

 [ 
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276.006.patch

> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch, 
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch, 
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch, 
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch, 
> YARN-7276.006.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the 
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7276) Federation Router Web Service fixes

[JIRA]

 [ 
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.002.patch

> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch, 
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch, 
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch, 
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the 
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7276) Federation Router Web Service fixes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214124#comment-16214124
 ] 

Hadoop QA commented on YARN-7276:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
24s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 17s{color} 
| {color:red} hadoop-yarn-server-router in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 12s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router: 
The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
19s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
16s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 19s{color} 
| {color:red} hadoop-yarn-server-router in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 36s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:17213a0 |
| JIRA Issue | YARN-7276 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12893433/YARN-7276-branch-2.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 1fe935826328 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2 / 9bd77a6 |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-YARN-Build/18075/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
 |
| compile | 
https://builds.apache.org/job/PreCommit-YARN-Build/18075/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
 |
| javac | 
https://builds.apache.org/job/PreCommit-YARN-Build/18075/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/18075/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
 |
| mvnsite | 

[jira] [Updated] (YARN-7276) Federation Router Web Service fixes

[JIRA]

 [ 
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.001.patch

> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch, 
> YARN-7276-branch-2.001.patch, YARN-7276.000.patch, YARN-7276.001.patch, 
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch, 
> YARN-7276.005.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the 
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7276) Federation Router Web Service fixes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214103#comment-16214103
 ] 

Hadoop QA commented on YARN-7276:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
52s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
23s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
15s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
14s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 14s{color} 
| {color:red} hadoop-yarn-server-router in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 12s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router: 
The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
16s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
14s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} 
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 14s{color} 
| {color:red} hadoop-yarn-server-router in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m  9s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:17213a0 |
| JIRA Issue | YARN-7276 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12893431/YARN-7276-branch-2.000.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux bae0cb8c7aeb 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2 / 9bd77a6 |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-YARN-Build/18074/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
 |
| compile | 
https://builds.apache.org/job/PreCommit-YARN-Build/18074/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
 |
| javac | 
https://builds.apache.org/job/PreCommit-YARN-Build/18074/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/18074/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
 |
| mvnsite | 

[jira] [Updated] (YARN-7276) Federation Router Web Service fixes

[JIRA]

 [ 
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.000.patch

> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch, YARN-7276.000.patch, 
> YARN-7276.001.patch, YARN-7276.002.patch, YARN-7276.003.patch, 
> YARN-7276.004.patch, YARN-7276.005.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the 
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7102) NM heartbeat stuck when responseId overflows MAX_INT

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214081#comment-16214081
 ] 

Hadoop QA commented on YARN-7102:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 1s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 7s{color} | {color:green} root: The patch generated 0 new + 339 unchanged - 6 
fixed = 339 total (was 345) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 28s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 14s{color} 
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 
 1s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 19s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.TestResourceTrackerService |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue |
| Timed out junit tests | 
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore |
|   | org.apache.hadoop.yarn.sls.TestSLSRunner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:17213a0 |
| JIRA Issue | YARN-7102 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12893421/YARN-7102-branch-2.v9.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 7458790fb87b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2 / 9bd77a6 |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| unit | 

[jira] [Commented] (YARN-7102) NM heartbeat stuck when responseId overflows MAX_INT

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214040#comment-16214040
 ] 

Hadoop QA commented on YARN-7102:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
47s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
10s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 9s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
0s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
45s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
44s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
24s{color} | {color:green} root: The patch generated 0 new + 142 unchanged - 10 
fixed = 142 total (was 152) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 19s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
6s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
34s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m  8s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
|   | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:c2d96dd |
| JIRA Issue | YARN-7102 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12893417/YARN-7102-branch-2.8.v11.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 9b2e3e81d6fd 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2.8 / a83f87e |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/18072/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/18072/testReport/ |
| modules | C: 

[jira] [Updated] (YARN-7102) NM heartbeat stuck when responseId overflows MAX_INT

[Botong Huang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Botong Huang updated YARN-7102:
---
Attachment: YARN-7102-branch-2.v9.patch

Cannot repro any of the branch-2 patch failures... retry yetus

> NM heartbeat stuck when responseId overflows MAX_INT
> 
>
> Key: YARN-7102
> URL: https://issues.apache.org/jira/browse/YARN-7102
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Critical
> Attachments: YARN-7102-branch-2.8.v10.patch, 
> YARN-7102-branch-2.8.v11.patch, YARN-7102-branch-2.8.v9.patch, 
> YARN-7102-branch-2.v9.patch, YARN-7102-branch-2.v9.patch, 
> YARN-7102-branch-2.v9.patch, YARN-7102.v1.patch, YARN-7102.v2.patch, 
> YARN-7102.v3.patch, YARN-7102.v4.patch, YARN-7102.v5.patch, 
> YARN-7102.v6.patch, YARN-7102.v7.patch, YARN-7102.v8.patch, YARN-7102.v9.patch
>
>
> ResponseId overflow problem in NM-RM heartbeat. This is same as AM-RM 
> heartbeat in YARN-6640, please refer to YARN-6640 for details. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7102) NM heartbeat stuck when responseId overflows MAX_INT

[Botong Huang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Botong Huang updated YARN-7102:
---
Attachment: YARN-7102-branch-2.8.v11.patch

> NM heartbeat stuck when responseId overflows MAX_INT
> 
>
> Key: YARN-7102
> URL: https://issues.apache.org/jira/browse/YARN-7102
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Critical
> Attachments: YARN-7102-branch-2.8.v10.patch, 
> YARN-7102-branch-2.8.v11.patch, YARN-7102-branch-2.8.v9.patch, 
> YARN-7102-branch-2.v9.patch, YARN-7102-branch-2.v9.patch, YARN-7102.v1.patch, 
> YARN-7102.v2.patch, YARN-7102.v3.patch, YARN-7102.v4.patch, 
> YARN-7102.v5.patch, YARN-7102.v6.patch, YARN-7102.v7.patch, 
> YARN-7102.v8.patch, YARN-7102.v9.patch
>
>
> ResponseId overflow problem in NM-RM heartbeat. This is same as AM-RM 
> heartbeat in YARN-6640, please refer to YARN-6640 for details. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-7102) NM heartbeat stuck when responseId overflows MAX_INT

[Botong Huang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213737#comment-16213737
 ] 

Botong Huang edited comment on YARN-7102 at 10/21/17 2:57 PM:
--

Thanks [~jlowe] for the double check! Let me update the patches. 


was (Author: botong):
Thanks [~jlowe] for the double check! When I did the cherry-pick for branch-2 
it doesn't have any conflict. I think it is the auto merge that messed the 
annotation up. Somehow the Jenkins still didn't run for branch-2 though...

> NM heartbeat stuck when responseId overflows MAX_INT
> 
>
> Key: YARN-7102
> URL: https://issues.apache.org/jira/browse/YARN-7102
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Critical
> Attachments: YARN-7102-branch-2.8.v10.patch, 
> YARN-7102-branch-2.8.v9.patch, YARN-7102-branch-2.v9.patch, 
> YARN-7102-branch-2.v9.patch, YARN-7102.v1.patch, YARN-7102.v2.patch, 
> YARN-7102.v3.patch, YARN-7102.v4.patch, YARN-7102.v5.patch, 
> YARN-7102.v6.patch, YARN-7102.v7.patch, YARN-7102.v8.patch, YARN-7102.v9.patch
>
>
> ResponseId overflow problem in NM-RM heartbeat. This is same as AM-RM 
> heartbeat in YARN-6640, please refer to YARN-6640 for details. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7343) Add a junit test for ContainerScheduler recovery

[Sampada Dehankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sampada Dehankar updated YARN-7343:
---
Attachment: YARN-7343.001.patch

> Add a junit test for ContainerScheduler recovery
> 
>
> Key: YARN-7343
> URL: https://issues.apache.org/jira/browse/YARN-7343
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: kartheek muthyala
>Assignee: Sampada Dehankar
>Priority: Minor
> Attachments: YARN-7343.001.patch
>
>
> With queuing at NM, Container recovery becomes interesting. Add a junit test 
> for recovering containers in different states. This should test the recovery 
> with the ContainerScheduler class that was introduced for enabling container 
> queuing on contention of resources. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7102) NM heartbeat stuck when responseId overflows MAX_INT

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213825#comment-16213825
 ] 

Hadoop QA commented on YARN-7102:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  3m 
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
 2s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
53s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
58s{color} | {color:green} root: The patch generated 0 new + 339 unchanged - 6 
fixed = 339 total (was 345) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m  3s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 46s{color} 
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
41s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}190m  9s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.TestResourceTrackerService |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue |
|   | hadoop.yarn.sls.nodemanager.TestNMSimulator |
|   | hadoop.yarn.sls.appmaster.TestAMSimulator |
|   | hadoop.yarn.sls.TestReservationSystemInvariants |
| Timed out junit tests | 
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA |
|   | org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore 
|
|   | org.apache.hadoop.yarn.sls.TestSLSRunner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:17213a0 |
| JIRA Issue | YARN-7102 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12893286/YARN-7102-branch-2.v9.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux ae7142939cf8 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Commented] (YARN-7339) LocalityMulticastAMRMProxyPolicy should handle cancel request properly

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213770#comment-16213770
 ] 

Hadoop QA commented on YARN-7339:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 37s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m  6s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
45s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m  6s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:ca8ddc6 |
| JIRA Issue | YARN-7339 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12893399/YARN-7339-v6.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 072c18eb4146 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 248d9b6 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/18071/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/18071/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> LocalityMulticastAMRMProxyPolicy should handle cancel request properly
> --
>
> Key: YARN-7339
> URL: 

[jira] [Commented] (YARN-7376) YARN top ACLs

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213759#comment-16213759
 ] 

Hadoop QA commented on YARN-7376:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 35s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 43s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
45s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m 
20s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m  9s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:ca8ddc6 |
| JIRA Issue | YARN-7376 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12893394/YARN-7376.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  xml  |
| uname | Linux 3bf92bd7770b 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 

[jira] [Created] (YARN-7378) Documentation changes post branch-2 merge

[Varun Saxena (JIRA)]

Varun Saxena created YARN-7378:
--

 Summary: Documentation changes post branch-2 merge
 Key: YARN-7378
 URL: https://issues.apache.org/jira/browse/YARN-7378
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Saxena






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3993) Change to use the AM flag in ContainerContext determine AM container

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3993:
---
Fix Version/s: 2.9.0

> Change to use the AM flag in ContainerContext determine AM container
> 
>
> Key: YARN-3993
> URL: https://issues.apache.org/jira/browse/YARN-3993
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Zhijie Shen
>Assignee: Sunil G
>  Labels: newbie
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: 0001-YARN-3993-YARN-2928.patch, 
> YARN-3993-YARN-2928.0001.patch
>
>
> After YARN-3116, we will have a flag in ContainerContext to determine if the 
> container is AM or not in aux service. We need to change accordingly to make 
> use of this feature instead of depending on container ID.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5102) timeline service build fails with java 8

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5102:
---
Fix Version/s: 2.9.0

> timeline service build fails with java 8
> 
>
> Key: YARN-5102
> URL: https://issues.apache.org/jira/browse/YARN-5102
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Blocker
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5102-YARN-2928.01.patch
>
>
> The build fails with java 8:
> {noformat}
> [WARNING] 
> Dependency convergence error for jdk.tools:jdk.tools:1.8 paths to dependency 
> are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
>   +-org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT
> +-jdk.tools:jdk.tools:1.8
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
>   +-org.apache.hbase:hbase-common:1.0.1
> +-org.apache.hbase:hbase-annotations:1.0.1
>   +-jdk.tools:jdk.tools:1.7
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence 
> failed with message:
> Failed while enforcing releasability the error(s) are [
> Dependency convergence error for jdk.tools:jdk.tools:1.8 paths to dependency 
> are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
>   +-org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT
> +-jdk.tools:jdk.tools:1.8
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
>   +-org.apache.hbase:hbase-common:1.0.1
> +-org.apache.hbase:hbase-annotations:1.0.1
>   +-jdk.tools:jdk.tools:1.7
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3862) Support for fetching specific configs and metrics based on prefixes

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3862:
---
Fix Version/s: 2.9.0

> Support for fetching specific configs and metrics based on prefixes
> ---
>
> Key: YARN-3862
> URL: https://issues.apache.org/jira/browse/YARN-3862
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3862-YARN-2928.wip.01.patch, 
> YARN-3862-YARN-2928.wip.02.patch, YARN-3862-feature-YARN-2928.005.patch, 
> YARN-3862-feature-YARN-2928.04.patch, YARN-3862-feature-YARN-2928.wip.03.patch
>
>
> Currently, we will retrieve all the contents of the field if that field is 
> specified in the query API. In case of configs and metrics, this can become a 
> lot of data even though the user doesn't need it. So we need to provide a way 
> to query only a set of configs or metrics.
> As a comma spearated list of configs/metrics to be returned will be quite 
> cumbersome to specify, we have to support either of the following options :
> # Prefix match
> # Regex
> # Group the configs/metrics and query that group.
> We also need a facility to specify a metric time window to return metrics in 
> a that window. This may be useful in plotting graphs 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3461) Consolidate flow name/version/run defaults

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3461:
---
Fix Version/s: 2.9.0

> Consolidate flow name/version/run defaults
> --
>
> Key: YARN-3461
> URL: https://issues.apache.org/jira/browse/YARN-3461
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Sangjin Lee
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3461-YARN-2928.01.patch, 
> YARN-3461-YARN-2928.02.patch, YARN-3461-YARN-2928.03.patch
>
>
> In YARN-3391, it's not resolved what should be the defaults for flow 
> name/version/run. Let's continue the discussion here and unblock YARN-3391 
> from moving forward.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4711) NM is going down with NPE's due to single thread processing of events by Timeline client

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4711:
---
Fix Version/s: 2.9.0

> NM is going down with NPE's due to single thread processing of events by 
> Timeline client
> 
>
> Key: YARN-4711
> URL: https://issues.apache.org/jira/browse/YARN-4711
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Critical
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: 4711Analysis.txt, YARN-4711-YARN-2928.v1.001.patch, 
> YARN-4711-YARN-2928.v1.002.patch
>
>
> After YARN-3367, while testing the latest 2928 branch came across few NPEs 
> due to which NM is shutting down.
> {code}
> 2016-02-21 23:19:54,078 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: 
> Error in dispatcher thread
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ContainerEventHandler.handle(NMTimelinePublisher.java:306)
> at 
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ContainerEventHandler.handle(NMTimelinePublisher.java:296)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> {code}
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.putEntity(NMTimelinePublisher.java:213)
> at 
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.publishContainerFinishedEvent(NMTimelinePublisher.java:192)
> at 
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.access$400(NMTimelinePublisher.java:63)
> at 
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ApplicationEventHandler.handle(NMTimelinePublisher.java:289)
> at 
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ApplicationEventHandler.handle(NMTimelinePublisher.java:280)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> On analysis found that the there was delay in processing of events, as after 
> YARN-3367 all the events were getting processed by a single thread inside the 
> timeline client. 
> Additionally found one scenario where there is possibility of NPE:
> * TimelineEntity.toString() when {{real}} is not null



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7038) [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-7038:
---
Fix Version/s: 2.9.0

> [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when 
> service authorization is enabled
> --
>
> Key: YARN-7038
> URL: https://issues.apache.org/jira/browse/YARN-7038
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: YARN-5355
>Reporter: Rohith Sharma K S
>Assignee: Varun Saxena
>  Labels: yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-7038-YARN-5355.01.patch, 
> YARN-7038-YARN-5355.02.patch, YARN-7038-YARN-5355.03.patch
>
>
> Below error appears in the log when authorization is enabled.
> {noformat}
> 2017-08-17 11:16:40,664 ERROR collector.NodeTimelineCollectorManager 
> (NodeTimelineCollectorManager.java:doPostPut(227)) - Failed to communicate 
> with NM Collector Service for application_1502964541476_0001
> 2017-08-17 11:16:40,665 WARN  containermanager.AuxServices 
> (AuxServices.java:logWarningWhenAuxServiceThrowExceptions(283)) - The 
> auxService name is timeline_collector and it got an error at event: 
> CONTAINER_INIT
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
> org.apache.hadoop.security.authorize.AuthorizationException: Protocol 
> interface org.apache.hadoop.yarn.server.api.CollectorNodemanagerProtocolPB is 
> not known.
> at 
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.putIfAbsent(TimelineCollectorManager.java:146)
> at 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5109) timestamps are stored unencoded causing parse errors

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5109:
---
Fix Version/s: 2.9.0

> timestamps are stored unencoded causing parse errors
> 
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5109-YARN-2928.003.patch, 
> YARN-5109-YARN-2928.01.patch, YARN-5109-YARN-2928.02.patch, 
> YARN-5109-YARN-2928.03.patch, YARN-5109-YARN-2928.04.patch, 
> YARN-5109-YARN-2928.05.patch, YARN-5109-YARN-2928.06.patch, 
> YARN-5109-YARN-2928.07.patch, YARN-5109-YARN-2928.08.patch
>
>
> When we store timestamps (for example as part of the row key or part of the 
> column name for an event), the bytes are used as is without any encoding. If 
> the byte value happens to contain a separator character we use (e.g. "!" or 
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
>  incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event) 
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event 
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5142) fix findbugs warnings/errors for hadoop-yarn-server-timelineservice-hbase-tests

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5142:
---
Fix Version/s: 2.9.0

> fix findbugs warnings/errors for 
> hadoop-yarn-server-timelineservice-hbase-tests
> ---
>
> Key: YARN-5142
> URL: https://issues.apache.org/jira/browse/YARN-5142
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5142-YARN-2928.01.patch
>
>
> Fix the errors/warnings reported for 
> hadoop-yarn-server-timelineservice-hbase-tests once YARN-5138 is in



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6146) Add Builder methods for TimelineEntityFilters

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6146:
---
Fix Version/s: 2.9.0

> Add Builder methods for TimelineEntityFilters
> -
>
> Key: YARN-6146
> URL: https://issues.apache.org/jira/browse/YARN-6146
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Haibo Chen
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-6146-YARN-5355.01.patch, 
> YARN-6146-YARN-5355.02.patch, YARN-6146-YARN-5355.03.patch, 
> YARN-6146.01.patch, YARN-6146.02.patch, YARN-6146.03.patch
>
>
> The timeline filters are evolving and can be add more and more filters. It is 
> better to start using Builder methods rather than changing constructor every 
> time for adding new filters. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6555) Store application flow context in NM state store for work-preserving restart

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6555:
---
Fix Version/s: 2.9.0

> Store application flow context in NM state store for work-preserving restart
> 
>
> Key: YARN-6555
> URL: https://issues.apache.org/jira/browse/YARN-6555
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4
>Reporter: Vrushali C
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4
>
> Attachments: YARN-6555.001.patch, YARN-6555.002.patch, 
> YARN-6555.003.patch
>
>
> If timeline service v2 is enabled and NM is restarted with recovery enabled, 
> then NM fails to start and throws an error as  "flow context can't be null".
> This is happening because the flow context did not exist before but now that 
> timeline service v2 is enabled, ApplicationImpl expects it to exist. 
> This would also happen even if flow context existed before but since we are 
> not persisting it / reading it during 
> ContainerManagerImpl#recoverApplication, it does not get passed in to 
> ApplicationImpl.
> full stack trace
> {code}
> 2017-05-03 21:51:52,178 FATAL 
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting 
> NodeManager
> java.lang.IllegalArgumentException: flow context cannot be null
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:104)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:90)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverApplication(ContainerManagerImpl.java:318)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:280)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:267)
> at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at 
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at 
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:276)
> at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at 
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:588)
> at 
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:649)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3816) [Aggregation] App-level aggregation and accumulation for YARN system metrics

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3816:
---
Fix Version/s: 2.9.0

> [Aggregation] App-level aggregation and accumulation for YARN system metrics
> 
>
> Key: YARN-3816
> URL: https://issues.apache.org/jira/browse/YARN-3816
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Junping Du
>Assignee: Li Lu
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: Application Level Aggregation of Timeline Data.pdf, 
> YARN-3816-YARN-2928-v1.patch, YARN-3816-YARN-2928-v2.1.patch, 
> YARN-3816-YARN-2928-v2.2.patch, YARN-3816-YARN-2928-v2.3.patch, 
> YARN-3816-YARN-2928-v2.patch, YARN-3816-YARN-2928-v3.1.patch, 
> YARN-3816-YARN-2928-v3.patch, YARN-3816-YARN-2928-v4.patch, 
> YARN-3816-YARN-2928-v5.patch, YARN-3816-YARN-2928-v6.patch, 
> YARN-3816-YARN-2928-v7.patch, YARN-3816-YARN-2928-v8.patch, 
> YARN-3816-YARN-2928-v9.patch, YARN-3816-feature-YARN-2928.v4.1.patch, 
> YARN-3816-poc-v1.patch, YARN-3816-poc-v2.patch
>
>
> We need application level aggregation of Timeline data:
> - To present end user aggregated states for each application, include: 
> resource (CPU, Memory) consumption across all containers, number of 
> containers launched/completed/failed, etc. We need this for apps while they 
> are running as well as when they are done.
> - Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be 
> aggregated to show details of states in framework level.
> - Other level (Flow/User/Queue) aggregation can be more efficient to be based 
> on Application-level aggregations rather than raw entity-level data as much 
> less raws need to scan (with filter out non-aggregated entities, like: 
> events, configurations, etc.).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6134:
---
Fix Version/s: 2.9.0

> [ATSv2 Security] Regenerate delegation token for app just before token 
> expires if app collector is active
> -
>
> Key: YARN-6134
> URL: https://issues.apache.org/jira/browse/YARN-6134
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-6134-YARN-5355.01.patch, 
> YARN-6134-YARN-5355.02.patch, YARN-6134-YARN-5355.03.patch, 
> YARN-6134-YARN-5355.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3125) [Event producers] Change distributed shell to use new timeline service

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3125:
---
Fix Version/s: 2.9.0

> [Event producers] Change distributed shell to use new timeline service
> --
>
> Key: YARN-3125
> URL: https://issues.apache.org/jira/browse/YARN-3125
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Junping Du
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3125.patch, YARN-3125_UT-022615.patch, 
> YARN-3125_UT-022715.patch, YARN-3125v2.patch, YARN-3125v3.patch
>
>
> We can start with changing distributed shell to use new timeline service once 
> the framework is completed, in which way we can quickly verify the next gen 
> is working fine end-to-end.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3901) Populate flow run data in the flow_run & flow activity tables

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3901:
---
Fix Version/s: 2.9.0

> Populate flow run data in the flow_run & flow activity tables
> -
>
> Key: YARN-3901
> URL: https://issues.apache.org/jira/browse/YARN-3901
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3901-YARN-2928.1.patch, 
> YARN-3901-YARN-2928.10.patch, YARN-3901-YARN-2928.2.patch, 
> YARN-3901-YARN-2928.3.patch, YARN-3901-YARN-2928.4.patch, 
> YARN-3901-YARN-2928.5.patch, YARN-3901-YARN-2928.6.patch, 
> YARN-3901-YARN-2928.7.patch, YARN-3901-YARN-2928.8.patch, 
> YARN-3901-YARN-2928.9.patch
>
>
> As per the schema proposed in YARN-3815 in 
> https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf
> filing jira to track creation and population of data in the flow run table. 
> Some points that are being  considered:
> - Stores per flow run information aggregated across applications, flow version
> RM’s collector writes to on app creation and app completion
> - Per App collector writes to it for metric updates at a slower frequency 
> than the metric updates to application table
> primary key: cluster ! user ! flow ! flow run id
> - Only the latest version of flow-level aggregated metrics will be kept, even 
> if the entity and application level keep a timeseries.
> - The running_apps column will be incremented on app creation, and 
> decremented on app completion.
> - For min_start_time the RM writer will simply write a value with the tag for 
> the applicationId. A coprocessor will return the min value of all written 
> values. - 
> - Upon flush and compactions, the min value between all the cells of this 
> column will be written to the cell without any tag (empty tag) and all the 
> other cells will be discarded.
> - Ditto for the max_end_time, but then the max will be kept.
> - Tags are represented as #type:value. The type can be not set (0), or can 
> indicate running (1) or complete (2). In those cases (for metrics) only 
> complete app metrics are collapsed on compaction.
> - The m! values are aggregated (summed) upon read. Only when applications are 
> completed (indicated by tag type 2) can the values be collapsed.
> - The application ids that have completed and been aggregated into the flow 
> numbers are retained in a separate column for historical tracking: we don’t 
> want to re-aggregate for those upon replay
> 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4455) Support fetching metrics by time range

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4455:
---
Fix Version/s: 2.9.0

> Support fetching metrics by time range
> --
>
> Key: YARN-4455
> URL: https://issues.apache.org/jira/browse/YARN-4455
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-5355
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: YARN-5355, yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-4455-YARN-5355.01.patch, 
> YARN-4455-YARN-5355.02.patch, YARN-4455-YARN-5355.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4765) Split TestHBaseTimelineStorage into multiple test classes

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4765:
---
Fix Version/s: 2.9.0

> Split TestHBaseTimelineStorage into multiple test classes
> -
>
> Key: YARN-4765
> URL: https://issues.apache.org/jira/browse/YARN-4765
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: YARN-5355, atsv2-hbase, oct16-medium
> Fix For: 2.9.0, 3.0.0-alpha2, YARN-5355
>
> Attachments: YARN-4765-YARN-5355.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6159) Documentation changes for TimelineV2Client

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6159:
---
Fix Version/s: 2.9.0

> Documentation changes for TimelineV2Client
> --
>
> Key: YARN-6159
> URL: https://issues.apache.org/jira/browse/YARN-6159
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Varun Saxena
>Assignee: Naganarasimha G R
>Priority: Minor
> Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4
>
> Attachments: TimelineServiceV2.html, YARN-6159.v1.001.patch, 
> YARN-6159.v1.002.patch, YARN-6159.v1.003.patch, YARN-6159.v1.004.patch
>
>
> Make documentation changes for TimelineV2Client i.e. to reflect changes made 
> in client API in YARN-4675.
> Also in TimelineServiceV2.md, under section Publishing application specific 
> data, we have the following code snippet. Here, 
> {{timelineClient.putEntitiesAsync(entity);}} should be 
> {{client.putEntitiesAsync(entity);}} instead.
> {code}
> // Create and start the Timeline client v.2
> TimelineClient client = TimelineClient.createTimelineClient(appId);
> client.init(conf);
> client.start();
> try {
>   TimelineEntity myEntity = new TimelineEntity();
>   myEntity.setEntityType("MY_APPLICATION");
>   myEntity.setEntityId("MyApp1")
>   // Compose other entity info
>   // Blocking write
>   client.putEntities(entity);
>   TimelineEntity myEntity2 = new TimelineEntity();
>   // Compose other info
>   // Non-blocking write
>   timelineClient.putEntitiesAsync(entity);
> } catch (IOException e) {
>   // Handle the exception
> } catch (RuntimeException e) {
> {code}
> Below can also be changed to client to keep it consistent.
> {code}
> amRMClient.registerTimelineClient(timelineClient);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3041) [Data Model] create overall data objects of TS next gen

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3041:
---
Fix Version/s: 2.9.0

> [Data Model] create overall data objects of TS next gen
> ---
>
> Key: YARN-3041
> URL: https://issues.apache.org/jira/browse/YARN-3041
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Zhijie Shen
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, 
> YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.5.patch, 
> YARN-3041.preliminary.001.patch
>
>
> Per design in YARN-2928, create the ATS entity and events API.
> Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, 
> flow, flow run, YARN app, ...).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5156) YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5156:
---
Fix Version/s: 2.9.0

> YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state
> -
>
> Key: YARN-5156
> URL: https://issues.apache.org/jira/browse/YARN-5156
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Li Lu
>Assignee: Vrushali C
>  Labels: YARN-5355
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1
>
> Attachments: YARN-5156-YARN-2928.01.patch, 
> YARN-5156-YARN-5355.01.patch, YARN-5156-YARN-5355.02.patch
>
>
> On container finished, we're reporting "YARN_CONTAINER_STATE: "RUNNING"". Do 
> we design this deliberately or it's a bug? 
> {code}
> {
> metrics: [ ],
> events: [
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: 1464213765890,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: 1464213761133,
> info: { }
> },
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: 1464213761132,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: 1464213761132,
> info: { }
> }
> ],
> id: "container_e15_1464213707405_0001_01_18",
> type: "YARN_CONTAINER",
> createdtime: 1464213761132,
> info: {
> YARN_CONTAINER_ALLOCATED_PRIORITY: "20",
> YARN_CONTAINER_ALLOCATED_VCORE: 1,
> YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS: "10.22.16.164:0",
> UID: 
> "yarn_cluster!application_1464213707405_0001!YARN_CONTAINER!container_e15_1464213707405_0001_01_18",
> YARN_CONTAINER_ALLOCATED_HOST: "10.22.16.164",
> YARN_CONTAINER_ALLOCATED_MEMORY: 1024,
> SYSTEM_INFO_PARENT_ENTITY: {
> type: "YARN_APPLICATION_ATTEMPT",
> id: "appattempt_1464213707405_0001_01"
> },
> YARN_CONTAINER_ALLOCATED_PORT: 64694
> },
> configs: { },
> isrelatedto: { },
> relatesto: { }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6237) Move UID constant to TimelineReaderUtils

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6237:
---
Fix Version/s: 2.9.0

> Move UID constant to TimelineReaderUtils
> 
>
> Key: YARN-6237
> URL: https://issues.apache.org/jira/browse/YARN-6237
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: newbie
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-6237-YARN-5355.0001.patch
>
>
> UID constant is kept in TimelineReader Manager. This can be moved to 
> TimelineReaderUtils which can keep track of all reader constants. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4409) Fix javadoc and checkstyle issues in timelineservice code

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4409:
---
Fix Version/s: 2.9.0

> Fix javadoc and checkstyle issues in timelineservice code
> -
>
> Key: YARN-4409
> URL: https://issues.apache.org/jira/browse/YARN-4409
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4409-YARN-2928.01.patch, 
> YARN-4409-YARN-2928.02.patch, YARN-4409-YARN-2928.03.patch
>
>
> There are a large number of javadoc and checkstyle issues currently open in 
> timelineservice code. We need to fix them before we merge it into trunk.
> Refer to 
> https://issues.apache.org/jira/browse/YARN-3862?focusedCommentId=15035267=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15035267
> We still have 94 open checkstyle issues and javadocs failing for Java 8.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3276) Refactor and fix null casting in some map cast for TimelineEntity (old and new) and fix findbug warnings

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3276:
---
Fix Version/s: 2.9.0

> Refactor and fix null casting in some map cast for TimelineEntity (old and 
> new) and fix findbug warnings
> 
>
> Key: YARN-3276
> URL: https://issues.apache.org/jira/browse/YARN-3276
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Junping Du
>Assignee: Junping Du
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3276-YARN-2928.v3.patch, 
> YARN-3276-YARN-2928.v4.patch, YARN-3276-YARN-2928.v5-fix-checkstyle.patch, 
> YARN-3276-YARN-2928.v5.patch, YARN-3276-YARN-2928.v6.patch, 
> YARN-3276-v2.patch, YARN-3276-v3.patch, YARN-3276.patch
>
>
> Per discussion in YARN-3087, we need to refactor some similar logic to cast 
> map to hashmap and get rid of NPE issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4297) TestJobHistoryEventHandler and TestRMContainerAllocator failing on YARN-2928 branch

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4297:
---
Fix Version/s: 2.9.0

> TestJobHistoryEventHandler and TestRMContainerAllocator failing on YARN-2928 
> branch
> ---
>
> Key: YARN-4297
> URL: https://issues.apache.org/jira/browse/YARN-4297
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4297-YARN-2928.01.patch, 
> YARN-4297-feature-YARN-2928.02.patch, YARN-4297-feature-YARN-2928.03.patch
>
>
> {noformat}
> Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 16.09 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
> testTimelineEventHandling(org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler)
>   Time elapsed: 0.11 sec  <<< ERROR!
> java.lang.ClassCastException: 
> org.apache.hadoop.mapreduce.v2.app.AppContext$$EnhancerByMockitoWithCGLIB$$95d3ddbe
>  cannot be cast to 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$RunningAppContext
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceInit(JobHistoryEventHandler.java:271)
>   at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler.testTimelineEventHandling(TestJobHistoryEventHandler.java:495)
> {noformat}
> {noformat}
> testRMContainerAllocatorResendsRequestsOnRMRestart(org.apache.hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator)
>   Time elapsed: 2.649 sec  <<< ERROR!
> java.lang.ClassCastException: 
> org.apache.hadoop.mapreduce.v2.app.AppContext$$EnhancerByMockitoWithCGLIB$$8e08559a
>  cannot be cast to 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$RunningAppContext
>   at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:802)
>   at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:269)
> Tests in error: 
>   TestRMContainerAllocator.testExcessReduceContainerAssign:669 » ClassCast 
> org.a...
>   TestRMContainerAllocator.testReportedAppProgress:970 » NullPointer
>   TestRMContainerAllocator.testBlackListedNodesWithSchedulingToThatNode:1578 
> » ClassCast
>   TestRMContainerAllocator.testBlackListedNodes:1292 » ClassCast 
> org.apache.hado...
>   TestRMContainerAllocator.testAMRMTokenUpdate:2691 » ClassCast 
> org.apache.hadoo...
>   TestRMContainerAllocator.testMapReduceAllocationWithNodeLabelExpression:722 
> » ClassCast
>   TestRMContainerAllocator.testReducerRampdownDiagnostics:443 » ClassCast 
> org.ap...
>   TestRMContainerAllocator.testReportedAppProgressWithOnlyMaps:1118 » 
> NullPointer
>   TestRMContainerAllocator.testMapReduceScheduling:819 » ClassCast 
> org.apache.ha...
>   TestRMContainerAllocator.testResource:390 » ClassCast 
> org.apache.hadoop.mapred...
>   TestRMContainerAllocator.testUpdatedNodes:1190 » ClassCast 
> org.apache.hadoop.m...
>   TestRMContainerAllocator.testCompletedTasksRecalculateSchedule:2249 » 
> ClassCast
>   TestRMContainerAllocator.testConcurrentTaskLimits:2779 » ClassCast 
> org.apache
>   TestRMContainerAllocator.testSimple:219 » ClassCast 
> org.apache.hadoop.mapreduc...
>   
> TestRMContainerAllocator.testIgnoreBlacklisting:1378->getContainerOnHost:1511 
> » ClassCast
>   TestRMContainerAllocator.testMapNodeLocality:310 » ClassCast 
> org.apache.hadoop...
>   
> TestRMContainerAllocator.testRMContainerAllocatorResendsRequestsOnRMRestart:2489
>  » ClassCast
> Tests run: 26, Failures: 0, Errors: 17, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6424) TimelineCollector is not stopped when an app finishes in RM

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6424:
---
Fix Version/s: 2.9.0

> TimelineCollector is not stopped when an app finishes in RM
> ---
>
> Key: YARN-6424
> URL: https://issues.apache.org/jira/browse/YARN-6424
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: 3.0.0-alpha2
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4
>
> Attachments: YARN-6424.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3390) Reuse TimelineCollectorManager for RM

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3390:
---
Fix Version/s: 2.9.0

> Reuse TimelineCollectorManager for RM
> -
>
> Key: YARN-3390
> URL: https://issues.apache.org/jira/browse/YARN-3390
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3390.1.patch, YARN-3390.2.patch, YARN-3390.3.patch, 
> YARN-3390.4.patch
>
>
> RMTimelineCollector should have the context info of each app whose entity  
> has been put



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3908) Bugs in HBaseTimelineWriterImpl

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3908:
---
Fix Version/s: 2.9.0

> Bugs in HBaseTimelineWriterImpl
> ---
>
> Key: YARN-3908
> URL: https://issues.apache.org/jira/browse/YARN-3908
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Vrushali C
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3908-YARN-2928.001.patch, 
> YARN-3908-YARN-2928.002.patch, YARN-3908-YARN-2928.003.patch, 
> YARN-3908-YARN-2928.004.patch, YARN-3908-YARN-2928.004.patch, 
> YARN-3908-YARN-2928.005.patch
>
>
> 1. In HBaseTimelineWriterImpl, the info column family contains the basic 
> fields of a timeline entity plus events. However, entity#info map is not 
> stored at all.
> 2 event#timestamp is also not persisted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5378) Accommodate app-id->cluster mapping

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5378:
---
Fix Version/s: 2.9.0

> Accommodate app-id->cluster mapping
> ---
>
> Key: YARN-5378
> URL: https://issues.apache.org/jira/browse/YARN-5378
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Joep Rottinghuis
>Assignee: Sangjin Lee
>  Labels: yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-5378-YARN-5355.01.patch, 
> YARN-5378-YARN-5355.02.patch, YARN-5378-YARN-5355.03.patch
>
>
> In discussion with [~sjlee0], [~vrushalic], [~subru], and [~curino] a 
> use-case came up to be able to map from application-id to cluster-id in 
> context of federation for Yarn.
> What happens is that a "random" cluster in the federation is asked to 
> generate an app-id and then potentially a different cluster can be the "home" 
> cluster for the AM. Furthermore, tasks can then run in yet other clusters.
> In order to be able to pull up the logical home cluster on which the 
> application ran, there needs to be a mapping from application-id to 
> cluster-id. This mapping is available in the federated Yarn case only during 
> the active live of the application.
> A similar situation is common in our larger production environment. Somebody 
> will complain about a slow job, some failure or whatever. If we're lucky we 
> have an application-id. When we ask the user which cluster they ran on, 
> they'll typically answer with the machine from where they launched the job 
> (many users are unaware of the underlying physical clusters). This leaves us 
> to spelunk through various RM ui's to find a matching epoch in the 
> application ID. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5170) Eliminate singleton converters and static method access

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5170:
---
Fix Version/s: 2.9.0

> Eliminate singleton converters and static method access
> ---
>
> Key: YARN-5170
> URL: https://issues.apache.org/jira/browse/YARN-5170
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Joep Rottinghuis
>Assignee: Joep Rottinghuis
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5170-YARN-2928.01.patch, 
> YARN-5170-YARN-2928.02.patch, YARN-5170-YARN-2928.03.patch, 
> YARN-5170-YARN-2928.04.patch, YARN-5170-YARN-2928.05.patch, 
> YARN-5170-YARN-2928.06.patch, YARN-5170-YARN-2928.07.patch, 
> YARN-5170-YARN-2928.08.patch, YARN-5170-YARN-2928.09.patch, 
> YARN-5170-YARN-2928.10.patch, YARN-5170-YARN-2928.11.patch, 
> YARN-5170-YARN-2928.12.patch, YARN-5170-YARN-2928.13.patch
>
>
> As part of YARN-5109 we introduced several KeyConverter classes.
> To stay consistent with the existing LongConverter in the sample patch I 
> created I made these other converter classes singleton as well.
> In conversation with [~sjlee0] who has a general dislike of singletons, we 
> discussed it is best to get rid of these singletons and make them simply 
> instance variables.
> There are other classes where the keys have static methods referring to a 
> singleton converter.
> Moreover, it turns out that due to code evolution we end up creating the same 
> keys several times.
> So general approach is to not re-instantiate rowkeys, converters when not 
> needed.
> I would like to create the byte[] rowKey in the RowKey classes their 
> constructor, but that would leak an incomplete object to the converter.
> There are a few method in TimelineStorageUtils that are used only once, or 
> only by one class, as part of this refactor I'll move these to keep the 
> "Utils" class as small as possible and keep them for truly generally used 
> utils that don't really belong anywhere else.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4622) TestDistributedShell fails for v2 test cases after modifications for 1.5

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4622:
---
Fix Version/s: 2.9.0

> TestDistributedShell fails for v2 test cases after modifications for 1.5
> 
>
> Key: YARN-4622
> URL: https://issues.apache.org/jira/browse/YARN-4622
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>  Labels: test
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4622-YARN-2928.v1.001.patch
>
>
> TestDistributedShell fails for v2 test cases : 
> *testDSShellWithoutDomainV2DefaultFlow and 
> testDSShellWithoutDomainV2CustomizedFlow* after trunk rebase with 
> modifications for 1.5,
> {code}
> Caused by: 
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): 
> java.lang.NullPointerException
>   at 
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
>   at com.google.common.base.Joiner.toString(Joiner.java:532)
>   at com.google.common.base.Joiner.appendTo(Joiner.java:124)
>   at com.google.common.base.Joiner.appendTo(Joiner.java:181)
>   at com.google.common.base.Joiner.join(Joiner.java:237)
>   at com.google.common.base.Joiner.join(Joiner.java:226)
>   at com.google.common.base.Joiner.join(Joiner.java:253)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.constructResURI(TimelineClientImpl.java:726)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceStart(TimelineClientImpl.java:336)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.createAndStartTimelineClient(ApplicationImpl.java:149)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:113)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainerInternal(ContainerManagerImpl.java:971)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:830)
>   at 
> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:65)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3264) [Storage implementation] Create backing storage write interface and a POC only file based storage implementation

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3264:
---
Fix Version/s: 2.9.0

> [Storage implementation] Create backing storage write interface and  a POC 
> only file based storage implementation
> -
>
> Key: YARN-3264
> URL: https://issues.apache.org/jira/browse/YARN-3264
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3264.001.patch, YARN-3264.002.patch, 
> YARN-3264.003.patch, YARN-3264.004.patch, YARN-3264.005.patch, 
> YARN-3264.006.patch, YARN-3264.007.patch, YARN-3264.008.patch
>
>
> For the PoC, need to create a backend impl for file based storage of entities 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5243) fix several rebase and other miscellaneous issues before merge

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5243:
---
Fix Version/s: 2.9.0

> fix several rebase and other miscellaneous issues before merge
> --
>
> Key: YARN-5243
> URL: https://issues.apache.org/jira/browse/YARN-5243
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5243-YARN-2928.01.patch, 
> YARN-5243-YARN-2928.02.patch, YARN-5243-YARN-2928.03.patch
>
>
> I have come across a couple of miscellaneous issues while inspecting the 
> diffs against the trunk.
> We also need to review one last time (probably after the final rebase) to 
> ensure the timeline services v.2 leaves no impact when disabled.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5052) [Documentation] Update timeline service v2 documentation to capture information about filters

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5052:
---
Fix Version/s: 2.9.0

> [Documentation] Update timeline service v2 documentation to capture 
> information about filters
> -
>
> Key: YARN-5052
> URL: https://issues.apache.org/jira/browse/YARN-5052
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: Apache Hadoop 3.0.0-SNAPSHOT – The YARN Timeline Service 
> v.pdf, Hierarchy.png, The YARN Timeline Service v2.02.pdf, 
> YARN-5052-YARN-2928.01.patch, YARN-5052-YARN-2928.02.patch
>
>
> Since YARN-4447 has gone in, we can update our documentation to capture 
> information about usage of filters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4712) CPU Usage Metric is not captured properly in YARN-2928

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4712:
---
Fix Version/s: 2.9.0

> CPU Usage Metric is not captured properly in YARN-2928
> --
>
> Key: YARN-4712
> URL: https://issues.apache.org/jira/browse/YARN-4712
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4712-YARN-2928.v1.001.patch, 
> YARN-4712-YARN-2928.v1.002.patch, YARN-4712-YARN-2928.v1.003.patch, 
> YARN-4712-YARN-2928.v1.004.patch, YARN-4712-YARN-2928.v1.005.patch, 
> YARN-4712-YARN-2928.v1.006.patch
>
>
> There are 2 issues with CPU usage collection 
> * I was able to observe that that many times CPU usage got from 
> {{pTree.getCpuUsagePercent()}} is 
> ResourceCalculatorProcessTree.UNAVAILABLE(i.e. -1) but ContainersMonitor do 
> the calculation  i.e. {{cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore 
> /resourceCalculatorPlugin.getNumProcessors()}} because of which UNAVAILABLE 
> check in {{NMTimelinePublisher.reportContainerResourceUsage}} is not 
> encountered. so proper checks needs to be handled
> * {{EntityColumnPrefix.METRIC}} uses always LongConverter but 
> ContainerMonitor is publishing decimal values for the CPU usage.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4102) Add a "skip existing table" mode for timeline schema creator

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4102:
---
Fix Version/s: 2.9.0

> Add a "skip existing table" mode for timeline schema creator
> 
>
> Key: YARN-4102
> URL: https://issues.apache.org/jira/browse/YARN-4102
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Li Lu
>Assignee: Li Lu
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4102-YARN-2928.001.patch, 
> YARN-4102-YARN-2928.002.patch, YARN-4102-YARN-2928.003.patch, 
> YARN-4102-YARN-2928.004.patch
>
>
> When debugging timeline POCs, we may need to create hbase tables that are 
> added in some ongoing patches. Right now, our schema creator will exit when 
> it hits one existing table. While this is a correct behavior with end users, 
> this introduces much trouble in debugging POCs: every time we have to disable 
> all existing tables, drop them, run the schema creator to generate all 
> tables, and regenerate all test data. 
> Maybe we'd like to add an "incremental" mode so that the creator will only 
> create non-existing tables? This is pretty handy in deploying our POCs. Of 
> course, consistency has to be kept in mind across tables. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3706) Generalize native HBase writer for additional tables

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3706:
---
Fix Version/s: 2.9.0

> Generalize native HBase writer for additional tables
> 
>
> Key: YARN-3706
> URL: https://issues.apache.org/jira/browse/YARN-3706
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Joep Rottinghuis
>Assignee: Joep Rottinghuis
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3706-YARN-2928.001.patch, 
> YARN-3706-YARN-2928.010.patch, YARN-3706-YARN-2928.011.patch, 
> YARN-3706-YARN-2928.012.patch, YARN-3706-YARN-2928.013.patch, 
> YARN-3706-YARN-2928.014.patch, YARN-3706-YARN-2928.015.patch, 
> YARN-3726-YARN-2928.002.patch, YARN-3726-YARN-2928.003.patch, 
> YARN-3726-YARN-2928.004.patch, YARN-3726-YARN-2928.005.patch, 
> YARN-3726-YARN-2928.006.patch, YARN-3726-YARN-2928.007.patch, 
> YARN-3726-YARN-2928.008.patch, YARN-3726-YARN-2928.009.patch
>
>
> When reviewing YARN-3411 we noticed that we could change the class hierarchy 
> a little in order to accommodate additional tables easily.
> In order to get ready for benchmark testing we left the original layout in 
> place, as performance would not be impacted by the code hierarchy.
> Here is a separate jira to address the hierarchy.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6064) Support fromId for flowRuns and flow/flowRun apps REST API's

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6064:
---
Fix Version/s: 2.9.0

> Support fromId for flowRuns and flow/flowRun apps REST API's
> 
>
> Key: YARN-6064
> URL: https://issues.apache.org/jira/browse/YARN-6064
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-6064-YARN-5355.0001.patch, 
> YARN-6064-YARN-5355.0002.patch, YARN-6064-YARN-5355.0003.patch, 
> YARN-6064-YARN-5355.0004.patch, YARN-6064-YARN-5355.addendum.patch
>
>
> Splitting out JIRA YARN-6027 for pagination support for flowRuns, flow apps 
> and flow run apps. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3034:
---
Fix Version/s: 2.9.0

> [Collector wireup] Implement RM starting its timeline collector
> ---
>
> Key: YARN-3034
> URL: https://issues.apache.org/jira/browse/YARN-3034
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Naganarasimha G R
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3024.20150324-1.patch, YARN-3034-20150312-1.patch, 
> YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, 
> YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch
>
>
> Per design in YARN-2928, implement resource managers starting their own ATS 
> writers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5095) flow activities and flow runs are populated with wrong timestamp when RM restarts w/ recovery enabled

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5095:
---
Fix Version/s: 2.9.0

> flow activities and flow runs are populated with wrong timestamp when RM 
> restarts w/ recovery enabled
> -
>
> Key: YARN-5095
> URL: https://issues.apache.org/jira/browse/YARN-5095
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5095-YARN-2928.01.patch, 
> YARN-5095-YARN-2928.02.patch, YARN-5095-YARN-2928.03.patch
>
>
> I have the RM recovery enabled. I see that upon restart the RM populates 
> records into flow activity and flow runs but with *wrong* timestamps. What I 
> mean by the timestamp is the part of the row key:
> - flow activity: row created with the day of the RM restart
> - flow run: row created with the RM start time as the "run id"
> The following illustrates an example flow run:
> {noformat}
> metrics: [ ],
> events: [ ],
> id: "sjlee@Sleep job/1463433569917",
> type: "YARN_FLOW_RUN",
> createdtime: 1463422860987,
> info: {
> UID: "yarn_cluster!sjlee!Sleep job!1463433569917",
> SYSTEM_INFO_FLOW_RUN_ID: 1463433569917,
> SYSTEM_INFO_FLOW_NAME: "Sleep job",
> SYSTEM_INFO_FLOW_RUN_END_TIME: 1463422865033,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> {noformat}
> The created time and the end time are correct (i.e. original time), whereas 
> the timestamp in the row key (= run id: 1463433569917) is actually later than 
> the end time and coincides with the RM restart.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3721) build is broken on YARN-2928 branch due to possible dependency cycle

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3721:
---
Fix Version/s: 2.9.0

> build is broken on YARN-2928 branch due to possible dependency cycle
> 
>
> Key: YARN-3721
> URL: https://issues.apache.org/jira/browse/YARN-3721
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Li Lu
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3721-YARN-2928.001.patch, 
> YARN-3721-YARN-2928.002.patch, YARN-3721-YARN-2928.002.patch
>
>
> The build is broken on the YARN-2928 branch at the 
> hadoop-yarn-server-timelineservice module. It's been broken for a while, but 
> we didn't notice it because the build happens to work despite this if the 
> maven local cache is not cleared.
> To reproduce, remove all hadoop (3.0.0-SNAPSHOT) artifacts from your maven 
> local cache and build it.
> Almost certainly it was introduced by YARN-3529.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3049) [Storage Implementation] Implement storage reader interface to fetch raw data from HBase backend

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3049:
---
Fix Version/s: 2.9.0

> [Storage Implementation] Implement storage reader interface to fetch raw data 
> from HBase backend
> 
>
> Key: YARN-3049
> URL: https://issues.apache.org/jira/browse/YARN-3049
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Zhijie Shen
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3049-WIP.1.patch, YARN-3049-WIP.2.patch, 
> YARN-3049-WIP.3.patch, YARN-3049-YARN-2928.2.patch, 
> YARN-3049-YARN-2928.3.patch, YARN-3049-YARN-2928.4.patch, 
> YARN-3049-YARN-2928.5.patch, YARN-3049-YARN-2928.6.patch, 
> YARN-3049-YARN-2928.7.patch
>
>
> Implement existing ATS queries with the new ATS reader design.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5699:
---
Fix Version/s: 2.9.0

> Retrospect yarn entity fields which are publishing in events info fields.
> -
>
> Key: YARN-5699
> URL: https://issues.apache.org/jira/browse/YARN-5699
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, 
> 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch, 
> 0003-YARN-5699.YARN-5355.patch, 0003-YARN-5699.patch
>
>
> Currently, all the container information are published at 2 places. Some of 
> them are at entity info(top-level) and some are  at event info. 
> For containers, some of the event info should be published at container info 
> level. For example : container exist status, container state, createdTime, 
> finished time. These are general information to container required for 
> container-report. So it is better to publish at top level info field. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7041) Nodemanager NPE running jobs with security off

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-7041:
---
Fix Version/s: 2.9.0

> Nodemanager NPE running jobs with security off
> --
>
> Key: YARN-7041
> URL: https://issues.apache.org/jira/browse/YARN-7041
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-5355
>Reporter: Aaron Gresch
>Assignee: Varun Saxena
>  Labels: yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-7041-YARN-5355.01.patch
>
>
> 2017-08-17 09:47:18,127 ERROR [Node Status Updater] 
> nodemanager.NodeStatusUpdaterImpl: Caught exception in status-updater
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatRequestPBImpl.convertToProtoFormat(NodeHeartbeatRequestPBImpl.java:326)
> at 
> org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatRequestPBImpl.addRegisteringCollectorsToProto(NodeHeartbeatRequestPBImpl.java:174)
> at 
> org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatRequestPBImpl.mergeLocalToBuilder(NodeHeartbeatRequestPBImpl.java:122)
> at 
> org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatRequestPBImpl.mergeLocalToProto(NodeHeartbeatRequestPBImpl.java:183)
> at 
> org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatRequestPBImpl.getProto(NodeHeartbeatRequestPBImpl.java:77)
> at 
> org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.nodeHeartbeat(ResourceTrackerPBClientImpl.java:83)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:411)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:348)
> at com.sun.proxy.$Proxy77.nodeHeartbeat(Unknown Source)
> at 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:799)
> at java.lang.Thread.run(Thread.java:745)
> Aug 17, 2017 9:47:18 AM com.sun.jersey.api.core.PackagesResourceConfig init
> INFO: Scanning for root resource and provider classes in the packages:
>   org.apache.hadoop.yarn.server.timelineservice.collector
>   org.apache.hadoop.yarn.webapp
> I believe this is due to YARN-6130, commented on that JIRA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6658) Remove columnFor() methods of Columns in HBaseTimeline backend

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6658:
---
Fix Version/s: 2.9.0

> Remove columnFor() methods of Columns in HBaseTimeline backend 
> ---
>
> Key: YARN-6658
> URL: https://issues.apache.org/jira/browse/YARN-6658
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha3
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>  Labels: atsv2-hbase
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-6658.00.patch
>
>
> The columnFor() methods are all dead. We can remove them.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3634) TestMRTimelineEventHandling and TestApplication are broken

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3634:
---
Fix Version/s: 2.9.0

> TestMRTimelineEventHandling and TestApplication are broken
> --
>
> Key: YARN-3634
> URL: https://issues.apache.org/jira/browse/YARN-3634
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3634-YARN-2928.001.patch, 
> YARN-3634-YARN-2928.002.patch, YARN-3634-YARN-2928.003.patch, 
> YARN-3634-YARN-2928.004.patch
>
>
> TestMRTimelineEventHandling is broken. Relevant error message:
> {noformat}
> 2015-05-12 06:28:56,415 INFO  [AsyncDispatcher event handler] ipc.Client 
> (Client.java:handleConnectionFailure(882)) - Retrying connect to server: 
> asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 0 time(s); retry 
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 
> MILLISECONDS)
> 2015-05-12 06:28:57,416 INFO  [AsyncDispatcher event handler] ipc.Client 
> (Client.java:handleConnectionFailure(882)) - Retrying connect to server: 
> asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 1 time(s); retry 
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 
> MILLISECONDS)
> 2015-05-12 06:28:58,416 INFO  [AsyncDispatcher event handler] ipc.Client 
> (Client.java:handleConnectionFailure(882)) - Retrying connect to server: 
> asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 2 time(s); retry 
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 
> MILLISECONDS)
> 2015-05-12 06:28:59,417 INFO  [AsyncDispatcher event handler] ipc.Client 
> (Client.java:handleConnectionFailure(882)) - Retrying connect to server: 
> asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 3 time(s); retry 
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 
> MILLISECONDS)
> 2015-05-12 06:29:00,418 INFO  [AsyncDispatcher event handler] ipc.Client 
> (Client.java:handleConnectionFailure(882)) - Retrying connect to server: 
> asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 4 time(s); retry 
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 
> MILLISECONDS)
> 2015-05-12 06:29:01,419 INFO  [AsyncDispatcher event handler] ipc.Client 
> (Client.java:handleConnectionFailure(882)) - Retrying connect to server: 
> asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 5 time(s); retry 
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 
> MILLISECONDS)
> 2015-05-12 06:29:02,420 INFO  [AsyncDispatcher event handler] ipc.Client 
> (Client.java:handleConnectionFailure(882)) - Retrying connect to server: 
> asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 6 time(s); retry 
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 
> MILLISECONDS)
> 2015-05-12 06:29:03,420 INFO  [AsyncDispatcher event handler] ipc.Client 
> (Client.java:handleConnectionFailure(882)) - Retrying connect to server: 
> asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 7 time(s); retry 
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 
> MILLISECONDS)
> 2015-05-12 06:29:04,421 INFO  [AsyncDispatcher event handler] ipc.Client 
> (Client.java:handleConnectionFailure(882)) - Retrying connect to server: 
> asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 8 time(s); retry 
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 
> MILLISECONDS)
> 2015-05-12 06:29:05,422 INFO  [AsyncDispatcher event handler] ipc.Client 
> (Client.java:handleConnectionFailure(882)) - Retrying connect to server: 
> asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 9 time(s); retry 
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 
> MILLISECONDS)
> 2015-05-12 06:29:05,424 ERROR [AsyncDispatcher event handler] 
> collector.NodeTimelineCollectorManager 
> (NodeTimelineCollectorManager.java:postPut(121)) - Failed to communicate with 
> NM Collector Service for application_1431412130291_0001
> 2015-05-12 06:29:05,425 WARN  [AsyncDispatcher event handler] 
> containermanager.AuxServices 
> (AuxServices.java:logWarningWhenAuxServiceThrowExceptions(261)) - The 
> auxService name is timeline_collector and it got an error at event: 
> CONTAINER_INIT
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
> java.net.ConnectException: Call From asf904.gq1.ygridcore.net/67.195.81.148 
> to asf904.gq1.ygridcore.net:0 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see: 

[jira] [Updated] (YARN-6455) Enhance the timelinewriter.flush() race condition fix

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6455:
---
Fix Version/s: 2.9.0

> Enhance the timelinewriter.flush() race condition fix
> -
>
> Key: YARN-6455
> URL: https://issues.apache.org/jira/browse/YARN-6455
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.0.0-alpha2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4
>
> Attachments: YARN-6455.00.patch
>
>
> YARN-6376 fixes timelinewriter.flush() race condition among concurrent 
> putEntities() calls and periodical flush by TimelineCollectorManager by 
> synchronizing on the writer object. 
> Synchronizing on the writer is still a little brittle there, because there is 
> a getWriter method which lets callers access the writer without synchronizing 
> on it. AppLevelTimelineCollector#AppLevelAggregator#agregate() does this in 
> line 152: getWriter().write(...)  In this case it doesn't flush, but if that 
> were to be added, that would re-introduce the race fixed in YARN-6376.
> Instead of exposing the writer, perhaps it would be better to have the 
> sub-classes call #putEntities instead. It defers to the private 
> writeTimelineEntities which does the same work to get the context:
> TimelineCollectorContext context = getTimelineEntityContext();



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3949) ensure timely flush of timeline writes

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3949:
---
Fix Version/s: 2.9.0

> ensure timely flush of timeline writes
> --
>
> Key: YARN-3949
> URL: https://issues.apache.org/jira/browse/YARN-3949
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3949-YARN-2928.001.patch, 
> YARN-3949-YARN-2928.002.patch, YARN-3949-YARN-2928.002.patch, 
> YARN-3949-YARN-2928.003.patch, YARN-3949-YARN-2928.004.patch, 
> YARN-3949-YARN-2928.004.patch
>
>
> Currently flushing of timeline writes is not really handled. For example, 
> {{HBaseTimelineWriterImpl}} relies on HBase's {{BufferedMutator}} to batch 
> and write puts asynchronously. However, {{BufferedMutator}} may not flush 
> them to HBase unless the internal buffer fills up.
> We do need a flush functionality first to ensure that data are written in a 
> reasonably timely manner, and to be able to ensure some critical writes are 
> done synchronously (e.g. key lifecycle events).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4447) Provide a mechanism to represent complex filters and parse them at the REST layer

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4447:
---
Fix Version/s: 2.9.0

> Provide a mechanism to represent complex filters and parse them at the REST 
> layer 
> --
>
> Key: YARN-4447
> URL: https://issues.apache.org/jira/browse/YARN-4447
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: Timeline-Filters.pdf, YARN-4447-YARN-2928.01.patch, 
> YARN-4447-YARN-2928.02.patch, YARN-4447-YARN-2928.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4350) TestDistributedShell fails for V2 scenarios

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4350:
---
Fix Version/s: 2.9.0

> TestDistributedShell fails for V2 scenarios
> ---
>
> Key: YARN-4350
> URL: https://issues.apache.org/jira/browse/YARN-4350
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Naganarasimha G R
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4350-feature-YARN-2928.001.patch, 
> YARN-4350-feature-YARN-2928.002.patch, YARN-4350-feature-YARN-2928.003.patch
>
>
> Currently TestDistributedShell does not pass on the feature-YARN-2928 branch. 
> There seem to be 2 distinct issues.
> (1) testDSShellWithoutDomainV2* tests fail sporadically
> These test fail more often than not if tested by themselves:
> {noformat}
> testDSShellWithoutDomainV2DefaultFlow(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
>   Time elapsed: 30.998 sec  <<< FAILURE!
> java.lang.AssertionError: Application created event should be published 
> atleast once expected:<1> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at 
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.checkTimelineV2(TestDistributedShell.java:451)
>   at 
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:326)
>   at 
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow(TestDistributedShell.java:207)
> {noformat}
> They start happening after YARN-4129. I suspect this might have to do with 
> some timing issue.
> (2) the whole test times out
> If you run the whole TestDistributedShell test, it times out without fail. 
> This may or may not have to do with the port change introduced by YARN-2859 
> (just a hunch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3906) split the application table from the entity table

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3906:
---
Fix Version/s: 2.9.0

> split the application table from the entity table
> -
>
> Key: YARN-3906
> URL: https://issues.apache.org/jira/browse/YARN-3906
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3906-YARN-2928.001.patch, 
> YARN-3906-YARN-2928.002.patch, YARN-3906-YARN-2928.003.patch, 
> YARN-3906-YARN-2928.004.patch, YARN-3906-YARN-2928.005.patch, 
> YARN-3906-YARN-2928.006.patch, YARN-3906-YARN-2928.007.patch
>
>
> Per discussions on YARN-3815, we need to split the application entities from 
> the main entity table into its own table (application).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6801) NPE in RM while setting collectors map in NodeHeartbeatResponse

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6801:
---
Fix Version/s: 2.9.0

> NPE in RM while setting collectors map in NodeHeartbeatResponse
> ---
>
> Key: YARN-6801
> URL: https://issues.apache.org/jira/browse/YARN-6801
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-5355, YARN-5355-branch-2
>Reporter: Vrushali C
>Assignee: Vrushali C
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-6801-YARN-5355.001.patch
>
>
> Null Pointer Exception seen in 
> ResourceTrackerService#setAppCollectorsMapToResponse call 
> {code}
> 2017-06-22 22:24:01,437 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 49 on 8031, call 
> org.apache.hadoop.yarn.server.api.ResourceTrackerPB.nodeHeartbeat from 
> 10.35.172.116:44399 Call#3929 Retry#0
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.setAppCollectorsMapToResponse(ResourceTrackerService.java:467)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.nodeHeartbeat(ResourceTrackerService.java:447)
> at 
> org.apache.hadoop.yarn.server.api.impl.pb.service.ResourceTrackerPBServiceImpl.nodeHeartbeat(ResourceTrackerPBServiceImpl.java:68)
> at 
> org.apache.hadoop.yarn.proto.ResourceTracker$ResourceTrackerService$2.callBlockingMethod(ResourceTracker.java:81)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2084)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2080)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1645)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2078)
> {code}
> It correlates to RM invoking setAppCollectorsMapToResponse and calling 
> {code}
>   AppCollectorData appCollectorData = 
> rmApps.get(appId).getCollectorData();
> {code}
> If the app object is not present in the list of running app ids, then this 
> will throw NPE.
> Filing jira to fix it. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3367) Replace starting a separate thread for post entity with event loop in TimelineClient

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3367:
---
Fix Version/s: 2.9.0

> Replace starting a separate thread for post entity with event loop in 
> TimelineClient
> 
>
> Key: YARN-3367
> URL: https://issues.apache.org/jira/browse/YARN-3367
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Junping Du
>Assignee: Naganarasimha G R
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3367-YARN-2928.v1.005.patch, 
> YARN-3367-YARN-2928.v1.006.patch, YARN-3367-YARN-2928.v1.007.patch, 
> YARN-3367-YARN-2928.v1.008.patch, YARN-3367-YARN-2928.v1.009.patch, 
> YARN-3367-YARN-2928.v1.010.patch, YARN-3367-YARN-2928.v1.011.patch, 
> YARN-3367-YARN-2928.v1.012.patch, YARN-3367-YARN-2928.v1.013.patch, 
> YARN-3367-YARN-2928.v1.014.patch, YARN-3367-feature-YARN-2928.003.patch, 
> YARN-3367-feature-YARN-2928.v1.002.patch, 
> YARN-3367-feature-YARN-2928.v1.004.patch, YARN-3367.YARN-2928.001.patch, 
> sjlee-suggestion.patch
>
>
> Since YARN-3039, we add loop in TimelineClient to wait for 
> collectorServiceAddress ready before posting any entity. In consumer of  
> TimelineClient (like AM), we are starting a new thread for each call to get 
> rid of potential deadlock in main thread. This way has at least 3 major 
> defects:
> 1. The consumer need some additional code to wrap a thread before calling 
> putEntities() in TimelineClient.
> 2. It cost many thread resources which is unnecessary.
> 3. The sequence of events could be out of order because each posting 
> operation thread get out of waiting loop randomly.
> We should have something like event loop in TimelineClient side, 
> putEntities() only put related entities into a queue of entities and a 
> separated thread handle to deliver entities in queue to collector via REST 
> call.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6850) Ensure that supplemented timestamp is stored only for flow run metrics

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6850:
---
Fix Version/s: 2.9.0

> Ensure that supplemented timestamp is stored only for flow run metrics
> --
>
> Key: YARN-6850
> URL: https://issues.apache.org/jira/browse/YARN-6850
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Varun Saxena
>  Labels: atsv2-hbase, yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-6850-YARN-5355.01.patch
>
>
> In timeline service v2,  ColumnHelper#getPutTimestamp supplements the 
> timestamp and is called by ColumnHelper#store. This is not conditional and 
> called for every put.
> We need to ensure that the cell timestamps for metrics in entity and 
> application (and sub application) tables are "correct" timestamps since we 
> will be enabling TTLs for these cells. 
> The supplemented timestamp is to be used only in the flow run table by the 
> coprocessor which intercepts all reads & writes to cells in this table. It 
> looks at the supplemented timestamp to figure out which app id this 
> particular cell belongs to. This is done in order to ensure no collision 
> occurs when two apps belonging to same flow run write the same metric at the 
> same timestamp. 
> Discovered in the discussion in YARN-4455 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5638) Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5638:
---
Fix Version/s: 2.9.0

> Introduce a collector timestamp to uniquely identify collectors creation 
> order in collector discovery
> -
>
> Key: YARN-5638
> URL: https://issues.apache.org/jira/browse/YARN-5638
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Li Lu
>Assignee: Li Lu
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1
>
> Attachments: YARN-5638-YARN-5355.v4.patch, 
> YARN-5638-YARN-5355.v5.patch, YARN-5638-trunk.v1.patch, 
> YARN-5638-trunk.v2.patch, YARN-5638-trunk.v3.patch
>
>
> As discussed in YARN-3359, we need to further identify timeline collectors' 
> creation order to rebuild collector discovery data in the RM. This JIRA 
> proposes to use  to order collectors 
> for each application in the RM. This timestamp can then be used when a 
> standby RM becomes active and rebuild collector discovery data. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6604) Allow metric TTL for Application table to be specified through cmd

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6604:
---
Fix Version/s: 2.9.0

> Allow metric TTL for Application table to be specified through cmd
> --
>
> Key: YARN-6604
> URL: https://issues.apache.org/jira/browse/YARN-6604
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.0.0-alpha2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>  Labels: atsv2-hbase
> Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4
>
> Attachments: YARN-6604.00.patch
>
>
> We should allow metrics TTL in application table to be specified in schema 
> cmd, as we do with metrics TTL in entity table



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6874) Supplement timestamp for min start/max end time columns in flow run table to avoid overwrite

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6874:
---
Fix Version/s: 2.9.0

> Supplement timestamp for min start/max end time columns in flow run table to 
> avoid overwrite
> 
>
> Key: YARN-6874
> URL: https://issues.apache.org/jira/browse/YARN-6874
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Vrushali C
>  Labels: atsv2-hbase
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-6874-YARN-5355.0001.patch
>
>
> Following test case is failing in YARN-5355 branch.
> This is coming because we are not supplementing the timestamp for 
> FlowRunColumn i.e. min_start_time and max_end_time columns, post YARN-6850 
> which can lead to a clash, if 2 writes for app created events happen at the 
> same time, which is true for this test case.
> To fix this, we need to pass true flag into ColumnHelper constructor. I did 
> encounter this failure once earlier too.
> {noformat}
> testWriteFlowRunMinMax(org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun)
>   Time elapsed: 0.088 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<142502690> but was:<1425026901000>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun.testWriteFlowRunMinMax(TestHBaseStorageFlowRun.java:237)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3984) Rethink event column key issue

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3984:
---
Fix Version/s: 2.9.0

> Rethink event column key issue
> --
>
> Key: YARN-3984
> URL: https://issues.apache.org/jira/browse/YARN-3984
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Vrushali C
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3984-YARN-2928.001.patch
>
>
> Currently, the event column key is event_id?info_key?timestamp, which is not 
> so friendly to fetching all the events of an entity and sorting them in a 
> chronologic order. IMHO, timestamp?event_id?info_key may be a better key 
> schema. I open this jira to continue the discussion about it which was 
> commented on YARN-3908.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4174) Fix javadoc warnings floating up from hbase

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4174:
---
Fix Version/s: 2.9.0

> Fix javadoc warnings floating up from hbase 
> 
>
> Key: YARN-4174
> URL: https://issues.apache.org/jira/browse/YARN-4174
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Vrushali C
>Assignee: Sangjin Lee
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha1
>
>
> As part of the patch for YARN-3901, [~sjlee0]  observed some (~200) javadoc 
> warnings that are coming from hbase classes. 
> We tried a bunch of things like making the FlowRunCoprocessor class non 
> public and excluding the package from the pom. If the class in made non 
> public, the table creation has an exception.
> {code}
> 206 warnings
> [WARNING] Javadoc Warnings
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestWALObserver.class):
>  warning: Cannot find annotation method 'value()' in type 'Category': class 
> file for org.junit.experimental.categories.Category not found
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRowProcessorEndpoint.class):
>  warning: Cannot find annotation method 'value()' in type 'Category'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerObserver.class):
>  warning: Cannot find annotation method 'value()' in type 'Category'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.class):
>  warning: Cannot find annotation method 'value()' in type 'Category'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.class):
>  warning: Cannot find annotation method 'timeout()' in type 'Test': class 
> file for org.junit.Test not found
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.class):
>  warning: Cannot find annotation method 'value()' in type 'Category'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.class):
>  warning: Cannot find annotation method 'timeout()' in type 'Test'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.class):
>  warning: Cannot find annotation method 'timeout()' in type 'Test'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorEndpoint.class):
>  warning: Cannot find annotation method 'value()' in type 'Category'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.class):
>  warning: Cannot find annotation method 'value()' in type 'Category'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.class):
>  warning: Cannot find annotation method 'value()' in type 'Category'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.class):
>  warning: Cannot find annotation method 'value()' in type 'Category'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.class):
>  warning: Cannot find annotation method 'timeout()' in type 'Test'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.class):
>  warning: Cannot find annotation method 'timeout()' in type 'Test'
> [WARNING] 
> /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.class):
>  warning: Cannot find annotation method 'timeout()' in 

[jira] [Updated] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6170:
---
Fix Version/s: 2.9.0

> TimelineReaderServer should wait to join with HttpServer2
> -
>
> Key: YARN-6170
> URL: https://issues.apache.org/jira/browse/YARN-6170
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Affects Versions: 3.0.0-alpha2, YARN-5355
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Fix For: 2.9.0, YARN-5355-branch-2, 3.0.0-alpha4
>
> Attachments: YARN-6170.01.patch
>
>
> While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I 
> noticed that the timeline reader daemon would promptly shut down upon start. 
> It turns out that in the 2.6.0 code line at least there are only daemon 
> threads left once the main method returns. That causes the JVM to shut down.
> The right pattern to start an embedded jetty web server is to call 
> {{Server.start()}} followed by {{Server.join()}}. That way, the server stays 
> up reliably no matter what other threads get created.
> It works on YARN-5355 only because there *happens* to be one other non-daemon 
> thread. We should add the {{join()}} call to be always correct.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3039) [Collector wireup] Implement timeline app-level collector service discovery

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3039:
---
Fix Version/s: 2.9.0

> [Collector wireup] Implement timeline app-level collector service discovery
> ---
>
> Key: YARN-3039
> URL: https://issues.apache.org/jira/browse/YARN-3039
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Junping Du
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: Service Binding for applicationaggregator of ATS 
> (draft).pdf, Service Discovery For Application Aggregator of ATS (v2).pdf, 
> YARN-3039-no-test.patch, YARN-3039-v2-incomplete.patch, 
> YARN-3039-v3-core-changes-only.patch, YARN-3039-v4.patch, YARN-3039-v5.patch, 
> YARN-3039-v6.patch, YARN-3039-v7.patch, YARN-3039-v8.patch, YARN-3039.9.patch
>
>
> Per design in YARN-2928, implement ATS writer service discovery. This is 
> essential for off-node clients to send writes to the right ATS writer. This 
> should also handle the case of AM failures.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5229) Refactor #isApplicationEntity and #getApplicationEvent from HBaseTimelineWriterImpl

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5229:
---
Fix Version/s: 2.9.0

> Refactor #isApplicationEntity and #getApplicationEvent from 
> HBaseTimelineWriterImpl
> ---
>
> Key: YARN-5229
> URL: https://issues.apache.org/jira/browse/YARN-5229
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928, YARN-5355
>Reporter: Joep Rottinghuis
>Assignee: Vrushali C
>Priority: Minor
>  Labels: YARN-5355, atsv2-hbase
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1
>
> Attachments: YARN-229-YARN-5355.01.patch, 
> YARN-5229-YARN-2928.01.patch, YARN-5229-YARN-2928.02.patch, 
> YARN-5229-YARN-2928.03.patch, YARN-5229-YARN-2928.04.patch
>
>
> As per [~gtCarrera9] commented in YARN-5170
> bq. In HBaseTimelineWriterImpl isApplicationEntity and getApplicationEvent 
> seem to be awkward. Looks more like something related to TimelineEntity or 
> ApplicationEntity
> In YARN-5170 we just made the method private, and in this separate jira we 
> can refactor these methods to TimelineEntity or ApplicationEntity.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6342) Make TimelineV2Client's drain timeout after stop configurable

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6342:
---
Fix Version/s: 2.9.0

> Make TimelineV2Client's drain timeout after stop configurable
> -
>
> Key: YARN-6342
> URL: https://issues.apache.org/jira/browse/YARN-6342
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Haibo Chen
> Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4
>
> Attachments: YARN-6342.00.patch, YARN-6342.01.patch, 
> YARN-6342.02.patch
>
>
> Found these with [~rohithsharma] while browsing the code
> - In stop: it calls shutdownNow which doens't wait for pending tasks, should 
> it use shutdown instead ?
> {code}
> public void stop() {
>   LOG.info("Stopping TimelineClient.");
>   executor.shutdownNow();
>   try {
> executor.awaitTermination(DRAIN_TIME_PERIOD, TimeUnit.MILLISECONDS);
>   } catch (InterruptedException e) {
> {code}
> - In TimelineClientImpl#createRunnable:
> If any exception happens when publish one entity 
> (publishWithoutBlockingOnQueue), the thread exists. I think it should try 
> best effort to continue publishing the timeline entities, one failure should 
> not cause all followup entities not published.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5070) upgrade HBase version for first merge

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5070:
---
Fix Version/s: 2.9.0

> upgrade HBase version for first merge
> -
>
> Key: YARN-5070
> URL: https://issues.apache.org/jira/browse/YARN-5070
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>Priority: Critical
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5070-YARN-2928.01.patch, 
> YARN-5070-YARN-2928.02.patch, YARN-5070-YARN-2928.03.patch, 
> YARN-5070-YARN-2928.04.patch, YARN-5070-YARN-2928.05.patch, 
> YARN-5070-YARN-2928.06.patch, YARN-5070-YARN-2928.07.patch
>
>
> Currently we set the HBase version for the timeline service storage to 1.0.1. 
> This is a fairly old version, and there are reasons to upgrade to a newer 
> version. We should upgrade it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3240) [Data Mode] Implement client API to put generic entities

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3240:
---
Fix Version/s: 2.9.0

> [Data Mode] Implement client API to put generic entities
> 
>
> Key: YARN-3240
> URL: https://issues.apache.org/jira/browse/YARN-3240
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3240.1.patch, YARN-3240.2.patch, YARN-3240.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4450) TestTimelineAuthenticationFilter and TestYarnConfigurationFields fail

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4450:
---
Fix Version/s: 2.9.0

> TestTimelineAuthenticationFilter and TestYarnConfigurationFields fail
> -
>
> Key: YARN-4450
> URL: https://issues.apache.org/jira/browse/YARN-4450
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
> Environment: jenkins
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4450-feature-YARN-2928.01.patch
>
>
> When I run the unit tests against the current branch, 
> TestTimelineAuthenticationFilter and TestYarnConfigurationFields fail:
> {noformat}
>   TestTimelineAuthenticationFilter.testDelegationTokenOperations:251 » 
> NullPointer
>   TestTimelineAuthenticationFilter.testDelegationTokenOperations:251 » 
> NullPointer
>  
> TestYarnConfigurationFields>TestConfigurationFieldsBase.testCompareConfigurationClassAgainstXml:429
>  class org.apache.hadoop.yarn.conf.YarnConfiguration has 1 variables missing 
> in yarn-default.xml
> {noformat}
> The latter failure is caused by YARN-4356 (when we deprecated 
> RM_SYSTEM_METRICS_PUBLISHER_ENABLED), and the former an older issue that was 
> caused when a later use of field {{resURI}} was added in trunk.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6563) ConcurrentModificationException in TimelineCollectorManager while stopping RM

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6563:
---
Fix Version/s: 2.9.0

> ConcurrentModificationException in TimelineCollectorManager while stopping RM
> -
>
> Key: YARN-6563
> URL: https://issues.apache.org/jira/browse/YARN-6563
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Haibo Chen
> Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4
>
> Attachments: YARN-6563.00.patch
>
>
> It is seen that ConcurrentModificationException while stopping RM when ATSv2 
> enabled. 
> {noformat}
> 2017-05-05 15:04:11,563 WARN org.apache.hadoop.service.CompositeService: When 
> stopping the service 
> org.apache.hadoop.yarn.server.resourcemanager.timelineservice.RMTimelineCollectorManager
>  : java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
>   at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.serviceStop(TimelineCollectorManager.java:222)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
>   at 
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
>   at 
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
>   at 
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
>   at 
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1285)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3431) Sub resources of timeline entity needs to be passed to a separate endpoint.

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3431:
---
Fix Version/s: 2.9.0

> Sub resources of timeline entity needs to be passed to a separate endpoint.
> ---
>
> Key: YARN-3431
> URL: https://issues.apache.org/jira/browse/YARN-3431
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-3431.1.patch, YARN-3431.2.patch, YARN-3431.3.patch, 
> YARN-3431.4.patch, YARN-3431.5.patch, YARN-3431.6.patch, YARN-3431.7.patch
>
>
> We have TimelineEntity and some other entities as subclass that inherit from 
> it. However, we only have a single endpoint, which consume TimelineEntity 
> rather than sub-classes and this endpoint will check the incoming request 
> body contains exactly TimelineEntity object. However, the json data which is 
> serialized from sub-class object seems not to be treated as an TimelineEntity 
> object, and won't be deserialized into the corresponding sub-class object 
> which cause deserialization failure as some discussions in YARN-3334 : 
> https://issues.apache.org/jira/browse/YARN-3334?focusedCommentId=14391059=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14391059.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6074) FlowRunEntity does not deserialize long values correctly

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6074:
---
Fix Version/s: 2.9.0

> FlowRunEntity does not deserialize long values correctly
> 
>
> Key: YARN-6074
> URL: https://issues.apache.org/jira/browse/YARN-6074
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 2.9.0, 3.0.0-alpha2, YARN-5355, YARN-5355-branch-2
>
> Attachments: YARN-6074.patch
>
>
> I see that FlowRunEntity methods *getRunId()* and *getMaxEndTime()* does not 
> deserialize in efficient way which causes class cast exception based on the 
> number.
> {code}
>   public long getRunId() {
> Object runId = getInfo().get(FLOW_RUN_ID_INFO_KEY);
> return runId == null ? 0L : (Long) runId;
>   }
> {code} 
> and 
> {code}
>   public long getMaxEndTime() {
> Object time = getInfo().get(FLOW_RUN_END_TIME);
> return time == null ? 0L : (Long)time;
>   }
> {code} 
> The reason for class caste exception is Json has data type Number which 
> includes all java primitive types. So, if number with in the range of Integer 
> max, then Object is converted to Integer which fails  to type cast to Long. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4062) Add the flush and compaction functionality via coprocessors and scanners for flow run table

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4062:
---
Fix Version/s: 2.9.0

> Add the flush and compaction functionality via coprocessors and scanners for 
> flow run table
> ---
>
> Key: YARN-4062
> URL: https://issues.apache.org/jira/browse/YARN-4062
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4062-YARN-2928.04.patch, 
> YARN-4062-YARN-2928.05.patch, YARN-4062-YARN-2928.06.patch, 
> YARN-4062-YARN-2928.07.patch, YARN-4062-YARN-2928.08.patch, 
> YARN-4062-YARN-2928.09.patch, YARN-4062-YARN-2928.1.patch, 
> YARN-4062-feature-YARN-2928.01.patch, YARN-4062-feature-YARN-2928.02.patch, 
> YARN-4062-feature-YARN-2928.03.patch
>
>
> As part of YARN-3901, coprocessor and scanner is being added for storing into 
> the flow_run table. It also needs a flush & compaction processing in the 
> coprocessor and perhaps a new scanner to deal with the data during flushing 
> and compaction stages. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6638:
---
Fix Version/s: 2.9.0

> [ATSv2 Security] Timeline reader side changes for loading auth filters and 
> principals
> -
>
> Key: YARN-6638
> URL: https://issues.apache.org/jira/browse/YARN-6638
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-6638-YARN-5355.01.patch, 
> YARN-6638-YARN-5355.02.patch, YARN-6638-YARN-5355.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6820) Restrict read access to timelineservice v2 data

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6820:
---
Fix Version/s: 2.9.0

> Restrict read access to timelineservice v2 data 
> 
>
> Key: YARN-6820
> URL: https://issues.apache.org/jira/browse/YARN-6820
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Vrushali C
>Assignee: Vrushali C
>  Labels: yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-6820-YARN-5355.0001.patch, 
> YARN-6820-YARN-5355.002.patch, YARN-6820-YARN-5355.003.patch, 
> YARN-6820-YARN-5355.004.patch, YARN-6820-YARN-5355.005.patch, 
> YARN-6820-YARN-5355_branch_2.patch
>
>
> Need to provide a way to restrict read access in ATSv2. Not all users should 
> be able to read all entities. On the flip side, some folks may not need any 
> read restrictions, so we need to provide a way to disable this access 
> restriction as well. 
> Initially this access restriction could be done in a simple way via a 
> whitelist of users allowed to read data. That set of users can read all data, 
> no other user can read any data. Can be turned off for all users to read all 
> data.
> Could be stored in a "domain" table in hbase perhaps. Or a configuration 
> setting for the cluster. Or something else that's simple enough. ATSv1 has a 
> concept of domain for isolating users for reading. Would be good to keep that 
> in consideration. 
> In ATSv1, domain offers a namespace for Timeline server allowing users to 
> host multiple entities, isolating them from other users and applications. A 
> “Domain” in ATSV1 primarily stores owner info, read and& write ACL 
> information, created and modified time stamp information. Each Domain is 
> identified by an ID which must be unique across all users in the YARN cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6323) Rolling upgrade/config change is broken on timeline v2.

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6323:
---
Fix Version/s: 2.9.0

> Rolling upgrade/config change is broken on timeline v2. 
> 
>
> Key: YARN-6323
> URL: https://issues.apache.org/jira/browse/YARN-6323
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Li Lu
>Assignee: Vrushali C
>  Labels: yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355_branch2
>
> Attachments: YARN-6323-YARN-5355.0001.patch, 
> YARN-6323-YARN-5355.0002.patch, YARN-6323.001.patch
>
>
> Found this issue when deploying on real clusters. If there are apps running 
> when we enable timeline v2 (with work preserving restart enabled), node 
> managers will fail to start due to missing app context data. We should 
> probably assign some default names to these "left over" apps. I believe it's 
> suboptimal to let users clean up the whole cluster before enabling timeline 
> v2. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6414) ATSv2 HBase related tests fail due to guava version upgrade

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6414:
---
Fix Version/s: 2.9.0

> ATSv2 HBase related tests fail due to guava version upgrade
> ---
>
> Key: YARN-6414
> URL: https://issues.apache.org/jira/browse/YARN-6414
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: 3.0.0-alpha4
> Environment: Ubuntu 14.04 
> x86, ppc64le
> $ java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: Sonia Garudi
>Assignee: Haibo Chen
>  Labels: ppc64le, x86_64
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: YARN-6414.00.patch, YARN-6414.01.patch
>
>
> Test failures seen in Hadoop YARN Timeline Service HBase tests project with 
> following error :
> {code}
> java.lang.NoClassDefFoundError: com/google/common/io/LimitInputStream
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:223)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:913)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:899)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:722)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:660)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:279)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:529)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:585)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:751)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:735)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:998)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:869)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:704)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:642)
> at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:590)
> at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:987)
> at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:868)
> at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:862)
> at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
> at 
> org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setup(TestTimelineReaderWebServicesHBaseStorage.java:88)
> {code}
> Below are the failing tests :
> {code}
> TestTimelineReaderWebServicesHBaseStorage.setup:88 » NoClassDefFound 
> com/googl...
>   TestHBaseTimelineStorageEntities.setupBeforeClass:100 » NoClassDefFound 
> com/go...
>   TestHBaseStorageFlowActivity.setupBeforeClass:72 » NoClassDefFound 
> com/google/...
>   TestHBaseStorageFlowRunCompaction.setupBeforeClass:84 » NoClassDefFound 
> com/go...
>   TestHBaseStorageFlowRun.setupBeforeClass:86 » NoClassDefFound 
> com/google/commo...
>   TestHBaseTimelineStorageApps.setupBeforeClass:93 » NoClassDefFound 
> com/google/...
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5097) NPE in Separator.joinEncoded()

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5097:
---
Fix Version/s: 2.9.0

> NPE in Separator.joinEncoded()
> --
>
> Key: YARN-5097
> URL: https://issues.apache.org/jira/browse/YARN-5097
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>Priority: Critical
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5097-YARN-2928.01.patch, 
> YARN-5097-YARN-2928.02.patch, YARN-5097-YARN-2928.03.patch, 
> YARN-5097-YARN-2928.04.patch
>
>
> Both in the RM log and the NM log, I see the following exception thrown. 
> First for RM,
> {noformat}
> 2016-05-16 14:19:29,930 ERROR 
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector: 
> Error aggregating timeline metrics
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.Separator.joinEncoded(Separator.java:249)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.application.ApplicationRowKey.getRowKey(ApplicationRowKey.java:110)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl.write(HBaseTimelineWriterImpl.java:131)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.collector.AppLevelTimelineCollector$AppLevelAggregator.run(AppLevelTimelineCollector.java:136)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at 
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>   at java.lang.Thread.run(Thread.java:722)
> {noformat}
> In the NM log, I see a similar exception:
> {noformat}
> 2016-05-16 14:54:23,116 ERROR 
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector: 
> Error aggregating timeline metrics
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.Separator.joinEncoded(Separator.java:249)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.application.ApplicationRowKey.getRowKey(ApplicationRowKey.java:110)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl.write(HBaseTimelineWriterImpl.java:131)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.collector.AppLevelTimelineCollector$AppLevelAggregator.run(AppLevelTimelineCollector.java:136)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4460) [Bug fix] RM fails to start when SMP is enabled

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4460:
---
Fix Version/s: 2.9.0

> [Bug fix] RM fails to start when SMP is enabled
> ---
>
> Key: YARN-4460
> URL: https://issues.apache.org/jira/browse/YARN-4460
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Li Lu
>Assignee: Li Lu
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4460-feature-YARN-2928.001.patch
>
>
> When SMP is enabled, the RM starts with the following fatal message: 
> {code}
> FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(189)) - Error in 
> dispatcher thread true
> java.lang.Exception: No handler for registered for class 
> org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractSystemMetricsPubli
> sher$SystemMetricsEventType
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:185)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
>at java.lang.Thread.run(Thread.java:745)
> {code}
> We should register event handlers in service init stage in 
> TimelineServiceV2Publisher to fix this problem. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4445) Unify the term flowId and flowName in timeline v2 codebase

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4445:
---
Fix Version/s: 2.9.0

> Unify the term flowId and flowName in timeline v2 codebase
> --
>
> Key: YARN-4445
> URL: https://issues.apache.org/jira/browse/YARN-4445
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Li Lu
>Assignee: Zhan Zhang
>  Labels: refactor
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4445-feature-YARN-2928.001.patch, YARN-4445.patch
>
>
> Flow names are not sufficient to identify a flow. I noticed we used both 
> "flowName" and "flowId" to point to the same thing. We need to unify them to 
> flowName. Otherwise, front end users may think flow id is a top level concept 
> and try to directly locate a flow by its flow id.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5980) Update documentation for single node hbase deploy

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5980:
---
Fix Version/s: 2.9.0

> Update documentation for single node hbase deploy
> -
>
> Key: YARN-5980
> URL: https://issues.apache.org/jira/browse/YARN-5980
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>  Labels: atsv2-hbase, yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1
>
> Attachments: YARN-5980.001.patch, YARN-5980.002.patch, 
> YARN-5980.003.patch, YARN-5980.004.patch
>
>
> Per HBASE-17272, a single node hbase deployment (single jvm running daemons + 
> hdfs writes) will be added to hbase shortly. 
> We should update the timeline service documentation in the setup/deployment 
> context accordingly, this will help users who are a bit wary of hbase 
> deployments help get started with timeline service more easily.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4210) HBase reader throws NPE if Get returns no rows

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4210:
---
Fix Version/s: 2.9.0

> HBase reader throws NPE if Get returns no rows
> --
>
> Key: YARN-4210
> URL: https://issues.apache.org/jira/browse/YARN-4210
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4210-YARN-2928.01.patch, 
> YARN-4210-YARN-2928.02.patch, YARN-4210-YARN-2928.03.patch
>
>
> This JIRA handles multiple issues.
> * If HBase Get does not fetch any rows for the query, we still try to parse 
> the result and read fields. This leads to NPE while reading metrics. We 
> should not attempt to read anything if no row is returned i.e. result is 
> empty.
> Found during web UI poc testing. 
> {noformat}
> 2015-09-29 20:22:32,027 ERROR [95336304@qtp-1814206058-0] 
> reader.TimelineReaderWebServices 
> (TimelineReaderWebServices.java:handleException(199)) - Error while 
> processing REST request
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper.readResultsWithTimestamps(ColumnHelper.java:176)
> at 
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunColumnPrefix.readResultsWithTimestamps(FlowRunColumnPrefix.java:182)
> at 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineEntityReader.readMetrics(TimelineEntityReader.java:212)
> at 
> org.apache.hadoop.yarn.server.timelineservice.storage.FlowRunEntityReader.parseEntity(FlowRunEntityReader.java:136)
> at 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineEntityReader.readEntity(TimelineEntityReader.java:137)
> at 
> org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.getEntity(HBaseTimelineReaderImpl.java:72)
> at 
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderManager.getEntity(TimelineReaderManager.java:93)
> at 
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getFlowRun(TimelineReaderWebServices.java:403)
> {noformat}
> * ResultScanner is not closed in HBase Reader.
> * Exception encountered while reading start and end time in 
> FlowRunEntityReader



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5291) Store node information for finished containers in timeline v2

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5291:
---
Fix Version/s: 2.9.0

> Store node information for finished containers in timeline v2
> -
>
> Key: YARN-5291
> URL: https://issues.apache.org/jira/browse/YARN-5291
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Li Lu
> Fix For: 2.9.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4238) correctly set createdTime and remove modifiedTime when publishing entities

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-4238:
---
Fix Version/s: 2.9.0

> correctly set createdTime and remove modifiedTime when publishing entities
> --
>
> Key: YARN-4238
> URL: https://issues.apache.org/jira/browse/YARN-4238
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4238-YARN-2928.01.patch, 
> YARN-4238-YARN-2928.04.patch, YARN-4238-YARN-2928.05.patch, 
> YARN-4238-feature-YARN-2928.002.patch, YARN-4238-feature-YARN-2928.003.patch
>
>
> While publishing entities from RM and elsewhere we are not sending created 
> time. For instance, created time in TimelineServiceV2Publisher class and for 
> other entities in other such similar classes is not updated. We can easily 
> update created time when sending application created event. Likewise for 
> modification time on every write.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5096) timelinereader has a lot of logging that's not useful

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5096:
---
Fix Version/s: 2.9.0

> timelinereader has a lot of logging that's not useful
> -
>
> Key: YARN-5096
> URL: https://issues.apache.org/jira/browse/YARN-5096
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>  Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5096-YARN-2928.01.patch
>
>
> After running about a dozen or so requests, the timelinereader log is filled 
> with the following logging entries:
> {noformat}
> 2016-05-16 15:59:13,364 INFO 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: 
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: 
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: 
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: 
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: 
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: 
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: 
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: 
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: 
> null prefix was specified; returning all columns
> {noformat}
> There were some ~ 3,000 such logging entries. It's too excessive.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org