[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369746#comment-15369746
]
Hudson commented on YARN-3792:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10074/])
YARN-3792. Test case failures in TestDistributedShell and some issue (sjlee:
rev 22e7ae57715cedb1dcba736e357e8daaf5133e5c)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/NodeTimelineCollectorManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/timelineservice/RMTimelineCollectorManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/PerNodeTimelineCollectorsAuxService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShellWithNodeLabels.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
> Test case failures in TestDistributedShell and some issue fixes related to
> ATSV2
>
>
> Key: YARN-3792
> URL: https://issues.apache.org/jira/browse/YARN-3792
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Fix For: YARN-2928
>
> Attachments: YARN-3792-YARN-2928.001.patch,
> YARN-3792-YARN-2928.002.patch, YARN-3792-YARN-2928.003.patch,
> YARN-3792-YARN-2928.004.patch
>
>
> # encountered [testcase
> failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
> which was happening even without the patch modifications in YARN-3044
> TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
> TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
> TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
> # Remove unused {{enableATSV1}} in testDisstributedShell
> # container metrics needs to be published only for v2 test cases of
> testDisstributedShell
> # Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
> service was not configured and {{TimelineClient.putObjects}} was getting
> invoked.
> # Race condition for the Application events to published and test case
> verification for RM's ApplicationFinished Timeline Events
> # Application Tags for converted to lowercase in
> ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
> detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596925#comment-14596925
]
Sangjin Lee commented on YARN-3792:
---
The latest patch LGTM. Once the jenkins comes back, I'll go ahead and merge it.
Folks, do let me know soon if you have any other feedback. Thanks!
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch,
YARN-3792-YARN-2928.002.patch, YARN-3792-YARN-2928.003.patch,
YARN-3792-YARN-2928.004.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596730#comment-14596730
]
Sangjin Lee commented on YARN-3792:
---
Thanks [~Naganarasimha] for the update!
+1 on the test failure. It appears to be an issue unrelated to the timeline
service.
It does seem like the whitespace is related to the patch (or in the vicinity of
the patch). Could you kindly do a quick change to remove those extra spaces?
Also, for findbugs, I ran findbugs against those two projects (distributed
shell and resource manager). I do see several findbugs warnings, and they are
not introduced by this patch but do appear to be related to the YARN-2928 work.
distributed shell:
{code}
file
classname='org.apache.hadoop.yarn.applications.distributedshell.Client'BugInstance
type='DM_BOXED_PRIMITIVE_FOR_PARSING' priority='High' category='PERFORMANCE'
message='Boxing/unboxing to parse a primitive
org.apache.hadoop.yarn.applications.distributedshell.Client.init(String[])'
lineNumber='466'//file
{code}
resource manager:
{code}
file
classname='org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractTimelineServicePublisher'BugInstance
type='BC_UNCONFIRMED_CAST' priority='Normal' category='STYLE'
message='Unchecked/unconfirmed cast from
org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsEvent to
org.apache.hadoop.yarn.server.resourcemanager.metrics.AppAttemptFinishedEvent
in
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractTimelineServicePublisher.handle(SystemMetricsEvent)'
lineNumber='79'/BugInstance type='BC_UNCONFIRMED_CAST' priority='Normal'
category='STYLE' message='Unchecked/unconfirmed cast from
org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsEvent to
org.apache.hadoop.yarn.server.resourcemanager.metrics.AppAttemptRegisteredEvent
in
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractTimelineServicePublisher.handle(SystemMetricsEvent)'
lineNumber='76'/BugInstance type='BC_UNCONFIRMED_CAST' priority='Normal'
category='STYLE' message='Unchecked/unconfirmed cast from
org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsEvent to
org.apache.hadoop.yarn.server.resourcemanager.metrics.ApplicationACLsUpdatedEvent
in
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractTimelineServicePublisher.handle(SystemMetricsEvent)'
lineNumber='73'/BugInstance type='BC_UNCONFIRMED_CAST' priority='Normal'
category='STYLE' message='Unchecked/unconfirmed cast from
org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsEvent to
org.apache.hadoop.yarn.server.resourcemanager.metrics.ApplicationCreatedEvent
in
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractTimelineServicePublisher.handle(SystemMetricsEvent)'
lineNumber='67'/BugInstance type='BC_UNCONFIRMED_CAST' priority='Normal'
category='STYLE' message='Unchecked/unconfirmed cast from
org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsEvent to
org.apache.hadoop.yarn.server.resourcemanager.metrics.ApplicationFinishedEvent
in
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractTimelineServicePublisher.handle(SystemMetricsEvent)'
lineNumber='70'/BugInstance type='BC_UNCONFIRMED_CAST' priority='Normal'
category='STYLE' message='Unchecked/unconfirmed cast from
org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsEvent to
org.apache.hadoop.yarn.server.resourcemanager.metrics.ContainerCreatedEvent in
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractTimelineServicePublisher.handle(SystemMetricsEvent)'
lineNumber='82'/BugInstance type='BC_UNCONFIRMED_CAST' priority='Normal'
category='STYLE' message='Unchecked/unconfirmed cast from
org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsEvent to
org.apache.hadoop.yarn.server.resourcemanager.metrics.ContainerFinishedEvent in
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractTimelineServicePublisher.handle(SystemMetricsEvent)'
lineNumber='85'//file{code}
It would be nice to address them (at least the one on Client.java) here, but if
you're not inclined, we could do it later... Let me know how you want to
proceed.
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch,
YARN-3792-YARN-2928.002.patch, YARN-3792-YARN-2928.003.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596994#comment-14596994
]
Hadoop QA commented on YARN-3792:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 29s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 49s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 4s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 38s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 43s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 40s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 5m 59s | The patch appears to introduce 7
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 8m 10s | Tests passed in
hadoop-yarn-applications-distributedshell. |
| {color:green}+1{color} | yarn tests | 1m 58s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 11s | Tests passed in
hadoop-yarn-server-nodemanager. |
| {color:red}-1{color} | yarn tests | 51m 49s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 1m 17s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 115m 20s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| Failed unit tests |
hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12741171/YARN-3792-YARN-2928.004.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / 8c036a1 |
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8319/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
|
| hadoop-yarn-applications-distributedshell test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8319/artifact/patchprocess/testrun_hadoop-yarn-applications-distributedshell.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8319/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8319/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8319/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8319/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8319/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8319/console |
This message was automatically generated.
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch,
YARN-3792-YARN-2928.002.patch, YARN-3792-YARN-2928.003.patch,
YARN-3792-YARN-2928.004.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594455#comment-14594455
]
Naganarasimha G R commented on YARN-3792:
-
Hi [~sjlee0],
Missed to check your comment yesterday, will get this done asap and oops i
missed ur second comment to fix earlier...
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch,
YARN-3792-YARN-2928.002.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594778#comment-14594778
]
Hadoop QA commented on YARN-3792:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 30s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 56s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 57s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 40s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 46s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 42s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 6m 6s | The patch appears to introduce 8
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 8m 11s | Tests passed in
hadoop-yarn-applications-distributedshell. |
| {color:green}+1{color} | yarn tests | 2m 3s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 8s | Tests passed in
hadoop-yarn-server-nodemanager. |
| {color:red}-1{color} | yarn tests | 52m 43s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 1m 22s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 116m 36s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-applications-distributedshell |
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| Failed unit tests |
hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12740818/YARN-3792-YARN-2928.003.patch
|
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | YARN-2928 / 8c036a1 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8300/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8300/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-applications-distributedshell.html
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8300/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
|
| hadoop-yarn-applications-distributedshell test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8300/artifact/patchprocess/testrun_hadoop-yarn-applications-distributedshell.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8300/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8300/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8300/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8300/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8300/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8300/console |
This message was automatically generated.
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch,
YARN-3792-YARN-2928.002.patch, YARN-3792-YARN-2928.003.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594821#comment-14594821
]
Naganarasimha G R commented on YARN-3792:
-
* Test case reported is not due to this patch and already YARN-3790 has been
raised to address it.
* white space is not caused by this patch
* incorrect findbugs alert, report has no issues
[~sjlee0], i think its good state now !
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch,
YARN-3792-YARN-2928.002.patch, YARN-3792-YARN-2928.003.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594025#comment-14594025
]
Sangjin Lee commented on YARN-3792:
---
I tested the latest patch, and at least on my local environment all the unit
tests seem to pass.
It looks good to me overall. I have a couple of small nits. It'd be great if
you could take a look at them and address them quickly.
(TestDistributedShellWithNodeLabels.java)
- l. 47: spacing
(TimelineClientImpl.java)
- l.385: let's use a more standard java style condition: i.e.
{{timelineServiceAddress == null}}
We'll also need to see the jenkins result (looks like there is a large queue so
it might take a while).
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch,
YARN-3792-YARN-2928.002.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594265#comment-14594265
]
Hadoop QA commented on YARN-3792:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 59s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 57s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 6s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 37s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 2s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 41s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 6m 2s | The patch appears to introduce 8
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 8m 12s | Tests passed in
hadoop-yarn-applications-distributedshell. |
| {color:green}+1{color} | yarn tests | 1m 59s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 17s | Tests passed in
hadoop-yarn-server-nodemanager. |
| {color:red}-1{color} | yarn tests | 33m 29s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:red}-1{color} | yarn tests | 0m 14s | Tests failed in
hadoop-yarn-server-timelineservice. |
| | | 95m 43s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-applications-distributedshell |
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
|
| | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
| |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates |
| |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId |
| | hadoop.yarn.server.resourcemanager.TestRM |
| | hadoop.yarn.server.resourcemanager.TestApplicationACLs |
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| |
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisherForV2 |
| | hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth |
| | hadoop.yarn.server.resourcemanager.resourcetracker.TestRMNMRPCResponseId |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
| | hadoop.yarn.server.resourcemanager.TestRMHA |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs |
| | hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
|
| | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings |
| | hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils |
| | hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA |
| | hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
|
| Failed build | hadoop-yarn-server-timelineservice |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12740626/YARN-3792-YARN-2928.002.patch
|
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | YARN-2928 / 8c036a1 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8291/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8291/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-applications-distributedshell.html
|
| Findbugs warnings |
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14588465#comment-14588465
]
Naganarasimha G R commented on YARN-3792:
-
bq. I'm not sure why that is the case. Maybe they didn't want to run the base
tests? Unless that changes, I guess we'll have to check for null. This looks
pretty brittle however. Sigh.
Well can make {{TestDistributedShell.setupInternal}} to take additional
argument as method name so that caller can pass the method name and its less
brittle?
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14588901#comment-14588901
]
Sangjin Lee commented on YARN-3792:
---
Yes that might be an idea.
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14588441#comment-14588441
]
Sangjin Lee commented on YARN-3792:
---
{quote}
Other approach i can think of is to make the test sleep for 500 seconds 3~4
times or till the desired result is got (i think this is the approach which is
followed in most of the test cases which are highly async), thoughts ?
{quote}
Given the asynchronous nature (and lack of strong coordination between
actions), I'm +1 on looping a few times.
{quote}
Well atleast in my environment(latest ubuntu) even after JAVA_HOME was set
properly in shell, test cases when executed from eclipse were failing because
of the JAVA_HOME was not availble and it took a while for me to figure out
doing this. Hence i thought it would be usefull information for others who r
testing for the first time, If you guys feel not necessary i can remove it.
{quote}
IMO the comment seems a little out of place. I would prefer not having this
comment here, but I'd like to hear what others think.
{quote}
yes its req, as mentioned earlier for
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression was
failing because method name rule will not set be set in
TestDistributedShell.setupInternal
{quote}
Thanks for reminding me. I didn't realize that
{{TestDistributedShellWithNodeLabels}} does *NOT* extend
{{TestDistributedShell}}. I'm not sure why that is the case. Maybe they didn't
want to run the base tests? Unless that changes, I guess we'll have to check
for null. This looks pretty brittle however. Sigh.
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14587612#comment-14587612
]
Naganarasimha G R commented on YARN-3792:
-
bq. I still see one test failing:
I had sufficiently tested in my laptop but still seems to be happening :(
bq. l.376: I don't really like the sleep call as it is not completely
deterministic; could there be a way to make this completely deterministic
(using things like CountDownLatch, etc.)?
Test failing also is due to the fact that there is a race condition between the
application finished being published and test case checking this published
event in the file. AFAIK using CountDownLatch might not be possible as its
happening asynchrously in the Mini YARN cluster and the test case needs to wait
till published event is written to the file. In the test case once Application
status(YarnApplicationState) and FinalApplicationStatus indicates the app is
over we are checking for the published events, even then as events are
published through dispatcher its not guranteed to be finished. Other approach i
can think of is to make the test sleep for 500 seconds 3~4 times or till the
desired result is got (i think this is the approach which is followed in most
of the test cases which are highly async), thoughts ? I am open for other
approaches too :)
bq. l.71-75: Is this comment necessary here? I'm not sure if we want to add a
generic comment like this to a specific test...
Well atleast in my environment(latest ubuntu) even after JAVA_HOME was set
properly in shell, test cases when executed from eclipse were failing because
of the JAVA_HOME was not availble and it took a while for me to figure out
doing this. Hence i thought it would be usefull information for others who r
testing for the first time, If you guys feel not necessary i can remove it.
bq. l.106: Are the checks for null necessary? I thought that the test name was
populated by junit and made available to test methods. Do things fail if we do
not check for null?
yes its req, as mentioned earlier for
??{{TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression}} was
failing because method name rule will not set be set in
TestDistributedShell.setupInternal??
Can take care of the remaining comments
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14585676#comment-14585676
]
Naganarasimha G R commented on YARN-3792:
-
* Test case reported is not due to this patch and already YARN-3790 has been
raised to address it.
* white space is not caused by this patch
* incorrect findbugs alert, report has no issues.
Patch is good enough to review !
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14586128#comment-14586128
]
Junping Du commented on YARN-3792:
--
Thanks [~Naganarasimha] for delivering the patch to fix it! Will review your
patch soon.
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14587183#comment-14587183
]
Sangjin Lee commented on YARN-3792:
---
Thanks [~Naganarasimha] for identifying the issues and providing a patch! I
applied the patch on top of the current YARN-2928 branch, rebuilt, and ran the
TestDistributedShell test locally. I still see one test failing:
{noformat}
---
T E S T S
---
Running
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 581.546 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
testDSShellWithoutDomainV2CustomizedFlow(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
Time elapsed: 29.651 sec FAILURE!
java.lang.AssertionError: Application finished event should be published
atleast once expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.verifyStringExistsSpecifiedTimes(TestDistributedShell.java:483)
at
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.checkTimelineV2(TestDistributedShell.java:431)
at
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:323)
at
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow(TestDistributedShell.java:209)
Results :
Failed tests:
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow:209-testDSShell:323-checkTimelineV2:431-verifyStringExistsSpecifiedTimes:483
Application finished event should be published atleast once expected:1 but
was:0
Tests run: 13, Failures: 1, Errors: 0, Skipped: 0
{noformat}
Have you seen this? Could you kindly look into that? I'll also see if this is
reproducible on my end.
Some quick comments:
(TestDistributedShell.java)
- l.71-75: Is this comment necessary here? I'm not sure if we want to add a
generic comment like this to a specific test...
- l.106: Are the checks for null necessary? I thought that the test name was
populated by junit and made available to test methods. Do things fail if we do
not check for null?
- l.376: I don't really like the sleep call as it is not completely
deterministic; could there be a way to make this completely deterministic
(using things like CountDownLatch, etc.)?
(TimelineClientImpl.java)
- l.385: nit: the C-style conditional check is not necessary; I would suggest a
more natural check of {{(timelineServiceAddress == null)}}
(ContainersMonitorImpl.java)
- l.96: It is unrelated to this patch itself, but should we rename the variable
name threadPool? It is a completely generic name. We should rename it to
something like timelineWriterThreadPool or something to that effect. Let me
know if you have a suggestion.
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14585263#comment-14585263
]
Naganarasimha G R commented on YARN-3792:
-
Uploading the patch with the fixes for the mentioned issues.
Analysis for few issues which might require review/discussion :
* Issue1 : {{TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow}}
was failing when all the test cases was run because of the ExecutorService was
statically initialized and was shutdown on the service stop in the initial run
{{TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow}} was failing
because of NodeTimelineCollectorManager was singleton service used in
PerNodeTimelineCollectorsAuxService, i feel singleton reference was not
required as it was used only by the NodeTimelineCollectorManager. /cc [~djp],
please review the modifications are fine for this check in specific.
{{TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression}} was
failing because method name rule will not set be set in
TestDistributedShell.setupInternal.
* Issue 4: Hope the fix given in {{TimelineClientImpl.putObjects}} is
sufficient for this.
* Issue 6: Not sure why tags are converted to lower case in
{{ApplicationSubmissionContextPBImpl.setApplicationTags}}. Currently have done
the modifications at our end {{RMTimelineCollectorManager.postPut}}. But seems
this needs to be corrected in ApplicationSubmissionContextPBImpl, based on
suggestions can rework on this. Also tried to check the history/log was not
able to find the source of this change.
Other trivial issues has been handled.
[~zjshen], [~djp] [~sjlee0], please review...
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression
# Remove unused {{enableATSV1}} in testDisstributedShell
# container metrics needs to be published only for v2 test cases of
testDisstributedShell
# Nullpointer was thrown in TimelineClientImpl.constructResURI when Aux
service was not configured and {{TimelineClient.putObjects}} was getting
invoked.
# Race condition for the Application events to published and test case
verification for RM's ApplicationFinished Timeline Events
# Application Tags for converted to lowercase in
ApplicationSubmissionContextPBimpl, hence RMTimelinecollector was not able to
detect to custom flow details of the app
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3792) Test case failures in TestDistributedShell and some issue fixes related to ATSV2
[
https://issues.apache.org/jira/browse/YARN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14585304#comment-14585304
]
Hadoop QA commented on YARN-3792:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 2s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 45s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 52s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 37s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 40s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 41s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 5m 51s | The patch appears to introduce 8
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 8m 9s | Tests passed in
hadoop-yarn-applications-distributedshell. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 15s | Tests passed in
hadoop-yarn-server-nodemanager. |
| {color:red}-1{color} | yarn tests | 51m 4s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 1m 15s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 113m 36s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-applications-distributedshell |
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| Failed unit tests |
hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12739519/YARN-3792-YARN-2928.001.patch
|
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | YARN-2928 / 17842a3 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8248/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8248/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-applications-distributedshell.html
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8248/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
|
| hadoop-yarn-applications-distributedshell test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8248/artifact/patchprocess/testrun_hadoop-yarn-applications-distributedshell.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8248/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8248/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8248/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8248/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8248/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8248/console |
This message was automatically generated.
Test case failures in TestDistributedShell and some issue fixes related to
ATSV2
Key: YARN-3792
URL: https://issues.apache.org/jira/browse/YARN-3792
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Attachments: YARN-3792-YARN-2928.001.patch
# encountered [testcase
failures|https://builds.apache.org/job/PreCommit-YARN-Build/8233/testReport/]
which was happening even without the patch modifications in YARN-3044
TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow
