[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[
https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591501#comment-16591501
]
genericqa commented on YARN-8632:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
14s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8632 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936981/YARN-8632.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c61d94021620 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 96c4575 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21674/testReport/ |
| Max. process+thread count | 446 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21674/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> No data in file realtimetrack.json after running
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[ https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591435#comment-16591435 ] Xianghao Lu commented on YARN-8632: --- [~yufeigu] Thanks for your review! Patch for 2.7 is not necessary for me. Attatched YARN-8632.004.patch to avoid npe. > No data in file realtimetrack.json after running SchedulerLoadSimulator > --- > > Key: YARN-8632 > URL: https://issues.apache.org/jira/browse/YARN-8632 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler-load-simulator >Reporter: Xianghao Lu >Assignee: Xianghao Lu >Priority: Major > Attachments: YARN-8632-branch-2.7.2.001.patch, YARN-8632.001.patch, > YARN-8632.002.patch, YARN-8632.003.patch, YARN-8632.004.patch > > > Recently, I have been using > [SchedulerLoadSimulator|https://hadoop.apache.org/docs/r2.7.2/hadoop-sls/SchedulerLoadSimulator.html] > to validate the impact of changes on my FairScheduler. I encountered some > problems. > Firstly, I fix a npe bug with the patch in > https://issues.apache.org/jira/browse/YARN-4302 > Secondly, everything seems to be ok, but I just get "[]" in file > realtimetrack.json. Finally, I find the MetricsLogRunnable thread will exit > because of npe, > the reason is "wrapper.getQueueSet()" is still null when executing "String > metrics = web.generateRealTimeTrackingMetrics();" > So, we should put "String metrics = web.generateRealTimeTrackingMetrics();" > in try section to avoid MetricsLogRunnable thread exit with unexpected > exception. > My hadoop version is 2.7.2, it seems that hadoop trunk branch also has the > second problem and I have made a patch to solve it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[ https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590570#comment-16590570 ] Yufei Gu commented on YARN-8632: [~luxianghao], thanks for the patch. Nice finding. +1 for the patch v3. Will commit later. Do you need a patch for 2.7? > No data in file realtimetrack.json after running SchedulerLoadSimulator > --- > > Key: YARN-8632 > URL: https://issues.apache.org/jira/browse/YARN-8632 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler-load-simulator >Reporter: Xianghao Lu >Assignee: Xianghao Lu >Priority: Major > Attachments: YARN-8632-branch-2.7.2.001.patch, YARN-8632.001.patch, > YARN-8632.002.patch, YARN-8632.003.patch > > > Recently, I have been using > [SchedulerLoadSimulator|https://hadoop.apache.org/docs/r2.7.2/hadoop-sls/SchedulerLoadSimulator.html] > to validate the impact of changes on my FairScheduler. I encountered some > problems. > Firstly, I fix a npe bug with the patch in > https://issues.apache.org/jira/browse/YARN-4302 > Secondly, everything seems to be ok, but I just get "[]" in file > realtimetrack.json. Finally, I find the MetricsLogRunnable thread will exit > because of npe, > the reason is "wrapper.getQueueSet()" is still null when executing "String > metrics = web.generateRealTimeTrackingMetrics();" > So, we should put "String metrics = web.generateRealTimeTrackingMetrics();" > in try section to avoid MetricsLogRunnable thread exit with unexpected > exception. > My hadoop version is 2.7.2, it seems that hadoop trunk branch also has the > second problem and I have made a patch to solve it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[
https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590028#comment-16590028
]
genericqa commented on YARN-8632:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 20s{color}
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.sls.TestSLSRunner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8632 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936796/YARN-8632.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e40660a5fc15 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1ac0144 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/21665/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21665/testReport/ |
| Max. process+thread count | 449 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls |
| Console output |
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[ https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589918#comment-16589918 ] Xianghao Lu commented on YARN-8632: --- It seems that "setUncaughtExceptionHandler" is not suitable in thread pool, refer to https://issues.apache.org/jira/browse/HADOOP-12748 [http://literatejava.com/threading/silent-thread-death-unhandled-exceptions/|http://example.com] Using try catch block to catch Exception may not be a good method but it does solve the problem in this jira, but I also find it is used in [AppLevelAggregator|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/AppLevelTimelineCollectorWithAgg.java#L139] [EntityLogScanner|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java#L865], etc. which are related to function scheduleAtFixedRate. Fortunately, I noticed HADOOP-12749 HADOOP-14966 provide a nice way to log uncaught exception in thread pool, so the patch of branch brunk will be very easy. Attatched YARN-8632.003.patch > No data in file realtimetrack.json after running SchedulerLoadSimulator > --- > > Key: YARN-8632 > URL: https://issues.apache.org/jira/browse/YARN-8632 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler-load-simulator >Reporter: Xianghao Lu >Assignee: Xianghao Lu >Priority: Major > Attachments: YARN-8632-branch-2.7.2.001.patch, YARN-8632.001.patch, > YARN-8632.002.patch, YARN-8632.003.patch > > > Recently, I have been using > [SchedulerLoadSimulator|https://hadoop.apache.org/docs/r2.7.2/hadoop-sls/SchedulerLoadSimulator.html] > to validate the impact of changes on my FairScheduler. I encountered some > problems. > Firstly, I fix a npe bug with the patch in > https://issues.apache.org/jira/browse/YARN-4302 > Secondly, everything seems to be ok, but I just get "[]" in file > realtimetrack.json. Finally, I find the MetricsLogRunnable thread will exit > because of npe, > the reason is "wrapper.getQueueSet()" is still null when executing "String > metrics = web.generateRealTimeTrackingMetrics();" > So, we should put "String metrics = web.generateRealTimeTrackingMetrics();" > in try section to avoid MetricsLogRunnable thread exit with unexpected > exception. > My hadoop version is 2.7.2, it seems that hadoop trunk branch also has the > second problem and I have made a patch to solve it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[
https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16586344#comment-16586344
]
Yufei Gu commented on YARN-8632:
For that sake, we need to "setUncaughtExceptionHandler" for the thread, and
provide a handler. Catching every exception in {{run()}} isn't enough.
> No data in file realtimetrack.json after running SchedulerLoadSimulator
> ---
>
> Key: YARN-8632
> URL: https://issues.apache.org/jira/browse/YARN-8632
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler-load-simulator
>Reporter: Xianghao Lu
>Assignee: Xianghao Lu
>Priority: Major
> Attachments: YARN-8632-branch-2.7.2.001.patch, YARN-8632.001.patch,
> YARN-8632.002.patch
>
>
> Recently, I have been using
> [SchedulerLoadSimulator|https://hadoop.apache.org/docs/r2.7.2/hadoop-sls/SchedulerLoadSimulator.html]
> to validate the impact of changes on my FairScheduler. I encountered some
> problems.
> Firstly, I fix a npe bug with the patch in
> https://issues.apache.org/jira/browse/YARN-4302
> Secondly, everything seems to be ok, but I just get "[]" in file
> realtimetrack.json. Finally, I find the MetricsLogRunnable thread will exit
> because of npe,
> the reason is "wrapper.getQueueSet()" is still null when executing "String
> metrics = web.generateRealTimeTrackingMetrics();"
> So, we should put "String metrics = web.generateRealTimeTrackingMetrics();"
> in try section to avoid MetricsLogRunnable thread exit with unexpected
> exception.
> My hadoop version is 2.7.2, it seems that hadoop trunk branch also has the
> second problem and I have made a patch to solve it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[
https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583824#comment-16583824
]
genericqa commented on YARN-8632:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 35m
4s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
20s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8632 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936000/YARN-8632.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b4cc79be12ab 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fa121eb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| mvninstall |
https://builds.apache.org/job/PreCommit-YARN-Build/21618/artifact/out/branch-mvninstall-root.txt
|
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21618/testReport/ |
| Max. process+thread count | 448 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21618/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[
https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583766#comment-16583766
]
Xianghao Lu commented on YARN-8632:
---
Thanks for your review!
{quote}
It is not a good practice to catch runtime exception. Normally we should let
the program exit if a runtime exception happens. Create a new exception if you
think it is necessary and throw and catch it explicitly.
{quote}
I agree, but we should make log to remind user no matter which kind of
exception occurs in the periodic scheduling thread. As stated in the
description, I just get "[]" in file realtimetrack.json without any exception
message, I didn’t know what happened until I debugged the code.
Attatched YARN-8632.002.patch according to comments
> No data in file realtimetrack.json after running SchedulerLoadSimulator
> ---
>
> Key: YARN-8632
> URL: https://issues.apache.org/jira/browse/YARN-8632
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler-load-simulator
>Reporter: Xianghao Lu
>Assignee: Xianghao Lu
>Priority: Major
> Attachments: YARN-8632-branch-2.7.2.001.patch, YARN-8632.001.patch
>
>
> Recently, I have beenning using
> [SchedulerLoadSimulator|https://hadoop.apache.org/docs/r2.7.2/hadoop-sls/SchedulerLoadSimulator.html]
> to validate the impact of changes on my FairScheduler. I encountered some
> problems.
> Firstly, I fix a npe bug with the patch in
> https://issues.apache.org/jira/browse/YARN-4302
> Secondly, Everything seems to be ok, but I just get "[]" in file
> realtimetrack.json. Finally, I find the MetricsLogRunnable thread will exit
> because of npe,
> the reason is "wrapper.getQueueSet()" is still null when executing "String
> metrics = web.generateRealTimeTrackingMetrics();"
> So, we should put "String metrics = web.generateRealTimeTrackingMetrics();"
> in try section to avoid MetricsLogRunnable thread exit with unexpected
> exception.
> My hadoop version is 2.7.2, it seems that hadoop trunk branch also has the
> second problem and I have made a patch to solve it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[
https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578945#comment-16578945
]
Yufei Gu commented on YARN-8632:
Thanks for the patch. Some comments:
# It is not a good practice to catch runtime exception. Normally we should let
the program exit if a runtime exception happens. Create a new exception if you
think it is necessary and throw and catch it explicitly.
# Use {{LOG.info("message", e)}} instead of {{e.printStackTrace();}}
# {{(SchedulerWrapper)scheduler;}} needs a space before "scheduler"
# Create a unitest if possible.
> No data in file realtimetrack.json after running SchedulerLoadSimulator
> ---
>
> Key: YARN-8632
> URL: https://issues.apache.org/jira/browse/YARN-8632
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler-load-simulator
>Reporter: Xianghao Lu
>Assignee: Xianghao Lu
>Priority: Major
> Attachments: YARN-8632-branch-2.7.2.001.patch, YARN-8632.001.patch
>
>
> Recently, I have beenning using
> [SchedulerLoadSimulator|https://hadoop.apache.org/docs/r2.7.2/hadoop-sls/SchedulerLoadSimulator.html]
> to validate the impact of changes on my FairScheduler. I encountered some
> problems.
> Firstly, I fix a npe bug with the patch in
> https://issues.apache.org/jira/browse/YARN-4302
> Secondly, Everything seems to be ok, but I just get "[]" in file
> realtimetrack.json. Finally, I find the MetricsLogRunnable thread will exit
> because of npe,
> the reason is "wrapper.getQueueSet()" is still null when executing "String
> metrics = web.generateRealTimeTrackingMetrics();"
> So, we should put "String metrics = web.generateRealTimeTrackingMetrics();"
> in try section to avoid MetricsLogRunnable thread exit with unexpected
> exception.
> My hadoop version is 2.7.2, it seems that hadoop trunk branch also has the
> second problem and I have made a patch to solve it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[
https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578898#comment-16578898
]
genericqa commented on YARN-8632:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
23s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8632 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935358/YARN-8632.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f923d6ae1d25 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b4031a8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21589/testReport/ |
| Max. process+thread count | 457 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21589/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> No data in file realtimetrack.json after running SchedulerLoadSimulator
>
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[ https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578158#comment-16578158 ] Xianghao Lu commented on YARN-8632: --- Sorry for unclear information! I have updated the patch, the target of YARN-8632-branch-2.7.2.001.patch is branch-2.7.2, and the target of YARN-8632.001.patch is branch trunk. > No data in file realtimetrack.json after running SchedulerLoadSimulator > --- > > Key: YARN-8632 > URL: https://issues.apache.org/jira/browse/YARN-8632 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler-load-simulator >Reporter: Xianghao Lu >Assignee: Xianghao Lu >Priority: Major > Attachments: YARN-8632-branch-2.7.2.001.patch, YARN-8632.001.patch > > > Recently, I have beenning using > [SchedulerLoadSimulator|https://hadoop.apache.org/docs/r2.7.2/hadoop-sls/SchedulerLoadSimulator.html] > to validate the impact of changes on my FairScheduler. I encountered some > problems. > Firstly, I fix a npe bug with the patch in > https://issues.apache.org/jira/browse/YARN-4302 > Secondly, Everything seems to be ok, but I just get "[]" in file > realtimetrack.json. Finally, I find the MetricsLogRunnable thread will exit > because of npe, > the reason is "wrapper.getQueueSet()" is still null when executing "String > metrics = web.generateRealTimeTrackingMetrics();" > So, we should put "String metrics = web.generateRealTimeTrackingMetrics();" > in try section to avoid MetricsLogRunnable thread exit with unexpected > exception. > My hadoop version is 2.7.2, it seems that hadoop trunk branch also has the > second problem and I have made a patch to solve it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[ https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576875#comment-16576875 ] Yufei Gu commented on YARN-8632: Your patch doesn't apply to trunk? You said the bug is in trunk as well, can you provide a patch for the trunk? What is the version does your patch target? 2.7.2? > No data in file realtimetrack.json after running SchedulerLoadSimulator > --- > > Key: YARN-8632 > URL: https://issues.apache.org/jira/browse/YARN-8632 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler-load-simulator >Reporter: Xianghao Lu >Assignee: Xianghao Lu >Priority: Major > Attachments: YARN-8632.001.patch > > > Recently, I have beenning using > [SchedulerLoadSimulator|https://hadoop.apache.org/docs/r2.7.2/hadoop-sls/SchedulerLoadSimulator.html] > to validate the impact of changes on my FairScheduler. I encountered some > problems. > Firstly, I fix a npe bug with the patch in > https://issues.apache.org/jira/browse/YARN-4302 > Secondly, Everything seems to be ok, but I just get "[]" in file > realtimetrack.json. Finally, I find the MetricsLogRunnable thread will exit > because of npe, > the reason is "wrapper.getQueueSet()" is still null when executing "String > metrics = web.generateRealTimeTrackingMetrics();" > So, we should put "String metrics = web.generateRealTimeTrackingMetrics();" > in try section to avoid MetricsLogRunnable thread exit with unexpected > exception. > My hadoop version is 2.7.2, it seems that hadoop trunk branch also has the > second problem and I have made a patch to solve it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[ https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1657#comment-1657 ] Yufei Gu commented on YARN-8632: Added you to the contributor list and assign this to you. Will review later. > No data in file realtimetrack.json after running SchedulerLoadSimulator > --- > > Key: YARN-8632 > URL: https://issues.apache.org/jira/browse/YARN-8632 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler-load-simulator >Reporter: Xianghao Lu >Assignee: Xianghao Lu >Priority: Major > Attachments: YARN-8632.001.patch > > > Recently, I have beenning using > [SchedulerLoadSimulator|https://hadoop.apache.org/docs/r2.7.2/hadoop-sls/SchedulerLoadSimulator.html] > to validate the impact of changes on my FairScheduler. I encountered some > problems. > Firstly, I fix a npe bug with the patch in > https://issues.apache.org/jira/browse/YARN-4302 > Secondly, Everything seems to be ok, but I just get "[]" in file > realtimetrack.json. Finally, I find the MetricsLogRunnable thread will exit > because of npe, > the reason is "wrapper.getQueueSet()" is still null when executing "String > metrics = web.generateRealTimeTrackingMetrics();" > So, we should put "String metrics = web.generateRealTimeTrackingMetrics();" > in try section to avoid MetricsLogRunnable thread exit with unexpected > exception. > My hadoop version is 2.7.2, it seems that hadoop trunk branch also has the > second problem and I have made a patch to solve it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8632) No data in file realtimetrack.json after running SchedulerLoadSimulator
[ https://issues.apache.org/jira/browse/YARN-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576121#comment-16576121 ] Xianghao Lu commented on YARN-8632: --- [~ywskycn] or [~yufeigu] Woud you like to review the patch? Thanks! > No data in file realtimetrack.json after running SchedulerLoadSimulator > --- > > Key: YARN-8632 > URL: https://issues.apache.org/jira/browse/YARN-8632 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler-load-simulator >Reporter: Xianghao Lu >Priority: Major > Attachments: YARN-8632.001.patch > > > Recently, I have beenning using > [SchedulerLoadSimulator|https://hadoop.apache.org/docs/r2.7.2/hadoop-sls/SchedulerLoadSimulator.html] > to validate the impact of changes on my FairScheduler. I encountered some > problems. > Firstly, I fix a npe bug with the patch in > https://issues.apache.org/jira/browse/YARN-4302 > Secondly, Everything seems to be ok, but I just get "[]" in file > realtimetrack.json. Finally, I find the MetricsLogRunnable thread will exit > because of npe, > the reason is "wrapper.getQueueSet()" is still null when executing "String > metrics = web.generateRealTimeTrackingMetrics();" > So, we should put "String metrics = web.generateRealTimeTrackingMetrics();" > in try section to avoid MetricsLogRunnable thread exit with unexpected > exception. > My hadoop version is 2.7.2, it seems that hadoop trunk branch also has the > second problem and I have made a patch to solve it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
