[jira] [Resolved] (YARN-7151) Compilation error in YARN!
[
https://issues.apache.org/jira/browse/YARN-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S resolved YARN-7151.
-
Resolution: Not A Problem
I had an extra file in my works space that causing compilation error. Closing
as Not a Problem!
> Compilation error in YARN!
> --
>
> Key: YARN-7151
> URL: https://issues.apache.org/jira/browse/YARN-7151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Blocker
>
> The compilation error while building trunk. Does anyone else facing this
> issue?
> {noformat}
> [INFO] -
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /Users/rohithsharmaks/repository/apache/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogService.java:[360,64]
> cannot find symbol
> symbol: method
> outputAggregatedContainerLog(org.apache.hadoop.conf.Configuration,org.apache.hadoop.yarn.api.records.ApplicationId,java.lang.String,java.lang.String,java.lang.String,java.lang.String,long,java.io.OutputStream,byte[])
> location: class org.apache.hadoop.yarn.logaggregation.LogToolUtils
> [ERROR]
> /Users/rohithsharmaks/repository/apache/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogService.java:[389,79]
> cannot find symbol
> symbol: method
> getContainerLogMetaFromRemoteFS(org.apache.hadoop.conf.Configuration,org.apache.hadoop.yarn.api.records.ApplicationId,java.lang.String,java.lang.String,java.lang.String)
> location: class org.apache.hadoop.yarn.logaggregation.LogToolUtils
> [INFO] 2 errors
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7151) Compilation error in YARN!
[
https://issues.apache.org/jira/browse/YARN-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152143#comment-16152143
]
Rohith Sharma K S commented on YARN-7151:
-
YARN-6877 removes LogToolUtils#getContainerLogMetaFromRemoteFS and
LogToolUtils#outputAggregatedContainerLog methods.
> Compilation error in YARN!
> --
>
> Key: YARN-7151
> URL: https://issues.apache.org/jira/browse/YARN-7151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Blocker
>
> The compilation error while building trunk. Does anyone else facing this
> issue?
> {noformat}
> [INFO] -
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /Users/rohithsharmaks/repository/apache/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogService.java:[360,64]
> cannot find symbol
> symbol: method
> outputAggregatedContainerLog(org.apache.hadoop.conf.Configuration,org.apache.hadoop.yarn.api.records.ApplicationId,java.lang.String,java.lang.String,java.lang.String,java.lang.String,long,java.io.OutputStream,byte[])
> location: class org.apache.hadoop.yarn.logaggregation.LogToolUtils
> [ERROR]
> /Users/rohithsharmaks/repository/apache/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogService.java:[389,79]
> cannot find symbol
> symbol: method
> getContainerLogMetaFromRemoteFS(org.apache.hadoop.conf.Configuration,org.apache.hadoop.yarn.api.records.ApplicationId,java.lang.String,java.lang.String,java.lang.String)
> location: class org.apache.hadoop.yarn.logaggregation.LogToolUtils
> [INFO] 2 errors
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7151) Compilation error in YARN!
Rohith Sharma K S created YARN-7151:
---
Summary: Compilation error in YARN!
Key: YARN-7151
URL: https://issues.apache.org/jira/browse/YARN-7151
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
Priority: Blocker
The compilation error while building trunk. Does anyone else facing this issue?
{noformat}
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/Users/rohithsharmaks/repository/apache/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogService.java:[360,64]
cannot find symbol
symbol: method
outputAggregatedContainerLog(org.apache.hadoop.conf.Configuration,org.apache.hadoop.yarn.api.records.ApplicationId,java.lang.String,java.lang.String,java.lang.String,java.lang.String,long,java.io.OutputStream,byte[])
location: class org.apache.hadoop.yarn.logaggregation.LogToolUtils
[ERROR]
/Users/rohithsharmaks/repository/apache/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogService.java:[389,79]
cannot find symbol
symbol: method
getContainerLogMetaFromRemoteFS(org.apache.hadoop.conf.Configuration,org.apache.hadoop.yarn.api.records.ApplicationId,java.lang.String,java.lang.String,java.lang.String)
location: class org.apache.hadoop.yarn.logaggregation.LogToolUtils
[INFO] 2 errors
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152116#comment-16152116 ] Wangda Tan edited comment on YARN-7136 at 9/4/17 4:46 AM: -- Thanks [~templedf] for review. All good points, for two following items: bq. In ResourceInformation should the constants be called URIs? Aren't they just names? I think we can support the URI like "yarn.io/gpu" or "example.com/resource-x" as the original design. So do you think if the "URI" is more appropriate than "name"? bq. is it really worth adding the length to the signature? You're retrieving the value in all the other methods, so why not there as well? Since get length retrieves a volatile field, so we should avoid such call as much as possible. What I try to do is only get it once for every public method. Will update patch once you post rest comments. was (Author: leftnoteasy): Thanks [~templedf] for review. All good points, bq. In ResourceInformation should the constants be called URIs? Aren't they just names? I think we can support the URI like "yarn.io/gpu" or "example.com/resource-x" as the original design. So do you think if the "URI" is more appropriate than "name"? bq. is it really worth adding the length to the signature? You're retrieving the value in all the other methods, so why not there as well? Since get length retrieves a volatile field, so we should avoid such call as much as possible. What I try to do is only get it once for every public method. > Additional Performance Improvement for Resource Profile Feature > --- > > Key: YARN-7136 > URL: https://issues.apache.org/jira/browse/YARN-7136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, > YARN-7136.YARN-3926.002.patch, YARN-7136.YARN-3926.003.patch, > YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch, > YARN-7136.YARN-3926.006.patch, YARN-7136.YARN-3926.007.patch > > > This JIRA is plan to add following misc perf improvements: > 1) Use final int in Resources/ResourceCalculator to cache > #known-resource-types. (Significant improvement). > 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length > every time. (Significant improvement). > 3) Avoid setUnit validation (which is a HashSet lookup) when initialize > default Memory/VCores ResourceInformation (Significant improvement). > 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some > improvement). > 5) Removed readOnlyResources in BaseResource. (Minor improvement). > 6) Removed enum: MandatoryResources, use final integer instead. (Minor > improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152116#comment-16152116 ] Wangda Tan commented on YARN-7136: -- Thanks [~templedf] for review. All good points, bq. In ResourceInformation should the constants be called URIs? Aren't they just names? I think we can support the URI like "yarn.io/gpu" or "example.com/resource-x" as the original design. So do you think if the "URI" is more appropriate than "name"? bq. is it really worth adding the length to the signature? You're retrieving the value in all the other methods, so why not there as well? Since get length retrieves a volatile field, so we should avoid such call as much as possible. What I try to do is only get it once for every public method. > Additional Performance Improvement for Resource Profile Feature > --- > > Key: YARN-7136 > URL: https://issues.apache.org/jira/browse/YARN-7136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, > YARN-7136.YARN-3926.002.patch, YARN-7136.YARN-3926.003.patch, > YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch, > YARN-7136.YARN-3926.006.patch, YARN-7136.YARN-3926.007.patch > > > This JIRA is plan to add following misc perf improvements: > 1) Use final int in Resources/ResourceCalculator to cache > #known-resource-types. (Significant improvement). > 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length > every time. (Significant improvement). > 3) Avoid setUnit validation (which is a HashSet lookup) when initialize > default Memory/VCores ResourceInformation (Significant improvement). > 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some > improvement). > 5) Removed readOnlyResources in BaseResource. (Minor improvement). > 6) Removed enum: MandatoryResources, use final integer instead. (Minor > improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6855) CLI Proto Modifications to support Node Attributes
[
https://issues.apache.org/jira/browse/YARN-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152113#comment-16152113
]
Hadoop QA commented on YARN-6855:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
53s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
43s{color} | {color:red} hadoop-yarn-common in trunk failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 56s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 12 new + 95 unchanged - 0 fixed = 107 total (was 95) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
42s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
29s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
43s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 55s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} hadoop-yarn-server-router in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6855 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885178/YARN-6855-YARN-3409.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 6d4aedbc059b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017
[jira] [Commented] (YARN-7104) Improvement of Heatmap
[
https://issues.apache.org/jira/browse/YARN-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152108#comment-16152108
]
Hadoop QA commented on YARN-7104:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7104 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885187/yarn-7104.003.patch |
| Optional Tests | asflicense |
| uname | Linux cb49e9fdb9e2 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 633c1ea |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17274/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improvement of Heatmap
> --
>
> Key: YARN-7104
> URL: https://issues.apache.org/jira/browse/YARN-7104
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Da Ding
>Assignee: Da Ding
> Attachments: Screen Shot 2017-08-25 at 8.26.16 PM.png, Screen Shot
> 2017-08-29 at 4.43.25 PM.png, Screen Shot 2017-08-31 at 8.58.36 PM.png,
> Screen Shot 2017-08-31 at 8.58.43 PM.png, yarn-7104.001.patch,
> yarn-7104.002.patch, yarn-7104.003.patch
>
>
> 1. Changed color gradient of heat map
> 2. Modified the tooltip style to have better UI
> 3. Adjusted display style of reck
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7104) Improvement of Heatmap
[ https://issues.apache.org/jira/browse/YARN-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152104#comment-16152104 ] Da Ding commented on YARN-7104: --- [~sunilg], patch rebased. Please let me know if it works. Thanks! > Improvement of Heatmap > -- > > Key: YARN-7104 > URL: https://issues.apache.org/jira/browse/YARN-7104 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Da Ding >Assignee: Da Ding > Attachments: Screen Shot 2017-08-25 at 8.26.16 PM.png, Screen Shot > 2017-08-29 at 4.43.25 PM.png, Screen Shot 2017-08-31 at 8.58.36 PM.png, > Screen Shot 2017-08-31 at 8.58.43 PM.png, yarn-7104.001.patch, > yarn-7104.002.patch, yarn-7104.003.patch > > > 1. Changed color gradient of heat map > 2. Modified the tooltip style to have better UI > 3. Adjusted display style of reck -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7104) Improvement of Heatmap
[ https://issues.apache.org/jira/browse/YARN-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Ding updated YARN-7104: -- Attachment: yarn-7104.003.patch > Improvement of Heatmap > -- > > Key: YARN-7104 > URL: https://issues.apache.org/jira/browse/YARN-7104 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Da Ding >Assignee: Da Ding > Attachments: Screen Shot 2017-08-25 at 8.26.16 PM.png, Screen Shot > 2017-08-29 at 4.43.25 PM.png, Screen Shot 2017-08-31 at 8.58.36 PM.png, > Screen Shot 2017-08-31 at 8.58.43 PM.png, yarn-7104.001.patch, > yarn-7104.002.patch, yarn-7104.003.patch > > > 1. Changed color gradient of heat map > 2. Modified the tooltip style to have better UI > 3. Adjusted display style of reck -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6855) CLI Proto Modifications to support Node Attributes
[
https://issues.apache.org/jira/browse/YARN-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-6855:
Attachment: YARN-6855-YARN-3409.006.patch
As per offline discussion with [~sunilg], have rebased the {{YARN-3409}} branch
to the latest trunk code and rebased the patch again.
Thanks [~bibinchundatt] for the review comments ! have addressed it though in
most of the other PBImpls do the same, because though {{map.addAll}} throws
exception we are clearing the map in the previous step which leaves it in
Inconsistent state.
> CLI Proto Modifications to support Node Attributes
> --
>
> Key: YARN-6855
> URL: https://issues.apache.org/jira/browse/YARN-6855
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, capacityscheduler, client
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-6855-YARN-3409.001.patch,
> YARN-6855-YARN-3409.002.patch, YARN-6855-YARN-3409.003.patch,
> YARN-6855-YARN-3409.004.patch, YARN-6855-YARN-3409.005.patch,
> YARN-6855-YARN-3409.006.patch
>
>
> This jira focuses only on the proto modifications required for the CLI
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7150) Yarn crash [max number of completed apps kept in memory met]
[
https://issues.apache.org/jira/browse/YARN-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151883#comment-16151883
]
anikad ayman commented on YARN-7150:
Thank you [~templedf] for your answer :)
In fact, the problem is more complicated just after this message in the log,
all the processing craches => That means the two jobs were running very slowly
for long time ( more than 7 hours) without incresming in the state, and
several jobs was in waiting state => Just after killing the 2 running jobs, all
the jobs started to run and ended very fastly.
We had this issue several times before ( at least 1 time in every mounth )
thing that makes a lot of problems in our production envirenment !
Do you think that increasing maxCompletedAppsInStateStore will solve the
problem ?
> Yarn crash [max number of completed apps kept in memory met]
>
>
> Key: YARN-7150
> URL: https://issues.apache.org/jira/browse/YARN-7150
> Project: Hadoop YARN
> Issue Type: Bug
> Environment: Production
>Reporter: anikad ayman
>
> During MapReduce processing of several jobs, Yarn did crash and the
> processing of jobs had stopped.
> I successed to back the processing after killing jobs which were running
> (2jobs).
> In the logs I find from the beginning of the crash :
> {code:java}
> *Max number of completed apps kept in state store met*:
> maxCompletedAppsInStateStore = 1, removing app
> application_1500982512144_26754 from state store.
> 2017-08-25 03:50:05,799 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: *Application
> should be expired, max number of completed apps kept in memory met*:
> maxCompletedAppsInMemory = 1, removing app
> application_1500982512144_26754 from memory
> {code}
> After that , this message shows up several times in the log :
> {code:java}
> Large response size 4742320 for call
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications
> {code}
> Have you any explication and solution of this issue ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151842#comment-16151842 ] YunFan Zhou commented on YARN-7107: --- Hi, [~templedf] I think we can split the queue into three types: 1. Normal queue(The queue we currently use). 2. Disabled queue(The attributes are exactly the same as you think). 3. Exclusive queue(Make all sibling queues disabled). I think this can perfectly meet my requirements, What do you think? > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > Attachments: YARN-7107.001.preview.patch > > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151841#comment-16151841 ] YunFan Zhou commented on YARN-7107: --- Thank [~templedf] for your review, I benefited a lot from your advice. Your solution is so perfect for: * Do not allocate resources to disabled queue. * The disabled status of the queue can be viewed with the CLI way. Basically, can meet our requirements. But the current solution is more difficult to solve the problem we face in our production environment. Because in our production, we have hundreds of queue, but in the emergency case, we only want to make sure a few queues (because of this few queues be used to produce important reports) can be normally assigned resources. If we use the way you suggest, we have to set the remaining hundreds of queues to be disabled. This makes the operation very cumbersome. We need a better way to solve this problem， what do you think? > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > Attachments: YARN-7107.001.preview.patch > > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[
https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151833#comment-16151833
]
Daniel Templeton commented on YARN-7107:
Thanks for the patch. I love that you started with unit tests. We got off on
a tangent about override queues (my fault) and never got around to discussing
how to implement disabling a queue. To my understanding, disabling a queue
should still allow applications to be submitted to the queue, but it should
prevent them from running, even if the cluster is otherwise idle.
The patch you've posted almost does that, except that the jobs in disabled
queues will run if there's nothing else in the cluster that wants resources.
Instead of turning down the min share and weight (which will also cause
reporting to look odd), I'd add a new property to {{FSQueue}} called
{{disabled}}. In {{FSQueue.assignContainerPreCheck()}}, I'd immediately return
false is {{disabled}} is {{true}}. In {{FSQueue.getQueueInfo()}}, I'd also add
{{disabled}} to the reported state. As a performance optimization, I'd also
modify {{FSParentQueue}} to keep a disabled queues in a list separate from
{{childQueues}}, and then in {{FSParentQueue.getQueueInfo()}}, I'd iterate
through both {{childQueues}} and the list of disabled queues. I think that's
all that would be needed in order to completely shutdown execution from the
disabled queues. (Actually, the {{disabled}} is probably overkill, but it's
nice to have it for reporting.)
What do you think?
> add ability in Fair Scheduler to configure whether disable a queue
> --
>
> Key: YARN-7107
> URL: https://issues.apache.org/jira/browse/YARN-7107
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: YunFan Zhou
>Assignee: YunFan Zhou
> Labels: fairscheduler
> Attachments: YARN-7107.001.preview.patch
>
>
> In a production environment, emergency situations (such as the need to
> calculate the important reports) as soon as possible we need to disable all
> other queues, only allows the *RM* 's resources assigned to emergency queue
> and other queue only at the end of the urgent tasks before allowing them to
> be scheduled properly.
> At present, our approach is to write a script, in the case of an emergency
> manual changes all other queues' *minResources *and *maxResources * to *0mb,
> 0vcores* and then rebase it.This is very troublesome and easy to make
> mistakes.
> So we need to add a configuration in the *FairScheduler* configuration to
> indicate whether the queue is disabled, and if it is disabled, then *RM *will
> not allocate resources to the queue.
> * The child queue will integrate this property of the parent queue.
> * If the child queue is configured with this property, the value of the child
> queue configuration overrides the attributes of the parent queue.
> * The default value of the root queue is *enabled*.
> This will satisfy our needs, and I think other users will encounter such a
> scenario.I think this is very applicable to everyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[
https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151830#comment-16151830
]
Hadoop QA commented on YARN-7107:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-7107 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7107 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885140/YARN-7107.001.preview.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17272/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> add ability in Fair Scheduler to configure whether disable a queue
> --
>
> Key: YARN-7107
> URL: https://issues.apache.org/jira/browse/YARN-7107
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: YunFan Zhou
>Assignee: YunFan Zhou
> Labels: fairscheduler
> Attachments: YARN-7107.001.preview.patch
>
>
> In a production environment, emergency situations (such as the need to
> calculate the important reports) as soon as possible we need to disable all
> other queues, only allows the *RM* 's resources assigned to emergency queue
> and other queue only at the end of the urgent tasks before allowing them to
> be scheduled properly.
> At present, our approach is to write a script, in the case of an emergency
> manual changes all other queues' *minResources *and *maxResources * to *0mb,
> 0vcores* and then rebase it.This is very troublesome and easy to make
> mistakes.
> So we need to add a configuration in the *FairScheduler* configuration to
> indicate whether the queue is disabled, and if it is disabled, then *RM *will
> not allocate resources to the queue.
> * The child queue will integrate this property of the parent queue.
> * If the child queue is configured with this property, the value of the child
> queue configuration overrides the attributes of the parent queue.
> * The default value of the root queue is *enabled*.
> This will satisfy our needs, and I think other users will encounter such a
> scenario.I think this is very applicable to everyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151827#comment-16151827 ] YunFan Zhou commented on YARN-7107: --- [~templedf] Please help me review my code, thank you. > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > Attachments: YARN-7107.001.preview.patch > > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YunFan Zhou updated YARN-7107: -- Attachment: (was: YARN-7107.001.preview.patch) > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > Attachments: YARN-7107.001.preview.patch > > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YunFan Zhou updated YARN-7107: -- Attachment: YARN-7107.001.preview.patch > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > Attachments: YARN-7107.001.preview.patch, YARN-7107.001.preview.patch > > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7150) Yarn crash [max number of completed apps kept in memory met]
[
https://issues.apache.org/jira/browse/YARN-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151825#comment-16151825
]
Daniel Templeton commented on YARN-7150:
That's expected behavior. YARN has a configurable maximum number of jobs it
will keep in memory and recover from the state store after a failure. Once
that limit is exceeded, you can find the applications that are no longer
available from the resource manager in the application history server. Is
there some other issue other than the log about hitting the max number of jobs?
If not, I'll close this as not an issue.
> Yarn crash [max number of completed apps kept in memory met]
>
>
> Key: YARN-7150
> URL: https://issues.apache.org/jira/browse/YARN-7150
> Project: Hadoop YARN
> Issue Type: Bug
> Environment: Production
>Reporter: anikad ayman
>
> During MapReduce processing of several jobs, Yarn did crash and the
> processing of jobs had stopped.
> I successed to back the processing after killing jobs which were running
> (2jobs).
> In the logs I find from the beginning of the crash :
> {code:java}
> *Max number of completed apps kept in state store met*:
> maxCompletedAppsInStateStore = 1, removing app
> application_1500982512144_26754 from state store.
> 2017-08-25 03:50:05,799 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: *Application
> should be expired, max number of completed apps kept in memory met*:
> maxCompletedAppsInMemory = 1, removing app
> application_1500982512144_26754 from memory
> {code}
> After that , this message shows up several times in the log :
> {code:java}
> Large response size 4742320 for call
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications
> {code}
> Have you any explication and solution of this issue ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[
https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151823#comment-16151823
]
Hadoop QA commented on YARN-7107:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-7107 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7107 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885138/YARN-7107.001.preview.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17271/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> add ability in Fair Scheduler to configure whether disable a queue
> --
>
> Key: YARN-7107
> URL: https://issues.apache.org/jira/browse/YARN-7107
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: YunFan Zhou
>Assignee: YunFan Zhou
> Labels: fairscheduler
> Attachments: YARN-7107.001.preview.patch
>
>
> In a production environment, emergency situations (such as the need to
> calculate the important reports) as soon as possible we need to disable all
> other queues, only allows the *RM* 's resources assigned to emergency queue
> and other queue only at the end of the urgent tasks before allowing them to
> be scheduled properly.
> At present, our approach is to write a script, in the case of an emergency
> manual changes all other queues' *minResources *and *maxResources * to *0mb,
> 0vcores* and then rebase it.This is very troublesome and easy to make
> mistakes.
> So we need to add a configuration in the *FairScheduler* configuration to
> indicate whether the queue is disabled, and if it is disabled, then *RM *will
> not allocate resources to the queue.
> * The child queue will integrate this property of the parent queue.
> * If the child queue is configured with this property, the value of the child
> queue configuration overrides the attributes of the parent queue.
> * The default value of the root queue is *enabled*.
> This will satisfy our needs, and I think other users will encounter such a
> scenario.I think this is very applicable to everyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YunFan Zhou updated YARN-7107: -- Attachment: YARN-7107.001.preview.patch > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > Attachments: YARN-7107.001.preview.patch > > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YunFan Zhou updated YARN-7107: -- Attachment: (was: YARN-7107.001.preview.patch) > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > Attachments: YARN-7107.001.preview.patch > > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6523) RM requires large memory in sending out security tokens as part of Node Heartbeat in large cluster
[ https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151817#comment-16151817 ] Manikandan R commented on YARN-6523: [~Naganarasimha] I made an attempt to understand the discussion in this JIRA and come up with below steps to address this. Please review and share your thoughts. Can I work on this? A) Sequence No flow: 1. Introduce a atomic long variable in RMContextImpl to hold this sequence no 2. Make sure the above sequence no current value passed to Nodes as part of node registration process through response 3. Increment above sequence no as and when there is any update in delegation tokens, specifically in DelegationTokenRenewer#requestNewHdfsDelegationTokenAsProxyUser 4. ResourceTrackerService#nodeHeartbeat would use the above sequence number to decide whether to update SystemCredentialsForApps in NodeHeartbeatResponse or not by comparing it with the number received as part of NodeHeartbeatRequest. 5. NodeHeartbeatRequest will start having sequence no as part of the request. This requires a change in corresponding proto class. 6. NodeHeartbeatResponse will start having sequence no as part of the response. This requires a change in corresponding proto class. B) Caching system credentials objects Will go through the code and share my proposal to achieve the same. > RM requires large memory in sending out security tokens as part of Node > Heartbeat in large cluster > -- > > Key: YARN-6523 > URL: https://issues.apache.org/jira/browse/YARN-6523 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Affects Versions: 2.8.0, 2.7.3 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Critical > > Currently as part of heartbeat response RM sets all application's tokens > though all applications might not be active on the node. On top of it > NodeHeartbeatResponsePBImpl converts tokens for each app into > SystemCredentialsForAppsProto. Hence for each node and each heartbeat too > many SystemCredentialsForAppsProto objects were getting created. > We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with > 8GB RAM configured for RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7150) Yarn crash [max number of completed apps kept in memory met]
[
https://issues.apache.org/jira/browse/YARN-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anikad ayman updated YARN-7150:
---
Description:
During MapReduce processing of several jobs, Yarn did crash and the processing
of jobs had stopped.
I successed to back the processing after killing jobs which were running
(2jobs).
In the logs I find from the beginning of the crash :
{code:java}
*Max number of completed apps kept in state store met*:
maxCompletedAppsInStateStore = 1, removing app
application_1500982512144_26754 from state store.
2017-08-25 03:50:05,799 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: *Application should
be expired, max number of completed apps kept in memory met*:
maxCompletedAppsInMemory = 1, removing app application_1500982512144_26754
from memory
{code}
After that , this message shows up several times in the log :
{code:java}
Large response size 4742320 for call
org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications
{code}
Have you any explication and solution of this issue ?
was:
During MapReduce processing of several jobs, Yarn did crash and the processing
of jobs had stopped.
I successed to back the processing after killing jobs which were running
(2jobs).
In the logs I find from the beginning of the crash :
{code:java}
*Max number of completed apps kept in state store met*:
maxCompletedAppsInStateStore = 1, removing app
application_1500982512144_26754 from state store.
2017-08-25 03:50:05,799 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: *Application should
be expired, max number of completed apps kept in memory met*:
maxCompletedAppsInMemory = 1, removing app application_1500982512144_26754
from memory
{code}
After that , this messages shows up several times in the log :
{code:java}
Large response size 4742320 for call
org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications
{code}
Have you any explication and solution of this issue ?
> Yarn crash [max number of completed apps kept in memory met]
>
>
> Key: YARN-7150
> URL: https://issues.apache.org/jira/browse/YARN-7150
> Project: Hadoop YARN
> Issue Type: Bug
> Environment: Production
>Reporter: anikad ayman
>
> During MapReduce processing of several jobs, Yarn did crash and the
> processing of jobs had stopped.
> I successed to back the processing after killing jobs which were running
> (2jobs).
> In the logs I find from the beginning of the crash :
> {code:java}
> *Max number of completed apps kept in state store met*:
> maxCompletedAppsInStateStore = 1, removing app
> application_1500982512144_26754 from state store.
> 2017-08-25 03:50:05,799 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: *Application
> should be expired, max number of completed apps kept in memory met*:
> maxCompletedAppsInMemory = 1, removing app
> application_1500982512144_26754 from memory
> {code}
> After that , this message shows up several times in the log :
> {code:java}
> Large response size 4742320 for call
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications
> {code}
> Have you any explication and solution of this issue ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7150) Yarn crash [max number of completed apps kept in memory met]
anikad ayman created YARN-7150:
--
Summary: Yarn crash [max number of completed apps kept in memory
met]
Key: YARN-7150
URL: https://issues.apache.org/jira/browse/YARN-7150
Project: Hadoop YARN
Issue Type: Bug
Environment: Production
Reporter: anikad ayman
During MapReduce processing of several jobs, Yarn did crash and the processing
of jobs had stopped.
I successed to back the processing after killing jobs which were running
(2jobs).
In the logs I find from the beginning of the crash :
{code:java}
*Max number of completed apps kept in state store met*:
maxCompletedAppsInStateStore = 1, removing app
application_1500982512144_26754 from state store.
2017-08-25 03:50:05,799 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: *Application should
be expired, max number of completed apps kept in memory met*:
maxCompletedAppsInMemory = 1, removing app application_1500982512144_26754
from memory
{code}
After that , this messages shows up several times in the log :
{code:java}
Large response size 4742320 for call
org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications
{code}
Have you any explication and solution of this issue ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-65) Reduce RM app memory footprint once app has completed
[
https://issues.apache.org/jira/browse/YARN-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151704#comment-16151704
]
Naganarasimha G R commented on YARN-65:
---
Agree with [~rohithsharma],
We should not modify each individual test class and the approach will be prone
to have errors when new test cases are added.
So i would suggest to clone (either using clone method or having explicit code)
{{appState.setApplicationSubmissionContext}} inside {{MemoryRMStateStore}}
while adding or updating the ApplicationStateData.
> Reduce RM app memory footprint once app has completed
> -
>
> Key: YARN-65
> URL: https://issues.apache.org/jira/browse/YARN-65
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 0.23.3
>Reporter: Jason Lowe
>Assignee: Manikandan R
> Attachments: YARN-65.001.patch, YARN-65.002.patch, YARN-65.003.patch,
> YARN-65.004.patch, YARN-65.005.patch, YARN-65.006.patch, YARN-65.007.patch,
> YARN-65.008.patch, YARN-65.009.patch, YARN-65.010.patch
>
>
> The ResourceManager holds onto a configurable number of completed
> applications (yarn.resource.max-completed-applications, defaults to 1),
> and the memory footprint of these completed applications can be significant.
> For example, the {{submissionContext}} in RMAppImpl contains references to
> protocolbuffer objects and other items that probably aren't necessary to keep
> around once the application has completed. We could significantly reduce the
> memory footprint of the RM by releasing objects that are no longer necessary
> once an application completes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
