[jira] [Created] (YARN-7076) yarn application -list -appTypes is not working
Jian He created YARN-7076: - Summary: yarn application -list -appTypes is not working Key: YARN-7076 URL: https://issues.apache.org/jira/browse/YARN-7076 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Priority: Blocker Fix For: 2.9.0, 3.0.0-beta1 yarn application -list -appTypes is not working Looks like it's because the ApplicationCLI pass in the appType as uppercase, but ClientRMService#getApplications is case sensitive, so if user submits an app with lowercase appType, it wont work -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137934#comment-16137934
]
Varun Saxena edited comment on YARN-5355 at 8/23/17 5:45 AM:
-
bq. Once merge vote is finishes, we can rebase again and run jenkins before
merge.
Yeah that's the plan. We may have to rebase multiple times before final merge
as even during these 7 days, conflicts may emerge.
bq. I would suggest not to run tests again and again because everyday one or
more commits will be keep happening which causes build failure. However we have
got one report with couple of test failures.
The last build was invoked because I had fixed checkstyle and whitespace issues.
was (Author: varun_saxena):
bq. Once merge vote is finishes, we can rebase again and run jenkins before
merge.
Yeah that's the plan. We may have to rebase multiple times before final merge
as even during these 7 days, conflicts may emerge.
> YARN Timeline Service v.2: alpha 2
> --
>
> Key: YARN-5355
> URL: https://issues.apache.org/jira/browse/YARN-5355
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>Priority: Critical
> Attachments: Timeline Service v2_ Ideas for Next Steps.pdf,
> YARN-5355.01.patch, YARN-5355.02.patch, YARN-5355.03.patch,
> YARN-5355-branch-2.01.patch
>
>
> This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service
> v.2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any
> subtask work on this JIRA will be committed to those 2 branches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137934#comment-16137934
]
Varun Saxena commented on YARN-5355:
bq. Once merge vote is finishes, we can rebase again and run jenkins before
merge.
Yeah that's the plan. We may have to rebase multiple times before final merge
as even during these 7 days, conflicts may emerge.
> YARN Timeline Service v.2: alpha 2
> --
>
> Key: YARN-5355
> URL: https://issues.apache.org/jira/browse/YARN-5355
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>Priority: Critical
> Attachments: Timeline Service v2_ Ideas for Next Steps.pdf,
> YARN-5355.01.patch, YARN-5355.02.patch, YARN-5355.03.patch,
> YARN-5355-branch-2.01.patch
>
>
> This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service
> v.2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any
> subtask work on this JIRA will be committed to those 2 branches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137924#comment-16137924
]
Varun Saxena commented on YARN-5355:
Seems to be some problem. Let me reinvoke the build
> YARN Timeline Service v.2: alpha 2
> --
>
> Key: YARN-5355
> URL: https://issues.apache.org/jira/browse/YARN-5355
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>Priority: Critical
> Attachments: Timeline Service v2_ Ideas for Next Steps.pdf,
> YARN-5355.01.patch, YARN-5355.02.patch, YARN-5355.03.patch,
> YARN-5355-branch-2.01.patch
>
>
> This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service
> v.2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any
> subtask work on this JIRA will be committed to those 2 branches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137923#comment-16137923
]
Rohith Sharma K S commented on YARN-5355:
-
I would suggest not to run tests again and again because everyday one or more
commits will be keep happening which causes build failure. However we have got
one report with couple of test failures.
Once merge vote is finishes, we can rebase again and run jenkins before merge.
> YARN Timeline Service v.2: alpha 2
> --
>
> Key: YARN-5355
> URL: https://issues.apache.org/jira/browse/YARN-5355
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>Priority: Critical
> Attachments: Timeline Service v2_ Ideas for Next Steps.pdf,
> YARN-5355.01.patch, YARN-5355.02.patch, YARN-5355.03.patch,
> YARN-5355-branch-2.01.patch
>
>
> This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service
> v.2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any
> subtask work on this JIRA will be committed to those 2 branches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137867#comment-16137867
]
Hadoop QA commented on YARN-5355:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 59 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 12m
52s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
53s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
38s{color} | {color:red} hadoop-yarn-server-applicationhistoryservice in the
patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
33s{color} | {color:red} hadoop-yarn-server-timelineservice in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
45s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
32s{color} | {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
39s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
28s{color} | {color:red} hadoop-yarn-server-timelineservice-hbase in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
28s{color} | {color:red} hadoop-yarn-server-timelineservice-hbase-tests in the
patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
32s{color} | {color:red} hadoop-yarn-applications-distributedshell in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
43s{color} | {color:red} hadoop-mapreduce-client-core in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
31s{color} | {color:red} hadoop-mapreduce-client-app in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
40s{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 16m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
4m 4s{color} |
[jira] [Commented] (YARN-7050) Post cleanup after YARN-6903
[
https://issues.apache.org/jira/browse/YARN-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137834#comment-16137834
]
Hadoop QA commented on YARN-7050:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 119 new or modified
test files. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
48s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
53s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
11s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
3s{color} | {color:green} yarn-native-services passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
4s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
in yarn-native-services has 7 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
56s{color} | {color:red} hadoop-yarn in yarn-native-services failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-slider in yarn-native-services failed.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services
failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
27s{color} | {color:green} root generated 0 new + 1313 unchanged - 27 fixed =
1313 total (was 1340) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 35s{color} | {color:orange} root: The patch generated 742 new + 415
unchanged - 3239 fixed = 1157 total (was 3654) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} pylint {color} | {color:green} 0m
2s{color} | {color:green} The patch generated 0 new + 0 unchanged - 110 fixed =
0 total (was 110) {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
24s{color} | {color:green} The patch generated 0 new + 0 unchanged - 1 fixed =
0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
12s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 84 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-yarn-project/hadoop-yarn
[jira] [Updated] (YARN-7075) [YARN-3368] Improvement of Web UI
[ https://issues.apache.org/jira/browse/YARN-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Ding updated YARN-7075: -- Attachment: Screen Shot 2017-08-22 at 8.36.07 PM.png Preview > [YARN-3368] Improvement of Web UI > -- > > Key: YARN-7075 > URL: https://issues.apache.org/jira/browse/YARN-7075 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Da Ding >Assignee: Da Ding > Attachments: Screen Shot 2017-08-22 at 8.36.07 PM.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7075) [YARN-3368] Improvement of Web UI
Da Ding created YARN-7075: - Summary: [YARN-3368] Improvement of Web UI Key: YARN-7075 URL: https://issues.apache.org/jira/browse/YARN-7075 Project: Hadoop YARN Issue Type: Sub-task Reporter: Da Ding Assignee: Da Ding -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6640) AM heartbeat stuck when responseId overflows MAX_INT
[
https://issues.apache.org/jira/browse/YARN-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137825#comment-16137825
]
Botong Huang commented on YARN-6640:
Hi [~wangda], I think [~jlowe]'s approach already handles the case
request.responseId > lastResponseId. I will update the patch soon. Thanks for
the comments!
> AM heartbeat stuck when responseId overflows MAX_INT
> -
>
> Key: YARN-6640
> URL: https://issues.apache.org/jira/browse/YARN-6640
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Blocker
> Attachments: YARN-6640.v1.patch
>
>
> The current code in {{ApplicationMasterService}}:
> if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {/* old
> heartbeat */ return lastResponse;}
> else if (request.getResponseId() + 1 < lastResponse.getResponseId()) { throw
> ... }
> process the heartbeat...
> When a heartbeat comes in, in usual case we are expecting
> request.getResponseId() == lastResponse.getResponseId(). The “if“ is for the
> duplicate heartbeat that’s one step old, the “else if” is to throw and
> complain for heartbeats more than two steps old, otherwise we accept the new
> heartbeat and process it.
> So the bug is: when lastResponse.getResponseId() == MAX_INT, the newest
> heartbeat comes in with responseId == MAX_INT. However reponseId + 1 will be
> MIN_INT, and we will fall into the “else if” case and RM will throw. Then we
> are stuck here…
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7053) Move curator transaction support to ZKCuratorManager
[ https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137805#comment-16137805 ] Jonathan Hung commented on YARN-7053: - Thanks [~subru] for the commit/reviews! > Move curator transaction support to ZKCuratorManager > > > Key: YARN-7053 > URL: https://issues.apache.org/jira/browse/YARN-7053 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-7053.001.patch, YARN-7053.002.patch, > YARN-7053.003.patch, YARN-7053.004.patch, YARN-7053.005.patch > > > HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore > has some curator transaction support which can be reused, so this can be > moved to ZKCuratorManager as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6251) Add support for async Container release to prevent deadlock during container updates
[ https://issues.apache.org/jira/browse/YARN-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137804#comment-16137804 ] Wangda Tan commented on YARN-6251: -- Committing, thanks Arun. > Add support for async Container release to prevent deadlock during container > updates > > > Key: YARN-6251 > URL: https://issues.apache.org/jira/browse/YARN-6251 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6251.001.patch, YARN-6251.002.patch, > YARN-6251.003.patch, YARN-6251.004.patch, YARN-6251.005.patch, > YARN-6251.006.patch, YARN-6251.007.patch, YARN-6251.008.patch > > > Opening to track a locking issue that was uncovered when running a custom SLS > AMSimulator. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6640) AM heartbeat stuck when responseId overflows MAX_INT
[
https://issues.apache.org/jira/browse/YARN-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137803#comment-16137803
]
Wangda Tan commented on YARN-6640:
--
Agree with [~jlowe] provided approach.
Jason could you also share your thoughts for below question?
bq. Another potential problem is, we only fail when request.responseId <
lastResponse.responseId - 1, I think we should also fail when
request.responseId > lastResponseId.
> AM heartbeat stuck when responseId overflows MAX_INT
> -
>
> Key: YARN-6640
> URL: https://issues.apache.org/jira/browse/YARN-6640
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Blocker
> Attachments: YARN-6640.v1.patch
>
>
> The current code in {{ApplicationMasterService}}:
> if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {/* old
> heartbeat */ return lastResponse;}
> else if (request.getResponseId() + 1 < lastResponse.getResponseId()) { throw
> ... }
> process the heartbeat...
> When a heartbeat comes in, in usual case we are expecting
> request.getResponseId() == lastResponse.getResponseId(). The “if“ is for the
> duplicate heartbeat that’s one step old, the “else if” is to throw and
> complain for heartbeats more than two steps old, otherwise we accept the new
> heartbeat and process it.
> So the bug is: when lastResponse.getResponseId() == MAX_INT, the newest
> heartbeat comes in with responseId == MAX_INT. However reponseId + 1 will be
> MIN_INT, and we will fall into the “else if” case and RM will throw. Then we
> are stuck here…
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7053) Move curator transaction support to ZKCuratorManager
[ https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137801#comment-16137801 ] Hudson commented on YARN-7053: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12229 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12229/]) YARN-7053. Move curator transaction support to ZKCuratorManager. (subru: rev 4249172e1419acdb2b69ae3db43dc59da2aa2e03) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestZKCuratorManager.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java > Move curator transaction support to ZKCuratorManager > > > Key: YARN-7053 > URL: https://issues.apache.org/jira/browse/YARN-7053 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-7053.001.patch, YARN-7053.002.patch, > YARN-7053.003.patch, YARN-7053.004.patch, YARN-7053.005.patch > > > HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore > has some curator transaction support which can be reused, so this can be > moved to ZKCuratorManager as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7053) Move curator transaction support to ZKCuratorManager
[
https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137765#comment-16137765
]
Jonathan Hung commented on YARN-7053:
-
{{TestSFTPFileSystem.testGetAccessTime}},
{{TestOpportunisticContainerAllocatorAMService.testContainerAutoUpdateContainer}},
{{TestSchedulingMonitor.testRMStarts}},
{{TestFSAppStarvation.testPreemptionEnabled}} pass locally for me.
{{TestDelegationTokenRenewer.testCancelWithMultipleAppSubmissions}}
and {{TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable}}
fails with or without the change.
> Move curator transaction support to ZKCuratorManager
>
>
> Key: YARN-7053
> URL: https://issues.apache.org/jira/browse/YARN-7053
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-7053.001.patch, YARN-7053.002.patch,
> YARN-7053.003.patch, YARN-7053.004.patch, YARN-7053.005.patch
>
>
> HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore
> has some curator transaction support which can be reused, so this can be
> moved to ZKCuratorManager as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7070) some of local cache files for yarn can't be deleted
[ https://issues.apache.org/jira/browse/YARN-7070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Changyao Ye updated YARN-7070: -- Attachment: application_1501810184023_55949.log > some of local cache files for yarn can't be deleted > --- > > Key: YARN-7070 > URL: https://issues.apache.org/jira/browse/YARN-7070 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.1 > Environment: Hadoop 2.8.1 >Reporter: Changyao Ye > Attachments: application_1501810184023_55949.log > > > We have found some of cache files(in > /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache) for yarn on > nodemanager cannot be deleted properly. The directories are like > following(blockmgr***) > = > # ls -ltr application_1501810184023_55949 > total 120 > drwx--x--- 2 hdfs yarn 4096 Aug 22 04:29 filecache > drwxr-s--- 2 hdfs yarn 4096 Aug 22 04:56 > blockmgr-881fab2c-fba4-4bb1-8dd9-5ab35a512df7 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 04:56 > blockmgr-bf8a19f5-e9ae-4269-a0ef-b27d0f9c17e7 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 04:58 > blockmgr-f3437e8d-9595-4898-8bda-92ebff3ada1d > drwxr-s--- 18 hdfs yarn 4096 Aug 22 05:01 > blockmgr-930c0cd8-1d31-4cdb-a244-f6ad4bf74bff > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:13 > blockmgr-83fc0702-ac40-4743-812a-7d488e92004e > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:13 > blockmgr-f6cfe045-12c3-41d6-b77e-aa5200daeb6a > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:13 > blockmgr-53dcb4ea-ba5d-4b8b-859b-805b9303a149 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-0c0c4bb9-ef5e-4ca1-8d23-ce5cd58d0a75 > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:13 > blockmgr-557d0f39-67d2-491a-9307-12fc1d724380 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-fbc87680-4df7-498e-bf6d-456a5aea4fc9 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-53ee8251-fac1-4f62-82c2-5e970f0d86ec > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:14 > blockmgr-5a8bc187-abcf-482d-9da5-e8c4647d4731 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:14 > blockmgr-251c3a99-cd85-442a-8945-52c344c0d861 > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:14 > blockmgr-c352c1ad-15dc-456b-8b62-5b83b9950494 > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:15 > blockmgr-b4f01347-4b51-4b35-8146-2aa840084c2b > drwxr-s--- 14 hdfs yarn 4096 Aug 22 05:15 > blockmgr-0095d26c-c134-48b4-82a6-e8ae02f0189c > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:15 > blockmgr-28a31574-61ae-459f-be3a-8608892246d7 > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-c0cd0df9-b355-4209-b6aa-b549a1fa36eb > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-a2730abb-9517-461e-bedf-d9a2dcef373f > drwxr-s--- 14 hdfs yarn 4096 Aug 22 05:15 > blockmgr-91dd2e1a-6bc2-4429-8b71-2f4240987159 > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:15 > blockmgr-f4e3a586-8817-45ea-a197-9fdbb3d91946 > drwxr-s--- 15 hdfs yarn 4096 Aug 22 05:15 > blockmgr-ba2c605e-89d8-4f7c-b42c-6ed4ba6bf4ea > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-2ae72383-5f72-4002-84a7-e6335b8c2b6c > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:15 > blockmgr-6c5e260f-d3c7-4af6-91c1-168c73343f2d > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-2e9923b1-281c-4a9d-8069-6c5430bd5fc3 > drwxr-s--- 18 hdfs yarn 4096 Aug 22 05:15 > blockmgr-cc3f1406-d8a2-4bf5-a276-8f7aed75c513 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-975bcce0-84b2-4590-880b-bf182d76e319 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-ce82cb63-5998-4227-b85e-77f1c633db43 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-592af4aa-3c89-4081-8746-29b99f2220b1 > = > We also applied patches YARN-4594, YARN-4731, but nothing changed. > YARN-4594 https://issues.apache.org/jira/browse/YARN-4594 > YARN-4731 https://issues.apache.org/jira/browse/YARN-4731 > Any advice will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6251) Add support for async Container release to prevent deadlock during container updates
[ https://issues.apache.org/jira/browse/YARN-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137755#comment-16137755 ] Arun Suresh commented on YARN-6251: --- The remaining test case failures are unrelated.. > Add support for async Container release to prevent deadlock during container > updates > > > Key: YARN-6251 > URL: https://issues.apache.org/jira/browse/YARN-6251 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6251.001.patch, YARN-6251.002.patch, > YARN-6251.003.patch, YARN-6251.004.patch, YARN-6251.005.patch, > YARN-6251.006.patch, YARN-6251.007.patch, YARN-6251.008.patch > > > Opening to track a locking issue that was uncovered when running a custom SLS > AMSimulator. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7070) some of local cache files for yarn can't be deleted
[ https://issues.apache.org/jira/browse/YARN-7070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Changyao Ye updated YARN-7070: -- Attachment: (was: application_1501810184023_55949.log) > some of local cache files for yarn can't be deleted > --- > > Key: YARN-7070 > URL: https://issues.apache.org/jira/browse/YARN-7070 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.1 > Environment: Hadoop 2.8.1 >Reporter: Changyao Ye > > We have found some of cache files(in > /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache) for yarn on > nodemanager cannot be deleted properly. The directories are like > following(blockmgr***) > = > # ls -ltr application_1501810184023_55949 > total 120 > drwx--x--- 2 hdfs yarn 4096 Aug 22 04:29 filecache > drwxr-s--- 2 hdfs yarn 4096 Aug 22 04:56 > blockmgr-881fab2c-fba4-4bb1-8dd9-5ab35a512df7 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 04:56 > blockmgr-bf8a19f5-e9ae-4269-a0ef-b27d0f9c17e7 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 04:58 > blockmgr-f3437e8d-9595-4898-8bda-92ebff3ada1d > drwxr-s--- 18 hdfs yarn 4096 Aug 22 05:01 > blockmgr-930c0cd8-1d31-4cdb-a244-f6ad4bf74bff > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:13 > blockmgr-83fc0702-ac40-4743-812a-7d488e92004e > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:13 > blockmgr-f6cfe045-12c3-41d6-b77e-aa5200daeb6a > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:13 > blockmgr-53dcb4ea-ba5d-4b8b-859b-805b9303a149 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-0c0c4bb9-ef5e-4ca1-8d23-ce5cd58d0a75 > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:13 > blockmgr-557d0f39-67d2-491a-9307-12fc1d724380 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-fbc87680-4df7-498e-bf6d-456a5aea4fc9 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-53ee8251-fac1-4f62-82c2-5e970f0d86ec > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:14 > blockmgr-5a8bc187-abcf-482d-9da5-e8c4647d4731 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:14 > blockmgr-251c3a99-cd85-442a-8945-52c344c0d861 > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:14 > blockmgr-c352c1ad-15dc-456b-8b62-5b83b9950494 > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:15 > blockmgr-b4f01347-4b51-4b35-8146-2aa840084c2b > drwxr-s--- 14 hdfs yarn 4096 Aug 22 05:15 > blockmgr-0095d26c-c134-48b4-82a6-e8ae02f0189c > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:15 > blockmgr-28a31574-61ae-459f-be3a-8608892246d7 > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-c0cd0df9-b355-4209-b6aa-b549a1fa36eb > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-a2730abb-9517-461e-bedf-d9a2dcef373f > drwxr-s--- 14 hdfs yarn 4096 Aug 22 05:15 > blockmgr-91dd2e1a-6bc2-4429-8b71-2f4240987159 > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:15 > blockmgr-f4e3a586-8817-45ea-a197-9fdbb3d91946 > drwxr-s--- 15 hdfs yarn 4096 Aug 22 05:15 > blockmgr-ba2c605e-89d8-4f7c-b42c-6ed4ba6bf4ea > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-2ae72383-5f72-4002-84a7-e6335b8c2b6c > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:15 > blockmgr-6c5e260f-d3c7-4af6-91c1-168c73343f2d > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-2e9923b1-281c-4a9d-8069-6c5430bd5fc3 > drwxr-s--- 18 hdfs yarn 4096 Aug 22 05:15 > blockmgr-cc3f1406-d8a2-4bf5-a276-8f7aed75c513 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-975bcce0-84b2-4590-880b-bf182d76e319 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-ce82cb63-5998-4227-b85e-77f1c633db43 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-592af4aa-3c89-4081-8746-29b99f2220b1 > = > We also applied patches YARN-4594, YARN-4731, but nothing changed. > YARN-4594 https://issues.apache.org/jira/browse/YARN-4594 > YARN-4731 https://issues.apache.org/jira/browse/YARN-4731 > Any advice will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7070) some of local cache files for yarn can't be deleted
[ https://issues.apache.org/jira/browse/YARN-7070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137740#comment-16137740 ] Changyao Ye commented on YARN-7070: --- Hi Lowe Thanks for your comments. I checked the log of the application again. It seems that the application finished successfully without any WARNs and ERRORs. I found that those blockmgr*** directories refer to shuffle.ExternalShuffleBlockResolver, maybe something wrong happened during the shuffle process. I attached the log file and It would be very appreciated if you could take a look at it. > some of local cache files for yarn can't be deleted > --- > > Key: YARN-7070 > URL: https://issues.apache.org/jira/browse/YARN-7070 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.1 > Environment: Hadoop 2.8.1 >Reporter: Changyao Ye > Attachments: application_1501810184023_55949.log > > > We have found some of cache files(in > /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache) for yarn on > nodemanager cannot be deleted properly. The directories are like > following(blockmgr***) > = > # ls -ltr application_1501810184023_55949 > total 120 > drwx--x--- 2 hdfs yarn 4096 Aug 22 04:29 filecache > drwxr-s--- 2 hdfs yarn 4096 Aug 22 04:56 > blockmgr-881fab2c-fba4-4bb1-8dd9-5ab35a512df7 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 04:56 > blockmgr-bf8a19f5-e9ae-4269-a0ef-b27d0f9c17e7 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 04:58 > blockmgr-f3437e8d-9595-4898-8bda-92ebff3ada1d > drwxr-s--- 18 hdfs yarn 4096 Aug 22 05:01 > blockmgr-930c0cd8-1d31-4cdb-a244-f6ad4bf74bff > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:13 > blockmgr-83fc0702-ac40-4743-812a-7d488e92004e > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:13 > blockmgr-f6cfe045-12c3-41d6-b77e-aa5200daeb6a > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:13 > blockmgr-53dcb4ea-ba5d-4b8b-859b-805b9303a149 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-0c0c4bb9-ef5e-4ca1-8d23-ce5cd58d0a75 > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:13 > blockmgr-557d0f39-67d2-491a-9307-12fc1d724380 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-fbc87680-4df7-498e-bf6d-456a5aea4fc9 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-53ee8251-fac1-4f62-82c2-5e970f0d86ec > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:14 > blockmgr-5a8bc187-abcf-482d-9da5-e8c4647d4731 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:14 > blockmgr-251c3a99-cd85-442a-8945-52c344c0d861 > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:14 > blockmgr-c352c1ad-15dc-456b-8b62-5b83b9950494 > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:15 > blockmgr-b4f01347-4b51-4b35-8146-2aa840084c2b > drwxr-s--- 14 hdfs yarn 4096 Aug 22 05:15 > blockmgr-0095d26c-c134-48b4-82a6-e8ae02f0189c > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:15 > blockmgr-28a31574-61ae-459f-be3a-8608892246d7 > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-c0cd0df9-b355-4209-b6aa-b549a1fa36eb > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-a2730abb-9517-461e-bedf-d9a2dcef373f > drwxr-s--- 14 hdfs yarn 4096 Aug 22 05:15 > blockmgr-91dd2e1a-6bc2-4429-8b71-2f4240987159 > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:15 > blockmgr-f4e3a586-8817-45ea-a197-9fdbb3d91946 > drwxr-s--- 15 hdfs yarn 4096 Aug 22 05:15 > blockmgr-ba2c605e-89d8-4f7c-b42c-6ed4ba6bf4ea > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-2ae72383-5f72-4002-84a7-e6335b8c2b6c > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:15 > blockmgr-6c5e260f-d3c7-4af6-91c1-168c73343f2d > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-2e9923b1-281c-4a9d-8069-6c5430bd5fc3 > drwxr-s--- 18 hdfs yarn 4096 Aug 22 05:15 > blockmgr-cc3f1406-d8a2-4bf5-a276-8f7aed75c513 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-975bcce0-84b2-4590-880b-bf182d76e319 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-ce82cb63-5998-4227-b85e-77f1c633db43 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-592af4aa-3c89-4081-8746-29b99f2220b1 > = > We also applied patches YARN-4594, YARN-4731, but nothing changed. > YARN-4594 https://issues.apache.org/jira/browse/YARN-4594 > YARN-4731 https://issues.apache.org/jira/browse/YARN-4731 > Any advice will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7070) some of local cache files for yarn can't be deleted
[ https://issues.apache.org/jira/browse/YARN-7070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Changyao Ye updated YARN-7070: -- Attachment: application_1501810184023_55949.log > some of local cache files for yarn can't be deleted > --- > > Key: YARN-7070 > URL: https://issues.apache.org/jira/browse/YARN-7070 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.1 > Environment: Hadoop 2.8.1 >Reporter: Changyao Ye > Attachments: application_1501810184023_55949.log > > > We have found some of cache files(in > /tmp/hadoop-yarn/nm-local-dir/usercache/hdfs/appcache) for yarn on > nodemanager cannot be deleted properly. The directories are like > following(blockmgr***) > = > # ls -ltr application_1501810184023_55949 > total 120 > drwx--x--- 2 hdfs yarn 4096 Aug 22 04:29 filecache > drwxr-s--- 2 hdfs yarn 4096 Aug 22 04:56 > blockmgr-881fab2c-fba4-4bb1-8dd9-5ab35a512df7 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 04:56 > blockmgr-bf8a19f5-e9ae-4269-a0ef-b27d0f9c17e7 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 04:58 > blockmgr-f3437e8d-9595-4898-8bda-92ebff3ada1d > drwxr-s--- 18 hdfs yarn 4096 Aug 22 05:01 > blockmgr-930c0cd8-1d31-4cdb-a244-f6ad4bf74bff > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:13 > blockmgr-83fc0702-ac40-4743-812a-7d488e92004e > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:13 > blockmgr-f6cfe045-12c3-41d6-b77e-aa5200daeb6a > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:13 > blockmgr-53dcb4ea-ba5d-4b8b-859b-805b9303a149 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-0c0c4bb9-ef5e-4ca1-8d23-ce5cd58d0a75 > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:13 > blockmgr-557d0f39-67d2-491a-9307-12fc1d724380 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-fbc87680-4df7-498e-bf6d-456a5aea4fc9 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:13 > blockmgr-53ee8251-fac1-4f62-82c2-5e970f0d86ec > drwxr-s--- 9 hdfs yarn 4096 Aug 22 05:14 > blockmgr-5a8bc187-abcf-482d-9da5-e8c4647d4731 > drwxr-s--- 10 hdfs yarn 4096 Aug 22 05:14 > blockmgr-251c3a99-cd85-442a-8945-52c344c0d861 > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:14 > blockmgr-c352c1ad-15dc-456b-8b62-5b83b9950494 > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:15 > blockmgr-b4f01347-4b51-4b35-8146-2aa840084c2b > drwxr-s--- 14 hdfs yarn 4096 Aug 22 05:15 > blockmgr-0095d26c-c134-48b4-82a6-e8ae02f0189c > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:15 > blockmgr-28a31574-61ae-459f-be3a-8608892246d7 > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-c0cd0df9-b355-4209-b6aa-b549a1fa36eb > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-a2730abb-9517-461e-bedf-d9a2dcef373f > drwxr-s--- 14 hdfs yarn 4096 Aug 22 05:15 > blockmgr-91dd2e1a-6bc2-4429-8b71-2f4240987159 > drwxr-s--- 12 hdfs yarn 4096 Aug 22 05:15 > blockmgr-f4e3a586-8817-45ea-a197-9fdbb3d91946 > drwxr-s--- 15 hdfs yarn 4096 Aug 22 05:15 > blockmgr-ba2c605e-89d8-4f7c-b42c-6ed4ba6bf4ea > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-2ae72383-5f72-4002-84a7-e6335b8c2b6c > drwxr-s--- 13 hdfs yarn 4096 Aug 22 05:15 > blockmgr-6c5e260f-d3c7-4af6-91c1-168c73343f2d > drwxr-s--- 16 hdfs yarn 4096 Aug 22 05:15 > blockmgr-2e9923b1-281c-4a9d-8069-6c5430bd5fc3 > drwxr-s--- 18 hdfs yarn 4096 Aug 22 05:15 > blockmgr-cc3f1406-d8a2-4bf5-a276-8f7aed75c513 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-975bcce0-84b2-4590-880b-bf182d76e319 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-ce82cb63-5998-4227-b85e-77f1c633db43 > drwxr-s--- 11 hdfs yarn 4096 Aug 22 05:15 > blockmgr-592af4aa-3c89-4081-8746-29b99f2220b1 > = > We also applied patches YARN-4594, YARN-4731, but nothing changed. > YARN-4594 https://issues.apache.org/jira/browse/YARN-4594 > YARN-4731 https://issues.apache.org/jira/browse/YARN-4731 > Any advice will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7053) Move curator transaction support to ZKCuratorManager
[
https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137734#comment-16137734
]
Hadoop QA commented on YARN-7053:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 18s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 10s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}132m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem |
| |
hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService
|
| | hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7053 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883210/YARN-7053.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 23caccc104a6 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c379310 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (YARN-7074) Fix NM state store update comment
[
https://issues.apache.org/jira/browse/YARN-7074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137730#comment-16137730
]
Hadoop QA commented on YARN-7074:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
48s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
48s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7074 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883230/YARN-7074.v1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a951f4074825 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c379310 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/17084/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17084/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17084/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix NM state store update comment
> -
>
> Key: YARN-7074
> URL:
[jira] [Commented] (YARN-7053) Move curator transaction support to ZKCuratorManager
[
https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137701#comment-16137701
]
Hadoop QA commented on YARN-7053:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 53s{color} | {color:orange} root: The patch generated 6 new + 1 unchanged -
0 fixed = 7 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 20s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
54s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 45m 2s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
| | hadoop.fs.shell.TestCopyFromLocal |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7053 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883201/YARN-7053.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a2265463890c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c379310 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/17080/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit |
[jira] [Comment Edited] (YARN-6798) Fix NM startup failure with old state store due to version mismatch
[
https://issues.apache.org/jira/browse/YARN-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137679#comment-16137679
]
Botong Huang edited comment on YARN-6798 at 8/23/17 12:35 AM:
--
Thanks [~kasha] for catching it. I've created YARN-7074 to fix the typo.
was (Author: botong):
Thanks [~kasha] for catching it. I've create YARN-7074 to fix the typo.
> Fix NM startup failure with old state store due to version mismatch
> ---
>
> Key: YARN-6798
> URL: https://issues.apache.org/jira/browse/YARN-6798
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha4
>Reporter: Ray Chiang
>Assignee: Botong Huang
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6798.v1.patch, YARN-6798.v2.patch
>
>
> YARN-6703 rolled back the state store version number for the RM from 2.0 to
> 1.4.
> YARN-6127 bumped the version for the NM to 3.0
> private static final Version CURRENT_VERSION_INFO =
> Version.newInstance(3, 0);
> YARN-5049 bumped the version for the NM to 2.0
> private static final Version CURRENT_VERSION_INFO =
> Version.newInstance(2, 0);
> During an upgrade, all NMs died after upgrading a C6 cluster from alpha2 to
> alpha4.
> {noformat}
> 2017-07-07 15:48:17,259 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> org.apache.hadoop.service.ServiceStateException: java.io.IOException:
> Incompatible version for NM state: expecting NM state version 3.0, but
> loading version 2.0
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:246)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:307)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:748)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:809)
> Caused by: java.io.IOException: Incompatible version for NM state: expecting
> NM state version 3.0, but loading version 2.0
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.checkVersion(NMLeveldbStateStoreService.java:1454)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:1308)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:307)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 5 more
> 2017-07-07 15:48:17,277 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NodeManager at xxx.gce.cloudera.com/aa.bb.cc.dd
> /
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6798) Fix NM startup failure with old state store due to version mismatch
[
https://issues.apache.org/jira/browse/YARN-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137679#comment-16137679
]
Botong Huang commented on YARN-6798:
Thanks [~kasha] for catching it. I've create YARN-7074 to fix the typo.
> Fix NM startup failure with old state store due to version mismatch
> ---
>
> Key: YARN-6798
> URL: https://issues.apache.org/jira/browse/YARN-6798
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha4
>Reporter: Ray Chiang
>Assignee: Botong Huang
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6798.v1.patch, YARN-6798.v2.patch
>
>
> YARN-6703 rolled back the state store version number for the RM from 2.0 to
> 1.4.
> YARN-6127 bumped the version for the NM to 3.0
> private static final Version CURRENT_VERSION_INFO =
> Version.newInstance(3, 0);
> YARN-5049 bumped the version for the NM to 2.0
> private static final Version CURRENT_VERSION_INFO =
> Version.newInstance(2, 0);
> During an upgrade, all NMs died after upgrading a C6 cluster from alpha2 to
> alpha4.
> {noformat}
> 2017-07-07 15:48:17,259 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> org.apache.hadoop.service.ServiceStateException: java.io.IOException:
> Incompatible version for NM state: expecting NM state version 3.0, but
> loading version 2.0
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:246)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:307)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:748)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:809)
> Caused by: java.io.IOException: Incompatible version for NM state: expecting
> NM state version 3.0, but loading version 2.0
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.checkVersion(NMLeveldbStateStoreService.java:1454)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:1308)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:307)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 5 more
> 2017-07-07 15:48:17,277 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NodeManager at xxx.gce.cloudera.com/aa.bb.cc.dd
> /
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7074) Fix NM state store update comment
[ https://issues.apache.org/jira/browse/YARN-7074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-7074: --- Description: A follow up of YARN-6798 to fix a typo. (was: A follow up of YARN-6798. ) > Fix NM state store update comment > - > > Key: YARN-7074 > URL: https://issues.apache.org/jira/browse/YARN-7074 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-7074.v1.patch > > > A follow up of YARN-6798 to fix a typo. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7074) Fix NM state store update comment
[ https://issues.apache.org/jira/browse/YARN-7074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-7074: --- Attachment: YARN-7074.v1.patch > Fix NM state store update comment > - > > Key: YARN-7074 > URL: https://issues.apache.org/jira/browse/YARN-7074 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-7074.v1.patch > > > A follow up of YARN-6798. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7074) Fix NM state store update comment
[ https://issues.apache.org/jira/browse/YARN-7074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-7074: --- Component/s: nodemanager > Fix NM state store update comment > - > > Key: YARN-7074 > URL: https://issues.apache.org/jira/browse/YARN-7074 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > > A follow up of YARN-6798. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7074) Fix NM state store update comment
[ https://issues.apache.org/jira/browse/YARN-7074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-7074: --- Description: A follow up of YARN-6798. > Fix NM state store update comment > - > > Key: YARN-7074 > URL: https://issues.apache.org/jira/browse/YARN-7074 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > > A follow up of YARN-6798. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7074) Fix NM state store update comment
Botong Huang created YARN-7074: -- Summary: Fix NM state store update comment Key: YARN-7074 URL: https://issues.apache.org/jira/browse/YARN-7074 Project: Hadoop YARN Issue Type: Bug Reporter: Botong Huang Assignee: Botong Huang Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7050) Post cleanup after YARN-6903
[ https://issues.apache.org/jira/browse/YARN-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7050: -- Attachment: YARN-7050.yarn-native-services.07.patch > Post cleanup after YARN-6903 > > > Key: YARN-7050 > URL: https://issues.apache.org/jira/browse/YARN-7050 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7050.yarn-native-services.01.patch, > YARN-7050.yarn-native-services.02.patch, > YARN-7050.yarn-native-services.03.patch, > YARN-7050.yarn-native-services.04.patch, > YARN-7050.yarn-native-services.05.patch, > YARN-7050.yarn-native-services.06.patch, > YARN-7050.yarn-native-services.07.patch > > > This jira tries to remove some old code, and moves dependency classes to the > new package, and also some other side changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7050) Post cleanup after YARN-6903
[ https://issues.apache.org/jira/browse/YARN-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137663#comment-16137663 ] Jian He commented on YARN-7050: --- I've renamed slideram-log4j.properties to serviceam-log4j.properties SliderKeys.java is renamed to YarnServiceConstants.java I also tried to fix the user-facing namings, for the internal ones, we can do them separately. > Post cleanup after YARN-6903 > > > Key: YARN-7050 > URL: https://issues.apache.org/jira/browse/YARN-7050 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7050.yarn-native-services.01.patch, > YARN-7050.yarn-native-services.02.patch, > YARN-7050.yarn-native-services.03.patch, > YARN-7050.yarn-native-services.04.patch, > YARN-7050.yarn-native-services.05.patch, > YARN-7050.yarn-native-services.06.patch > > > This jira tries to remove some old code, and moves dependency classes to the > new package, and also some other side changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7073) Rest API site documentation
[ https://issues.apache.org/jira/browse/YARN-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha updated YARN-7073: Component/s: (was: yarn-native-services) > Rest API site documentation > --- > > Key: YARN-7073 > URL: https://issues.apache.org/jira/browse/YARN-7073 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation, site >Reporter: Gour Saha >Assignee: Gour Saha > > Commit site documentation for REST API service, generated from the swagger > definition as a MD file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7073) Rest API site documentation
[ https://issues.apache.org/jira/browse/YARN-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha updated YARN-7073: Target Version/s: yarn-native-services > Rest API site documentation > --- > > Key: YARN-7073 > URL: https://issues.apache.org/jira/browse/YARN-7073 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation, site >Reporter: Gour Saha >Assignee: Gour Saha > > Commit site documentation for REST API service, generated from the swagger > definition as a MD file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7073) Rest API site documentation
Gour Saha created YARN-7073: --- Summary: Rest API site documentation Key: YARN-7073 URL: https://issues.apache.org/jira/browse/YARN-7073 Project: Hadoop YARN Issue Type: Sub-task Components: documentation, site, yarn-native-services Reporter: Gour Saha Commit site documentation for REST API service, generated from the swagger definition as a MD file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7073) Rest API site documentation
[ https://issues.apache.org/jira/browse/YARN-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha reassigned YARN-7073: --- Assignee: Gour Saha > Rest API site documentation > --- > > Key: YARN-7073 > URL: https://issues.apache.org/jira/browse/YARN-7073 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation, site, yarn-native-services >Reporter: Gour Saha >Assignee: Gour Saha > > Commit site documentation for REST API service, generated from the swagger > definition as a MD file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6251) Add support for async Container release to prevent deadlock during container updates
[
https://issues.apache.org/jira/browse/YARN-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137624#comment-16137624
]
Hadoop QA commented on YARN-6251:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 4 new + 277 unchanged - 2 fixed = 281 total (was 279)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 48s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
|
| | org.apache.hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6251 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883200/YARN-6251.008.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c8614a49 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c379310 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/17079/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17079/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
[jira] [Commented] (YARN-6640) AM heartbeat stuck when responseId overflows MAX_INT
[
https://issues.apache.org/jira/browse/YARN-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137597#comment-16137597
]
Hadoop QA commented on YARN-6640:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 45m 47s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6640 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12871045/YARN-6640.v1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ffcba91d5067 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c379310 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17077/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17077/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17077/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AM heartbeat stuck when responseId overflows MAX_INT
> -
>
> Key: YARN-6640
> URL: https://issues.apache.org/jira/browse/YARN-6640
> Project: Hadoop YARN
>
[jira] [Commented] (YARN-6798) Fix NM startup failure with old state store due to version mismatch
[
https://issues.apache.org/jira/browse/YARN-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137593#comment-16137593
]
Karthik Kambatla commented on YARN-6798:
Nit pick on the patch: the second line says {{1.2 to 1.2}}. The intention was
likely {{1.1 to 1.2}}?
> Fix NM startup failure with old state store due to version mismatch
> ---
>
> Key: YARN-6798
> URL: https://issues.apache.org/jira/browse/YARN-6798
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha4
>Reporter: Ray Chiang
>Assignee: Botong Huang
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6798.v1.patch, YARN-6798.v2.patch
>
>
> YARN-6703 rolled back the state store version number for the RM from 2.0 to
> 1.4.
> YARN-6127 bumped the version for the NM to 3.0
> private static final Version CURRENT_VERSION_INFO =
> Version.newInstance(3, 0);
> YARN-5049 bumped the version for the NM to 2.0
> private static final Version CURRENT_VERSION_INFO =
> Version.newInstance(2, 0);
> During an upgrade, all NMs died after upgrading a C6 cluster from alpha2 to
> alpha4.
> {noformat}
> 2017-07-07 15:48:17,259 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> org.apache.hadoop.service.ServiceStateException: java.io.IOException:
> Incompatible version for NM state: expecting NM state version 3.0, but
> loading version 2.0
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:246)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:307)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:748)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:809)
> Caused by: java.io.IOException: Incompatible version for NM state: expecting
> NM state version 3.0, but loading version 2.0
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.checkVersion(NMLeveldbStateStoreService.java:1454)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:1308)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:307)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 5 more
> 2017-07-07 15:48:17,277 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NodeManager at xxx.gce.cloudera.com/aa.bb.cc.dd
> /
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7053) Move curator transaction support to ZKCuratorManager
[
https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137587#comment-16137587
]
Jonathan Hung commented on YARN-7053:
-
Oops, OK. Attached 005 which reverts these changes.
Also for {{assertTrue(Arrays.equals(setData, curator.getData(node1)));}}, in
this transaction I'm changing node1's data and deleting node2, so node1 still
exists and I'm verifying the data was set properly.
> Move curator transaction support to ZKCuratorManager
>
>
> Key: YARN-7053
> URL: https://issues.apache.org/jira/browse/YARN-7053
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-7053.001.patch, YARN-7053.002.patch,
> YARN-7053.003.patch, YARN-7053.004.patch, YARN-7053.005.patch
>
>
> HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore
> has some curator transaction support which can be reused, so this can be
> moved to ZKCuratorManager as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7013) merge related work for YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7013: - Attachment: YARN-7013.004.patch > merge related work for YARN-3926 branch > --- > > Key: YARN-7013 > URL: https://issues.apache.org/jira/browse/YARN-7013 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7013.001.patch, YARN-7013.002.patch, > YARN-7013.003.patch, YARN-7013.004.patch > > > To run jenkins for whole branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7053) Move curator transaction support to ZKCuratorManager
[ https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-7053: Attachment: YARN-7053.005.patch > Move curator transaction support to ZKCuratorManager > > > Key: YARN-7053 > URL: https://issues.apache.org/jira/browse/YARN-7053 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-7053.001.patch, YARN-7053.002.patch, > YARN-7053.003.patch, YARN-7053.004.patch, YARN-7053.005.patch > > > HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore > has some curator transaction support which can be reused, so this can be > moved to ZKCuratorManager as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6876) Create an abstract log writer for extendability
[
https://issues.apache.org/jira/browse/YARN-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137577#comment-16137577
]
Hadoop QA commented on YARN-6876:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
35s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 13m
30s{color} | {color:red} hadoop-yarn in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
45s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
8s{color} | {color:green} hadoop-yarn-project_hadoop-yarn generated 0 new + 79
unchanged - 7 fixed = 79 total (was 86) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 30 new + 517 unchanged - 10 fixed = 547 total (was 527) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 30s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
48s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
3s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 45m 43s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| Timed out junit tests |
org.apache.hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation |
| | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6876 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883183/YARN-6876-trunk.005.patch
|
| Optional Tests | asflicense compile javac javadoc
[jira] [Updated] (YARN-7072) Add a new log aggregation file format controller
[ https://issues.apache.org/jira/browse/YARN-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-7072: Attachment: YARN-7072-trunk.001.patch > Add a new log aggregation file format controller > > > Key: YARN-7072 > URL: https://issues.apache.org/jira/browse/YARN-7072 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-7072-trunk.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7072) Add a new log aggregation file format controller
Xuan Gong created YARN-7072: --- Summary: Add a new log aggregation file format controller Key: YARN-7072 URL: https://issues.apache.org/jira/browse/YARN-7072 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow users to enter containers as UID:GID pair instead of by username
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137571#comment-16137571
]
Daniel Templeton commented on YARN-4266:
Instead of forking out _id_, should be use the {{GroupMappingServiceProvider}}
instead? That would let an admin drive the mapping through LDAP instead.
> Allow users to enter containers as UID:GID pair instead of by username
> --
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: luhuichun
> Attachments: YARN-4266.001.patch, YARN-4266.001.patch,
> YARN-4266.002.patch, YARN-4266.003.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7039) Fix javac and javadoc errors in YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137570#comment-16137570 ] Wangda Tan commented on YARN-7039: -- Looks good, +1. Committing. > Fix javac and javadoc errors in YARN-3926 branch > > > Key: YARN-7039 > URL: https://issues.apache.org/jira/browse/YARN-7039 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-3926 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7039.YARN-3926.001.patch, > YARN-7039.YARN-3926.002.patch > > > Fix javac and doc errors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137552#comment-16137552
]
Hadoop QA commented on YARN-5355:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 59 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
8s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
53s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 36s{color} | {color:orange} root: The patch generated 41 new + 2185
unchanged - 167 fixed = 2226 total (was 2352) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 14m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} |
[jira] [Commented] (YARN-7053) Move curator transaction support to ZKCuratorManager
[
https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137546#comment-16137546
]
Subru Krishnan commented on YARN-7053:
--
Thanks [~jhung] for updating the patch. Let's look at how the tests go for v4.
We do *not* need fencing in {{ZookeeperFederationStateStore}} as we rely on RM
for that. That's why after looking at the code I wanted to skip this. So you'll
have to revert the changes to {{ZookeeperFederationStateStore}} (sorry).
In {{TestZKCuratorManager::testTransaction}}, this should be false after
deletion right?
{quote} assertTrue(Arrays.equals(setData, curator.getData(node1)));{quote}
> Move curator transaction support to ZKCuratorManager
>
>
> Key: YARN-7053
> URL: https://issues.apache.org/jira/browse/YARN-7053
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-7053.001.patch, YARN-7053.002.patch,
> YARN-7053.003.patch, YARN-7053.004.patch
>
>
> HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore
> has some curator transaction support which can be reused, so this can be
> moved to ZKCuratorManager as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6798) Fix NM startup failure with old state store due to version mismatch
[
https://issues.apache.org/jira/browse/YARN-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137537#comment-16137537
]
Subru Krishnan commented on YARN-6798:
--
Backported this to branch-2 based on [~kasha]'s feedback
[here|https://issues.apache.org/jira/browse/YARN-6127?focusedCommentId=16136099]
on YARN-6127.
> Fix NM startup failure with old state store due to version mismatch
> ---
>
> Key: YARN-6798
> URL: https://issues.apache.org/jira/browse/YARN-6798
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha4
>Reporter: Ray Chiang
>Assignee: Botong Huang
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6798.v1.patch, YARN-6798.v2.patch
>
>
> YARN-6703 rolled back the state store version number for the RM from 2.0 to
> 1.4.
> YARN-6127 bumped the version for the NM to 3.0
> private static final Version CURRENT_VERSION_INFO =
> Version.newInstance(3, 0);
> YARN-5049 bumped the version for the NM to 2.0
> private static final Version CURRENT_VERSION_INFO =
> Version.newInstance(2, 0);
> During an upgrade, all NMs died after upgrading a C6 cluster from alpha2 to
> alpha4.
> {noformat}
> 2017-07-07 15:48:17,259 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> org.apache.hadoop.service.ServiceStateException: java.io.IOException:
> Incompatible version for NM state: expecting NM state version 3.0, but
> loading version 2.0
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:246)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:307)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:748)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:809)
> Caused by: java.io.IOException: Incompatible version for NM state: expecting
> NM state version 3.0, but loading version 2.0
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.checkVersion(NMLeveldbStateStoreService.java:1454)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:1308)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:307)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 5 more
> 2017-07-07 15:48:17,277 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NodeManager at xxx.gce.cloudera.com/aa.bb.cc.dd
> /
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6798) Fix NM startup failure with old state store due to version mismatch
[
https://issues.apache.org/jira/browse/YARN-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-6798:
-
Fix Version/s: 2.9.0
> Fix NM startup failure with old state store due to version mismatch
> ---
>
> Key: YARN-6798
> URL: https://issues.apache.org/jira/browse/YARN-6798
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha4
>Reporter: Ray Chiang
>Assignee: Botong Huang
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6798.v1.patch, YARN-6798.v2.patch
>
>
> YARN-6703 rolled back the state store version number for the RM from 2.0 to
> 1.4.
> YARN-6127 bumped the version for the NM to 3.0
> private static final Version CURRENT_VERSION_INFO =
> Version.newInstance(3, 0);
> YARN-5049 bumped the version for the NM to 2.0
> private static final Version CURRENT_VERSION_INFO =
> Version.newInstance(2, 0);
> During an upgrade, all NMs died after upgrading a C6 cluster from alpha2 to
> alpha4.
> {noformat}
> 2017-07-07 15:48:17,259 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> org.apache.hadoop.service.ServiceStateException: java.io.IOException:
> Incompatible version for NM state: expecting NM state version 3.0, but
> loading version 2.0
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:246)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:307)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:748)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:809)
> Caused by: java.io.IOException: Incompatible version for NM state: expecting
> NM state version 3.0, but loading version 2.0
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.checkVersion(NMLeveldbStateStoreService.java:1454)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:1308)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:307)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 5 more
> 2017-07-07 15:48:17,277 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NodeManager at xxx.gce.cloudera.com/aa.bb.cc.dd
> /
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7039) Fix javac and javadoc errors in YARN-3926 branch
[
https://issues.apache.org/jira/browse/YARN-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137530#comment-16137530
]
Hadoop QA commented on YARN-7039:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
34s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
58s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
25s{color} | {color:green} YARN-3926 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-build-tools {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
51s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-3926 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
55s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
53s{color} | {color:green} root generated 0 new + 1297 unchanged - 8 fixed =
1297 total (was 1305) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 54s{color} | {color:orange} root: The patch generated 3 new + 264 unchanged
- 53 fixed = 267 total (was 317) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-build-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-build-tools in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
56s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
29s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 17s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
0s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
57s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate
[jira] [Commented] (YARN-7053) Move curator transaction support to ZKCuratorManager
[
https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137526#comment-16137526
]
Jonathan Hung commented on YARN-7053:
-
{{TestZKFailoverController}} and {{TestSubmitApplicationWithRMHA}} pass locally
for me. The {{TestKDiag}} tests don't seem to be related (kerberos-related
failures). {{TestContainerAllocation}} fails with or without the change.
> Move curator transaction support to ZKCuratorManager
>
>
> Key: YARN-7053
> URL: https://issues.apache.org/jira/browse/YARN-7053
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-7053.001.patch, YARN-7053.002.patch,
> YARN-7053.003.patch, YARN-7053.004.patch
>
>
> HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore
> has some curator transaction support which can be reused, so this can be
> moved to ZKCuratorManager as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6640) AM heartbeat stuck when responseId overflows MAX_INT
[
https://issues.apache.org/jira/browse/YARN-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated YARN-6640:
-
Target Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
Fix Version/s: (was: 2.8.2)
(was: 3.0.0-beta1)
(was: 2.9.0)
I agree we don't need to change from int to long here, we can just manage the
wrap. It would be simpler if we didn't have to manage the special case of the
response being -1 to indicate we've never heard from this AM. Otherwise we
would only need to handle to valid cases: current == last or current + 1 ==
last. Anything else would be an error.
Given we need to reserve a value for "not registered" we can use Wangda's idea
to explicitly manage the looping at some convenient positive value rather than
allowing the response ID to ever go negative after it has registered. Then we
leave all the negative values for other "special values" if we ever need them.
It's not like we need all these possible values to know when we're out of sync
given we only expect this value or the previous.
For example, I think we could do something like the following (note: haven't
tested, straight off the top of my head):
{code}
if (((request.getResponseId() + 1) & Integer.MAX_VALUE) ==
lastResponse.getResponseId()) {
/* old heartbeat */
return lastResponse;
} else if (request.getResponseId() != lastResponse.getResponseId()) {
String message =
"Invalid responseId in AllocateRequest from application attempt: "
+ appAttemptId + ", expect responseId to be "
+ (lastResponse.getResponseId() + 1);
throw new InvalidApplicationMasterRequestException(message);
}
[...]
response.setResponseId((lastResponse.getResponseId() + 1) &
Integer.MAX_VALUE);
{code}
There should be a little helper function that takes the current response ID and
returns the next one to make it easier to read.
> AM heartbeat stuck when responseId overflows MAX_INT
> -
>
> Key: YARN-6640
> URL: https://issues.apache.org/jira/browse/YARN-6640
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Blocker
> Attachments: YARN-6640.v1.patch
>
>
> The current code in {{ApplicationMasterService}}:
> if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {/* old
> heartbeat */ return lastResponse;}
> else if (request.getResponseId() + 1 < lastResponse.getResponseId()) { throw
> ... }
> process the heartbeat...
> When a heartbeat comes in, in usual case we are expecting
> request.getResponseId() == lastResponse.getResponseId(). The “if“ is for the
> duplicate heartbeat that’s one step old, the “else if” is to throw and
> complain for heartbeats more than two steps old, otherwise we accept the new
> heartbeat and process it.
> So the bug is: when lastResponse.getResponseId() == MAX_INT, the newest
> heartbeat comes in with responseId == MAX_INT. However reponseId + 1 will be
> MIN_INT, and we will fall into the “else if” case and RM will throw. Then we
> are stuck here…
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137515#comment-16137515
]
Eric Badger commented on YARN-6623:
---
Hey [~vvasudev], I’ve finished my comments on the production code portion of
the patch. This doesn’t include test code. I think I’ll wait for your response
to this before I review the tests. I tried to be quite thorough and so I have
quite a few comments. I hope that they are all correct interpretations, but
it’s quite a big patch so I might have misunderstood some stuff and/or missed
some stuff.
Should we fix the no newline at the end of file warnings? The apply tool
complains about them.
{noformat:title=DockerCommand:getCommandArguments()}
public Map getCommandArguments() {
return Collections.unmodifiableMap(commandArguments);
}
{noformat}
This will return the command as well as the arguments. Unless we are
considering the {{/usr/docker}} to be the actual command and {{inspect}} to be
one of the arguments. If that’s what we’re expecting to happen here, then the
name is a little bit misleading. This might be more of a problem with how
{{commandArguments}} is named than how this function is named.
{noformat:title=container-executor.c:construct_docker_command()}
+char *construct_docker_command(const char *command_file) {
+ int ret = 0;
+ char *buffer = (char *) malloc(EXECUTOR_PATH_MAX * sizeof(char));
{noformat}
This should use {{_SC_ARG_MAX}} as we did in YARN-6988
{{size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024);}}
Also, why not use {{calloc()}} instead of {{malloc()}} and then {{memset()}}?
{noformat:title=container-executor.c:run_docker()}
+ docker_command = construct_docker_command(command_file);
+ docker_binary = get_docker_binary();
{noformat}
I don’t see these getting freed. Am I missing this invocation somewhere?
{noformat:title=container-executor.c:run_docker()}
+ memset(docker_command_with_binary, 0, EXECUTOR_PATH_MAX);
{noformat}
Is this necessary? We allocate the memory with {{calloc()}} which already
clears all of the memory upon allocation.
{noformat:title=container-executor.h}
// get the executable's filename
char* get_executable(char *argv0);
{noformat}
Do we need this declaration (in container-executor.h) since we have moved that
declaration into get_executable.h? We should also add get_executable.h in the
appropriate places. Looks like main.c and test-container-executor.c both call
{{get_executable}}.
{noformat:title=main.c:assert_valid_setup()}
-fprintf(ERRORFILE,"realpath of executable: %s\n",
- errno != 0 ? strerror(errno) : "unknown");
-flush_and_close_log_files();
-exit(-1);
+fprintf(ERRORFILE, "realpath of executable: %s\n",
+errno != 0 ? strerror(errno) : "unknown");
+exit(INVALID_CONFIG_FILE);
{noformat}
Why don’t we want to flush the log files anymore?
{noformat:title=util.c:alloc_and_clear_memory()}
+void* alloc_and_clear_memory(size_t num, size_t size) {
+ void *ret = calloc(num, size);
+ if (ret == NULL) {
+exit(OUT_OF_MEMORY);
+ }
+ return ret;
+}
{noformat}
Should we inline this? Also, if we’re going to use this function, then all
calloc calls should be replaced with this (like the ones I mentioned above)
{noformat:title=util.h}
// DOCKER_CONTAINER_NAME_INVALID = 41,
{noformat}
Should add {{(NOT USED)}} to follow convention
{noformat:title=docker-util.c:read_and_verify_command_file()}
if (command == NULL || (strcmp(command, docker_command) != 0)) {
ret = INCORRECT_COMMAND;
}
{noformat}
Is {{command}} guaranteed to be null-terminated? It comes from the
configuration file, which is a Java creation and I don’t think Java
null-terminates. This comment is probably relevant for quite a few other places
that are doing string operations. We need to be very safe about this or else we
risk possibly overrunning into random regions of the heap. A safe alternative
would be to use the “n” version of all the string operations. This patch uses a
mixed bag of the regular versions and their accompanying “n” versions. I don’t
quite understand the reasoning behind the usage of each. If we guarantee that
the string is null terminated (and always null terminated) then we don’t need
the “n” versions. But even if we guarantee that the input string is null
terminated, it may accidentally have the null character chopped off by an off
by 1 error in a strdup or something like that. So my preference here would be
to use the “n” versions of all of the string functions. Thoughts?
{noformat:title=docker-util.c:read_and_verify_command_file()}
+ if (current_len + string_len < bufflen - 1) {
+strncpy(buff + current_len, string, bufflen - current_len);
+buff[current_len + string_len] = '\0';
+return 0;
+ }
{noformat}
Here it is copying over {{bufflen - current_len}} bytes, when we really only
[jira] [Commented] (YARN-7053) Move curator transaction support to ZKCuratorManager
[
https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137497#comment-16137497
]
Jonathan Hung commented on YARN-7053:
-
Thanks [~subru], actually I just uploaded 004 patch which addresses the
{{ZookeeperFederationStateStore#put}} issue. Also added {{setData}} and
{{delete}} to {{ZKCuratorManager}} tests.
Not sure if the other test failures are related. Will look into it
> Move curator transaction support to ZKCuratorManager
>
>
> Key: YARN-7053
> URL: https://issues.apache.org/jira/browse/YARN-7053
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-7053.001.patch, YARN-7053.002.patch,
> YARN-7053.003.patch, YARN-7053.004.patch
>
>
> HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore
> has some curator transaction support which can be reused, so this can be
> moved to ZKCuratorManager as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7053) Move curator transaction support to ZKCuratorManager
[ https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-7053: Attachment: YARN-7053.004.patch > Move curator transaction support to ZKCuratorManager > > > Key: YARN-7053 > URL: https://issues.apache.org/jira/browse/YARN-7053 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-7053.001.patch, YARN-7053.002.patch, > YARN-7053.003.patch, YARN-7053.004.patch > > > HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore > has some curator transaction support which can be reused, so this can be > moved to ZKCuratorManager as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6251) Add support for async Container release to prevent deadlock during container updates
[
https://issues.apache.org/jira/browse/YARN-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-6251:
--
Attachment: YARN-6251.008.patch
The {{TestOpportunisticContainerAllocatorAMService}} testcase failure is not
really related, but am updating the patch to make it little less flaky.
> Add support for async Container release to prevent deadlock during container
> updates
>
>
> Key: YARN-6251
> URL: https://issues.apache.org/jira/browse/YARN-6251
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6251.001.patch, YARN-6251.002.patch,
> YARN-6251.003.patch, YARN-6251.004.patch, YARN-6251.005.patch,
> YARN-6251.006.patch, YARN-6251.007.patch, YARN-6251.008.patch
>
>
> Opening to track a locking issue that was uncovered when running a custom SLS
> AMSimulator.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6640) AM heartbeat stuck when responseId overflows MAX_INT
[
https://issues.apache.org/jira/browse/YARN-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137475#comment-16137475
]
Wangda Tan commented on YARN-6640:
--
Thanks [~botong] for reporting and working on this case. I think this is pretty
severe issue which we should fix and backport to previous releases. Marked as
blocker for all > 2.8 releases.
We have two choices, one is change type of response id from int to long. I
personally don't prefer that because even long could be exhausted if they have
an app runs for hundred years :), and it has compatibility issue as well.
I prefer to reuse int like what you did in the patch, if value equals MAX_INT,
we will set it back to 0 and handle the special checking logic. I don't suggest
have a special reserved-id, this makes code become confusing.
Another potential problem is, we only fail when request.responseId <
lastResponse.responseId - 1, I think we should also fail when
{{request.responseId > lastResponseId}}.
Thoughts?
+ [~jlowe].
> AM heartbeat stuck when responseId overflows MAX_INT
> -
>
> Key: YARN-6640
> URL: https://issues.apache.org/jira/browse/YARN-6640
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: YARN-6640.v1.patch
>
>
> The current code in {{ApplicationMasterService}}:
> if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {/* old
> heartbeat */ return lastResponse;}
> else if (request.getResponseId() + 1 < lastResponse.getResponseId()) { throw
> ... }
> process the heartbeat...
> When a heartbeat comes in, in usual case we are expecting
> request.getResponseId() == lastResponse.getResponseId(). The “if“ is for the
> duplicate heartbeat that’s one step old, the “else if” is to throw and
> complain for heartbeats more than two steps old, otherwise we accept the new
> heartbeat and process it.
> So the bug is: when lastResponse.getResponseId() == MAX_INT, the newest
> heartbeat comes in with responseId == MAX_INT. However reponseId + 1 will be
> MIN_INT, and we will fall into the “else if” case and RM will throw. Then we
> are stuck here…
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7066: Issue Type: Sub-task (was: New Feature) Parent: YARN-3611 > Add ability to specify volumes to mount for DockerContainerRuntime > -- > > Key: YARN-7066 > URL: https://issues.apache.org/jira/browse/YARN-7066 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Affects Versions: 3.0.0-beta1 >Reporter: Eric Yang > > Yarnfile describes environment, docker image, and configuration template for > launching docker containers in YARN. It would be nice to have ability to > specify the volumes to mount. This can be used in combination to > AMBARI-21748 to mount HDFS as data directories to docker containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6640) AM heartbeat stuck when responseId overflows MAX_INT
[
https://issues.apache.org/jira/browse/YARN-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-6640:
-
Priority: Blocker (was: Minor)
> AM heartbeat stuck when responseId overflows MAX_INT
> -
>
> Key: YARN-6640
> URL: https://issues.apache.org/jira/browse/YARN-6640
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: YARN-6640.v1.patch
>
>
> The current code in {{ApplicationMasterService}}:
> if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {/* old
> heartbeat */ return lastResponse;}
> else if (request.getResponseId() + 1 < lastResponse.getResponseId()) { throw
> ... }
> process the heartbeat...
> When a heartbeat comes in, in usual case we are expecting
> request.getResponseId() == lastResponse.getResponseId(). The “if“ is for the
> duplicate heartbeat that’s one step old, the “else if” is to throw and
> complain for heartbeats more than two steps old, otherwise we accept the new
> heartbeat and process it.
> So the bug is: when lastResponse.getResponseId() == MAX_INT, the newest
> heartbeat comes in with responseId == MAX_INT. However reponseId + 1 will be
> MIN_INT, and we will fall into the “else if” case and RM will throw. Then we
> are stuck here…
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6640) AM heartbeat stuck when responseId overflows MAX_INT
[
https://issues.apache.org/jira/browse/YARN-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-6640:
-
Fix Version/s: 2.8.2
3.0.0-beta1
2.9.0
> AM heartbeat stuck when responseId overflows MAX_INT
> -
>
> Key: YARN-6640
> URL: https://issues.apache.org/jira/browse/YARN-6640
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: YARN-6640.v1.patch
>
>
> The current code in {{ApplicationMasterService}}:
> if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {/* old
> heartbeat */ return lastResponse;}
> else if (request.getResponseId() + 1 < lastResponse.getResponseId()) { throw
> ... }
> process the heartbeat...
> When a heartbeat comes in, in usual case we are expecting
> request.getResponseId() == lastResponse.getResponseId(). The “if“ is for the
> duplicate heartbeat that’s one step old, the “else if” is to throw and
> complain for heartbeats more than two steps old, otherwise we accept the new
> heartbeat and process it.
> So the bug is: when lastResponse.getResponseId() == MAX_INT, the newest
> heartbeat comes in with responseId == MAX_INT. However reponseId + 1 will be
> MIN_INT, and we will fall into the “else if” case and RM will throw. Then we
> are stuck here…
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5355:
---
Attachment: YARN-5355.03.patch
Fixing checkstyle and whitespace issues
> YARN Timeline Service v.2: alpha 2
> --
>
> Key: YARN-5355
> URL: https://issues.apache.org/jira/browse/YARN-5355
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>Priority: Critical
> Attachments: Timeline Service v2_ Ideas for Next Steps.pdf,
> YARN-5355.01.patch, YARN-5355.02.patch, YARN-5355.03.patch,
> YARN-5355-branch-2.01.patch
>
>
> This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service
> v.2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any
> subtask work on this JIRA will be committed to those 2 branches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow users to enter containers as UID:GID pair instead of by username
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137455#comment-16137455 ] Eric Yang commented on YARN-4266: - +1 works on my local build and pass test-patch test. > Allow users to enter containers as UID:GID pair instead of by username > -- > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: luhuichun > Attachments: YARN-4266.001.patch, YARN-4266.001.patch, > YARN-4266.002.patch, YARN-4266.003.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6960) definition of active queue allows idle long-running apps to distort fair shares
[
https://issues.apache.org/jira/browse/YARN-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137446#comment-16137446
]
Steven Rand commented on YARN-6960:
---
Thanks, Daniel. Having thought about this some more, I don't think that either
of the two patches I've posted is a good solution. In the first patch, inactive
queues have fair shares of zero, and AM containers are subject to preemption
even when running in high-priority queues. And in the second patch,
applications running in idle queues define what their fair shares are
irrespective of cluster-side settings, which doesn't make sense.
I'll think about this some more and try to come up with a better idea, but I'd
also be quite interested in hearing your opinion and those of others.
> definition of active queue allows idle long-running apps to distort fair
> shares
> ---
>
> Key: YARN-6960
> URL: https://issues.apache.org/jira/browse/YARN-6960
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.1, 3.0.0-alpha4
>Reporter: Steven Rand
>Assignee: Steven Rand
> Attachments: YARN-6960.001.patch, YARN-6960.002.patch
>
>
> YARN-2026 introduced the notion of only considering active queues when
> computing the fair share of each queue. The definition of an active queue is
> a queue with at least one runnable app:
> {code}
> public boolean isActive() {
> return getNumRunnableApps() > 0;
> }
> {code}
> One case that this definition of activity doesn't account for is that of
> long-running applications that scale dynamically. Such an application might
> request many containers when jobs are running, but scale down to very few
> containers, or only the AM container, when no jobs are running.
> Even when such an application has scaled down to a negligible amount of
> demand and utilization, the queue that it's in is still considered to be
> active, which defeats the purpose of YARN-2026. For example, consider this
> scenario:
> 1. We have queues {{root.a}}, {{root.b}}, {{root.c}}, and {{root.d}}, all of
> which have the same weight.
> 2. Queues {{root.a}} and {{root.b}} contain long-running applications that
> currently have only one container each (the AM).
> 3. An application in queue {{root.c}} starts, and uses the whole cluster
> except for the small amount in use by {{root.a}} and {{root.b}}. An
> application in {{root.d}} starts, and has a high enough demand to be able to
> use half of the cluster. Because all four queues are active, the app in
> {{root.d}} can only preempt the app in {{root.c}} up to roughly 25% of the
> cluster's resources, while the app in {{root.c}} keeps about 75%.
> Ideally in this example, the app in {{root.d}} would be able to preempt the
> app in {{root.c}} up to 50% of the cluster, which would be possible if the
> idle apps in {{root.a}} and {{root.b}} didn't cause those queues to be
> considered active.
> One way to address this is to update the definition of an active queue to be
> a queue containing 1 or more non-AM containers. This way if all apps in a
> queue scale down to only the AM, other queues' fair shares aren't affected.
> The benefit of this approach is that it's quite simple. The downside is that
> it doesn't account for apps that are idle and using almost no resources, but
> still have at least one non-AM container.
> There are a couple of other options that seem plausible to me, but they're
> much more complicated, and it seems to me that this proposal makes good
> progress while adding minimal extra complexity.
> Does this seem like a reasonable change? I'm certainly open to better ideas
> as well.
> Thanks,
> Steve
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6640) AM heartbeat stuck when responseId overflows MAX_INT
[
https://issues.apache.org/jira/browse/YARN-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137448#comment-16137448
]
Botong Huang commented on YARN-6640:
[~wangda] ([~jianhe] and [~asuresh]), can you please take a look at this patch?
Thanks in advance!
> AM heartbeat stuck when responseId overflows MAX_INT
> -
>
> Key: YARN-6640
> URL: https://issues.apache.org/jira/browse/YARN-6640
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments: YARN-6640.v1.patch
>
>
> The current code in {{ApplicationMasterService}}:
> if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {/* old
> heartbeat */ return lastResponse;}
> else if (request.getResponseId() + 1 < lastResponse.getResponseId()) { throw
> ... }
> process the heartbeat...
> When a heartbeat comes in, in usual case we are expecting
> request.getResponseId() == lastResponse.getResponseId(). The “if“ is for the
> duplicate heartbeat that’s one step old, the “else if” is to throw and
> complain for heartbeats more than two steps old, otherwise we accept the new
> heartbeat and process it.
> So the bug is: when lastResponse.getResponseId() == MAX_INT, the newest
> heartbeat comes in with responseId == MAX_INT. However reponseId + 1 will be
> MIN_INT, and we will fall into the “else if” case and RM will throw. Then we
> are stuck here…
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated YARN-6988:
-
Fix Version/s: 2.8.2
2.9.0
Thanks, Eric!
+1 for the branch-2 and branch-2.8 patches as well. I committed these to
branch-2, branch-2.8 and branch-2.8.2.
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: YARN-6988.001.patch, YARN-6988.002.patch,
> YARN-6988-branch-2.002.patch, YARN-6988-branch-2.8.002.patch
>
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7053) Move curator transaction support to ZKCuratorManager
[
https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137439#comment-16137439
]
Subru Krishnan commented on YARN-7053:
--
[~jhung], I looked at {{ZookeeperFederationStateStore::put}}, it's fine to
defer that. But can you fix the test failures.
> Move curator transaction support to ZKCuratorManager
>
>
> Key: YARN-7053
> URL: https://issues.apache.org/jira/browse/YARN-7053
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-7053.001.patch, YARN-7053.002.patch,
> YARN-7053.003.patch
>
>
> HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore
> has some curator transaction support which can be reused, so this can be
> moved to ZKCuratorManager as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2127) Move YarnUncaughtExceptionHandler into Hadoop common
[
https://issues.apache.org/jira/browse/YARN-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137431#comment-16137431
]
Eric Payne commented on YARN-2127:
--
The unit tests are not failing for me in my local environment.
[~ste...@apache.org], will you be able to review the patch?
> Move YarnUncaughtExceptionHandler into Hadoop common
>
>
> Key: YARN-2127
> URL: https://issues.apache.org/jira/browse/YARN-2127
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 2.4.0
>Reporter: Steve Loughran
>Assignee: Eric Payne
>Priority: Minor
> Attachments: YARN-2127.001.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Create a superclass of {{YarnUncaughtExceptionHandler}} in the hadoop-common
> code (retaining the original for compatibility).
> This would be available for any hadoop application to use, and the YARN-679
> launcher could automatically set up the handler.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6736) Consider writing to both ats v1 & v2 from RM for smoother upgrades
[ https://issues.apache.org/jira/browse/YARN-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137424#comment-16137424 ] Aaron Gresch commented on YARN-6736: Same basic patch last two runs, with differing test failures. I'm seeing lots of flakiness locally. > Consider writing to both ats v1 & v2 from RM for smoother upgrades > -- > > Key: YARN-6736 > URL: https://issues.apache.org/jira/browse/YARN-6736 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Aaron Gresch > Attachments: YARN-6736-YARN-5355.001.patch, > YARN-6736-YARN-5355.002.patch, YARN-6736-YARN-5355.003.patch, > YARN-6736-YARN-5355.004.patch > > > When the cluster is being upgraded from atsv1 to v2, it may be good to have a > brief time period during which RM writes to both atsv1 and v2. This will help > frameworks like Tez migrate more smoothly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6127) Add support for work preserving NM restart when AMRMProxy is enabled
[
https://issues.apache.org/jira/browse/YARN-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137408#comment-16137408
]
Subru Krishnan commented on YARN-6127:
--
[~kasha], you raise a valid concern. We addressed this as part of YARN-6798, so
I'll try it to cherry-pick that back to branch-2 to make sure we don't break
rolling upgrades.
> Add support for work preserving NM restart when AMRMProxy is enabled
>
>
> Key: YARN-6127
> URL: https://issues.apache.org/jira/browse/YARN-6127
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: amrmproxy, nodemanager
>Reporter: Subru Krishnan
>Assignee: Botong Huang
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: YARN-6127-branch-2.v1.patch, YARN-6127.v1.patch,
> YARN-6127.v2.patch, YARN-6127.v3.patch, YARN-6127.v4.patch
>
>
> YARN-1336 added the ability to restart NM without loosing any running
> containers. In a Federated YARN environment, there's additional state in the
> {{AMRMProxy}} to allow for spanning across multiple sub-clusters, so we need
> to enhance {{AMRMProxy}} to support work-preserving restart.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6736) Consider writing to both ats v1 & v2 from RM for smoother upgrades
[
https://issues.apache.org/jira/browse/YARN-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137405#comment-16137405
]
Hadoop QA commented on YARN-6736:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
4s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
45s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
27s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 329 unchanged - 0 fixed = 330 total (was 329) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
35s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 58s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
27s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
56s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6736 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883163/YARN-6736-YARN-5355.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b39857f22054 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (YARN-6876) Create an abstract log writer for extendability
[ https://issues.apache.org/jira/browse/YARN-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137386#comment-16137386 ] Xuan Gong commented on YARN-6876: - Thanks for the review. [~djp] Uploaded a new patch to address all your comments > Create an abstract log writer for extendability > --- > > Key: YARN-6876 > URL: https://issues.apache.org/jira/browse/YARN-6876 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-6876-branch-2.001.patch, YARN-6876-trunk.001.patch, > YARN-6876-trunk.002.patch, YARN-6876-trunk.003.patch, > YARN-6876-trunk.004.patch, YARN-6876-trunk.005.patch > > > Currently, TFile log writer is used to aggregate log in YARN. We need to add > an abstract layer, and pick up the correct log writer based on the > configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6876) Create an abstract log writer for extendability
[ https://issues.apache.org/jira/browse/YARN-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-6876: Attachment: YARN-6876-trunk.005.patch > Create an abstract log writer for extendability > --- > > Key: YARN-6876 > URL: https://issues.apache.org/jira/browse/YARN-6876 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-6876-branch-2.001.patch, YARN-6876-trunk.001.patch, > YARN-6876-trunk.002.patch, YARN-6876-trunk.003.patch, > YARN-6876-trunk.004.patch, YARN-6876-trunk.005.patch > > > Currently, TFile log writer is used to aggregate log in YARN. We need to add > an abstract layer, and pick up the correct log writer based on the > configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137369#comment-16137369
]
Hadoop QA commented on YARN-2162:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 349 unchanged - 6 fixed = 354 total (was 355) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
12s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 48s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | new
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.ConfigurableResource(double[])
may expose internal representation by storing an externally mutable object
into ConfigurableResource.percentages At
ConfigurableResource.java:representation by storing an externally mutable
object into ConfigurableResource.percentages At
ConfigurableResource.java:[line 40] |
| Failed junit tests |
[jira] [Commented] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137362#comment-16137362
]
Varun Saxena commented on YARN-5355:
QA reports are quite positive.
Test failures in both the QA reports are unrelated except one failure for
TestTimelineAuthFilterForV2 which seems to be flaky.
Most of the checkstyle issues cannot be fixed as they relate to param num. Rest
of the issues and whitespace issues, I will fix and submit another patch.
> YARN Timeline Service v.2: alpha 2
> --
>
> Key: YARN-5355
> URL: https://issues.apache.org/jira/browse/YARN-5355
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>Priority: Critical
> Attachments: Timeline Service v2_ Ideas for Next Steps.pdf,
> YARN-5355.01.patch, YARN-5355.02.patch, YARN-5355-branch-2.01.patch
>
>
> This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service
> v.2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any
> subtask work on this JIRA will be committed to those 2 branches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient by caching resource usage
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137346#comment-16137346 ] Daniel Templeton commented on YARN-4090: Thanks for moving this JIRA forward, [~yufeigu]. This sounds like an important improvement that I'd like to see make it in by 3.0 beta 1. Beta 1 will be cut on 15th September 2017, so we'll need to move quickly. [~zsl2007], would it be alright with you if [~yufeigu] takes this JIRA over, or do you think you'll be able to wrap it up in the next 3 weeks? > Make Collections.sort() more efficient by caching resource usage > > > Key: YARN-4090 > URL: https://issues.apache.org/jira/browse/YARN-4090 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Xianyin Xin >Assignee: zhangshilong > Attachments: sampling1.jpg, sampling2.jpg, YARN-4090.001.patch, > YARN-4090.002.patch, YARN-4090.003.patch, YARN-4090.004.patch, > YARN-4090.005.patch, YARN-4090.006.patch, YARN-4090.007.patch, > YARN-4090-preview.patch, YARN-4090-TestResult.pdf > > > Collections.sort() consumes too much time in a scheduling round. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout
[ https://issues.apache.org/jira/browse/YARN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137326#comment-16137326 ] Ray Chiang commented on YARN-5536: -- I'm tracking blockers for beta1. What are the odds this is going to get done in time? > Multiple format support (JSON, etc.) for exclude node file in NM graceful > decommission with timeout > --- > > Key: YARN-5536 > URL: https://issues.apache.org/jira/browse/YARN-5536 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du >Priority: Blocker > > Per discussion in YARN-4676, we agree that multiple format (other than xml) > should be supported to decommission nodes with timeout values. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137315#comment-16137315
]
Hadoop QA commented on YARN-5355:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 59 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
20s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
56s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 3s{color} | {color:orange} root: The patch generated 41 new + 2183
unchanged - 166 fixed = 2224 total (was 2349) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 12m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 17m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} |
[jira] [Commented] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137305#comment-16137305
]
Hadoop QA commented on YARN-5355:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 59 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
45s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
56s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 39s{color} | {color:orange} root: The patch generated 41 new + 2182
unchanged - 167 fixed = 2223 total (was 2349) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 15m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} |
[jira] [Updated] (YARN-7039) Fix javac and javadoc errors in YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7039: -- Attachment: YARN-7039.YARN-3926.002.patch Fixing TestDistributedShell errors and deleting some unwanted files > Fix javac and javadoc errors in YARN-3926 branch > > > Key: YARN-7039 > URL: https://issues.apache.org/jira/browse/YARN-7039 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-3926 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7039.YARN-3926.001.patch, > YARN-7039.YARN-3926.002.patch > > > Fix javac and doc errors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7048) Fix tests faking kerberos to explicitly set ugi auth type
[ https://issues.apache.org/jira/browse/YARN-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137190#comment-16137190 ] Hudson commented on YARN-7048: -- ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #12227 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12227/]) YARN-7048. Fix tests faking kerberos to explicitly set ugi auth type. (jlowe: rev 657dd59cc840bce3426170baf68f9d46842f4f53) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java > Fix tests faking kerberos to explicitly set ugi auth type > - > > Key: YARN-7048 > URL: https://issues.apache.org/jira/browse/YARN-7048 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.9.0, 3.0.0-beta1, 2.8.2 > > Attachments: YARN-7048.patch > > > TestTokenClientRMService and TestRMDelegationTokens are faking kerberos > authentication. The remote user ugis are explicitly created as kerberos but > not the login user's ugi. Prior to HADOOP-9747 new ugi instances defaulted > to kerberos even if not kerberos. Now ugis default to kerberos only if > actually kerberos based which causes the login user based tests to fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[ https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-2162: --- Attachment: YARN-2162.003.patch > add ability in Fair Scheduler to optionally configure maxResources in terms > of percentage > - > > Key: YARN-2162 > URL: https://issues.apache.org/jira/browse/YARN-2162 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler, scheduler >Reporter: Ashwin Shankar >Assignee: Yufei Gu > Labels: scheduler > Attachments: YARN-2162.001.patch, YARN-2162.002.patch, > YARN-2162.003.patch > > > minResources and maxResources in fair scheduler configs are expressed in > terms of absolute numbers X mb, Y vcores. > As a result, when we expand or shrink our hadoop cluster, we need to > recalculate and change minResources/maxResources accordingly, which is pretty > inconvenient. > We can circumvent this problem if we can optionally configure these > properties in terms of percentage of cluster capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[
https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137180#comment-16137180
]
Hadoop QA commented on YARN-7062:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
44s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
55s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7062 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883153/YARN-7062.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 51406afc947f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 3efcd51 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/17071/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17071/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17071/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{"
> and "}}"
>
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137177#comment-16137177
]
ASF GitHub Bot commented on YARN-2162:
--
Github user flyrain commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134567787
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ResourceConfiguration.java
---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceType;
+
+import java.util.HashMap;
+
+/**
+ * A ResourceConfiguration represents an entity that is used to
configuration
+ * resources, such as maximum resources of a queue. It can be percentage of
+ * cluster resources or an absolute value.
+ */
+@Private
+@Unstable
+public class ResourceConfiguration {
+ private final boolean isPercentage;
+ private final Resource resource;
+ private final HashMap percentages;
+
+ public boolean isPercentage() {
+return isPercentage;
+ }
+
+ public ResourceConfiguration(HashMap percentages) {
+isPercentage = true;
+this.percentages = percentages;
+this.resource = null;
+ }
+
+ public ResourceConfiguration(Resource resource) {
+isPercentage = false;
+this.percentages = null;
+this.resource = resource;
+ }
+
+ public Resource getResource(Resource clusterResource) {
--- End diff --
I evaluated two ways you suggested. Seems neither of them is better than
the other. I rewrite the getResource a little bit. Let me know what you think.
Thanks.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6736) Consider writing to both ats v1 & v2 from RM for smoother upgrades
[ https://issues.apache.org/jira/browse/YARN-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Gresch updated YARN-6736: --- Attachment: YARN-6736-YARN-5355.004.patch > Consider writing to both ats v1 & v2 from RM for smoother upgrades > -- > > Key: YARN-6736 > URL: https://issues.apache.org/jira/browse/YARN-6736 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Aaron Gresch > Attachments: YARN-6736-YARN-5355.001.patch, > YARN-6736-YARN-5355.002.patch, YARN-6736-YARN-5355.003.patch, > YARN-6736-YARN-5355.004.patch > > > When the cluster is being upgraded from atsv1 to v2, it may be good to have a > brief time period during which RM writes to both atsv1 and v2. This will help > frameworks like Tez migrate more smoothly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7071) Add vcores and number of containers in web UI v2 node heat map
Abdullah Yousufi created YARN-7071: -- Summary: Add vcores and number of containers in web UI v2 node heat map Key: YARN-7071 URL: https://issues.apache.org/jira/browse/YARN-7071 Project: Hadoop YARN Issue Type: Improvement Components: yarn-ui-v2 Affects Versions: 3.0.0-alpha4 Reporter: Abdullah Yousufi Assignee: Abdullah Yousufi Currently, the node heat map displays memory usage per node. This change would add a dropdown to view cpu vcores or number of containers as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2416) InvalidStateTransitonException in ResourceManager if AMLauncher does not receive response for startContainers() call in time
[ https://issues.apache.org/jira/browse/YARN-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137154#comment-16137154 ] Hudson commented on YARN-2416: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12226 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12226/]) YARN-2416. InvalidStateTransitonException in ResourceManager if (jlowe: rev 3efcd51c3b3eb667d83e08b500bb7a7ea559fabe) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java > InvalidStateTransitonException in ResourceManager if AMLauncher does not > receive response for startContainers() call in time > > > Key: YARN-2416 > URL: https://issues.apache.org/jira/browse/YARN-2416 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Jian Fang >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 2.9.0, 3.0.0-beta1, 2.8.2 > > Attachments: YARN-2416.001.patch, YARN-2416.002.patch, > YARN-2416.003.patch > > > AMLauncher calls startContainers(allRequests) to launch a container for > application master. Normally, the call comes back immediately so that the > RMAppAttempt changes its state from ALLOCATED to LAUNCHED. > However, we do observed that in some cases, the RPC call came back very late > but the AM container was already started. Because the RMAppAttempt stuck in > ALLOCATED state, once resource manager received the REGISTERED event from the > application master, it threw InvalidStateTransitonException as follows. > 2014-07-05 08:59:05,021 ERROR > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: > Can't handle this event at current state > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > REGISTERED at ALLOCATED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:652) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:752) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:733) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106) > at java.lang.Thread.run(Thread.java:744) > For subsequent STATUS_UPDATE and CONTAINER_ALLOCATED events for this job, > resource manager kept throwing InvalidStateTransitonException. > 2014-07-05 08:59:06,152 ERROR > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: > Can't handle this event at current state > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > STATUS_UPDATE at ALLOCATED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:652) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:752) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:733) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173) > at >
[jira] [Commented] (YARN-6960) definition of active queue allows idle long-running apps to distort fair shares
[
https://issues.apache.org/jira/browse/YARN-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137152#comment-16137152
]
Daniel Templeton commented on YARN-6960:
I'll take a look when I get a chance.
> definition of active queue allows idle long-running apps to distort fair
> shares
> ---
>
> Key: YARN-6960
> URL: https://issues.apache.org/jira/browse/YARN-6960
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.1, 3.0.0-alpha4
>Reporter: Steven Rand
>Assignee: Steven Rand
> Attachments: YARN-6960.001.patch, YARN-6960.002.patch
>
>
> YARN-2026 introduced the notion of only considering active queues when
> computing the fair share of each queue. The definition of an active queue is
> a queue with at least one runnable app:
> {code}
> public boolean isActive() {
> return getNumRunnableApps() > 0;
> }
> {code}
> One case that this definition of activity doesn't account for is that of
> long-running applications that scale dynamically. Such an application might
> request many containers when jobs are running, but scale down to very few
> containers, or only the AM container, when no jobs are running.
> Even when such an application has scaled down to a negligible amount of
> demand and utilization, the queue that it's in is still considered to be
> active, which defeats the purpose of YARN-2026. For example, consider this
> scenario:
> 1. We have queues {{root.a}}, {{root.b}}, {{root.c}}, and {{root.d}}, all of
> which have the same weight.
> 2. Queues {{root.a}} and {{root.b}} contain long-running applications that
> currently have only one container each (the AM).
> 3. An application in queue {{root.c}} starts, and uses the whole cluster
> except for the small amount in use by {{root.a}} and {{root.b}}. An
> application in {{root.d}} starts, and has a high enough demand to be able to
> use half of the cluster. Because all four queues are active, the app in
> {{root.d}} can only preempt the app in {{root.c}} up to roughly 25% of the
> cluster's resources, while the app in {{root.c}} keeps about 75%.
> Ideally in this example, the app in {{root.d}} would be able to preempt the
> app in {{root.c}} up to 50% of the cluster, which would be possible if the
> idle apps in {{root.a}} and {{root.b}} didn't cause those queues to be
> considered active.
> One way to address this is to update the definition of an active queue to be
> a queue containing 1 or more non-AM containers. This way if all apps in a
> queue scale down to only the AM, other queues' fair shares aren't affected.
> The benefit of this approach is that it's quite simple. The downside is that
> it doesn't account for apps that are idle and using almost no resources, but
> still have at least one non-AM container.
> There are a couple of other options that seem plausible to me, but they're
> much more complicated, and it seems to me that this proposal makes good
> progress while adding minimal extra complexity.
> Does this seem like a reasonable change? I'm certainly open to better ideas
> as well.
> Thanks,
> Steve
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7039) Fix javac and javadoc errors in YARN-3926 branch
[
https://issues.apache.org/jira/browse/YARN-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137144#comment-16137144
]
Hadoop QA commented on YARN-7039:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
35s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
1s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
52s{color} | {color:green} YARN-3926 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
58s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-3926 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
52s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
38s{color} | {color:green} hadoop-yarn-project_hadoop-yarn generated 0 new + 92
unchanged - 8 fixed = 92 total (was 100) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 6s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 263 unchanged - 53 fixed = 266 total (was 316) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
43s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
29s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 14s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 31m 56s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 1s{color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}189m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.client.api.impl.TestNMClient |
| |
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels |
| |
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137142#comment-16137142
]
ASF GitHub Bot commented on YARN-2162:
--
Github user flyrain commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134562312
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSParentQueue.java
---
@@ -42,6 +43,8 @@ public void setUp() throws Exception {
AllocationConfiguration allocConf = new AllocationConfiguration(conf);
when(scheduler.getAllocationConfiguration()).thenReturn(allocConf);
when(scheduler.getConf()).thenReturn(conf);
+when(scheduler.getResourceCalculator()).thenReturn(
--- End diff --
It throws an exception like this if we don't do this:
```
java.lang.NullPointerException
at
org.apache.hadoop.yarn.util.resource.Resources.greaterThan(Resources.java:310)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.getMaxShare(FSQueue.java:171)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfiguration.initFSQueue(AllocationConfiguration.java:409)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.reinit(FSQueue.java:111)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.(FSQueue.java:96)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.(FSParentQueue.java:60)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.initialize(QueueManager.java:77)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSParentQueue.setUp(TestFSParentQueue.java:58)
```
This code path doesn't exist before my patch.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7048) Fix tests faking kerberos to explicitly set ugi auth type
[ https://issues.apache.org/jira/browse/YARN-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137138#comment-16137138 ] Jason Lowe commented on YARN-7048: -- The TestContainerAllocation failure is tracked by YARN-7044 and the TestDelegationTokenRenewer failure is tracked by YARN-5816. +1 lgtm. Committing this. > Fix tests faking kerberos to explicitly set ugi auth type > - > > Key: YARN-7048 > URL: https://issues.apache.org/jira/browse/YARN-7048 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: YARN-7048.patch > > > TestTokenClientRMService and TestRMDelegationTokens are faking kerberos > authentication. The remote user ugis are explicitly created as kerberos but > not the login user's ugi. Prior to HADOOP-9747 new ugi instances defaulted > to kerberos even if not kerberos. Now ugis default to kerberos only if > actually kerberos based which causes the login user based tests to fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6933) ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources() are duplicating work
[
https://issues.apache.org/jira/browse/YARN-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137136#comment-16137136
]
Daniel Templeton commented on YARN-6933:
Thanks for the patches, [~maniraj...@gmail.com]. I just have two tiny
quibbles. First, can we add a comment to explain why we're checking for
{{"memory"}} in addition to {{MEMORY}}, i.e. for historical reasons we still
support "memory" as a valid resource name. Second, for the exception error
message, can we mirror the messages for the {{MEMORY}} and {{VCORES}}
exceptions?
> ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources()
> are duplicating work
> ---
>
> Key: YARN-6933
> URL: https://issues.apache.org/jira/browse/YARN-6933
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
> Labels: newbie++
> Attachments: YARN-6933-YARN-3926.001.patch,
> YARN-6933-YARN-3926.002.patch, YARN-6933-YARN-3926.003.patch,
> YARN-6933-YARN-3926.004.patch, YARN-6933-YARN-3926.005.patch,
> YARN-6933-YARN-3926.006.patch
>
>
> Both are used to check that the mandatory resources were not redefined. Only
> one check is needed. I would recommend dropping {{DISALLOWED_RESOURCES}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[
https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137134#comment-16137134
]
Lingfeng Su commented on YARN-7062:
---
add a new patch for code style check, line-wrapping
> yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{"
> and "}}"
> ---
>
> Key: YARN-7062
> URL: https://issues.apache.org/jira/browse/YARN-7062
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Lingfeng Su
>Assignee: Lingfeng Su
> Labels: patch
> Attachments: YARN-7062.001.patch, YARN-7062.002.patch
>
>
> I passed a json string like "{User: Billy, {Age: 18}}" to main method args,
> when running spark jobs on yarn.
> It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age:
> 18" ("}}" to "")
> I found the final arg in launch_container.sh of yarn containers, as:
> --arg '{User: Billy, {Age: 18'
> {code:java}
> exec /bin/bash -c
> "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
> $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp
> -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01
> org.apache.spark.deploy.yarn.ApplicationMaster --class
> 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar
> file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18'
> --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1>
> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout
> 2>
> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr"
> {code}
> We could make some improvements
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment():
>
> {code:java}
> @VisibleForTesting
> public static String expandEnvironment(String var,
> Path containerLogDir) {
> var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
> containerLogDir.toString());
> var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
> File.pathSeparator);
> // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
> // as %VAR% and on Linux replaced as "$VAR"
> if (Shell.WINDOWS) {
> var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
> } else {
> var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
> var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");
> }
> return var;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137133#comment-16137133
]
ASF GitHub Bot commented on YARN-2162:
--
Github user flyrain commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134560857
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
---
@@ -235,21 +235,21 @@ public void testAllocationFileParsing() throws
Exception {
queueConf.getMinResources("root." +
YarnConfiguration.DEFAULT_QUEUE_NAME));
assertEquals(Resources.createResource(2048, 10),
-queueConf.getMaxResources("root.queueA"));
+queueConf.getMaxResources("root.queueA").getResource());
assertEquals(Resources.createResource(5120, 110),
-queueConf.getMaxResources("root.queueB"));
-assertEquals(Resources.createResource(5120, 0),
-queueConf.getMaxResources("root.queueC"));
+queueConf.getMaxResources("root.queueB").getResource());
+assertEquals(Resources.createResource(4096, 100),
--- End diff --
The logic to set max share to min share if max share is less than min share
has been moved to FSQueue#getMaxShare
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[
https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lingfeng Su updated YARN-7062:
--
Attachment: YARN-7062.002.patch
> yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{"
> and "}}"
> ---
>
> Key: YARN-7062
> URL: https://issues.apache.org/jira/browse/YARN-7062
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Lingfeng Su
>Assignee: Lingfeng Su
> Labels: patch
> Attachments: YARN-7062.001.patch, YARN-7062.002.patch
>
>
> I passed a json string like "{User: Billy, {Age: 18}}" to main method args,
> when running spark jobs on yarn.
> It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age:
> 18" ("}}" to "")
> I found the final arg in launch_container.sh of yarn containers, as:
> --arg '{User: Billy, {Age: 18'
> {code:java}
> exec /bin/bash -c
> "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
> $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp
> -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01
> org.apache.spark.deploy.yarn.ApplicationMaster --class
> 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar
> file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18'
> --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1>
> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout
> 2>
> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr"
> {code}
> We could make some improvements
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment():
>
> {code:java}
> @VisibleForTesting
> public static String expandEnvironment(String var,
> Path containerLogDir) {
> var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
> containerLogDir.toString());
> var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
> File.pathSeparator);
> // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
> // as %VAR% and on Linux replaced as "$VAR"
> if (Shell.WINDOWS) {
> var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
> } else {
> var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
> var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");
> }
> return var;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137126#comment-16137126
]
ASF GitHub Bot commented on YARN-2162:
--
Github user flyrain commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134559630
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ResourceConfiguration.java
---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceType;
+
+import java.util.HashMap;
+
+/**
+ * A ResourceConfiguration represents an entity that is used to
configuration
+ * resources, such as maximum resources of a queue. It can be percentage of
+ * cluster resources or an absolute value.
+ */
+@Private
+@Unstable
+public class ResourceConfiguration {
+ private final boolean isPercentage;
+ private final Resource resource;
+ private final HashMap percentages;
+
+ public boolean isPercentage() {
--- End diff --
Remove it in next patch.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137127#comment-16137127
]
ASF GitHub Bot commented on YARN-2162:
--
Github user flyrain commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134559721
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ResourceConfiguration.java
---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceType;
+
+import java.util.HashMap;
+
+/**
+ * A ResourceConfiguration represents an entity that is used to
configuration
+ * resources, such as maximum resources of a queue. It can be percentage of
+ * cluster resources or an absolute value.
+ */
+@Private
+@Unstable
+public class ResourceConfiguration {
+ private final boolean isPercentage;
--- End diff --
Remove the boolean in next patch
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2416) InvalidStateTransitonException in ResourceManager if AMLauncher does not receive response for startContainers() call in time
[ https://issues.apache.org/jira/browse/YARN-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137101#comment-16137101 ] Jason Lowe commented on YARN-2416: -- +1 for the latest patch. Committing this. > InvalidStateTransitonException in ResourceManager if AMLauncher does not > receive response for startContainers() call in time > > > Key: YARN-2416 > URL: https://issues.apache.org/jira/browse/YARN-2416 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Jian Fang >Assignee: Jonathan Eagles >Priority: Critical > Attachments: YARN-2416.001.patch, YARN-2416.002.patch, > YARN-2416.003.patch > > > AMLauncher calls startContainers(allRequests) to launch a container for > application master. Normally, the call comes back immediately so that the > RMAppAttempt changes its state from ALLOCATED to LAUNCHED. > However, we do observed that in some cases, the RPC call came back very late > but the AM container was already started. Because the RMAppAttempt stuck in > ALLOCATED state, once resource manager received the REGISTERED event from the > application master, it threw InvalidStateTransitonException as follows. > 2014-07-05 08:59:05,021 ERROR > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: > Can't handle this event at current state > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > REGISTERED at ALLOCATED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:652) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:752) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:733) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106) > at java.lang.Thread.run(Thread.java:744) > For subsequent STATUS_UPDATE and CONTAINER_ALLOCATED events for this job, > resource manager kept throwing InvalidStateTransitonException. > 2014-07-05 08:59:06,152 ERROR > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: > Can't handle this event at current state > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > STATUS_UPDATE at ALLOCATED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:652) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:752) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:733) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106) > at java.lang.Thread.run(Thread.java:744) > 2014-07-05 08:59:07,779 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_1404549222428_0001_02_02 Container Transitioned from NEW to > ALLOCATED > 2014-07-05 08:59:07,779 ERROR > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: > Can't handle this event at current state > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > CONTAINER_ALLOCATED at ALLOCATED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) >
[jira] [Assigned] (YARN-7052) RM SchedulingMonitor should use HadoopExecutors when creating ScheduledExecutorService
[
https://issues.apache.org/jira/browse/YARN-7052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne reassigned YARN-7052:
Assignee: Eric Payne
> RM SchedulingMonitor should use HadoopExecutors when creating
> ScheduledExecutorService
> --
>
> Key: YARN-7052
> URL: https://issues.apache.org/jira/browse/YARN-7052
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Eric Payne
>Assignee: Eric Payne
>
> In YARN-7051, we ran into a case where the preemption monitor thread hung
> with no indication of why. This was because the preemption monitor is started
> by the {{SchedulingExecutorService}} from {{SchedulingMonitor#serviceStart}},
> and then nothing ever gets the result of the future or allows it to throw an
> exception if needed.
> At least with {{HadoopExecutor}}, it will provide a
> {{HadoopScheduledThreadPoolExecutor}} that logs the exception if one happens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
