[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java
[
https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295948#comment-15295948
]
Xianyin Xin commented on YARN-4090:
---
Sorry for the delay [~yufeigu].
{quote}
do you mean "bring down" or "decrease" when you said "write
down" in this comment?
{quote}
Yes, i mean decrease, and changed the comments in the new patch.
{quote}
there are two spaces between "synchronized" and "void" in public synchronized
void incResourceUsage(Resourc
{quote}
Fixed.
> Make Collections.sort() more efficient in FSParentQueue.java
>
>
> Key: YARN-4090
> URL: https://issues.apache.org/jira/browse/YARN-4090
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Reporter: Xianyin Xin
>Assignee: Xianyin Xin
> Attachments: YARN-4090-TestResult.pdf, YARN-4090-preview.patch,
> YARN-4090.001.patch, sampling1.jpg, sampling2.jpg
>
>
> Collections.sort() consumes too much time in a scheduling round.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4435) Add RM Delegation Token DtFetcher Implementation for DtUtil
[
https://issues.apache.org/jira/browse/YARN-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295906#comment-15295906
]
Hadoop QA commented on YARN-4435:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: patch
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 2s
{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client
generated 2 new + 183 unchanged - 0 fixed = 185 total (was 183) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 52s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 36s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMProxy |
| | hadoop.yarn.client.api.impl.TestDistributedScheduling |
| | hadoop.yarn.client.TestGetGroups |
| Timed out junit tests | org.apache.hadoop.yarn.client.cli.TestYarnCLI |
| | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| | org.apache.hadoop.yarn.client.api.impl.TestYarnClient |
| | org.apache.hadoop.yarn.client.api.impl.TestNMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805568/YARN-4435.02.patch |
| JIRA Issue | YARN-4435 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux d7668db2ad8a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6161d9b |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Updated] (YARN-4435) Add RM Delegation Token DtFetcher Implementation for DtUtil
[ https://issues.apache.org/jira/browse/YARN-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Paduano updated YARN-4435: -- Attachment: YARN-4435.02.patch reimplement RmDtFetcher to use REST API add unit tests for TimelineDtFetcher and RmDtFetcher > Add RM Delegation Token DtFetcher Implementation for DtUtil > --- > > Key: YARN-4435 > URL: https://issues.apache.org/jira/browse/YARN-4435 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Matthew Paduano >Assignee: Matthew Paduano > Attachments: YARN-4435.00.patch.txt, YARN-4435.01.patch, > YARN-4435.02.patch, proposed_solution > > > Add a class to yarn project that implements the DtFetcher interface to return > a RM delegation token object. > I attached a proposed class implementation that does this, but it cannot be > added as a patch until the interface is merged in HADOOP-12563 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3971) Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery
[
https://issues.apache.org/jira/browse/YARN-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295869#comment-15295869
]
Naganarasimha G R commented on YARN-3971:
-
[~bibinchundatt], Checkstyle comments seems to be minor and can be corrected
while committing the patch and test case failure is not related to the patch ,
hence patch LGTM and committing it shortly if no further comments !
> Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel
> recovery
> --
>
> Key: YARN-3971
> URL: https://issues.apache.org/jira/browse/YARN-3971
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: 0001-YARN-3971.patch, 0002-YARN-3971.patch,
> 0003-YARN-3971.patch, 0004-YARN-3971.patch,
> 0005-YARN-3971.001.addendum.patch, 0005-YARN-3971.addendum.patch,
> 0005-YARN-3971.patch
>
>
> Steps to reproduce
> # Create label x,y
> # Delete label x,y
> # Create label x,y add capacity scheduler xml for labels x and y too
> # Restart RM
>
> Both RM will become Standby.
> Since below exception is thrown on {{FileSystemNodeLabelsStore#recover}}
> {code}
> 2015-07-23 14:03:33,627 INFO org.apache.hadoop.service.AbstractService:
> Service org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager failed in
> state STARTED; cause: java.io.IOException: Cannot remove label=x, because
> queue=a1 is using this label. Please remove label on queue before remove the
> label
> java.io.IOException: Cannot remove label=x, because queue=a1 is using this
> label. Please remove label on queue before remove the label
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager.checkRemoveFromClusterNodeLabelsOfQueue(RMNodeLabelsManager.java:104)
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager.removeFromClusterNodeLabels(RMNodeLabelsManager.java:118)
> at
> org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.recover(FileSystemNodeLabelsStore.java:221)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.initNodeLabelStore(CommonNodeLabelsManager.java:232)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStart(CommonNodeLabelsManager.java:245)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:587)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:964)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1005)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1001)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1001)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:832)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:422)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5082) ContainerId rapidly increased in fair scheduler if the num of node app reserved reached the limit
[
https://issues.apache.org/jira/browse/YARN-5082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295825#comment-15295825
]
sandflee commented on YARN-5082:
test failure are not related, cc [~asuresh] [~kasha] could you help to view
this?
> ContainerId rapidly increased in fair scheduler if the num of node app
> reserved reached the limit
> --
>
> Key: YARN-5082
> URL: https://issues.apache.org/jira/browse/YARN-5082
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: sandflee
>Assignee: sandflee
> Attachments: YARN-5082.01.patch, YARN-5082.02.patch
>
>
> see many logs like
> {quote}
> 16/05/14 01:07:58 DEBUG fair.FSAppAttempt: Not creating reservation as
> container container_1463159225729_0002_01_03 is not reservable
> 16/05/14 01:07:58 DEBUG fair.FSAppAttempt: Not creating reservation as
> container container_1463159225729_0002_01_04 is not reservable
> 16/05/14 01:07:58 DEBUG fair.FSAppAttempt: Not creating reservation as
> container container_1463159225729_0002_01_05 is not reservable
> 16/05/14 01:07:58 DEBUG fair.FSAppAttempt: Not creating reservation as
> container container_1463159225729_0002_01_06 is not reservable
> 16/05/14 01:07:58 DEBUG fair.FSAppAttempt: Not creating reservation as
> container container_1463159225729_0002_01_07 is not reservable
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5089) Improve "yarn log" command-line "logFiles" option to support regex
[
https://issues.apache.org/jira/browse/YARN-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295813#comment-15295813
]
Hadoop QA commented on YARN-5089:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 59s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 35s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 5 new +
137 unchanged - 5 fixed = 142 total (was 142) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 59s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 2 new + 592 unchanged - 0 fixed = 594 total (was 592) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 27s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 9s
{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 15s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 113m 12s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.TestGetGroups |
| | hadoop.yarn.client.api.impl.TestAMRMProxy |
| | hadoop.yarn.client.api.impl.TestDistributedScheduling |
| Timed out junit tests | org.apache.hadoop.yarn.client.cli.TestYarnCLI |
| | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| | org.apache.hadoop.yarn.client.api.impl.TestYarnClient |
| | org.apache.hadoop.yarn.client.api.impl.TestNMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
[jira] [Commented] (YARN-3971) Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery
[
https://issues.apache.org/jira/browse/YARN-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295812#comment-15295812
]
Hadoop QA commented on YARN-3971:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 3s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 36s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 4 new +
56 unchanged - 3 fixed = 60 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 32s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 9s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12797968/0005-YARN-3971.001.addendum.patch
|
| JIRA Issue | YARN-3971 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 12ab41d99a4f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6161d9b |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/11627/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
[jira] [Commented] (YARN-3971) Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery
[
https://issues.apache.org/jira/browse/YARN-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295785#comment-15295785
]
Naganarasimha G R commented on YARN-3971:
-
Had offline discussion with [~wangda], & [~bibinchundatt], as HADOOP-13035 is
not expected to come in near future, going ahead with patch
*0005-YARN-3971.001.addendum.patch* . Will initiate the jenkins again and once
the result is fine willl commit the patch !
> Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel
> recovery
> --
>
> Key: YARN-3971
> URL: https://issues.apache.org/jira/browse/YARN-3971
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: 0001-YARN-3971.patch, 0002-YARN-3971.patch,
> 0003-YARN-3971.patch, 0004-YARN-3971.patch,
> 0005-YARN-3971.001.addendum.patch, 0005-YARN-3971.addendum.patch,
> 0005-YARN-3971.patch
>
>
> Steps to reproduce
> # Create label x,y
> # Delete label x,y
> # Create label x,y add capacity scheduler xml for labels x and y too
> # Restart RM
>
> Both RM will become Standby.
> Since below exception is thrown on {{FileSystemNodeLabelsStore#recover}}
> {code}
> 2015-07-23 14:03:33,627 INFO org.apache.hadoop.service.AbstractService:
> Service org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager failed in
> state STARTED; cause: java.io.IOException: Cannot remove label=x, because
> queue=a1 is using this label. Please remove label on queue before remove the
> label
> java.io.IOException: Cannot remove label=x, because queue=a1 is using this
> label. Please remove label on queue before remove the label
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager.checkRemoveFromClusterNodeLabelsOfQueue(RMNodeLabelsManager.java:104)
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager.removeFromClusterNodeLabels(RMNodeLabelsManager.java:118)
> at
> org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.recover(FileSystemNodeLabelsStore.java:221)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.initNodeLabelStore(CommonNodeLabelsManager.java:232)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStart(CommonNodeLabelsManager.java:245)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:587)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:964)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1005)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1001)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1001)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:832)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:422)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5103) With NM recovery enabled, restarting NM multiple times results in AM restart
[
https://issues.apache.org/jira/browse/YARN-5103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295782#comment-15295782
]
Hadoop QA commented on YARN-5103:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 40s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
patch generated 1 new + 6 unchanged - 1 fixed = 7 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 11s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 7s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805557/YARN-5103-v2.patch |
| JIRA Issue | YARN-5103 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7fe8cc378812 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6161d9b |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/11626/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/11626/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/11626/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> With NM recovery enabled, restarting NM multiple times results in AM restart
>
[jira] [Updated] (YARN-5103) With NM recovery enabled, restarting NM multiple times results in AM restart
[
https://issues.apache.org/jira/browse/YARN-5103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-5103:
-
Attachment: YARN-5103-v2.patch
Thanks [~jlowe] for review and comments! v2 patch incorporate your comments
above.
About target branch, I agree it should be better to commit to branch-2.8 as
well - just update it.
> With NM recovery enabled, restarting NM multiple times results in AM restart
>
>
> Key: YARN-5103
> URL: https://issues.apache.org/jira/browse/YARN-5103
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Junping Du
>Priority: Critical
> Attachments: YARN-5103-demo.patch, YARN-5103-v2.patch, YARN-5103.patch
>
>
> AM is restarted when NM is restarted multiple times even though NM recovery
> is enabled.
> {Code:title=NM log on which AM attempt 1 was running }
> ERROR launcher.RecoveredContainerLaunch
> (RecoveredContainerLaunch.java:call(88)) - Unable to recover container
> container_e12_1463043063682_0002_01_01
> java.io.IOException: java.lang.InterruptedException
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:579)
> at org.apache.hadoop.util.Shell.run(Shell.java:487)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.signalContainer(LinuxContainerExecutor.java:478)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.isContainerProcessAlive(LinuxContainerExecutor.java:542)
> at
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor.reacquireContainer(ContainerExecutor.java:185)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.reacquireContainer(LinuxContainerExecutor.java:445)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:83)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:46)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {Code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5103) With NM recovery enabled, restarting NM multiple times results in AM restart
[
https://issues.apache.org/jira/browse/YARN-5103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-5103:
-
Target Version/s: 2.8.0 (was: 2.9.0)
> With NM recovery enabled, restarting NM multiple times results in AM restart
>
>
> Key: YARN-5103
> URL: https://issues.apache.org/jira/browse/YARN-5103
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Junping Du
>Priority: Critical
> Attachments: YARN-5103-demo.patch, YARN-5103-v2.patch, YARN-5103.patch
>
>
> AM is restarted when NM is restarted multiple times even though NM recovery
> is enabled.
> {Code:title=NM log on which AM attempt 1 was running }
> ERROR launcher.RecoveredContainerLaunch
> (RecoveredContainerLaunch.java:call(88)) - Unable to recover container
> container_e12_1463043063682_0002_01_01
> java.io.IOException: java.lang.InterruptedException
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:579)
> at org.apache.hadoop.util.Shell.run(Shell.java:487)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.signalContainer(LinuxContainerExecutor.java:478)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.isContainerProcessAlive(LinuxContainerExecutor.java:542)
> at
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor.reacquireContainer(ContainerExecutor.java:185)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.reacquireContainer(LinuxContainerExecutor.java:445)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:83)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:46)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {Code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4766) NM should not aggregate logs older than the retention policy
[ https://issues.apache.org/jira/browse/YARN-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295767#comment-15295767 ] Robert Kanter commented on YARN-4766: - [~haibochen], the 005 patch looks good to me. Can you take a look at the checkstyle and javadoc warnings? > NM should not aggregate logs older than the retention policy > > > Key: YARN-4766 > URL: https://issues.apache.org/jira/browse/YARN-4766 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation, nodemanager >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: yarn4766.001.patch, yarn4766.002.patch, > yarn4766.003.patch, yarn4766.004.patch, yarn4766.004.patch, yarn4766.005.patch > > > When a log aggregation fails on the NM the information is for the attempt is > kept in the recovery DB. Log aggregation can fail for multiple reasons which > are often related to HDFS space or permissions. > On restart the recovery DB is read and if an application attempt needs its > logs aggregated, the files are scheduled for aggregation without any checks. > The log files could be older than the retention limit in which case we should > not aggregate them but immediately mark them for deletion from the local file > system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5112) Excessive log warnings for directory permission issue on NM recovery.
[
https://issues.apache.org/jira/browse/YARN-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295766#comment-15295766
]
Hudson commented on YARN-5112:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #9840 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9840/])
YARN-5112. Excessive log warnings for directory permission issue on NM
(junping_du: rev 6161d9ba5230f553db5f5490dce67e2afd1e29ca)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java
> Excessive log warnings for directory permission issue on NM recovery.
> -
>
> Key: YARN-5112
> URL: https://issues.apache.org/jira/browse/YARN-5112
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Fix For: 2.9.0
>
> Attachments: YARN-5112.2.patch, YARN-5112.3.patch, YARN-5112.4.patch,
> YARN-5112.5.patch, YARN-5112.6.patch, YARN-5112.patch
>
>
> When there are a lot of apps to recover in NM store, NM prints these two
> lines for each app, which gets annoying.
> {code}
> 2015-10-13 01:58:40,277 WARN logaggregation.LogAggregationService
> (LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root
> Log Dir [/app-logs] already exist, but with incorrect permissions. Expected:
> [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple
> users.
> 336 2015-10-13 01:58:40,277 WARN logaggregation.AppLogAggregatorImpl
> (AppLogAggregatorImpl.java:(182)) - rollingMonitorInterval is set as
> -1. The log rolling mornitoring interval is disabled. The logs will be
> aggregated after this application is finished.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5088) Improve "yarn log" command-line to read the last K bytes for the log files
[ https://issues.apache.org/jira/browse/YARN-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5088: Attachment: YARN-5088.1.patch > Improve "yarn log" command-line to read the last K bytes for the log files > -- > > Key: YARN-5088 > URL: https://issues.apache.org/jira/browse/YARN-5088 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5088.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5089) Improve "yarn log" command-line "logFiles" option to support regex
[ https://issues.apache.org/jira/browse/YARN-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295765#comment-15295765 ] Xuan Gong commented on YARN-5089: - find the checkstyle and findbug issue > Improve "yarn log" command-line "logFiles" option to support regex > --- > > Key: YARN-5089 > URL: https://issues.apache.org/jira/browse/YARN-5089 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5089.1.patch, YARN-5089.2.patch, YARN-5089.3.patch, > YARN-5089.4.patch > > > Right now, we only support matching the exact file name. We need to support > Java regex as well -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5089) Improve "yarn log" command-line "logFiles" option to support regex
[ https://issues.apache.org/jira/browse/YARN-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5089: Attachment: YARN-5089.4.patch > Improve "yarn log" command-line "logFiles" option to support regex > --- > > Key: YARN-5089 > URL: https://issues.apache.org/jira/browse/YARN-5089 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5089.1.patch, YARN-5089.2.patch, YARN-5089.3.patch, > YARN-5089.4.patch > > > Right now, we only support matching the exact file name. We need to support > Java regex as well -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5089) Improve "yarn log" command-line "logFiles" option to support regex
[ https://issues.apache.org/jira/browse/YARN-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5089: Attachment: YARN-5089.3.patch > Improve "yarn log" command-line "logFiles" option to support regex > --- > > Key: YARN-5089 > URL: https://issues.apache.org/jira/browse/YARN-5089 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5089.1.patch, YARN-5089.2.patch, YARN-5089.3.patch > > > Right now, we only support matching the exact file name. We need to support > Java regex as well -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5112) Excessive log warnings for directory permission issue on NM recovery.
[
https://issues.apache.org/jira/browse/YARN-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-5112:
-
Summary: Excessive log warnings for directory permission issue on NM
recovery. (was: Excessive log warnings on NM recovery)
> Excessive log warnings for directory permission issue on NM recovery.
> -
>
> Key: YARN-5112
> URL: https://issues.apache.org/jira/browse/YARN-5112
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5112.2.patch, YARN-5112.3.patch, YARN-5112.4.patch,
> YARN-5112.5.patch, YARN-5112.6.patch, YARN-5112.patch
>
>
> When there are a lot of apps to recover in NM store, NM prints these two
> lines for each app, which gets annoying.
> {code}
> 2015-10-13 01:58:40,277 WARN logaggregation.LogAggregationService
> (LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root
> Log Dir [/app-logs] already exist, but with incorrect permissions. Expected:
> [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple
> users.
> 336 2015-10-13 01:58:40,277 WARN logaggregation.AppLogAggregatorImpl
> (AppLogAggregatorImpl.java:(182)) - rollingMonitorInterval is set as
> -1. The log rolling mornitoring interval is disabled. The logs will be
> aggregated after this application is finished.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5126) Remove CHANGES.txt from branch-2
[ https://issues.apache.org/jira/browse/YARN-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295741#comment-15295741 ] Arun Suresh commented on YARN-5126: --- +1, thanks for reporting [~ajisakaa] > Remove CHANGES.txt from branch-2 > > > Key: YARN-5126 > URL: https://issues.apache.org/jira/browse/YARN-5126 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Labels: newbie > Attachments: YARN-5126.branch-2.01.patch > > > CHANGES.txt was added accidentally when [cherry-picking > YARN-2882|https://github.com/apache/hadoop/commit/2c6c701affb99f250ec25abae7f76c44bc5b701a]. > We should remove the changelog. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5126) Remove CHANGES.txt from branch-2
[
https://issues.apache.org/jira/browse/YARN-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295737#comment-15295737
]
Hadoop QA commented on YARN-5126:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 24s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:babe025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805552/YARN-5126.branch-2.01.patch
|
| JIRA Issue | YARN-5126 |
| Optional Tests | asflicense |
| uname | Linux f63299b13ce8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / 094bc72 |
| modules | C: hadoop-yarn-project U: hadoop-yarn-project |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/11624/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Remove CHANGES.txt from branch-2
>
>
> Key: YARN-5126
> URL: https://issues.apache.org/jira/browse/YARN-5126
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Attachments: YARN-5126.branch-2.01.patch
>
>
> CHANGES.txt was added accidentally when [cherry-picking
> YARN-2882|https://github.com/apache/hadoop/commit/2c6c701affb99f250ec25abae7f76c44bc5b701a].
> We should remove the changelog.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5126) Remove CHANGES.txt from branch-2
[ https://issues.apache.org/jira/browse/YARN-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated YARN-5126: Attachment: YARN-5126.branch-2.01.patch > Remove CHANGES.txt from branch-2 > > > Key: YARN-5126 > URL: https://issues.apache.org/jira/browse/YARN-5126 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Akira AJISAKA > Labels: newbie > Attachments: YARN-5126.branch-2.01.patch > > > CHANGES.txt was added accidentally when [cherry-picking > YARN-2882|https://github.com/apache/hadoop/commit/2c6c701affb99f250ec25abae7f76c44bc5b701a]. > We should remove the changelog. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2877) Extend YARN to support distributed scheduling
[ https://issues.apache.org/jira/browse/YARN-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295735#comment-15295735 ] Akira AJISAKA commented on YARN-2877: - Hi [~asuresh], would you update the fix version field of the cherry-picked jiras? In addition, CHANGES.txt was added when cherry-picking YARN-2882. I filed YARN-5126 to remove it. > Extend YARN to support distributed scheduling > - > > Key: YARN-2877 > URL: https://issues.apache.org/jira/browse/YARN-2877 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, resourcemanager >Reporter: Sriram Rao >Assignee: Konstantinos Karanasos > Attachments: distributed-scheduling-design-doc_v1.pdf > > > This is an umbrella JIRA that proposes to extend YARN to support distributed > scheduling. Briefly, some of the motivations for distributed scheduling are > the following: > 1. Improve cluster utilization by opportunistically executing tasks otherwise > idle resources on individual machines. > 2. Reduce allocation latency. Tasks where the scheduling time dominates > (i.e., task execution time is much less compared to the time required for > obtaining a container from the RM). > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5126) Remove CHANGES.txt from branch-2
Akira AJISAKA created YARN-5126: --- Summary: Remove CHANGES.txt from branch-2 Key: YARN-5126 URL: https://issues.apache.org/jira/browse/YARN-5126 Project: Hadoop YARN Issue Type: Bug Reporter: Akira AJISAKA CHANGES.txt was added accidentally when [cherry-picking YARN-2882|https://github.com/apache/hadoop/commit/2c6c701affb99f250ec25abae7f76c44bc5b701a]. We should remove the changelog. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5125) Datanode and nodemanger appear in jps but arent start correctly.
Jon created YARN-5125: - Summary: Datanode and nodemanger appear in jps but arent start correctly. Key: YARN-5125 URL: https://issues.apache.org/jira/browse/YARN-5125 Project: Hadoop YARN Issue Type: Bug Reporter: Jon when I execute jps command datanode and nodemanagger appears, but it seems that is not starting correctly, because if I check logs it seems that they arent running correctly. Namenode log: 2016-05-22 11:40:37,725 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 2016-05-22 11:40:37,731 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode [] 2016-05-22 11:40:38,109 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2016-05-22 11:40:38,217 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2016-05-22 11:40:38,217 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started 2016-05-22 11:40:38,220 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is hdfs://masternode:9000 2016-05-22 11:40:38,221 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use masternode:9000 to access this namenode/service. 2016-05-22 11:40:38,412 INFO org.apache.hadoop.hdfs.DFSUtil: Starting Web-server for hdfs at: http://0.0.0.0:50070 2016-05-22 11:40:38,486 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-05-22 11:40:38,496 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2016-05-22 11:40:38,503 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.namenode is not defined 2016-05-22 11:40:38,509 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2016-05-22 11:40:38,513 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs 2016-05-22 11:40:38,514 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2016-05-22 11:40:38,514 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2016-05-22 11:40:38,543 INFO org.apache.hadoop.http.HttpServer2: Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hadoop.hdfs.web.AuthFilter) 2016-05-22 11:40:38,544 INFO org.apache.hadoop.http.HttpServer2: addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/* 2016-05-22 11:40:38,564 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 50070 2016-05-22 11:40:38,564 INFO org.mortbay.log: jetty-6.1.26 2016-05-22 11:40:38,751 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:50070 2016-05-22 11:40:38,785 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hadoopadmin/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 2016-05-22 11:40:38,785 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hadoopadmin/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 2016-05-22 11:40:38,785 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack of redundant storage directories! 2016-05-22 11:40:38,785 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of data loss due to lack of redundant storage directories! 2016-05-22 11:40:38,791 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hadoopadmin/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 2016-05-22 11:40:38,791 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hadoopadmin/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 2016-05-22 11:40:38,823 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: No KeyProvider found. 2016-05-22 11:40:38,824 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsLock is fair:true 2016-05-22 11:40:38,866 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 2016-05-22 11:40:38,867 INFO
[jira] [Comment Edited] (YARN-5123) SQL based RM state store
[ https://issues.apache.org/jira/browse/YARN-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295155#comment-15295155 ] Lavkesh Lahngir edited comment on YARN-5123 at 5/22/16 11:28 AM: - Few comments: 1. We applied this in hdp2.2.4 which is essentially hadoop-2.6.0 or similar. 2. Before applying this to higher versions, we need to check, if there are any API changes. 3. For this patch to work, Postgres jar should be present in the RM classpath. 4. See YarnConfiguration for the new configurations. 5. Make sure that the javac version is 1.7 or higher. was (Author: lavkesh): Few comments: 1. We applied this in hdp2.2.4 which is essentially hadoop-2.6.0 or similar. 2. Before applying this to higher versions, we need to check, if there are any API changes. 3. For this patch to work, postgres jar should be present in the RM classpath. 4. See YarnConfiguration for new configurations. > SQL based RM state store > > > Key: YARN-5123 > URL: https://issues.apache.org/jira/browse/YARN-5123 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.6.0 >Reporter: Lavkesh Lahngir > Attachments: sqlstatestore.patch > > > In our setup, zookeeper based RM state store didn't work. We ended up > implementing our own SQL based state store. Here is a patch, if anybody else > wants to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
