[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[
https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025714#comment-15025714
]
Hadoop QA commented on YARN-4365:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 4s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 50s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 18s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12774169/YARN-4365-3.patch |
| JIRA Issue | YARN-4365 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c0148886fec2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4ac6799 |
| findbugs | v3.0.0 |
| JDK v1.7.0_85 Test Results |
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[ https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025722#comment-15025722 ] Jason Lowe commented on YARN-4365: -- +1 latest patch lgtm. Committing this. > FileSystemNodeLabelStore should check for root dir existence on startup > --- > > Key: YARN-4365 > URL: https://issues.apache.org/jira/browse/YARN-4365 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Kuhu Shukla > Attachments: YARN-4365-1.patch, YARN-4365-2.patch, YARN-4365-3.patch > > > If the namenode is in safe mode for some reason then FileSystemNodeLabelStore > will prevent the RM from starting since it unconditionally tries to create > the root directory for the label store. If the root directory already exists > and no labels are changing then we shouldn't prevent the RM from starting > even if the namenode is in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[ https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025325#comment-15025325 ] Kuhu Shukla commented on YARN-4365: --- [~jlowe], Request for comments/review. Thanks a lot. > FileSystemNodeLabelStore should check for root dir existence on startup > --- > > Key: YARN-4365 > URL: https://issues.apache.org/jira/browse/YARN-4365 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Kuhu Shukla > Attachments: YARN-4365-1.patch, YARN-4365-2.patch > > > If the namenode is in safe mode for some reason then FileSystemNodeLabelStore > will prevent the RM from starting since it unconditionally tries to create > the root directory for the label store. If the root directory already exists > and no labels are changing then we shouldn't prevent the RM from starting > even if the namenode is in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[ https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025470#comment-15025470 ] Jason Lowe commented on YARN-4365: -- Rather than poking the store into the manager object, we don't need the manager object at all. We can just test the store object directly. Note how the test is simply using the manager object as a place to hold the store but only ever manipulates the store directly by reaching into the manager object. > FileSystemNodeLabelStore should check for root dir existence on startup > --- > > Key: YARN-4365 > URL: https://issues.apache.org/jira/browse/YARN-4365 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Kuhu Shukla > Attachments: YARN-4365-1.patch, YARN-4365-2.patch > > > If the namenode is in safe mode for some reason then FileSystemNodeLabelStore > will prevent the RM from starting since it unconditionally tries to create > the root directory for the label store. If the root directory already exists > and no labels are changing then we shouldn't prevent the RM from starting > even if the namenode is in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[
https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025311#comment-15025311
]
Hadoop QA commented on YARN-4365:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 54s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 14s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12774142/YARN-4365-2.patch |
| JIRA Issue | YARN-4365 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d650dc1aa099 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / db4cab2 |
| findbugs | v3.0.0 |
| JDK v1.7.0_85 Test Results |
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[
https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024844#comment-15024844
]
Jason Lowe commented on YARN-4365:
--
Thanks for the patch, Kuhu!
There appears to be a mix of overriding and mocking approaches in the test that
leads to a confusing test. For example, setFileSystem was promoted to
protected scope yet that's unnecessary in the current patch. The test is also
spying on the node label manager and mocking Configuration unnecessarily.
Instead of all the mocking and stubbing, I think it would be more
straightforward to simply override setFileSystem and have the test use a "real"
FileSystemNodeLabelsStore rather than a mocked one where we pass through
various methods. The only mock at that point would be the filesystem that
would be set in the overridden setFileSystem method.
There's also a misleading comment in the test:
{code}
// File Exists returns true the third time
Mockito.when(myStore.fs.exists(Mockito.any(Path.class))).thenReturn(false);
{code}
> FileSystemNodeLabelStore should check for root dir existence on startup
> ---
>
> Key: YARN-4365
> URL: https://issues.apache.org/jira/browse/YARN-4365
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.2
>Reporter: Jason Lowe
>Assignee: Kuhu Shukla
> Attachments: YARN-4365-1.patch
>
>
> If the namenode is in safe mode for some reason then FileSystemNodeLabelStore
> will prevent the RM from starting since it unconditionally tries to create
> the root directory for the label store. If the root directory already exists
> and no labels are changing then we shouldn't prevent the RM from starting
> even if the namenode is in safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[ https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024847#comment-15024847 ] Kuhu Shukla commented on YARN-4365: --- Thanks a lot Jason. Will update revised patch shorty. > FileSystemNodeLabelStore should check for root dir existence on startup > --- > > Key: YARN-4365 > URL: https://issues.apache.org/jira/browse/YARN-4365 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Kuhu Shukla > Attachments: YARN-4365-1.patch > > > If the namenode is in safe mode for some reason then FileSystemNodeLabelStore > will prevent the RM from starting since it unconditionally tries to create > the root directory for the label store. If the root directory already exists > and no labels are changing then we shouldn't prevent the RM from starting > even if the namenode is in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[ https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025842#comment-15025842 ] Hudson commented on YARN-4365: -- FAILURE: Integrated in Hadoop-trunk-Commit #8882 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8882/]) YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java > FileSystemNodeLabelStore should check for root dir existence on startup > --- > > Key: YARN-4365 > URL: https://issues.apache.org/jira/browse/YARN-4365 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Kuhu Shukla > Fix For: 2.6.3, 2.7.3 > > Attachments: YARN-4365-1.patch, YARN-4365-2.patch, YARN-4365-3.patch > > > If the namenode is in safe mode for some reason then FileSystemNodeLabelStore > will prevent the RM from starting since it unconditionally tries to create > the root directory for the label store. If the root directory already exists > and no labels are changing then we shouldn't prevent the RM from starting > even if the namenode is in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[ https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025891#comment-15025891 ] Hudson commented on YARN-4365: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2660 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2660/]) YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java > FileSystemNodeLabelStore should check for root dir existence on startup > --- > > Key: YARN-4365 > URL: https://issues.apache.org/jira/browse/YARN-4365 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Kuhu Shukla > Fix For: 2.6.3, 2.7.3 > > Attachments: YARN-4365-1.patch, YARN-4365-2.patch, YARN-4365-3.patch > > > If the namenode is in safe mode for some reason then FileSystemNodeLabelStore > will prevent the RM from starting since it unconditionally tries to create > the root directory for the label store. If the root directory already exists > and no labels are changing then we shouldn't prevent the RM from starting > even if the namenode is in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[ https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025915#comment-15025915 ] Hudson commented on YARN-4365: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #719 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/719/]) YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java > FileSystemNodeLabelStore should check for root dir existence on startup > --- > > Key: YARN-4365 > URL: https://issues.apache.org/jira/browse/YARN-4365 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Kuhu Shukla > Fix For: 2.6.3, 2.7.3 > > Attachments: YARN-4365-1.patch, YARN-4365-2.patch, YARN-4365-3.patch > > > If the namenode is in safe mode for some reason then FileSystemNodeLabelStore > will prevent the RM from starting since it unconditionally tries to create > the root directory for the label store. If the root directory already exists > and no labels are changing then we shouldn't prevent the RM from starting > even if the namenode is in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[ https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025921#comment-15025921 ] Hudson commented on YARN-4365: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1451 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1451/]) YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java > FileSystemNodeLabelStore should check for root dir existence on startup > --- > > Key: YARN-4365 > URL: https://issues.apache.org/jira/browse/YARN-4365 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Kuhu Shukla > Fix For: 2.6.3, 2.7.3 > > Attachments: YARN-4365-1.patch, YARN-4365-2.patch, YARN-4365-3.patch > > > If the namenode is in safe mode for some reason then FileSystemNodeLabelStore > will prevent the RM from starting since it unconditionally tries to create > the root directory for the label store. If the root directory already exists > and no labels are changing then we shouldn't prevent the RM from starting > even if the namenode is in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[ https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026070#comment-15026070 ] Hudson commented on YARN-4365: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #729 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/729/]) YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java > FileSystemNodeLabelStore should check for root dir existence on startup > --- > > Key: YARN-4365 > URL: https://issues.apache.org/jira/browse/YARN-4365 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Kuhu Shukla > Fix For: 2.6.3, 2.7.3 > > Attachments: YARN-4365-1.patch, YARN-4365-2.patch, YARN-4365-3.patch > > > If the namenode is in safe mode for some reason then FileSystemNodeLabelStore > will prevent the RM from starting since it unconditionally tries to create > the root directory for the label store. If the root directory already exists > and no labels are changing then we shouldn't prevent the RM from starting > even if the namenode is in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[ https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026143#comment-15026143 ] Hudson commented on YARN-4365: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2577 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2577/]) YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java > FileSystemNodeLabelStore should check for root dir existence on startup > --- > > Key: YARN-4365 > URL: https://issues.apache.org/jira/browse/YARN-4365 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Kuhu Shukla > Fix For: 2.6.3, 2.7.3 > > Attachments: YARN-4365-1.patch, YARN-4365-2.patch, YARN-4365-3.patch > > > If the namenode is in safe mode for some reason then FileSystemNodeLabelStore > will prevent the RM from starting since it unconditionally tries to create > the root directory for the label store. If the root directory already exists > and no labels are changing then we shouldn't prevent the RM from starting > even if the namenode is in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[ https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026198#comment-15026198 ] Hudson commented on YARN-4365: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #640 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/640/]) YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java * hadoop-yarn-project/CHANGES.txt > FileSystemNodeLabelStore should check for root dir existence on startup > --- > > Key: YARN-4365 > URL: https://issues.apache.org/jira/browse/YARN-4365 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Kuhu Shukla > Fix For: 2.6.3, 2.7.3 > > Attachments: YARN-4365-1.patch, YARN-4365-2.patch, YARN-4365-3.patch > > > If the namenode is in safe mode for some reason then FileSystemNodeLabelStore > will prevent the RM from starting since it unconditionally tries to create > the root directory for the label store. If the root directory already exists > and no labels are changing then we shouldn't prevent the RM from starting > even if the namenode is in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[
https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15022816#comment-15022816
]
Kuhu Shukla commented on YARN-4365:
---
Test failure is irreproducible locally and is unrelated to the patch as far as
I can see. Findbugs warnings are coming from
{{org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl}},
which are not related to this patch.
> FileSystemNodeLabelStore should check for root dir existence on startup
> ---
>
> Key: YARN-4365
> URL: https://issues.apache.org/jira/browse/YARN-4365
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.2
>Reporter: Jason Lowe
>Assignee: Kuhu Shukla
> Attachments: YARN-4365-1.patch
>
>
> If the namenode is in safe mode for some reason then FileSystemNodeLabelStore
> will prevent the RM from starting since it unconditionally tries to create
> the root directory for the label store. If the root directory already exists
> and no labels are changing then we shouldn't prevent the RM from starting
> even if the namenode is in safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[
https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018550#comment-15018550
]
Hadoop QA commented on YARN-4365:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 31s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
trunk has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 17s {color}
| {color:red} hadoop-yarn-common in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 24s {color}
| {color:red} hadoop-yarn-common in the patch failed with JDK v1.7.0_85.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.yarn.webapp.TestWebApp |
| JDK v1.7.0_85 Failed junit tests | hadoop.yarn.webapp.TestWebApp |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:date2015-11-20 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12773553/YARN-4365-1.patch |
| JIRA Issue | YARN-4365 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ff28fcdb5368 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64
[jira] [Commented] (YARN-4365) FileSystemNodeLabelStore should check for root dir existence on startup
[
https://issues.apache.org/jira/browse/YARN-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008956#comment-15008956
]
Jason Lowe commented on YARN-4365:
--
Sample stacktrace when the problem occurs:
{noformat}
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException):
Cannot create directory /path/to/labelstore. Name node is in safe mode.
It was turned on manually. Use "hdfs dfsadmin -safemode leave" to turn safe
mode off.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1322)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3887)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:988)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server.call(Server.java:2297)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:654)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:621)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1680)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2247)
at org.apache.hadoop.ipc.Client.call(Client.java:1458)
at org.apache.hadoop.ipc.Client.call(Client.java:1395)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy22.mkdirs(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:558)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy23.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3008)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2978)
at
org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
at
org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
at
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1877)
at
org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.init(FileSystemNodeLabelsStore.java:87)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.initNodeLabelStore(CommonNodeLabelsManager.java:219)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStart(CommonNodeLabelsManager.java:233)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:579)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:964)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1005)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1001)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1680)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1001)
at
