[jira] [Commented] (HDFS-11087) NamenodeFsck should check if the output writer is still writable.
[
https://issues.apache.org/jira/browse/HDFS-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669056#comment-15669056
]
Brahma Reddy Battula commented on HDFS-11087:
-
That will be good idea .Everybody will aware of it before committing to
branch-2.7 and we might not miss any entry.
bq.Should we consider adding a pre-commit hook?
can we do this..?
Even I sent [mail in
common-dev|http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201609.mbox/browser]
(sub: Updation of Change.txt in branch-2.7 on Sept-30),but I did not hear any
thoughts there.
If we need any discussion in mailing list ,may be we can use this..?
> NamenodeFsck should check if the output writer is still writable.
> -
>
> Key: HDFS-11087
> URL: https://issues.apache.org/jira/browse/HDFS-11087
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.5
>Reporter: Konstantin Shvachko
>Assignee: Erik Krogen
> Fix For: 2.8.0, 2.9.0, 2.7.4
>
> Attachments: HDFS-11087-branch-2.000.patch,
> HDFS-11087-branch-2.001.patch, HDFS-11087.branch-2.000.patch
>
>
> {{NamenodeFsck}} keeps running even after the client was interrupted. So if
> you start {{fsck /}} on a large namespace and kill the client, the NameNode
> will keep traversing the tree for hours although there is nobody to receive
> the result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11087) NamenodeFsck should check if the output writer is still writable.
[
https://issues.apache.org/jira/browse/HDFS-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667978#comment-15667978
]
Konstantin Shvachko commented on HDFS-11087:
Hey [~brahmareddy], looks like everybody is forgetting to update CHANGES.txt
for branch-2.7. Should we consider adding a pre-commit hook?
> NamenodeFsck should check if the output writer is still writable.
> -
>
> Key: HDFS-11087
> URL: https://issues.apache.org/jira/browse/HDFS-11087
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.5
>Reporter: Konstantin Shvachko
>Assignee: Erik Krogen
> Fix For: 2.8.0, 2.9.0, 2.7.4
>
> Attachments: HDFS-11087-branch-2.000.patch,
> HDFS-11087-branch-2.001.patch, HDFS-11087.branch-2.000.patch
>
>
> {{NamenodeFsck}} keeps running even after the client was interrupted. So if
> you start {{fsck /}} on a large namespace and kill the client, the NameNode
> will keep traversing the tree for hours although there is nobody to receive
> the result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11087) NamenodeFsck should check if the output writer is still writable.
[
https://issues.apache.org/jira/browse/HDFS-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1543#comment-1543
]
Brahma Reddy Battula commented on HDFS-11087:
-
Looks to be {{CHANGES.txt}} not updated for this. can you update..?
> NamenodeFsck should check if the output writer is still writable.
> -
>
> Key: HDFS-11087
> URL: https://issues.apache.org/jira/browse/HDFS-11087
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.5
>Reporter: Konstantin Shvachko
>Assignee: Erik Krogen
> Fix For: 2.8.0, 2.9.0, 2.7.4
>
> Attachments: HDFS-11087-branch-2.000.patch,
> HDFS-11087-branch-2.001.patch, HDFS-11087.branch-2.000.patch
>
>
> {{NamenodeFsck}} keeps running even after the client was interrupted. So if
> you start {{fsck /}} on a large namespace and kill the client, the NameNode
> will keep traversing the tree for hours although there is nobody to receive
> the result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11087) NamenodeFsck should check if the output writer is still writable.
[
https://issues.apache.org/jira/browse/HDFS-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15659012#comment-15659012
]
Hadoop QA commented on HDFS-11087:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 105 unchanged - 2 fixed = 105 total (was 107) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 45s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_101 Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot |
| | hadoop.tools.TestDelegationTokenFetcher |
| JDK v1.7.0_111 Failed junit tests | hadoop.tools.TestDelegationTokenFetcher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HDFS-11087 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838638/HDFS-11087-branch-2.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall
[jira] [Commented] (HDFS-11087) NamenodeFsck should check if the output writer is still writable.
[
https://issues.apache.org/jira/browse/HDFS-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658651#comment-15658651
]
Konstantin Shvachko commented on HDFS-11087:
This makes sense, Erik.
Could you please add a comment before {{checkError()}} based on its JavaDoc.
It is not clear that the intent here to flush the stream while checking the
error.
+1 on the patch modular the comment line. Will commit once add it.
> NamenodeFsck should check if the output writer is still writable.
> -
>
> Key: HDFS-11087
> URL: https://issues.apache.org/jira/browse/HDFS-11087
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.5
>Reporter: Konstantin Shvachko
>Assignee: Erik Krogen
> Attachments: HDFS-11087-branch-2.000.patch,
> HDFS-11087.branch-2.000.patch
>
>
> {{NamenodeFsck}} keeps running even after the client was interrupted. So if
> you start {{fsck /}} on a large namespace and kill the client, the NameNode
> will keep traversing the tree for hours although there is nobody to receive
> the result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11087) NamenodeFsck should check if the output writer is still writable.
[
https://issues.apache.org/jira/browse/HDFS-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652584#comment-15652584
]
Hadoop QA commented on HDFS-11087:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 105 unchanged - 2 fixed = 105 total (was 107) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 48m 16s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_111 Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| JDK v1.7.0_111 Failed junit tests |
hadoop.hdfs.server.datanode.TestFsDatasetCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HDFS-11087 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838251/HDFS-11087-branch-2.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6657d7c280bd 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26
[jira] [Commented] (HDFS-11087) NamenodeFsck should check if the output writer is still writable.
[
https://issues.apache.org/jira/browse/HDFS-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652333#comment-15652333
]
Erik Krogen commented on HDFS-11087:
Whoops, thank you!
> NamenodeFsck should check if the output writer is still writable.
> -
>
> Key: HDFS-11087
> URL: https://issues.apache.org/jira/browse/HDFS-11087
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.5
>Reporter: Konstantin Shvachko
>Assignee: Erik Krogen
> Attachments: HDFS-11087-branch-2.000.patch,
> HDFS-11087.branch-2.000.patch
>
>
> {{NamenodeFsck}} keeps running even after the client was interrupted. So if
> you start {{fsck /}} on a large namespace and kill the client, the NameNode
> will keep traversing the tree for hours although there is nobody to receive
> the result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11087) NamenodeFsck should check if the output writer is still writable.
[
https://issues.apache.org/jira/browse/HDFS-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652327#comment-15652327
]
Mingliang Liu commented on HDFS-11087:
--
HDFS-11087.branch-2.000.patch -> HDFS-11087-branch-2.000.patch
> NamenodeFsck should check if the output writer is still writable.
> -
>
> Key: HDFS-11087
> URL: https://issues.apache.org/jira/browse/HDFS-11087
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.5
>Reporter: Konstantin Shvachko
>Assignee: Erik Krogen
> Attachments: HDFS-11087.branch-2.000.patch
>
>
> {{NamenodeFsck}} keeps running even after the client was interrupted. So if
> you start {{fsck /}} on a large namespace and kill the client, the NameNode
> will keep traversing the tree for hours although there is nobody to receive
> the result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11087) NamenodeFsck should check if the output writer is still writable.
[
https://issues.apache.org/jira/browse/HDFS-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652321#comment-15652321
]
Hadoop QA commented on HDFS-11087:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 5m
34s{color} | {color:red} Docker failed to build yetus/hadoop:b59b8b7. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11087 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838247/HDFS-11087.branch-2.000.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17489/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> NamenodeFsck should check if the output writer is still writable.
> -
>
> Key: HDFS-11087
> URL: https://issues.apache.org/jira/browse/HDFS-11087
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.5
>Reporter: Konstantin Shvachko
>Assignee: Erik Krogen
> Attachments: HDFS-11087.branch-2.000.patch
>
>
> {{NamenodeFsck}} keeps running even after the client was interrupted. So if
> you start {{fsck /}} on a large namespace and kill the client, the NameNode
> will keep traversing the tree for hours although there is nobody to receive
> the result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11087) NamenodeFsck should check if the output writer is still writable.
[
https://issues.apache.org/jira/browse/HDFS-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627060#comment-15627060
]
Konstantin Shvachko commented on HDFS-11087:
The problem is that {{PrintWriter.ensureOpen()}} does not really check the
status of the underlying Writer to the HttpConnection, which has already been
closed.
This is probably Jetty 6 problem. On trunk with upgraded Jetty I see that
server side fsck fails after client is interrupted. So this is applicable to
all v.2 branches only.
I propose to fix this by calling {{PrintWriter.checkError()}} before each round
of printing in {{NamenodeFsck.check()}}.
> NamenodeFsck should check if the output writer is still writable.
> -
>
> Key: HDFS-11087
> URL: https://issues.apache.org/jira/browse/HDFS-11087
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.5
>Reporter: Konstantin Shvachko
>
> {{NamenodeFsck}} keeps running even after the client was interrupted. So if
> you start {{fsck /}} on a large namespace and kill the client, the NameNode
> will keep traversing the tree for hours although there is nobody to receive
> the result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
