[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549491#comment-15549491
]
Jing Zhao commented on HDFS-10826:
--
I've committed the patch into trunk. Thanks for the contribution,
[~tasanuma0829]. Thanks for the review, [~ajisakaa] and [~jojochuang].
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch,
> HDFS-10826.4.patch, HDFS-10826.5.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15547167#comment-15547167
]
Hadoop QA commented on HDFS-10826:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 77m
37s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10826 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831055/HDFS-10826.5.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bf8590116da0 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 44f48ee |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17009/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17009/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17009/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15547037#comment-15547037
]
Wei-Chiu Chuang commented on HDFS-10826:
The patch looks good to me too.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch,
> HDFS-10826.4.patch, HDFS-10826.5.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546996#comment-15546996
]
Jing Zhao commented on HDFS-10826:
--
The latest patch looks good to me. To do further code refactoring in HDFS-10933
also sounds good to me. Do you have further comments, [~jojochuang]?
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch,
> HDFS-10826.4.patch, HDFS-10826.5.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535092#comment-15535092
]
Hadoop QA commented on HDFS-10826:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m
21s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10826 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831055/HDFS-10826.5.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ca7ba174e8aa 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 10be459 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16942/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16942/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16942/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534943#comment-15534943
]
Takanobu Asanuma commented on HDFS-10826:
-
I created HDFS-10933 for refactoring {{TestFsck}}.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch,
> HDFS-10826.4.patch, HDFS-10826.5.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534077#comment-15534077
]
Wei-Chiu Chuang commented on HDFS-10826:
Hello [~tasanuma0829] thanks for your contribution and [~ajisakaa] and
[~jingzhao] for the review. I am late to review this patch but if you don't
mind there are a little minor improvement that can be made:
in TestFsck
{code}
// make an unrecoverable ec file with corrupted blocks
for(int i = 0; i < 4; i++) {
{code}
* Instead of hardcode "4", it would be more meaningful to use parityBlocks+1 I
assume?
{code}
// Read the file to trigger reportBadBlocks
try {
IOUtils.copyBytes(fs.open(file), new
IOUtils.NullOutputStream(), conf,
true);
} catch (IOException ie) {
// Ignore exception
}
{code}
* Would it be possible to verify the exception thrown is expected when // Read
the file to trigger reportBadBlocks?
{code}
if (fs != null) {
try {
fs.close();
} catch (Exception e) {
}
}
if (cluster != null) {
cluster.shutdown();
}
{code}
You should just let the exception be thrown instead of catch and ignore it.
A better improvement could use @Before @After annotation to initialize/close
cluster and fs object. Then you do not even need to try...catch.
{code:title=waitForUnrecoverableBlockGroup()}
catch (Exception e) {
FSImage.LOG.error("Exception caught", e);
Assert.fail("Caught unexpected exception.");
}
{code}
I wonder if there is a more appropriate log object than FSImage.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch,
> HDFS-10826.4.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534006#comment-15534006
]
Hadoop QA commented on HDFS-10826:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 61m
46s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10826 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830865/HDFS-10826.4.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux cb4ee846c65f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 236ac77 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16928/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16928/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16928/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15532283#comment-15532283
]
Takanobu Asanuma commented on HDFS-10826:
-
The failed tests are passed in my local computer.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch,
> HDFS-10826.4.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15532248#comment-15532248
]
Hadoop QA commented on HDFS-10826:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 38s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestCrcCorruption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10826 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830865/HDFS-10826.4.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 00e8daa47b11 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 47f8092 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16922/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16922/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16922/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16922/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> -
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530710#comment-15530710
]
Jing Zhao commented on HDFS-10826:
--
# The failure of TestLeaseRecoveryStriped seems related. Could you please take
a look, [~tasanuma0829]?
# {{countNodes}} has already been called in {{createLocatedBlock}}. We can
reuse the result.
{code}
1071final boolean isCorrupt;
1072if (blk.isStriped()) {
1073 BlockInfoStriped sblk = (BlockInfoStriped) blk;
1074 isCorrupt = numCorruptReplicas != 0 &&
1075 countNodes(blk).liveReplicas() < sblk.getRealDataBlockNum();
1076} else {
1077 isCorrupt = numCorruptReplicas != 0 && numCorruptReplicas ==
numNodes;
1078}
{code}
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch,
> HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529238#comment-15529238
]
Hadoop QA commented on HDFS-10826:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 503 unchanged - 1 fixed = 504 total (was 504) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestLeaseRecoveryStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10826 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830648/HDFS-10826.3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2130732a0ad9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 03f519a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16902/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16902/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16902/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16902/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> --
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15528613#comment-15528613
]
Takanobu Asanuma commented on HDFS-10826:
-
[~jingzhao]
Thank you for your review and proposal! I agree with that feature.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524408#comment-15524408
]
Jing Zhao commented on HDFS-10826:
--
Thanks for working on this, [~tasanuma0829]. And thanks for the review,
[~ajisakaa].
The patch looks good to me. One minor comment is about the change in
NameNodeFsck: currently we do not print detailed replica info for a missing
replicated block. However for a striped block with less than min-replication
number of internal blocks, we may still want to print out its internal block
information even though we can claim it as MISSING.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15522530#comment-15522530
]
Takanobu Asanuma commented on HDFS-10826:
-
[~ajisakaa]
Thank you for your kind review! I will upload a new patch soon.
> In the following code, why do you need to run fsck multiple times?
There is a delay after reportBadBlcoks(or removeDetanode) before the result of
fsck includes "UNRECOVERABLE BLOCK GROUPS". I will use GenericTestUtils.waitFor
in the next patch.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15522116#comment-15522116
]
Akira Ajisaka commented on HDFS-10826:
--
Mostly looks good to me. Some comments from me:
* Would you fix checkstyle warnings?
* Would you update the following comment?
{code:title=NamenodeFsck.java}
if (isMissing && !isCorrupt) {
// If the block is corrupted, it means all its available replicas are
// corrupted. We don't mark it as missing given these available replicas
// might still be accessible as the block might be incorrectly marked as
// corrupted by client machines.
{code}
* In the following code, why do you need to run fsck multiple times?
{code}
while (!outStr.contains("UNRECOVERABLE BLOCK GROUPS")) {
bStream = new ByteArrayOutputStream();
out = new PrintStream(bStream, true);
ToolRunner.run(new DFSck(conf, out), new String[]{ "/" });
outStr = bStream.toString();
}
{code}
this code can cause infinite loop, so I'm thinking we need to set a retry
limit. Maybe we can use GenericTestUtils.waitFor method for such situation.
* {{Thread.sleep}} is missing in the while loop in testFsckMissingECFile.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15521760#comment-15521760
]
Takanobu Asanuma commented on HDFS-10826:
-
All failed tests are passed in my laptop and seem not to be related to the
patch.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515621#comment-15515621
]
Hadoop QA commented on HDFS-10826:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 21 new + 503 unchanged - 1 fixed = 524 total (was 504) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 59s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestDatanodeRegistration |
| | hadoop.hdfs.TestMissingBlocksAlert |
| | hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10826 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829991/HDFS-10826.2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5958066f73e1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d85d9b2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16840/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16840/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16840/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16840/
