[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[ https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750496#comment-16750496 ] Hudson commented on HDFS-14061: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15813 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15813/]) HDFS-14061. Check if the cluster topology supports the EC policy before (weichiu: rev 951cdd7e4cbe68284620f6805f85c51301150c58) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/ECTopologyVerifier.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/ECAdmin.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestECAdmin.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Check if the cluster topology supports the EC policy before setting, enabling > or adding it > -- > > Key: HDFS-14061 > URL: https://issues.apache.org/jira/browse/HDFS-14061 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, hdfs >Affects Versions: 3.1.1 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14061.001.patch, HDFS-14061.002.patch, > HDFS-14061.003.patch, HDFS-14061.004.patch, HDFS-14061.005.patch > > > HDFS-12946 introduces a command for verifying if there are enough racks and > datanodes for the enabled erasure coding policies. > This verification could be executed for the erasure coding policy before > enabling, setting or adding it and a warning message could be written if the > verify fails, or the policy setting could be even failed in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[ https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750485#comment-16750485 ] Wei-Chiu Chuang commented on HDFS-14061: +1 on the 005 patch. Will commit shortly. > Check if the cluster topology supports the EC policy before setting, enabling > or adding it > -- > > Key: HDFS-14061 > URL: https://issues.apache.org/jira/browse/HDFS-14061 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, hdfs >Affects Versions: 3.1.1 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HDFS-14061.001.patch, HDFS-14061.002.patch, > HDFS-14061.003.patch, HDFS-14061.004.patch, HDFS-14061.005.patch > > > HDFS-12946 introduces a command for verifying if there are enough racks and > datanodes for the enabled erasure coding policies. > This verification could be executed for the erasure coding policy before > enabling, setting or adding it and a warning message could be written if the > verify fails, or the policy setting could be even failed in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[
https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742022#comment-16742022
]
Hadoop QA commented on HDFS-14061:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
|
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14061 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954811/HDFS-14061.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 416e55fea8db 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3bb745d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25979/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25979/testReport/ |
| Max. process+thread count | 3021 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[ https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16741918#comment-16741918 ] Kitti Nanasi commented on HDFS-14061: - Thanks for the review [~jojochuang]! I will clean up the log messages in HDFS-14125, so for now I will leave them as is. I addressed your comments in the newest patch. About using all DataNodes in the verification, we did not consider in -HDFS-12946- to use only a subset of all DataNodes, that would make the verification a bit stricter and I am fine with that as well, though it would not make much difference in my opinion. Let me know if you have any suggestions on that. > Check if the cluster topology supports the EC policy before setting, enabling > or adding it > -- > > Key: HDFS-14061 > URL: https://issues.apache.org/jira/browse/HDFS-14061 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, hdfs >Affects Versions: 3.1.1 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HDFS-14061.001.patch, HDFS-14061.002.patch, > HDFS-14061.003.patch, HDFS-14061.004.patch, HDFS-14061.005.patch > > > HDFS-12946 introduces a command for verifying if there are enough racks and > datanodes for the enabled erasure coding policies. > This verification could be executed for the erasure coding policy before > enabling, setting or adding it and a warning message could be written if the > verify fails, or the policy setting could be even failed in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[
https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740771#comment-16740771
]
Wei-Chiu Chuang commented on HDFS-14061:
Nit: better to use parameterized log format in slf4j. I see that you have
HDFS-14125 opened to address though.
{code}
System.out.println("Warning: The cluster setup does not support " +
{code}
Use System.err instead of System.out
{code}
final DatanodeInfo[] report = dfs.getClient().getNamenode()
.getDatanodeReport(HdfsConstants.DatanodeReportType.ALL);
{code}
So we're assuming we use all DataNodes in the cluster to verify if the EC
policy can be applied, regardless of whether they are dead/decomm'ed/in
maintenance. Probably makes sense. The code was inherited from HDFS-12946, and
I didn't check if this is considered. I'll just leave the note here.
Ideally getPolicy() should not return a null if policy name is not found. It
won't break the functionality here though because the policy is enabled prior
to that. So the policy is guaranteed to exist (unless a race condition where
admin removes the policy concurrently)
Test: FYI {{GenericTestUtils#SystemErrCapturer}} can be used for checking
syserr output. Something for you to consider next time.
> Check if the cluster topology supports the EC policy before setting, enabling
> or adding it
> --
>
> Key: HDFS-14061
> URL: https://issues.apache.org/jira/browse/HDFS-14061
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.1
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Attachments: HDFS-14061.001.patch, HDFS-14061.002.patch,
> HDFS-14061.003.patch, HDFS-14061.004.patch
>
>
> HDFS-12946 introduces a command for verifying if there are enough racks and
> datanodes for the enabled erasure coding policies.
> This verification could be executed for the erasure coding policy before
> enabling, setting or adding it and a warning message could be written if the
> verify fails, or the policy setting could be even failed in this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[
https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738226#comment-16738226
]
Hadoop QA commented on HDFS-14061:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 679 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
13s{color} | {color:red} The patch 768 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m
4s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}209m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
|
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14061 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954297/HDFS-14061.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b428615afcfd 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 709ddb1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/25938/artifact/out/whitespace-eol.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/25938/artifact/out/whitespace-tabs.txt
|
| unit |
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[ https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738031#comment-16738031 ] Kitti Nanasi commented on HDFS-14061: - Thanks [~shwetayakkali] for the comment! I added messages for asserts in patch 004. > Check if the cluster topology supports the EC policy before setting, enabling > or adding it > -- > > Key: HDFS-14061 > URL: https://issues.apache.org/jira/browse/HDFS-14061 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, hdfs >Affects Versions: 3.1.1 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HDFS-14061.001.patch, HDFS-14061.002.patch, > HDFS-14061.003.patch, HDFS-14061.004.patch > > > HDFS-12946 introduces a command for verifying if there are enough racks and > datanodes for the enabled erasure coding policies. > This verification could be executed for the erasure coding policy before > enabling, setting or adding it and a warning message could be written if the > verify fails, or the policy setting could be even failed in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[ https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737407#comment-16737407 ] Shweta commented on HDFS-14061: --- Thanks for the patch [~knanasi]. Looks good to me. Just a small improvement would be to add messages for assertEquals() for it to be more descriptive. Apart from that +1 (Non-binding). > Check if the cluster topology supports the EC policy before setting, enabling > or adding it > -- > > Key: HDFS-14061 > URL: https://issues.apache.org/jira/browse/HDFS-14061 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, hdfs >Affects Versions: 3.1.1 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HDFS-14061.001.patch, HDFS-14061.002.patch, > HDFS-14061.003.patch > > > HDFS-12946 introduces a command for verifying if there are enough racks and > datanodes for the enabled erasure coding policies. > This verification could be executed for the erasure coding policy before > enabling, setting or adding it and a warning message could be written if the > verify fails, or the policy setting could be even failed in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[
https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737311#comment-16737311
]
Hadoop QA commented on HDFS-14061:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m
33s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14061 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954168/HDFS-14061.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3fb1d4da2490 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 695e93c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25931/testReport/ |
| Max. process+thread count | 3336 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25931/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Check if the cluster topology supports the EC policy before setting, enabling
> or adding it
>
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[ https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737125#comment-16737125 ] Kitti Nanasi commented on HDFS-14061: - Thanks for the comment [~adam.antal]! I fixed the renames and added a new test. The new message is already tested in TestECAdmin, so I don't think it would add more value to also test it in TestErasureCodingCLI, and the number of racks and data nodes are more difficult to configure there. You are right, System.err in TestECAdmin.java was only used by patch v001, but I think it is worth to keep that check, because ECAdmin can write to System.err. > Check if the cluster topology supports the EC policy before setting, enabling > or adding it > -- > > Key: HDFS-14061 > URL: https://issues.apache.org/jira/browse/HDFS-14061 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, hdfs >Affects Versions: 3.1.1 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HDFS-14061.001.patch, HDFS-14061.002.patch, > HDFS-14061.003.patch > > > HDFS-12946 introduces a command for verifying if there are enough racks and > datanodes for the enabled erasure coding policies. > This verification could be executed for the erasure coding policy before > enabling, setting or adding it and a warning message could be written if the > verify fails, or the policy setting could be even failed in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[
https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737015#comment-16737015
]
Adam Antal commented on HDFS-14061:
---
Thank you for the patch [~knanasi].
- Being a bit more descriptive I would rename {{testEnablePolicy}}, for e.g.
to {{testEnablePolicyWarning}}. Also it might be worthwhile to test when the
number of racks equals exactly to the requirement, that this warning is NOT
raised.
- Also in {{FSNamesystem.java:getVerifyECWithTopologyResult()}}
{code:java}
ErasureCodingPolicy[] ecPolicies =
getErasureCodingPolicyManager().getCopyOfEnabledPolicies();
{code}
Since previously we listed all the policies, but now we handle the enabled
policies only, the name of the variable {{ecPolicies}} should be renamed to
{{enabledEcPolicies}} to reflect that.
- Although jenkins did not throw error, it may worth to review tests in
{{TestErasureCodingCLI}} as well (so the tests in the
{{testErasureCodingConf.xml}} file). Where the warning arises, a new comparator
should be added that checks (because it must be tested that the warning do come
up) and also optionally add a test there that would specifically checks for the
warning.
- I hope I did not miss anything, but is redirecting stderr in
{{TestECAdmin.java}} somewhat unrelated to this issue? You seemingly did not
add any more output to stderr - so to make sure: in {{ECAdmin.java:run()}} it
is {{System.out}} and not a {{System.err}} ?
> Check if the cluster topology supports the EC policy before setting, enabling
> or adding it
> --
>
> Key: HDFS-14061
> URL: https://issues.apache.org/jira/browse/HDFS-14061
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.1
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Attachments: HDFS-14061.001.patch, HDFS-14061.002.patch
>
>
> HDFS-12946 introduces a command for verifying if there are enough racks and
> datanodes for the enabled erasure coding policies.
> This verification could be executed for the erasure coding policy before
> enabling, setting or adding it and a warning message could be written if the
> verify fails, or the policy setting could be even failed in this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[
https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736099#comment-16736099
]
Hadoop QA commented on HDFS-14061:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14061 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954018/HDFS-14061.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8d4d96f70819 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d715233 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25926/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25926/testReport/ |
| Max. process+thread count | 4276 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25926/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[ https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735969#comment-16735969 ] Kitti Nanasi commented on HDFS-14061: - Thanks for the comment [~ayushtkn]! You are right, failing the policy setting might be too harsh. Why I wanted to do that was because when setting the policy and writing to the folder, the error message we get is quite misleading. But I think if that message is corrected, it will be fine, so I raised [HDFS-14187|https://issues.apache.org/jira/browse/HDFS-14187] for that. And in patch 002 I fixed your comments. > Check if the cluster topology supports the EC policy before setting, enabling > or adding it > -- > > Key: HDFS-14061 > URL: https://issues.apache.org/jira/browse/HDFS-14061 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, hdfs >Affects Versions: 3.1.1 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HDFS-14061.001.patch, HDFS-14061.002.patch > > > HDFS-12946 introduces a command for verifying if there are enough racks and > datanodes for the enabled erasure coding policies. > This verification could be executed for the erasure coding policy before > enabling, setting or adding it and a warning message could be written if the > verify fails, or the policy setting could be even failed in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[
https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734506#comment-16734506
]
Ayush Saxena commented on HDFS-14061:
-
Thanks [~knanasi] for the patch.
{quote}A warning is shown if the cluster topology verify fails for all enabled
policies, when enabling a policy.
{quote}
I guess if we are enabling RS(6,3) warning should be shown wrt to the policy
which we are enabling.We should not be bothered about the others not in that
transaction.
{quote}Policy setting fails if the cluster topology verification fails for the
policy
{quote}
IMO This will be quite harsh and guess might be considered as incompatible
too.Moreover for EC File say RS(6,3) I guess It is not mandatory that it should
have strictly 9 DN's for successful write. In cases it can be successful with
lesser numbers too.
I think a warning if the policy that we are enabling is not supported should be
well enough.
> Check if the cluster topology supports the EC policy before setting, enabling
> or adding it
> --
>
> Key: HDFS-14061
> URL: https://issues.apache.org/jira/browse/HDFS-14061
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.1
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Attachments: HDFS-14061.001.patch
>
>
> HDFS-12946 introduces a command for verifying if there are enough racks and
> datanodes for the enabled erasure coding policies.
> This verification could be executed for the erasure coding policy before
> enabling, setting or adding it and a warning message could be written if the
> verify fails, or the policy setting could be even failed in this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14061) Check if the cluster topology supports the EC policy before setting, enabling or adding it
[ https://issues.apache.org/jira/browse/HDFS-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734313#comment-16734313 ] Kitti Nanasi commented on HDFS-14061: - Patch 001 contains: * A warning is shown if the cluster topology verify fails for all enabled policies, when enabling a policy. * Policy setting fails if the cluster topology verification fails for the policy There could be one concern with the second one, if we want to set RS(6,3) policy, the verify will only succeed if there are at least 9 data nodes which may be a bit strict. Should we allow setting the policy with less data nodes than 9? I would like to hear some opinions about that. > Check if the cluster topology supports the EC policy before setting, enabling > or adding it > -- > > Key: HDFS-14061 > URL: https://issues.apache.org/jira/browse/HDFS-14061 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, hdfs >Affects Versions: 3.1.1 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HDFS-14061.001.patch > > > HDFS-12946 introduces a command for verifying if there are enough racks and > datanodes for the enabled erasure coding policies. > This verification could be executed for the erasure coding policy before > enabling, setting or adding it and a warning message could be written if the > verify fails, or the policy setting could be even failed in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
