[jira] [Commented] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[
https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772111#comment-15772111
]
Hadoop QA commented on HDFS-11186:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
14s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.namenode.ha.TestHAStateTransitions |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11186 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844032/HDFS-11186-HDFS-10285.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 183221dacf6b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10285 / c5876ee |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17945/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17945/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17945/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [SPS]: Daemon thread of SPS should start only in Active NN
>
[jira] [Commented] (HDFS-10917) Collect peer performance statistics on DataNode.
[
https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772064#comment-15772064
]
Hadoop QA commented on HDFS-10917:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 40s{color} | {color:orange} root: The patch generated 1 new + 678 unchanged
- 9 fixed = 679 total (was 687) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
25s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m
0s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10917 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844477/HDFS-10917.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux dcbe9ea4e37d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 22befbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17942/artifact/patchprocess/diff-checkstyle-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17942/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
[jira] [Commented] (HDFS-11150) [SPS]: Provide persistence when satisfying storage policy.
[ https://issues.apache.org/jira/browse/HDFS-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772058#comment-15772058 ] Yuanbo Liu commented on HDFS-11150: --- Since Uma and Rakesh have brought up good suggestions about test cases, I will create a new test java class to track all them down. > [SPS]: Provide persistence when satisfying storage policy. > -- > > Key: HDFS-11150 > URL: https://issues.apache.org/jira/browse/HDFS-11150 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Attachments: HDFS-11150-HDFS-10285.001.patch, > HDFS-11150-HDFS-10285.002.patch, editsStored, editsStored.xml > > > Provide persistence for SPS in case that Hadoop cluster crashes by accident. > Basically we need to change EditLog and FsImage here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10917) Collect peer performance statistics on DataNode.
[ https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10917: - Attachment: (was: HDFS-10917.005.patch) > Collect peer performance statistics on DataNode. > > > Key: HDFS-10917 > URL: https://issues.apache.org/jira/browse/HDFS-10917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10917.000.patch, HDFS-10917.001.patch, > HDFS-10917.002.patch, HDFS-10917.003.patch, HDFS-10917.004.patch, > HDFS-10917.005.patch > > > DataNodes already detect if replication pipeline operations are slow and log > warnings. For the purpose of analysis, performance metrics are desirable. > This proposes adding them on DataNodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10917) Collect peer performance statistics on DataNode.
[ https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10917: - Attachment: HDFS-10917.005.patch > Collect peer performance statistics on DataNode. > > > Key: HDFS-10917 > URL: https://issues.apache.org/jira/browse/HDFS-10917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10917.000.patch, HDFS-10917.001.patch, > HDFS-10917.002.patch, HDFS-10917.003.patch, HDFS-10917.004.patch, > HDFS-10917.005.patch > > > DataNodes already detect if replication pipeline operations are slow and log > warnings. For the purpose of analysis, performance metrics are desirable. > This proposes adding them on DataNodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11274) Datanode should only check the failed volume upon IO errors
[
https://issues.apache.org/jira/browse/HDFS-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772017#comment-15772017
]
Hadoop QA commented on HDFS-11274:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 8 new + 376 unchanged - 1 fixed = 384 total (was 377) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
11s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Redundant nullcheck of cause, which is known to be non-null in
org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run() Redundant
null check at DataNode.java:is known to be non-null in
org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run() Redundant
null check at DataNode.java:[line 2466] |
| Failed junit tests |
hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| Timed out junit tests |
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11274 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844514/HDFS-11274.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 17db01be85b3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 22befbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HDFS-11274) Datanode should only check the failed volume upon IO errors
[
https://issues.apache.org/jira/browse/HDFS-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771992#comment-15771992
]
Hadoop QA commented on HDFS-11274:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 8 new + 378 unchanged - 1 fixed = 386 total (was 379) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
50s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 10s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Redundant nullcheck of cause, which is known to be non-null in
org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run() Redundant
null check at DataNode.java:is known to be non-null in
org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run() Redundant
null check at DataNode.java:[line 2466] |
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11274 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844514/HDFS-11274.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4e8c2d77a261 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 22befbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17944/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17944/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
|
| unit |
[jira] [Commented] (HDFS-11274) Datanode should only check the failed volume upon IO errors
[ https://issues.apache.org/jira/browse/HDFS-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771965#comment-15771965 ] Arpit Agarwal commented on HDFS-11274: -- +1 pending Jenkins. > Datanode should only check the failed volume upon IO errors > > > Key: HDFS-11274 > URL: https://issues.apache.org/jira/browse/HDFS-11274 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11274.01.patch, HDFS-11274.02.patch > > > This is a performance improvement that is possible after HDFS-11182. The goal > is to trigger async volume check with throttling only on suspected volume > upon datanode file IO errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11252) TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with BindException
[ https://issues.apache.org/jira/browse/HDFS-11252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11252: - Attachment: HDFS-11252.002.patch Thanks [~brahmareddy] for the review and comments. The comment makes sense for me. New patch attached. > TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with > BindException > > > Key: HDFS-11252 > URL: https://issues.apache.org/jira/browse/HDFS-11252 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha2 >Reporter: Jason Lowe >Assignee: Yiqun Lin > Attachments: HDFS-11252.001.patch, HDFS-11252.002.patch > > > testTruncateWithDataNodesRestartImmediately can fail with a BindException. > The setup for TestFileTruncate has been fixed in the past to solve a bind > exception, but this is occurring after the minicluster comes up and the > datanodes are being restarted. Maybe there's a race condition there? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[ https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771907#comment-15771907 ] Rakesh R commented on HDFS-11186: - FYI, previous [build|https://builds.apache.org/job/PreCommit-HDFS-Build/17915/] was not successful. I've triggered build now. > [SPS]: Daemon thread of SPS should start only in Active NN > -- > > Key: HDFS-11186 > URL: https://issues.apache.org/jira/browse/HDFS-11186 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Wei Zhou >Assignee: Wei Zhou > Attachments: HDFS-11186-HDFS-10285.00.patch, > HDFS-11186-HDFS-10285.01.patch > > > As discussed in [HDFS-10885 > |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that > SPS is started only in Active NN. This JIRA is opened for discussion and > tracking. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11274) Datanode should only check the failed volume upon IO errors
[ https://issues.apache.org/jira/browse/HDFS-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11274: -- Status: Patch Available (was: Open) > Datanode should only check the failed volume upon IO errors > > > Key: HDFS-11274 > URL: https://issues.apache.org/jira/browse/HDFS-11274 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11274.01.patch, HDFS-11274.02.patch > > > This is a performance improvement that is possible after HDFS-11182. The goal > is to trigger async volume check with throttling only on suspected volume > upon datanode file IO errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11274) Datanode should only check the failed volume upon IO errors
[ https://issues.apache.org/jira/browse/HDFS-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11274: -- Attachment: HDFS-11274.02.patch > Datanode should only check the failed volume upon IO errors > > > Key: HDFS-11274 > URL: https://issues.apache.org/jira/browse/HDFS-11274 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11274.01.patch, HDFS-11274.02.patch > > > This is a performance improvement that is possible after HDFS-11182. The goal > is to trigger async volume check with throttling only on suspected volume > upon datanode file IO errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11252) TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with BindException
[
https://issues.apache.org/jira/browse/HDFS-11252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771767#comment-15771767
]
Brahma Reddy Battula commented on HDFS-11252:
-
[~linyiqun] thanks for working on this..As we discussed in HDFS-11134, we no
need to keep same port while restarting the DN..
Patch LGTM ,Apart from the following,if you agree. can extract the duplicate
code to one method like following..?
{code}
+ private void truncateAndRestartDN(Path p, int dn, int newLength)
+ throws IOException {
+try {
+ boolean isReady = fs.truncate(p, newLength);
+ assertFalse(isReady);
+} finally {
+ cluster.restartDataNode(dn, false, true);
+ cluster.waitActive();
+}
+ }
{code}
> TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with
> BindException
>
>
> Key: HDFS-11252
> URL: https://issues.apache.org/jira/browse/HDFS-11252
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Jason Lowe
>Assignee: Yiqun Lin
> Attachments: HDFS-11252.001.patch
>
>
> testTruncateWithDataNodesRestartImmediately can fail with a BindException.
> The setup for TestFileTruncate has been fixed in the past to solve a bind
> exception, but this is occurring after the minicluster comes up and the
> datanodes are being restarted. Maybe there's a race condition there?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771746#comment-15771746
]
Hadoop QA commented on HDFS-11270:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 70m
42s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11270 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844500/HDFS-11270.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3a7697633704 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 22befbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17941/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17941/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11270.001.patch,
[jira] [Commented] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771629#comment-15771629
]
Mingliang Liu commented on HDFS-11270:
--
+1 pending on Jenkins.
[~akur...@gmail.com], if you don't have further comments, I'll commit this next
week. Thanks,
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11270.001.patch, HDFS-11270.002.patch
>
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and that will let users know how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771629#comment-15771629
]
Mingliang Liu edited comment on HDFS-11270 at 12/23/16 2:10 AM:
+1 pending on Jenkins.
[~ajisakaa], if you don't have further comments, I'll commit this next week.
Thanks,
was (Author: liuml07):
+1 pending on Jenkins.
[~akur...@gmail.com], if you don't have further comments, I'll commit this next
week. Thanks,
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11270.001.patch, HDFS-11270.002.patch
>
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and that will let users know how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11270:
-
Attachment: HDFS-11270.002.patch
Thanks [~liuml07] for the review. The comment makes sense for me. New patch
attached.
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11270.001.patch, HDFS-11270.002.patch
>
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and that will let users know how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11032) [SPS]: Handling of block movement failure at the coordinator datanode
[ https://issues.apache.org/jira/browse/HDFS-11032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-11032: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-10285 Status: Resolved (was: Patch Available) I have just pushed this to branch > [SPS]: Handling of block movement failure at the coordinator datanode > - > > Key: HDFS-11032 > URL: https://issues.apache.org/jira/browse/HDFS-11032 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Rakesh R >Assignee: Rakesh R > Fix For: HDFS-10285 > > Attachments: HDFS-11032-HDFS-10285-00.patch, > HDFS-11032-HDFS-10285-01.patch > > > The idea of this jira is to discuss and implement an efficient failure(block > movement failure) handling logic at the datanode cooridnator. [Code > reference|https://github.com/apache/hadoop/blob/HDFS-10285/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java#L243]. > Following are the possible errors during block movement: > # Block pinned - no retries marked as success/no-retry to NN. It is not > possible to relocate this block to another datanode. > # Network errors(IOException) - no retries maked as failure/retry to NN. > # No disk space(IOException) - no retries maked as failure/retry to NN. > # Gen_Stamp mismatches - no retries marked as failure/retry to NN. Could be a > case that the file might have re-opened. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10917) Collect peer performance statistics on DataNode.
[ https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771521#comment-15771521 ] Xiaoyu Yao commented on HDFS-10917: --- [~xiaobingo], thanks for updating the patch along with the rationale behind the sampling design of tracking send packet latency. I'm OK with the current penultimate node sampling design and think it might be easier to evaluate the effectiveness if we could make the sampling strategy configurable. We can do flexible sampling with a followup JIRA. +1 for v5 patch, pending Jenkins. > Collect peer performance statistics on DataNode. > > > Key: HDFS-10917 > URL: https://issues.apache.org/jira/browse/HDFS-10917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10917.000.patch, HDFS-10917.001.patch, > HDFS-10917.002.patch, HDFS-10917.003.patch, HDFS-10917.004.patch, > HDFS-10917.005.patch > > > DataNodes already detect if replication pipeline operations are slow and log > warnings. For the purpose of analysis, performance metrics are desirable. > This proposes adding them on DataNodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11032) [SPS]: Handling of block movement failure at the coordinator datanode
[ https://issues.apache.org/jira/browse/HDFS-11032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771512#comment-15771512 ] Uma Maheswara Rao G commented on HDFS-11032: +1 on the patch. Changes looks good to me. > [SPS]: Handling of block movement failure at the coordinator datanode > - > > Key: HDFS-11032 > URL: https://issues.apache.org/jira/browse/HDFS-11032 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Rakesh R >Assignee: Rakesh R > Attachments: HDFS-11032-HDFS-10285-00.patch, > HDFS-11032-HDFS-10285-01.patch > > > The idea of this jira is to discuss and implement an efficient failure(block > movement failure) handling logic at the datanode cooridnator. [Code > reference|https://github.com/apache/hadoop/blob/HDFS-10285/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java#L243]. > Following are the possible errors during block movement: > # Block pinned - no retries marked as success/no-retry to NN. It is not > possible to relocate this block to another datanode. > # Network errors(IOException) - no retries maked as failure/retry to NN. > # No disk space(IOException) - no retries maked as failure/retry to NN. > # Gen_Stamp mismatches - no retries marked as failure/retry to NN. Could be a > case that the file might have re-opened. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11274) Datanode should only check the failed volume upon IO errors
[ https://issues.apache.org/jira/browse/HDFS-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11274: -- Attachment: HDFS-11274.01.patch > Datanode should only check the failed volume upon IO errors > > > Key: HDFS-11274 > URL: https://issues.apache.org/jira/browse/HDFS-11274 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11274.01.patch > > > This is a performance improvement that is possible after HDFS-11182. The goal > is to trigger async volume check with throttling only on suspected volume > upon datanode file IO errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11274) Datanode should only check the failed volume upon IO errors
Xiaoyu Yao created HDFS-11274: - Summary: Datanode should only check the failed volume upon IO errors Key: HDFS-11274 URL: https://issues.apache.org/jira/browse/HDFS-11274 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This is a performance improvement that is possible after HDFS-11182. The goal is to trigger async volume check with throttling only on suspected volume upon datanode file IO errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771396#comment-15771396
]
Hadoop QA commented on HDFS-10860:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 43s{color}
| {color:red} root generated 4 new + 690 unchanged - 0 fixed = 694 total (was
690) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} root: The patch generated 0 new + 95 unchanged - 4
fixed = 95 total (was 99) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
20s{color} | {color:green} The patch generated 0 new + 565 unchanged - 10 fixed
= 565 total (was 575) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
48s{color} | {color:green} hadoop-auth in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
20s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 68m
26s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
14s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The
[jira] [Commented] (HDFS-10917) Collect peer performance statistics on DataNode.
[ https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771233#comment-15771233 ] Xiaobing Zhou commented on HDFS-10917: -- Thanks [~xyao]. I posted v5 patch to address your comments #1, 2 and 3. Only tracking the latency of sending packet to the last node in pipeline is a conscious design choice. In the case of pipeline [dn0, dn1, dn2], 5ms latency from dn0 to dn1, 100ms from dn1 to dn2, NameNode claims dn2 is slow since it sees 100ms latency to dn2. Note that NameNode is not ware of pipeline structure in this context and only sees latency between two DataNodes. In another case of the same pipeline, 100ms latency from dn0 to dn1, 5ms from dn1 to dn2, NameNode will miss detecting dn1 being slow since it's not the last node. However the assumption is that in a busy enough cluster there are many other pipelines where dn1 is the last node, e.g. [dn3, dn4, dn1]. Also our tracking interval is relatively long enough (at least an hour) to improve the chances of the bad DataNodes being the last nodes in multiple pipelines. > Collect peer performance statistics on DataNode. > > > Key: HDFS-10917 > URL: https://issues.apache.org/jira/browse/HDFS-10917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10917.000.patch, HDFS-10917.001.patch, > HDFS-10917.002.patch, HDFS-10917.003.patch, HDFS-10917.004.patch, > HDFS-10917.005.patch > > > DataNodes already detect if replication pipeline operations are slow and log > warnings. For the purpose of analysis, performance metrics are desirable. > This proposes adding them on DataNodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10917) Collect peer performance statistics on DataNode.
[ https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10917: - Attachment: HDFS-10917.005.patch > Collect peer performance statistics on DataNode. > > > Key: HDFS-10917 > URL: https://issues.apache.org/jira/browse/HDFS-10917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10917.000.patch, HDFS-10917.001.patch, > HDFS-10917.002.patch, HDFS-10917.003.patch, HDFS-10917.004.patch, > HDFS-10917.005.patch > > > DataNodes already detect if replication pipeline operations are slow and log > warnings. For the purpose of analysis, performance metrics are desirable. > This proposes adding them on DataNodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11272) Missing whitespace in the message of TestFSDirAttrOp.java
[
https://issues.apache.org/jira/browse/HDFS-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771194#comment-15771194
]
Hadoop QA commented on HDFS-11272:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.tools.TestDFSAdminWithHA |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.TestLocalDFS |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.server.namenode.TestMetaSave |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11272 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844452/hdfs-11272.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8b882701fac2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 22befbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17938/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17938/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17938/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Missing whitespace in the message of TestFSDirAttrOp.java
>
[jira] [Commented] (HDFS-4025) QJM: Sychronize past log segments to JNs that missed them
[ https://issues.apache.org/jira/browse/HDFS-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771160#comment-15771160 ] Hanisha Koneru commented on HDFS-4025: -- I have submitted a patch for moving the transfer Edit Logs code from TransferFsImage to Util class (HDFS-11273) so that it can be re-used here. > QJM: Sychronize past log segments to JNs that missed them > - > > Key: HDFS-4025 > URL: https://issues.apache.org/jira/browse/HDFS-4025 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Hanisha Koneru > Fix For: QuorumJournalManager (HDFS-3077) > > Attachments: HDFS-4025.000.patch, HDFS-4025.001.patch, > HDFS-4025.002.patch, HDFS-4025.003.patch, HDFS-4025.004.patch, > HDFS-4025.005.patch > > > Currently, if a JournalManager crashes and misses some segment of logs, and > then comes back, it will be re-added as a valid part of the quorum on the > next log roll. However, it will not have a complete history of log segments > (i.e any individual JN may have gaps in its transaction history). This > mirrors the behavior of the NameNode when there are multiple local > directories specified. > However, it would be better if a background thread noticed these gaps and > "filled them in" by grabbing the segments from other JournalNodes. This > increases the resilience of the system when JournalNodes get reformatted or > otherwise lose their local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11273) Move TransferFsImage#doGetUrl function to a Util class
[ https://issues.apache.org/jira/browse/HDFS-11273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDFS-11273: -- Attachment: HDFS-11273.000.patch > Move TransferFsImage#doGetUrl function to a Util class > -- > > Key: HDFS-11273 > URL: https://issues.apache.org/jira/browse/HDFS-11273 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Attachments: HDFS-11273.000.patch > > > TransferFsImage#doGetUrl function is required for JournalNode syncing as > well. We can move the code to a Utility class to avoid duplication of code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11273) Move TransferFsImage#doGetUrl function to a Util class
Hanisha Koneru created HDFS-11273: - Summary: Move TransferFsImage#doGetUrl function to a Util class Key: HDFS-11273 URL: https://issues.apache.org/jira/browse/HDFS-11273 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Reporter: Hanisha Koneru Assignee: Hanisha Koneru TransferFsImage#doGetUrl function is required for JournalNode syncing as well. We can move the code to a Utility class to avoid duplication of code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10917) Collect peer performance statistics on DataNode.
[
https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771091#comment-15771091
]
Xiaoyu Yao commented on HDFS-10917:
---
Thanks [~xiaobingo] for working on this. Patch v04 looks good to me. I just
have a few questions:
BlockReceiver.java
1. Line 828: can you clarify the design behind tracking penultimate node only
in a pipeline line [dn0, dn1, dn2]?
It works well for example like: 5, 100, 105. What if we have latency of
pipeline like: 5, 115, 15? Do we miss detecting non-penultimate slow datanodes?
DataNodeMXBean.java
2. Line 130: we are getting the average info of SendPacketDownstream when
acting as penultimate (2nd to last) node.
RollingAverages.java
3. NIT: Line 63: should be .setNameFormat("RollingAverages-%d")
> Collect peer performance statistics on DataNode.
>
>
> Key: HDFS-10917
> URL: https://issues.apache.org/jira/browse/HDFS-10917
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10917.000.patch, HDFS-10917.001.patch,
> HDFS-10917.002.patch, HDFS-10917.003.patch, HDFS-10917.004.patch
>
>
> DataNodes already detect if replication pipeline operations are slow and log
> warnings. For the purpose of analysis, performance metrics are desirable.
> This proposes adding them on DataNodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10860:
--
Attachment: HDFS-10860.003.patch
Patch 003
- Xiao's comments in HADOOP-13597
TESTING DONE
- Bats regression tests https://github.com/jzhuge/hadoop-regression-tests in
unsecure and ssl mode
- Verify docs
> Switch HttpFS from Tomcat to Jetty
> --
>
> Key: HDFS-10860
> URL: https://issues.apache.org/jira/browse/HDFS-10860
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: httpfs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch,
> HDFS-10860.003.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have to change client code that much. It would
> require more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11272) Missing whitespace in the message of TestFSDirAttrOp.java
[
https://issues.apache.org/jira/browse/HDFS-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771032#comment-15771032
]
Mingliang Liu commented on HDFS-11272:
--
+1 pending on Jenkins.
For the last two assert, we can also add a bit more error message about why the
atime should have been updated. And also we can update the summary and
description of this JIRA like {{Refining the assert message in
TestFSDirAttrOp}} or similar.
> Missing whitespace in the message of TestFSDirAttrOp.java
> -
>
> Key: HDFS-11272
> URL: https://issues.apache.org/jira/browse/HDFS-11272
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.8.0
>Reporter: Akira Ajisaka
>Assignee: Jimmy Xiang
>Priority: Trivial
> Labels: newbie
> Attachments: hdfs-11272.patch
>
>
> {code:title=TestFSDirAttrOp.java}
> assertFalse("SetTimes should not update access time"
> + "because it's within the last precision interval",
> {code}
> A whitespace is missing between time and because.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11028) libhdfs++: FileHandleImpl::CancelOperations needs to be able to cancel pending connections
[
https://issues.apache.org/jira/browse/HDFS-11028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770999#comment-15770999
]
Hadoop QA commented on HDFS-11028:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
41s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
51s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
18s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
16s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
24s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 6m 24s{color} |
{color:red} hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_111 with JDK
v1.8.0_111 generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
23s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 6m 23s{color} |
{color:red} hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.7.0_121 with JDK
v1.7.0_121 generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
6s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
8s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
15s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with
JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:78fc6b6 |
| JIRA Issue | HDFS-11028 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/1287/HDFS-11028.HDFS-8707.000.patch
|
| Optional Tests | asflicense compile cc mvnsite javac unit javadoc
mvninstall |
| uname | Linux 45411b301334 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HDFS-11272) Missing whitespace in the message of TestFSDirAttrOp.java
[
https://issues.apache.org/jira/browse/HDFS-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HDFS-11272:
---
Assignee: Jimmy Xiang
Affects Version/s: 2.8.0
Status: Patch Available (was: Open)
> Missing whitespace in the message of TestFSDirAttrOp.java
> -
>
> Key: HDFS-11272
> URL: https://issues.apache.org/jira/browse/HDFS-11272
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.8.0
>Reporter: Akira Ajisaka
>Assignee: Jimmy Xiang
>Priority: Trivial
> Labels: newbie
> Attachments: hdfs-11272.patch
>
>
> {code:title=TestFSDirAttrOp.java}
> assertFalse("SetTimes should not update access time"
> + "because it's within the last precision interval",
> {code}
> A whitespace is missing between time and because.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11272) Missing whitespace in the message of TestFSDirAttrOp.java
[
https://issues.apache.org/jira/browse/HDFS-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HDFS-11272:
---
Attachment: hdfs-11272.patch
> Missing whitespace in the message of TestFSDirAttrOp.java
> -
>
> Key: HDFS-11272
> URL: https://issues.apache.org/jira/browse/HDFS-11272
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.8.0
>Reporter: Akira Ajisaka
>Priority: Trivial
> Labels: newbie
> Attachments: hdfs-11272.patch
>
>
> {code:title=TestFSDirAttrOp.java}
> assertFalse("SetTimes should not update access time"
> + "because it's within the last precision interval",
> {code}
> A whitespace is missing between time and because.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11271) Typo in NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770894#comment-15770894 ] Akira Ajisaka commented on HDFS-11271: -- +1 > Typo in NameNode UI > --- > > Key: HDFS-11271 > URL: https://issues.apache.org/jira/browse/HDFS-11271 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, ui >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Trivial > Attachments: HDFS-11271.001.patch > > > The HADOOP-11098 commit has a typo: > It should be "" instead of "" when max heap is -1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770866#comment-15770866
]
Mingliang Liu commented on HDFS-11270:
--
Thanks [~linyiqun]. I think the patch is good overall.
One minor comment is about
{quote}
-skipSharedEditsCheck option skips edits check, the edits check can ensure we
have enough edits already in the shared directory to start up, otherwise it
will aborts the download of fsimage file from active namenode.
{quote}
It'd be better to have two sentences as "otherwise" case is not very
straightforward. And perhaps we don't have to mention the implementation detail
of downloading fsimage from active namenode; it also skips the possible
upgrade. How about:
{quote}
-skipSharedEditsCheck option skips edits check which ensures that we have
enough edits already in the shared directory to start up from the last
checkpoint on the active.
{quote}
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11270.001.patch
>
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and that will let users know how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11258) File mtime change could not save to editlog
[
https://issues.apache.org/jira/browse/HDFS-11258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770825#comment-15770825
]
Akira Ajisaka commented on HDFS-11258:
--
{code}
assertFalse("SetTimes should not update access time"
+ "because it's within the last precision interval",
{code}
Trivial: A whitespace is missing between time and because. Filed HDFS-11272 for
the additional fix.
> File mtime change could not save to editlog
> ---
>
> Key: HDFS-11258
> URL: https://issues.apache.org/jira/browse/HDFS-11258
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Critical
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: hdfs-11258-addendum-branch2.patch, hdfs-11258.1.patch,
> hdfs-11258.2.patch, hdfs-11258.3.patch, hdfs-11258.4.patch
>
>
> When both mtime and atime are changed, and atime is not beyond the precision
> limit, the mtime change is not saved to edit logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11028) libhdfs++: FileHandleImpl::CancelOperations needs to be able to cancel pending connections
[ https://issues.apache.org/jira/browse/HDFS-11028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11028: --- Status: Patch Available (was: Open) > libhdfs++: FileHandleImpl::CancelOperations needs to be able to cancel > pending connections > -- > > Key: HDFS-11028 > URL: https://issues.apache.org/jira/browse/HDFS-11028 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-11028.HDFS-8707.000.patch > > > Cancel support is now reasonably robust except the case where a FileHandle > operation ends up causing the RpcEngine to try to create a new RpcConnection. > In HA configs it's common to have something like 10-20 failovers and a 20 > second failover delay (no exponential backoff just yet). This means that all > of the functions with synchronous interfaces can still block for many minutes > after an operation has been canceled, and often the cause of this is > something trivial like a bad config file. > The current design makes this sort of thing tricky to do because the > FileHandles need to be individually cancelable via CancelOperations, but they > share the RpcEngine that does the async magic. > Updated design: > Original design would end up forcing lots of reconnects. Not a huge issue on > an unauthenticated cluster but on a kerberized cluster this is a recipe for > Kerberos thinking we're attempting a replay attack. > User visible cancellation and internal resources cleanup are separable > issues. The former can be implemented by atomically swapping the callback of > the operation to be canceled with a no-op callback. The original callback is > then posted to the IoService with an OperationCanceled status and the user is > no longer blocked. For RPC cancels this is sufficient, it's not expensive to > keep a request around a little bit longer and when it's eventually invoked or > timed out it invokes the no-op callback and is ignored (other than a trace > level log notification). Connect cancels push a flag down into the RPC > engine to kill the connection and make sure it doesn't attempt to reconnect. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11028) libhdfs++: FileHandleImpl::CancelOperations needs to be able to cancel pending connections
[ https://issues.apache.org/jira/browse/HDFS-11028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11028: --- Attachment: HDFS-11028.HDFS-8707.000.patch Posting cancel mega-patch. Plenty of work to do, but wanted to put it up in case anyone was curious (will be away for 1-2 weeks). The bulk of the code being changed is just forwarding handles down to the async operations so they can be bound. Includes: -connection level cancels (tested with included cancelable connect tool) -individual rpc level cancels (not well tested) -exposing all cancels in the C api I'm going to try and split connect cancels from RPC cancels since it's similar but mostly separate code paths. RPC cancels will get a new jira. The C bindings will just be ripped out for now since I haven't tested them at all and they aren't a priority for my work yet. > libhdfs++: FileHandleImpl::CancelOperations needs to be able to cancel > pending connections > -- > > Key: HDFS-11028 > URL: https://issues.apache.org/jira/browse/HDFS-11028 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-11028.HDFS-8707.000.patch > > > Cancel support is now reasonably robust except the case where a FileHandle > operation ends up causing the RpcEngine to try to create a new RpcConnection. > In HA configs it's common to have something like 10-20 failovers and a 20 > second failover delay (no exponential backoff just yet). This means that all > of the functions with synchronous interfaces can still block for many minutes > after an operation has been canceled, and often the cause of this is > something trivial like a bad config file. > The current design makes this sort of thing tricky to do because the > FileHandles need to be individually cancelable via CancelOperations, but they > share the RpcEngine that does the async magic. > Updated design: > Original design would end up forcing lots of reconnects. Not a huge issue on > an unauthenticated cluster but on a kerberized cluster this is a recipe for > Kerberos thinking we're attempting a replay attack. > User visible cancellation and internal resources cleanup are separable > issues. The former can be implemented by atomically swapping the callback of > the operation to be canceled with a no-op callback. The original callback is > then posted to the IoService with an OperationCanceled status and the user is > no longer blocked. For RPC cancels this is sufficient, it's not expensive to > keep a request around a little bit longer and when it's eventually invoked or > timed out it invokes the no-op callback and is ignored (other than a trace > level log notification). Connect cancels push a flag down into the RPC > engine to kill the connection and make sure it doesn't attempt to reconnect. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11272) Missing whitespace in the message of TestFSDirAttrOp.java
Akira Ajisaka created HDFS-11272:
Summary: Missing whitespace in the message of TestFSDirAttrOp.java
Key: HDFS-11272
URL: https://issues.apache.org/jira/browse/HDFS-11272
Project: Hadoop HDFS
Issue Type: Test
Components: test
Reporter: Akira Ajisaka
Priority: Trivial
{code:title=TestFSDirAttrOp.java}
assertFalse("SetTimes should not update access time"
+ "because it's within the last precision interval",
{code}
A whitespace is missing between time and because.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11258) File mtime change could not save to editlog
[ https://issues.apache.org/jira/browse/HDFS-11258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HDFS-11258. -- Resolution: Fixed Committed the addendum patch to branch-2 and branch-2.8. Thanks! > File mtime change could not save to editlog > --- > > Key: HDFS-11258 > URL: https://issues.apache.org/jira/browse/HDFS-11258 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: hdfs-11258-addendum-branch2.patch, hdfs-11258.1.patch, > hdfs-11258.2.patch, hdfs-11258.3.patch, hdfs-11258.4.patch > > > When both mtime and atime are changed, and atime is not beyond the precision > limit, the mtime change is not saved to edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11258) File mtime change could not save to editlog
[ https://issues.apache.org/jira/browse/HDFS-11258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770800#comment-15770800 ] Akira Ajisaka commented on HDFS-11258: -- +1 for the addendum patch. Thanks [~jxiang] for the quick fix. > File mtime change could not save to editlog > --- > > Key: HDFS-11258 > URL: https://issues.apache.org/jira/browse/HDFS-11258 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: hdfs-11258-addendum-branch2.patch, hdfs-11258.1.patch, > hdfs-11258.2.patch, hdfs-11258.3.patch, hdfs-11258.4.patch > > > When both mtime and atime are changed, and atime is not beyond the precision > limit, the mtime change is not saved to edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11028) libhdfs++: FileHandleImpl::CancelOperations needs to be able to cancel pending connections
[ https://issues.apache.org/jira/browse/HDFS-11028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11028: --- Description: Cancel support is now reasonably robust except the case where a FileHandle operation ends up causing the RpcEngine to try to create a new RpcConnection. In HA configs it's common to have something like 10-20 failovers and a 20 second failover delay (no exponential backoff just yet). This means that all of the functions with synchronous interfaces can still block for many minutes after an operation has been canceled, and often the cause of this is something trivial like a bad config file. The current design makes this sort of thing tricky to do because the FileHandles need to be individually cancelable via CancelOperations, but they share the RpcEngine that does the async magic. Updated design: Original design would end up forcing lots of reconnects. Not a huge issue on an unauthenticated cluster but on a kerberized cluster this is a recipe for Kerberos thinking we're attempting a replay attack. User visible cancellation and internal resources cleanup are separable issues. The former can be implemented by atomically swapping the callback of the operation to be canceled with a no-op callback. The original callback is then posted to the IoService with an OperationCanceled status and the user is no longer blocked. For RPC cancels this is sufficient, it's not expensive to keep a request around a little bit longer and when it's eventually invoked or timed out it invokes the no-op callback and is ignored (other than a trace level log notification). Connect cancels push a flag down into the RPC engine to kill the connection and make sure it doesn't attempt to reconnect. was: Cancel support is now reasonably robust except the case where a FileHandle operation ends up causing the RpcEngine to try to create a new RpcConnection. In HA configs it's common to have something like 10-20 failovers and a 20 second failover delay (no exponential backoff just yet). This means that all of the functions with synchronous interfaces can still block for many minutes after an operation has been canceled, and often the cause of this is something trivial like a bad config file. The current design makes this sort of thing tricky to do because the FileHandles need to be individually cancelable via CancelOperations, but they share the RpcEngine that does the async magic. A non-exhaustive list of design assumptions: 1) multiple users will be doing stuff on the same FS in the same process, and some users might be a lot more impatient than others. This means that it's possible that progress is slow and they want to give up but it wasn't stalled and other users are still able to make progress. Side effects of a FileHandle::CancelOperations call should only be visible to the owner of that FH. 2) In most use cases the library is spending more time in the read path than namenode metadata operations. At any given time it's unlikely that there are a crazy amount of pending RPC requests though this certainly can happen (see [~anatoli.shein]'s awesome tools). Some sparse design plans to help out reviewers: 1a) RPC Request objects get something analogous to the ReaderGroup to track all pending requests associated with a FileHandle. As long as there is a transitive dependency on the FH from the request a flag can be pushed down. 1b) FileSystem operations also need the same support. Since they return their result directly there isn't an object to call a cancel method on. One approach here would be to pass in an optional flag (CancelHandle object). 2) Based on assumption 2 it's generally not unacceptably expensive to cancel and resend async RPC calls. Since the RpcConnection is shared for all pending requests it needs to be wiped out. This will cause all pending and on-the-fly requests to return asio::operation_aborted status. If the Request object doesn't have it's flag set to canceled it gets placed back in line using the same mechanism as common RPC errors. This retry does not count against the retry_count or failover_count since it's a side effect of the cancel. Nor should this cause the RpcEngine to attempt to fail over. > libhdfs++: FileHandleImpl::CancelOperations needs to be able to cancel > pending connections > -- > > Key: HDFS-11028 > URL: https://issues.apache.org/jira/browse/HDFS-11028 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > > Cancel support is now reasonably robust except the case where a FileHandle > operation ends up causing the RpcEngine to try to create a new RpcConnection. > In HA
[jira] [Updated] (HDFS-11266) libhdfs++: Redesign block reader with with simplicity and resource management in mind
[
https://issues.apache.org/jira/browse/HDFS-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-11266:
---
Status: Patch Available (was: Open)
> libhdfs++: Redesign block reader with with simplicity and resource management
> in mind
> -
>
> Key: HDFS-11266
> URL: https://issues.apache.org/jira/browse/HDFS-11266
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-11266.HDFS-8707.000.patch
>
>
> The goal here is to significantly simplify the block reader and make it much
> harder to introduce issues. There are plenty of examples of these issues in
> the subtasks of HDFS-8707, the one that finally motivated a reimplementation
> is HDFS-10931.
> Goals:
> -The read side protocol of the data transfer pipeline is fundamentally really
> simple (even if done asynchronously). The code should be equally simple.
> -Get the code in a state that should be easy enough to reason about with a
> solid understanding of HDFS and basic understanding of C++ and vice versa:
> improve comments and avoid using esoteric C++ constructs. This is a
> must-have in order to lower the bar to contribute.
> -Get rid of dependencies on the existing continuation stuff. Myself and
> others have spent far too much time debugging both the continuation code and
> bugs introduced because the continuation code was hard to reason about.
> Notable issues:
> -It's cool from a theoretical perspective, but after 18 months of working
> on this it's still unclear what problem the continuation pattern helped solve
> that callbacks couldn't.
> -They spend more time allocating memory than the rest of the code does
> doing real work - seriously, profile it. This can't be fixed because the
> Pipeline takes ownership of all Continuation objects and then deletes them.
> -The way the block reader really uses them is a hybrid of a state machine,
> continuations, and directly using asio callbacks to bounce between the two.
> Proposed approach:
> Still have a BlockReader class that owns a PacketReader class, the packet
> reader is analogous to the ReadPacketContinuation that the BlockReader builds
> now. The difference is that none of this will be stitched together at
> runtime using continuations, and once we have a block reader with a member
> packet reader that gets allocated up front. The PacketReader can be recycled
> in order to avoid allocations. The block reader is only responsible for
> requesting block info, after that it keeps invoking the PacketReader until
> enough data has been read.
> Async chaining:
> Move to a state machine based approach. This allows the readers to be pinned
> in memory, where each state is represented as a method. The asynchronous IO
> becomes the state transitions. A callback is supplied to the asio async call
> that jumps to the next state upon completion of the IO operation. Epsilon
> transitions will be fairly rare, but if we need them to temporarily drop a
> lock as is done in the RPC code io_service::post can be used rather than a
> call that actually does IO.
> I'm fairly confident in this approach since I used the same to implement
> various hardware async bus interfaces in VHDL to good effect i.e. high
> performance and easy to understand. An asio callback is roughly analogous to
> a signal in a sensitivity list as the methods are to process blocks.
> Example state machine that would send some stuff, then wait to get something
> back like what the current BlockReader::AsyncRequestBlock does using the
> approach described above.
> {code}
> class ExampleHandshake {
> // class would own any small buffers so they can be directly accessed
> public:
> void SendHandshake();
> private:
> void OnHandshakeSend();
> void OnHandShakeDone();
> asio::io_service service_;
> asio::ip::tcp::socket socket_;
> }
> void ExampleHandshake::SendHandshake() {
> // trampoline to jump into read state once write completes
> auto trampoline[this](asio::error_code ec, size_t sz) {
> //error checking here
>this->OnHandshakeSend();
> };
> asio::write(service_, socket_, asio buffer of data here, trampoline);
> }
> void ExampleHandshake::OnHandshakeSend() {
> // when read completes bounce into handler
> auto trampoline = [this](asio::error_code ec, size_t sz) {
> this->OnHandshakeDone();
> };
> asio::read(service_, socket_, asio buffer for received data, trampoline);
> }
> void ExampleHandshake::OnHandshakeDone() {
> //just finished sending request, and receiving response, go do something
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-11268) Persist erasure coding policy ID in FSImage
[ https://issues.apache.org/jira/browse/HDFS-11268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770761#comment-15770761 ] Arpit Agarwal commented on HDFS-11268: -- bq. In case if the ec policy on file is not the default ec policy, then the content of the file cannot be accessed correctly in this case. If this is true it sounds like a bug and not an improvement. > Persist erasure coding policy ID in FSImage > --- > > Key: HDFS-11268 > URL: https://issues.apache.org/jira/browse/HDFS-11268 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-must-do > > Currently, FSImage only has the information about whether the file is striped > or not. It doesn't save the erasure coding policy ID. Later, when the FSImage > is loaded to create the name space, the default system ec policy is used to > as file's ec policy. In case if the ec policy on file is not the default ec > policy, then the content of the file cannot be accessed correctly in this > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11271) Typo in NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770745#comment-15770745 ] Arpit Agarwal commented on HDFS-11271: -- +1 > Typo in NameNode UI > --- > > Key: HDFS-11271 > URL: https://issues.apache.org/jira/browse/HDFS-11271 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, ui >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Trivial > Attachments: HDFS-11271.001.patch > > > The HADOOP-11098 commit has a typo: > It should be "" instead of "" when max heap is -1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11258) File mtime change could not save to editlog
[ https://issues.apache.org/jira/browse/HDFS-11258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770512#comment-15770512 ] Jimmy Xiang commented on HDFS-11258: The addendum is good for both branch 2 and 2.8. > File mtime change could not save to editlog > --- > > Key: HDFS-11258 > URL: https://issues.apache.org/jira/browse/HDFS-11258 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: hdfs-11258-addendum-branch2.patch, hdfs-11258.1.patch, > hdfs-11258.2.patch, hdfs-11258.3.patch, hdfs-11258.4.patch > > > When both mtime and atime are changed, and atime is not beyond the precision > limit, the mtime change is not saved to edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11271) Typo in NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11271: --- Affects Version/s: 3.0.0-alpha1 Status: Patch Available (was: Open) > Typo in NameNode UI > --- > > Key: HDFS-11271 > URL: https://issues.apache.org/jira/browse/HDFS-11271 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, ui >Affects Versions: 3.0.0-alpha1, 2.8.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Trivial > Attachments: HDFS-11271.001.patch > > > The HADOOP-11098 commit has a typo: > It should be "" instead of "" when max heap is -1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11258) File mtime change could not save to editlog
[ https://issues.apache.org/jira/browse/HDFS-11258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HDFS-11258: --- Attachment: hdfs-11258-addendum-branch2.patch Attached an addendum for branch-2 to fix compilation. > File mtime change could not save to editlog > --- > > Key: HDFS-11258 > URL: https://issues.apache.org/jira/browse/HDFS-11258 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: hdfs-11258-addendum-branch2.patch, hdfs-11258.1.patch, > hdfs-11258.2.patch, hdfs-11258.3.patch, hdfs-11258.4.patch > > > When both mtime and atime are changed, and atime is not beyond the precision > limit, the mtime change is not saved to edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11271) Typo in NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770494#comment-15770494 ] Wei-Chiu Chuang commented on HDFS-11271: Hello [~ozawa] mind take a look at this trivial patch? Thanks! > Typo in NameNode UI > --- > > Key: HDFS-11271 > URL: https://issues.apache.org/jira/browse/HDFS-11271 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, ui >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Trivial > Attachments: HDFS-11271.001.patch > > > The HADOOP-11098 commit has a typo: > It should be "" instead of "" when max heap is -1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11271) Typo in NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11271: --- Attachment: HDFS-11271.001.patch > Typo in NameNode UI > --- > > Key: HDFS-11271 > URL: https://issues.apache.org/jira/browse/HDFS-11271 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, ui >Affects Versions: 2.8.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Trivial > Attachments: HDFS-11271.001.patch > > > The HADOOP-11098 commit has a typo: > It should be "" instead of "" when max heap is -1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11271) Typo in NameNode UI
Wei-Chiu Chuang created HDFS-11271: -- Summary: Typo in NameNode UI Key: HDFS-11271 URL: https://issues.apache.org/jira/browse/HDFS-11271 Project: Hadoop HDFS Issue Type: Bug Components: namenode, ui Affects Versions: 2.8.0 Reporter: Wei-Chiu Chuang Assignee: Wei-Chiu Chuang Priority: Trivial The HADOOP-11098 commit has a typo: It should be "" instead of "" when max heap is -1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11072) Add ability to unset and change directory EC policy
[
https://issues.apache.org/jira/browse/HDFS-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770272#comment-15770272
]
Rakesh R commented on HDFS-11072:
-
Thanks [~Sammi] for the analysis.
bq. 2. removeErasureCodingPolicy
Since we are already using {{set/unset}} API pattern for the storage policy,
I'd prefer to create {{#unsetErasureCodingPolicy()}} API instead of
{{#removeErasureCodingPolicy()}} API.
{quote}
3. setDefaultReplicationPolicy
set replication on directory. This is only useful when user wants the directory
from stop inheriting from it's parent's ec policy.
{quote}
I understand there will be many use cases, like distcp(Thanks [~andrew.wang]
for pointing out this) etc. IMHO, implementing this logic requires more focused
coding/unit testing and reviews. How about separating out this task alone and
pushing via HDFS-11082 jira. Please feel free to assign HDFS-11082 task to you.
> Add ability to unset and change directory EC policy
> ---
>
> Key: HDFS-11072
> URL: https://issues.apache.org/jira/browse/HDFS-11072
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11072-v1.patch, HDFS-11072-v2.patch,
> HDFS-11072-v3.patch, HDFS-11072-v4.patch
>
>
> Since the directory-level EC policy simply applies to files at create time,
> it makes sense to make it more similar to storage policies and allow changing
> and unsetting the policy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11248) [SPS]: Handle partial block location movements
[
https://issues.apache.org/jira/browse/HDFS-11248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770162#comment-15770162
]
Hadoop QA commented on HDFS-11248:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
30s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}117m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11248 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844377/HDFS-11248-HDFS-10285-03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 44ecb3650570 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10285 / 06a0cbf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17934/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17934/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console
[jira] [Updated] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11270:
-
Description: Currently there are missing three options introduction in
{{HDFSCommands.md}} of command {{hdfs namenode [-bootstrapStandby]}}. Now this
command can support {{-force}}, {{-nonInteractive}} and a additional option
{{-skipSharedEditsCheck}}. It will be better to document what's the meaning of
these options and that will let users know how to use it correctly. (was:
Currently there are missing three options introduction in {{HDFSCommands.md}}
of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can support
{{-force}}, {{-nonInteractive}} and a additional option
{{-skipSharedEditsCheck}}. It will be better to document what's the meaning of
these options and how to use it correctly.)
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11270.001.patch
>
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and that will let users know how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769853#comment-15769853
]
Yiqun Lin edited comment on HDFS-11270 at 12/22/16 11:51 AM:
-
Attach a patch for this. Gentlely ping [~ajisakaa], could you please help see
again, it's similar to the JIRA HDFS-11204, :). Thanks.
was (Author: linyiqun):
Attach a patch for this.
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11270.001.patch
>
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11270:
-
Attachment: HDFS-11270.001.patch
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11270.001.patch
>
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11270:
-
Status: Patch Available (was: Open)
Attach a patch for this.
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11270.001.patch
>
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11270) Document the missing options of NameNode bootstrap command
[
https://issues.apache.org/jira/browse/HDFS-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11270:
-
Description: Currently there are missing three options introduction in
{{HDFSCommands.md}} of command {{hdfs namenode [-bootstrapStandby]}}. Now this
command can support {{-force}}, {{-nonInteractive}} and a additional option
{{-skipSharedEditsCheck}}. It will be better to document what's the meaning of
these options and how to use it correctly. (was: Currently there are missing
three options introduction in {{HDFSCommands.md}} of command {{hdfs namenode
[-bootstrapStandby}}. Now this command can support {{-force}},
{{-nonInteractive}} and a additional option {{-skipSharedEditsCheck}}. It will
be better to document what's the meaning of these options and how to use it
correctly.)
> Document the missing options of NameNode bootstrap command
> --
>
> Key: HDFS-11270
> URL: https://issues.apache.org/jira/browse/HDFS-11270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
>
> Currently there are missing three options introduction in {{HDFSCommands.md}}
> of command {{hdfs namenode [-bootstrapStandby]}}. Now this command can
> support {{-force}}, {{-nonInteractive}} and a additional option
> {{-skipSharedEditsCheck}}. It will be better to document what's the meaning
> of these options and how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11270) Document the missing options of NameNode bootstrap command
Yiqun Lin created HDFS-11270:
Summary: Document the missing options of NameNode bootstrap command
Key: HDFS-11270
URL: https://issues.apache.org/jira/browse/HDFS-11270
Project: Hadoop HDFS
Issue Type: Bug
Components: documentation
Affects Versions: 3.0.0-alpha2
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor
Currently there are missing three options introduction in {{HDFSCommands.md}}
of command {{hdfs namenode [-bootstrapStandby}}. Now this command can support
{{-force}}, {{-nonInteractive}} and a additional option
{{-skipSharedEditsCheck}}. It will be better to document what's the meaning of
these options and how to use it correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10917) Collect peer performance statistics on DataNode.
[
https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769780#comment-15769780
]
Hadoop QA commented on HDFS-10917:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 39s{color} | {color:orange} root: The patch generated 1 new + 678 unchanged
- 9 fixed = 679 total (was 687) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
34s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m
48s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}124m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10917 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844367/HDFS-10917.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 710448cef6db 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 56a13a6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17931/artifact/patchprocess/diff-checkstyle-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17931/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
[jira] [Commented] (HDFS-11248) [SPS]: Handle partial block location movements
[
https://issues.apache.org/jira/browse/HDFS-11248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769708#comment-15769708
]
Rakesh R commented on HDFS-11248:
-
Thanks [~umamaheswararao] for the useful comments. Attached another patch
addressing these comments.
Also, incorporated one bug fix with this patch, where
{{#blockStorageMovementResultCheck()}} function is not removing the {{trackID}}
from {{#storageMovementAttemptedItems list}} if the blocks movement result is a
failure.
{{TestBlockStorageMovementAttemptedItems#testPartialBlockMovementShouldBeRetried4()}}
unit test is added to cover this case.
> [SPS]: Handle partial block location movements
> --
>
> Key: HDFS-11248
> URL: https://issues.apache.org/jira/browse/HDFS-11248
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-10285
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-11248-HDFS-10285-00.patch,
> HDFS-11248-HDFS-10285-01.patch, HDFS-11248-HDFS-10285-02.patch,
> HDFS-11248-HDFS-10285-03.patch
>
>
> This jira is to handle partial block location movements due to unavailability
> of target nodes for the matching storage type.
> For example, We have only A(disk,archive), B(disk) and C(disk,archive) are
> live nodes with A & C have archive storage type. Say, we have a block with
> locations {{A(disk), B(disk), C(disk)}}. Again assume, user changed the
> storage policy to COLD. Now, SPS internally starts preparing the src-target
> pairing like, {{src=> (A, B, C) and target=> (A, C)}} and sends
> BLOCK_STORAGE_MOVEMENT to the coordinator. SPS is skipping B as it doesn't
> have archive media to indicate that it should do retries to satisfy all block
> locations after some time. On receiving the movement command, coordinator
> will pair the src-target node to schedule actual physical movements like,
> {{movetask=> (A, A), (B, C)}}. Here ideally it should do {{(C, C)}} instead
> of {{(B, C)}} but mistakenly choosing the source C and creates problem.
> IMHO, the implicit assumptions of retry needed is creating confusions and
> leads to coding mistakes. One idea to fix this problem is to create a new
> flag {{retryNeeded}} flag to make it more readable. With this, SPS will
> prepare only the matching pair and dummy source slots will be avoided like,
> {{src=> (A, C) and target=> (A, C)}} and mark {{retryNeeded=true}} to convey
> the message that this {{trackId}} has only partial blocks movements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11248) [SPS]: Handle partial block location movements
[
https://issues.apache.org/jira/browse/HDFS-11248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-11248:
Attachment: HDFS-11248-HDFS-10285-03.patch
> [SPS]: Handle partial block location movements
> --
>
> Key: HDFS-11248
> URL: https://issues.apache.org/jira/browse/HDFS-11248
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-10285
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-11248-HDFS-10285-00.patch,
> HDFS-11248-HDFS-10285-01.patch, HDFS-11248-HDFS-10285-02.patch,
> HDFS-11248-HDFS-10285-03.patch
>
>
> This jira is to handle partial block location movements due to unavailability
> of target nodes for the matching storage type.
> For example, We have only A(disk,archive), B(disk) and C(disk,archive) are
> live nodes with A & C have archive storage type. Say, we have a block with
> locations {{A(disk), B(disk), C(disk)}}. Again assume, user changed the
> storage policy to COLD. Now, SPS internally starts preparing the src-target
> pairing like, {{src=> (A, B, C) and target=> (A, C)}} and sends
> BLOCK_STORAGE_MOVEMENT to the coordinator. SPS is skipping B as it doesn't
> have archive media to indicate that it should do retries to satisfy all block
> locations after some time. On receiving the movement command, coordinator
> will pair the src-target node to schedule actual physical movements like,
> {{movetask=> (A, A), (B, C)}}. Here ideally it should do {{(C, C)}} instead
> of {{(B, C)}} but mistakenly choosing the source C and creates problem.
> IMHO, the implicit assumptions of retry needed is creating confusions and
> leads to coding mistakes. One idea to fix this problem is to create a new
> flag {{retryNeeded}} flag to make it more readable. With this, SPS will
> prepare only the matching pair and dummy source slots will be avoided like,
> {{src=> (A, C) and target=> (A, C)}} and mark {{retryNeeded=true}} to convey
> the message that this {{trackId}} has only partial blocks movements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10917) Collect peer performance statistics on DataNode.
[ https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769517#comment-15769517 ] Xiaobing Zhou edited comment on HDFS-10917 at 12/22/16 8:56 AM: Posted v4 patch. # added license to two files. # fix check style issues. # added collecting thread local states. was (Author: xiaobingo): Posted v4 patch. # added license to two files. # fix check style issues. > Collect peer performance statistics on DataNode. > > > Key: HDFS-10917 > URL: https://issues.apache.org/jira/browse/HDFS-10917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10917.000.patch, HDFS-10917.001.patch, > HDFS-10917.002.patch, HDFS-10917.003.patch, HDFS-10917.004.patch > > > DataNodes already detect if replication pipeline operations are slow and log > warnings. For the purpose of analysis, performance metrics are desirable. > This proposes adding them on DataNodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10917) Collect peer performance statistics on DataNode.
[ https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769517#comment-15769517 ] Xiaobing Zhou commented on HDFS-10917: -- Posted v4 patch. # added license to two files. # fix check style issues. > Collect peer performance statistics on DataNode. > > > Key: HDFS-10917 > URL: https://issues.apache.org/jira/browse/HDFS-10917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10917.000.patch, HDFS-10917.001.patch, > HDFS-10917.002.patch, HDFS-10917.003.patch, HDFS-10917.004.patch > > > DataNodes already detect if replication pipeline operations are slow and log > warnings. For the purpose of analysis, performance metrics are desirable. > This proposes adding them on DataNodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10917) Collect peer performance statistics on DataNode.
[ https://issues.apache.org/jira/browse/HDFS-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10917: - Attachment: HDFS-10917.004.patch > Collect peer performance statistics on DataNode. > > > Key: HDFS-10917 > URL: https://issues.apache.org/jira/browse/HDFS-10917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10917.000.patch, HDFS-10917.001.patch, > HDFS-10917.002.patch, HDFS-10917.003.patch, HDFS-10917.004.patch > > > DataNodes already detect if replication pipeline operations are slow and log > warnings. For the purpose of analysis, performance metrics are desirable. > This proposes adding them on DataNodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11216) Add remoteBytesRead counter metrics for erasure coding reconstruction task
[ https://issues.apache.org/jira/browse/HDFS-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769491#comment-15769491 ] SammiChen commented on HDFS-11216: -- Thanks Kai for review and commit the patch! > Add remoteBytesRead counter metrics for erasure coding reconstruction task > -- > > Key: HDFS-11216 > URL: https://issues.apache.org/jira/browse/HDFS-11216 > Project: Hadoop HDFS > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-11216-v1.patch > > > Currently there is Data Node metrics for how many bytes read during EC > reconstruction task. It's better to distinguish the local read and remote > read byte counts separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
