[jira] [Commented] (HDFS-10715) NPE when applying AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405373#comment-15405373
]
Akira Ajisaka commented on HDFS-10715:
--
Mostly looks good to me. Some minor comments:
1.
{code}
BlockPlacementPolicy placementPolicy =
namenode.getNamesystem().getBlockManager().getBlockPlacementPolicy();
{code}
Can we reuse the static variable placementPolicy instead of local variable?
2. Would you fix the checkstyle warnings?
3.
{code}
for (Node node: dataNodes) {
allNodes.add(node);
}
{code}
can be replaced with {{Collections.addAll(allNodes, dataNodes);}}.
> NPE when applying AvailableSpaceBlockPlacementPolicy
>
>
> Key: HDFS-10715
> URL: https://issues.apache.org/jira/browse/HDFS-10715
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
> Environment: cdh5.8.0
>Reporter: Guangbin Zhu
> Attachments: HDFS-10715.001.patch, HDFS-10715.002.patch
>
>
> As HDFS-8131 introduced an AvailableSpaceBlockPlacementPolicy, but In some
> cases, it caused NPE.
> Here are my namenode daemon logs :
> 2016-08-02 13:05:03,271 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 13 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from
> 10.132.89.79:14001 Call#56 Retry#0
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.compareDataNode(AvailableSpaceBlockPlacementPolicy.java:95)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.chooseDataNode(AvailableSpaceBlockPlacementPolicy.java:80)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:691)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:665)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:572)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:457)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:367)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:242)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:114)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:130)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1606)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3315)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:679)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:214)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:489)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> I reviewed the source code, and found the bug in method chooseDataNode.
> clusterMap.chooseRandom may return null, which cannot compare using equals
> a.equals(b) method.
> Though this exception can be caught, and then retry another call. I think
> this bug should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10717) Fix findbugs warnings of hadoop-hdfs-client in branch-2
[
https://issues.apache.org/jira/browse/HDFS-10717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405337#comment-15405337
]
Hadoop QA commented on HDFS-10717:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
5s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in branch-2 has
7 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs-client generated 0
new + 0 unchanged - 7 fixed = 0 total (was 7) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
4s{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821766/HDFS-10717-branch-2.01.patch
|
| JIRA Issue | HDFS-10717 |
| Optional Tests | asflicense xml compile javac javadoc mvninstall
mvnsite unit findbugs checkstyle |
| uname | Linux 88928efe0ba8 3.13.0-36-lowlatency
[jira] [Commented] (HDFS-10715) NPE when applying AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405315#comment-15405315
]
Hadoop QA commented on HDFS-10715:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 30 unchanged - 0 fixed = 34 total (was 30) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.TestDecommissionWithStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821762/HDFS-10715.002.patch |
| JIRA Issue | HDFS-10715 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5c7715300666 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d28c2d9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16301/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16301/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16301/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16301/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> NPE when applying AvailableSpaceBlockPlacementPolicy
>
>
>
[jira] [Commented] (HDFS-10645) Make block report size as a metric and add this metric to datanode web ui
[ https://issues.apache.org/jira/browse/HDFS-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405313#comment-15405313 ] Yuanbo Liu commented on HDFS-10645: --- [~ajisakaa] Sure, uploaded v7 patch to address it. > Make block report size as a metric and add this metric to datanode web ui > - > > Key: HDFS-10645 > URL: https://issues.apache.org/jira/browse/HDFS-10645 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ui >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Attachments: HDFS-10645.001.patch, HDFS-10645.002.patch, > HDFS-10645.003.patch, HDFS-10645.004.patch, HDFS-10645.005.patch, > HDFS-10645.006.patch, HDFS-10645.007.patch, Selection_047.png, > Selection_048.png > > > Record block report size as a metric and show it on datanode UI. It's > important for administrators to know the bottleneck of block report, and the > metric is also a good tuning metric. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10645) Make block report size as a metric and add this metric to datanode web ui
[ https://issues.apache.org/jira/browse/HDFS-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanbo Liu updated HDFS-10645: -- Attachment: HDFS-10645.007.patch > Make block report size as a metric and add this metric to datanode web ui > - > > Key: HDFS-10645 > URL: https://issues.apache.org/jira/browse/HDFS-10645 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ui >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Attachments: HDFS-10645.001.patch, HDFS-10645.002.patch, > HDFS-10645.003.patch, HDFS-10645.004.patch, HDFS-10645.005.patch, > HDFS-10645.006.patch, HDFS-10645.007.patch, Selection_047.png, > Selection_048.png > > > Record block report size as a metric and show it on datanode UI. It's > important for administrators to know the bottleneck of block report, and the > metric is also a good tuning metric. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10717) Fix findbugs warnings of hadoop-hdfs-client in branch-2
[ https://issues.apache.org/jira/browse/HDFS-10717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-10717: - Target Version/s: 2.8.0 > Fix findbugs warnings of hadoop-hdfs-client in branch-2 > --- > > Key: HDFS-10717 > URL: https://issues.apache.org/jira/browse/HDFS-10717 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HDFS-10717-branch-2.01.patch, findbugsHtml.html > > > There are 7 warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10588) False alarm in datanode log - ERROR - Disk Balancer is not enabled
[
https://issues.apache.org/jira/browse/HDFS-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405295#comment-15405295
]
Hadoop QA commented on HDFS-10588:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 58s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.server.balancer.TestBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821759/HDFS-10588.005.patch |
| JIRA Issue | HDFS-10588 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux db5af903714d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d28c2d9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16300/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16300/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16300/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> False alarm in datanode log - ERROR - Disk Balancer is not enabled
> --
>
> Key: HDFS-10588
> URL:
[jira] [Updated] (HDFS-10717) Fix findbugs warnings of hadoop-hdfs-client in branch-2
[ https://issues.apache.org/jira/browse/HDFS-10717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-10717: - Status: Patch Available (was: Open) > Fix findbugs warnings of hadoop-hdfs-client in branch-2 > --- > > Key: HDFS-10717 > URL: https://issues.apache.org/jira/browse/HDFS-10717 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HDFS-10717-branch-2.01.patch, findbugsHtml.html > > > There are 7 warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10717) Fix findbugs warnings of hadoop-hdfs-client in branch-2
[ https://issues.apache.org/jira/browse/HDFS-10717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-10717: - Attachment: HDFS-10717-branch-2.01.patch > Fix findbugs warnings of hadoop-hdfs-client in branch-2 > --- > > Key: HDFS-10717 > URL: https://issues.apache.org/jira/browse/HDFS-10717 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HDFS-10717-branch-2.01.patch, findbugsHtml.html > > > There are 7 warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10717) Fix findbugs warnings of hadoop-hdfs-client in branch-2
[ https://issues.apache.org/jira/browse/HDFS-10717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned HDFS-10717: Assignee: Akira Ajisaka > Fix findbugs warnings of hadoop-hdfs-client in branch-2 > --- > > Key: HDFS-10717 > URL: https://issues.apache.org/jira/browse/HDFS-10717 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: findbugsHtml.html > > > There are 7 warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-10717) Fix findbugs warnings of hadoop-hdfs-client in branch-2
[ https://issues.apache.org/jira/browse/HDFS-10717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka moved HADOOP-13460 to HDFS-10717: --- Key: HDFS-10717 (was: HADOOP-13460) Project: Hadoop HDFS (was: Hadoop Common) > Fix findbugs warnings of hadoop-hdfs-client in branch-2 > --- > > Key: HDFS-10717 > URL: https://issues.apache.org/jira/browse/HDFS-10717 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Akira Ajisaka > Attachments: findbugsHtml.html > > > There are 7 warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10717) Fix findbugs warnings of hadoop-hdfs-client in branch-2
[ https://issues.apache.org/jira/browse/HDFS-10717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-10717: - Attachment: findbugsHtml.html > Fix findbugs warnings of hadoop-hdfs-client in branch-2 > --- > > Key: HDFS-10717 > URL: https://issues.apache.org/jira/browse/HDFS-10717 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Akira Ajisaka > Attachments: findbugsHtml.html > > > There are 7 warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10715) NPE when applying AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangbin Zhu updated HDFS-10715: Attachment: HDFS-10715.002.patch UT added > NPE when applying AvailableSpaceBlockPlacementPolicy > > > Key: HDFS-10715 > URL: https://issues.apache.org/jira/browse/HDFS-10715 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 > Environment: cdh5.8.0 >Reporter: Guangbin Zhu > Attachments: HDFS-10715.001.patch, HDFS-10715.002.patch > > > As HDFS-8131 introduced an AvailableSpaceBlockPlacementPolicy, but In some > cases, it caused NPE. > Here are my namenode daemon logs : > 2016-08-02 13:05:03,271 WARN org.apache.hadoop.ipc.Server: IPC Server handler > 13 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.132.89.79:14001 Call#56 Retry#0 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.compareDataNode(AvailableSpaceBlockPlacementPolicy.java:95) > at > org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.chooseDataNode(AvailableSpaceBlockPlacementPolicy.java:80) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:691) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:665) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:572) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:457) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:367) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:242) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:114) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:130) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1606) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3315) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:679) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:214) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:489) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) > I reviewed the source code, and found the bug in method chooseDataNode. > clusterMap.chooseRandom may return null, which cannot compare using equals > a.equals(b) method. > Though this exception can be caught, and then retry another call. I think > this bug should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10588) False alarm in datanode log - ERROR - Disk Balancer is not enabled
[
https://issues.apache.org/jira/browse/HDFS-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-10588:
---
Attachment: HDFS-10588.005.patch
Revisited this issue, I think the simplest fix is to remove the error logging
from {{checkDiskBalancerEnabled}}, as the error is already wrapped into
{{DiskBalancerException}}, it could be handled accordingly by callers. Leave
this additional logging will cause the false alarm in logs when UI access
datanode JMX even when disk balacner is disabled.
> False alarm in datanode log - ERROR - Disk Balancer is not enabled
> --
>
> Key: HDFS-10588
> URL: https://issues.apache.org/jira/browse/HDFS-10588
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, hdfs
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-10588.001.patch, HDFS-10588.002.patch,
> HDFS-10588.003.patch, HDFS-10588.004.patch, HDFS-10588.005.patch
>
>
> Noticed error message in namenode log
> {code}2016-06-28 19:49:12,221 ERROR datanode.DiskBalancer
> (DiskBalancer.java:checkDiskBalancerEnabled(297)) - Disk Balancer is not
> enabled.
> {code}
> even with default configuration dfs.disk.balancer.enabled=false.This is
> triggered when accessing datanode web UI, because
> {{DataNode#getDiskBalancerStatus}} calls the check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10588) False alarm in datanode log - ERROR - Disk Balancer is not enabled
[
https://issues.apache.org/jira/browse/HDFS-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405238#comment-15405238
]
Weiwei Yang edited comment on HDFS-10588 at 8/3/16 3:16 AM:
Revisited this issue, I think the simplest fix is to remove the error logging
from {{checkDiskBalancerEnabled}}, as the error is already wrapped into
{{DiskBalancerException}}, it could be handled accordingly by callers. Leave
this additional logging will cause the false alarm in logs when UI access
datanode JMX even when disk balancer is disabled.
was (Author: cheersyang):
Revisited this issue, I think the simplest fix is to remove the error logging
from {{checkDiskBalancerEnabled}}, as the error is already wrapped into
{{DiskBalancerException}}, it could be handled accordingly by callers. Leave
this additional logging will cause the false alarm in logs when UI access
datanode JMX even when disk balacner is disabled.
> False alarm in datanode log - ERROR - Disk Balancer is not enabled
> --
>
> Key: HDFS-10588
> URL: https://issues.apache.org/jira/browse/HDFS-10588
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, hdfs
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-10588.001.patch, HDFS-10588.002.patch,
> HDFS-10588.003.patch, HDFS-10588.004.patch, HDFS-10588.005.patch
>
>
> Noticed error message in namenode log
> {code}2016-06-28 19:49:12,221 ERROR datanode.DiskBalancer
> (DiskBalancer.java:checkDiskBalancerEnabled(297)) - Disk Balancer is not
> enabled.
> {code}
> even with default configuration dfs.disk.balancer.enabled=false.This is
> triggered when accessing datanode web UI, because
> {{DataNode#getDiskBalancerStatus}} calls the check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-4176) EditLogTailer should call rollEdits with a timeout
[
https://issues.apache.org/jira/browse/HDFS-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405234#comment-15405234
]
Hadoop QA commented on HDFS-4176:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
5s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
23s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 49m
36s{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_101 Failed junit tests |
hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821706/HDFS-4176-branch-2.003.patch
|
| JIRA Issue | HDFS-4176 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 374c5269239f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HDFS-10588) False alarm in datanode log - ERROR - Disk Balancer is not enabled
[
https://issues.apache.org/jira/browse/HDFS-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-10588:
---
Summary: False alarm in datanode log - ERROR - Disk Balancer is not enabled
(was: False alarm in namenode log - ERROR - Disk Balancer is not enabled)
> False alarm in datanode log - ERROR - Disk Balancer is not enabled
> --
>
> Key: HDFS-10588
> URL: https://issues.apache.org/jira/browse/HDFS-10588
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, hdfs
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-10588.001.patch, HDFS-10588.002.patch,
> HDFS-10588.003.patch, HDFS-10588.004.patch
>
>
> Noticed error message in namenode log
> {code}2016-06-28 19:49:12,221 ERROR datanode.DiskBalancer
> (DiskBalancer.java:checkDiskBalancerEnabled(297)) - Disk Balancer is not
> enabled.
> {code}
> even with default configuration dfs.disk.balancer.enabled=false.This is
> triggered when accessing datanode web UI, because
> {{DataNode#getDiskBalancerStatus}} calls the check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10707) Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
[
https://issues.apache.org/jira/browse/HDFS-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405167#comment-15405167
]
Akira Ajisaka commented on HDFS-10707:
--
Thanks [~vincentpoon] for rebasing the patch! Minor nit:
{code:title=FSDirStatAndListingOp.java}
-final String startAfterString = new String(startAfter, Charsets.UTF_8);
+final String startAfterString = new String(startAfter,
StandardCharsets.UTF_8);
{code}
Would you render the line within 80 chars to fix the checkstyle warning? I'm +1
if that is addressed.
> Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
> -
>
> Key: HDFS-10707
> URL: https://issues.apache.org/jira/browse/HDFS-10707
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10707.2.patch, HDFS-10707.3.patch,
> HDFS-10707.branch-2.patch, HDFS-10707.patch
>
>
> org.apache.commons.io.Charsets is deprecated in favor of
> java.nio.charset.StandardCharsets
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HDFS-10636:
-
Attachment: HDFS-10636.004.patch
Fixed some findbugs, checkstyle.
[~jpallas] generally I agree on using {{Local\*}}, but it does match a
convention in Hadoop (e.g., {{LocalFileSystem}}, {{LocalRunner}},
{{LocalDirAllocator}}, {{ContainerLocalizer}}) to signify intra-node scope.
Calling these {{File\*}} matches the implementation, but that's incidental to
these being volumes owned/managed by this DN. That said, this convention isn't
used broadly in HDFS yet AFAIK, so if there's a better term we could use it.
* I really like the {{ReplicaBuilder}} pattern in this patch. It makes the code
easier to read. Breaking up the switch statement in {{build()}} avoids a
checkstyle complaint? This might be more legible if each (fairly long) case
stmt were in a private method.
* Is {{DataStorage::getTrashDirectoryForBlockFile}} now unused, except by
tests? {{BlockPoolSliceStorage::getTrashDirectory(File)}} could be inlined into
the new call, since the direct, {{File}} APIs are vestigial.
* It looks like most of the {{NativeIO}} calls are preserved, largely through
{{LocalReplica}}. The design here looks clean. In this section:
{noformat}
- Storage.nativeCopyFileUnbuffered(srcFile, dstFile, true);
+ // create date of the source replica is not longer preserved!
+ copyFileBuffered(srcReplica.getDataInputStream(0),
+ srcReplica.getBlockDataLength(), dstFile);
} catch (IOException e) {
- throw new IOException("Failed to copy " + srcFile + " to " + dstFile, e);
+ throw new IOException("Failed to copy " + srcReplica + " block file to "
+ + dstFile, e);
{noformat}
the call to {{Storage::nativeCopyFileUnbuffered}} is removed, so there's no
path to the {{NativeIO}} libraries. This should be maintained.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-4176) EditLogTailer should call rollEdits with a timeout
[
https://issues.apache.org/jira/browse/HDFS-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405104#comment-15405104
]
Hadoop QA commented on HDFS-4176:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 51m
18s{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}132m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_101 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821706/HDFS-4176-branch-2.003.patch
|
| JIRA Issue | HDFS-4176 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 77087e95dfe4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HDFS-10682) Replace FsDatasetImpl object lock with a separate lock object
[ https://issues.apache.org/jira/browse/HDFS-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-10682: -- Status: In Progress (was: Patch Available) > Replace FsDatasetImpl object lock with a separate lock object > - > > Key: HDFS-10682 > URL: https://issues.apache.org/jira/browse/HDFS-10682 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-10682.001.patch, HDFS-10682.002.patch, > HDFS-10682.003.patch, HDFS-10682.004.patch, HDFS-10682.005.patch, > HDFS-10682.006.patch, HDFS-10682.007.patch, HDFS-10682.008.patch > > > This Jira proposes to replace the FsDatasetImpl object lock with a separate > lock object. Doing so will make it easier to measure lock statistics like > lock held time and warn about potential lock contention due to slow disk > operations. > In the future we can also consider replacing the lock with a read-write lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10682) Replace FsDatasetImpl object lock with a separate lock object
[ https://issues.apache.org/jira/browse/HDFS-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-10682: -- Status: Patch Available (was: In Progress) > Replace FsDatasetImpl object lock with a separate lock object > - > > Key: HDFS-10682 > URL: https://issues.apache.org/jira/browse/HDFS-10682 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-10682.001.patch, HDFS-10682.002.patch, > HDFS-10682.003.patch, HDFS-10682.004.patch, HDFS-10682.005.patch, > HDFS-10682.006.patch, HDFS-10682.007.patch, HDFS-10682.008.patch > > > This Jira proposes to replace the FsDatasetImpl object lock with a separate > lock object. Doing so will make it easier to measure lock statistics like > lock held time and warn about potential lock contention due to slow disk > operations. > In the future we can also consider replacing the lock with a read-write lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10682) Replace FsDatasetImpl object lock with a separate lock object
[ https://issues.apache.org/jira/browse/HDFS-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-10682: -- Attachment: HDFS-10682.008.patch > Replace FsDatasetImpl object lock with a separate lock object > - > > Key: HDFS-10682 > URL: https://issues.apache.org/jira/browse/HDFS-10682 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-10682.001.patch, HDFS-10682.002.patch, > HDFS-10682.003.patch, HDFS-10682.004.patch, HDFS-10682.005.patch, > HDFS-10682.006.patch, HDFS-10682.007.patch, HDFS-10682.008.patch > > > This Jira proposes to replace the FsDatasetImpl object lock with a separate > lock object. Doing so will make it easier to measure lock statistics like > lock held time and warn about potential lock contention due to slow disk > operations. > In the future we can also consider replacing the lock with a read-write lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-742) A down DataNode makes Balancer to hang on repeatingly asking NameNode its partial block list
[
https://issues.apache.org/jira/browse/HDFS-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405001#comment-15405001
]
Hadoop QA commented on HDFS-742:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821694/HDFS-742.v1.trunk.patch
|
| JIRA Issue | HDFS-742 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 882346dd1655 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 954465e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16296/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16296/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16296/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> A down DataNode makes Balancer to hang on repeatingly asking NameNode its
> partial block list
>
>
> Key: HDFS-742
> URL:
[jira] [Commented] (HDFS-10682) Replace FsDatasetImpl object lock with a separate lock object
[
https://issues.apache.org/jira/browse/HDFS-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404974#comment-15404974
]
Hadoop QA commented on HDFS-10682:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 18 new + 530 unchanged - 11 fixed = 548 total (was 541) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 56m
54s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821688/HDFS-10682.007.patch |
| JIRA Issue | HDFS-10682 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1fadd43e7add 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 954465e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16295/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16295/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16295/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Replace FsDatasetImpl object lock with a separate lock object
> -
>
> Key: HDFS-10682
> URL: https://issues.apache.org/jira/browse/HDFS-10682
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Chen Liang
>
[jira] [Commented] (HDFS-9353) Code and comment mismatch in JavaKeyStoreProvider
[
https://issues.apache.org/jira/browse/HDFS-9353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404928#comment-15404928
]
Hudson commented on HDFS-9353:
--
FAILURE: Integrated in Hadoop-trunk-Commit #10197 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10197/])
HDFS-9353. Code and comment mismatch in JavaKeyStoreProvider. (Andras (arp: rev
d28c2d9f556372bce2d6a6d56fefabf0ba47094f)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ProviderUtils.java
> Code and comment mismatch in JavaKeyStoreProvider
> ---
>
> Key: HDFS-9353
> URL: https://issues.apache.org/jira/browse/HDFS-9353
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: nijel
>Assignee: Andras Bokor
>Priority: Trivial
> Fix For: 2.8.0
>
> Attachments: HDFS-9353.01.patch
>
>
> In
> org.apache.hadoop.crypto.key.JavaKeyStoreProvider.JavaKeyStoreProvider(URI
> uri, Configuration conf) throws IOException
> The comment mentioned is
> {code}
> // Get the password file from the conf, if not present from the user's
> // environment var
> {code}
> But the code takes the value form ENV first
> I think this make sense since the user can pass the ENV for a particular run.
> My suggestion is to change the comment
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-4176) EditLogTailer should call rollEdits with a timeout
[ https://issues.apache.org/jira/browse/HDFS-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-4176: Attachment: HDFS-4176-branch-2.003.patch > EditLogTailer should call rollEdits with a timeout > -- > > Key: HDFS-4176 > URL: https://issues.apache.org/jira/browse/HDFS-4176 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, namenode >Affects Versions: 2.0.2-alpha, 3.0.0-alpha1 >Reporter: Todd Lipcon >Assignee: Lei (Eddy) Xu > Attachments: HDFS-4176-branch-2.0.patch, > HDFS-4176-branch-2.003.patch, HDFS-4176-branch-2.1.patch, > HDFS-4176-branch-2.2.patch, HDFS-4176.00.patch, HDFS-4176.01.patch, > HDFS-4176.02.patch, HDFS-4176.03.patch, HDFS-4176.04.patch, namenode.jstack4 > > > When the EditLogTailer thread calls rollEdits() on the active NN via RPC, it > currently does so without a timeout. So, if the active NN has frozen (but not > actually crashed), this call can hang forever. This can then potentially > prevent the standby from becoming active. > This may actually considered a side effect of HADOOP-6762 -- if the RPC were > interruptible, that would also fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-4176) EditLogTailer should call rollEdits with a timeout
[ https://issues.apache.org/jira/browse/HDFS-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-4176: Attachment: (was: HDFS-4176-branch-2.3.patch) > EditLogTailer should call rollEdits with a timeout > -- > > Key: HDFS-4176 > URL: https://issues.apache.org/jira/browse/HDFS-4176 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, namenode >Affects Versions: 2.0.2-alpha, 3.0.0-alpha1 >Reporter: Todd Lipcon >Assignee: Lei (Eddy) Xu > Attachments: HDFS-4176-branch-2.0.patch, > HDFS-4176-branch-2.003.patch, HDFS-4176-branch-2.1.patch, > HDFS-4176-branch-2.2.patch, HDFS-4176.00.patch, HDFS-4176.01.patch, > HDFS-4176.02.patch, HDFS-4176.03.patch, HDFS-4176.04.patch, namenode.jstack4 > > > When the EditLogTailer thread calls rollEdits() on the active NN via RPC, it > currently does so without a timeout. So, if the active NN has frozen (but not > actually crashed), this call can hang forever. This can then potentially > prevent the standby from becoming active. > This may actually considered a side effect of HADOOP-6762 -- if the RPC were > interruptible, that would also fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404918#comment-15404918
]
Benoy Antony commented on HDFS-10477:
-
Didn't see Arpit's previous comments. If the recommended approach is set to
dfs.namenode.fslock.fair to false, then it may be good to sleep for a
millisecond between lock release and re-acquisition.
The patch looks good to me , +1
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
>
[jira] [Commented] (HDFS-10707) Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
[
https://issues.apache.org/jira/browse/HDFS-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404915#comment-15404915
]
Hadoop QA commented on HDFS-10707:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
23s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s{color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
57s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in branch-2 has
7 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
115 unchanged - 2 fixed = 116 total (was 117) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
5s{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 51m
23s{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
44s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
[jira] [Updated] (HDFS-9353) Code and comment mismatch in JavaKeyStoreProvider
[
https://issues.apache.org/jira/browse/HDFS-9353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-9353:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
+1 committed trunk through branch-2.8. Thank you for the contribution [~boky01].
> Code and comment mismatch in JavaKeyStoreProvider
> ---
>
> Key: HDFS-9353
> URL: https://issues.apache.org/jira/browse/HDFS-9353
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: nijel
>Assignee: Andras Bokor
>Priority: Trivial
> Fix For: 2.8.0
>
> Attachments: HDFS-9353.01.patch
>
>
> In
> org.apache.hadoop.crypto.key.JavaKeyStoreProvider.JavaKeyStoreProvider(URI
> uri, Configuration conf) throws IOException
> The comment mentioned is
> {code}
> // Get the password file from the conf, if not present from the user's
> // environment var
> {code}
> But the code takes the value form ENV first
> I think this make sense since the user can pass the ENV for a particular run.
> My suggestion is to change the comment
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404912#comment-15404912
]
Benoy Antony commented on HDFS-10477:
-
The patch looks good.
There is no need to sleep as the admin can control the lock fairness via
dfs.namenode.fslock.fair. The default value is true which means that the
earlier threads will acquire the lock.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
>
[jira] [Resolved] (HDFS-6687) nn.getNamesystem() may return NPE from JspHelper
[
https://issues.apache.org/jira/browse/HDFS-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee resolved HDFS-6687.
--
Resolution: Done
> nn.getNamesystem() may return NPE from JspHelper
>
>
> Key: HDFS-6687
> URL: https://issues.apache.org/jira/browse/HDFS-6687
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Mit Desai
>Assignee: Mit Desai
>
> In hadoop-2, the http server is started in the very early stage to show the
> progress. If the user tries to get the name system, it may not be completely
> up and the NN logs will have this kind of error.
> {noformat}
> 2014-07-14 15:49:03,521 [***] WARN
> resources.ExceptionHandler: INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.common.JspHelper.getTokenUGI(JspHelper.java:661)
> at
> org.apache.hadoop.hdfs.server.common.JspHelper.getUGI(JspHelper.java:604)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:53)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:41)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:84)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
> at com.yahoo.hadoop.GzipFilter.doFilter(GzipFilter.java:220)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1223)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at
[jira] [Commented] (HDFS-6687) nn.getNamesystem() may return NPE from JspHelper
[
https://issues.apache.org/jira/browse/HDFS-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404904#comment-15404904
]
Kihwal Lee commented on HDFS-6687:
--
The new namenode UI won't hit JSPs like it used to. I don't think this problem
exists anymore.
> nn.getNamesystem() may return NPE from JspHelper
>
>
> Key: HDFS-6687
> URL: https://issues.apache.org/jira/browse/HDFS-6687
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Mit Desai
>Assignee: Mit Desai
>
> In hadoop-2, the http server is started in the very early stage to show the
> progress. If the user tries to get the name system, it may not be completely
> up and the NN logs will have this kind of error.
> {noformat}
> 2014-07-14 15:49:03,521 [***] WARN
> resources.ExceptionHandler: INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.common.JspHelper.getTokenUGI(JspHelper.java:661)
> at
> org.apache.hadoop.hdfs.server.common.JspHelper.getUGI(JspHelper.java:604)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:53)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:41)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:84)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
> at com.yahoo.hadoop.GzipFilter.doFilter(GzipFilter.java:220)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1223)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
[jira] [Updated] (HDFS-742) A down DataNode makes Balancer to hang on repeatingly asking NameNode its partial block list
[ https://issues.apache.org/jira/browse/HDFS-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-742: Attachment: HDFS-742.v1.trunk.patch This somehow didn't get committed. Rebasing to trunk. > A down DataNode makes Balancer to hang on repeatingly asking NameNode its > partial block list > > > Key: HDFS-742 > URL: https://issues.apache.org/jira/browse/HDFS-742 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Reporter: Hairong Kuang >Assignee: Mit Desai >Priority: Minor > Attachments: HDFS-742-trunk.patch, HDFS-742.patch, > HDFS-742.v1.trunk.patch > > > We had a balancer that had not made any progress for a long time. It turned > out it was repeatingly asking Namenode for a partial block list of one > datanode, which was done while the balancer was running. > NameNode should notify Balancer that the datanode is not available and > Balancer should stop asking for the datanode's block list. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10682) Replace FsDatasetImpl object lock with a separate lock object
[ https://issues.apache.org/jira/browse/HDFS-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-10682: -- Status: Patch Available (was: In Progress) > Replace FsDatasetImpl object lock with a separate lock object > - > > Key: HDFS-10682 > URL: https://issues.apache.org/jira/browse/HDFS-10682 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-10682.001.patch, HDFS-10682.002.patch, > HDFS-10682.003.patch, HDFS-10682.004.patch, HDFS-10682.005.patch, > HDFS-10682.006.patch, HDFS-10682.007.patch > > > This Jira proposes to replace the FsDatasetImpl object lock with a separate > lock object. Doing so will make it easier to measure lock statistics like > lock held time and warn about potential lock contention due to slow disk > operations. > In the future we can also consider replacing the lock with a read-write lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10682) Replace FsDatasetImpl object lock with a separate lock object
[ https://issues.apache.org/jira/browse/HDFS-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-10682: -- Status: In Progress (was: Patch Available) > Replace FsDatasetImpl object lock with a separate lock object > - > > Key: HDFS-10682 > URL: https://issues.apache.org/jira/browse/HDFS-10682 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-10682.001.patch, HDFS-10682.002.patch, > HDFS-10682.003.patch, HDFS-10682.004.patch, HDFS-10682.005.patch, > HDFS-10682.006.patch, HDFS-10682.007.patch > > > This Jira proposes to replace the FsDatasetImpl object lock with a separate > lock object. Doing so will make it easier to measure lock statistics like > lock held time and warn about potential lock contention due to slow disk > operations. > In the future we can also consider replacing the lock with a read-write lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10682) Replace FsDatasetImpl object lock with a separate lock object
[ https://issues.apache.org/jira/browse/HDFS-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-10682: -- Attachment: HDFS-10682.007.patch > Replace FsDatasetImpl object lock with a separate lock object > - > > Key: HDFS-10682 > URL: https://issues.apache.org/jira/browse/HDFS-10682 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-10682.001.patch, HDFS-10682.002.patch, > HDFS-10682.003.patch, HDFS-10682.004.patch, HDFS-10682.005.patch, > HDFS-10682.006.patch, HDFS-10682.007.patch > > > This Jira proposes to replace the FsDatasetImpl object lock with a separate > lock object. Doing so will make it easier to measure lock statistics like > lock held time and warn about potential lock contention due to slow disk > operations. > In the future we can also consider replacing the lock with a read-write lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-8780) Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node.
[
https://issues.apache.org/jira/browse/HDFS-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger resolved HDFS-8780.
---
Resolution: Fixed
Thanks, [~shv]! I just confirmed that this fixes the test failure on branch-2.7
> Fetching live/dead datanode list with arg true for
> removeDecommissionNode,returns list with decom node.
> ---
>
> Key: HDFS-8780
> URL: https://issues.apache.org/jira/browse/HDFS-8780
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Fix For: 2.7.4
>
> Attachments: HDFS-8780-branch-2.7.patch, HDFS-8780.1.patch,
> HDFS-8780.2.patch, HDFS-8780.3.patch
>
>
> Current implementation:
> ==
> DatanodeManager#removeDecomNodeFromList() , Decommissioned node will be
> removed from dead/live node list only if below conditions are met
> I . If the Include list is not empty.
> II. If include and exclude list does not have decommissioned node and node
> state is decommissioned.
> {code}
> if (!hostFileManager.hasIncludes()) {
> return;
>}
> if ((!hostFileManager.isIncluded(node)) &&
> (!hostFileManager.isExcluded(node))
> && node.isDecommissioned()) {
> // Include list is not empty, an existing datanode does not appear
> // in both include or exclude lists and it has been decommissioned.
> // Remove it from the node list.
> it.remove();
> }
> {code}
> As mentioned in javadoc a datanode cannot be in "already decommissioned
> datanode state".
> Following the steps mentioned in javadoc datanode state is "dead" and not
> decommissioned.
> *Can we avoid the unnecessary checks and have check for the node is in
> decommissioned state then remove from node list. ?*
> Please provide your feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5805) TestCheckpoint.testCheckpoint fails intermittently on branch2
[
https://issues.apache.org/jira/browse/HDFS-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404771#comment-15404771
]
Hudson commented on HDFS-5805:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #10196 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10196/])
HDFS-5805. TestCheckpoint.testCheckpoint fails intermittently on (kihwal: rev
5e5b8793fba8e25aeba7a74878da4cf8e806f061)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
> TestCheckpoint.testCheckpoint fails intermittently on branch2
> -
>
> Key: HDFS-5805
> URL: https://issues.apache.org/jira/browse/HDFS-5805
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Mit Desai
>Assignee: Eric Badger
> Fix For: 2.8.0, 2.9.0, 3.0.0-alpha2
>
> Attachments: HDFS-5805.001.patch
>
>
> {noformat}
> java.lang.AssertionError: Bad value for metric GetEditAvgTime
> Expected: gt(0.0)
> got: <0.0>
> at org.junit.Assert.assertThat(Assert.java:780)
> at
> org.apache.hadoop.test.MetricsAsserts.assertGaugeGt(MetricsAsserts.java:341)
> at
> org.apache.hadoop.hdfs.server.namenode.TestCheckpoint.testCheckpoint(TestCheckpoint.java:1070)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10662) Optimize UTF8 string/byte conversions
[
https://issues.apache.org/jira/browse/HDFS-10662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404773#comment-15404773
]
Kihwal Lee commented on HDFS-10662:
---
Unfortunately, this just went in and the patch no longer applies.
{noformat}
commit a5fb298e56220a35d61b8d2bda716d8fb8ef8bb7
Author: Akira Ajisaka
Date: Tue Aug 2 17:07:59 2016 +0900
HDFS-10707. Replace org.apache.commons.io.Charsets with
java.nio.charset.StandardCharsets. Contributed by Vincent Poon.
{noformat}
> Optimize UTF8 string/byte conversions
> -
>
> Key: HDFS-10662
> URL: https://issues.apache.org/jira/browse/HDFS-10662
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10662.patch
>
>
> String/byte conversions may take either a Charset instance or its canonical
> name. One might think a Charset instance would be faster due to avoiding a
> lookup and instantiation of a Charset, but it's not. The canonical string
> name variants will cache the string encoder/decoder (obtained from a Charset)
> resulting in better performance.
> LOG4J2-935 describes a real-world performance boost. I micro-benched a
> marginal runtime improvement on jdk 7/8. However for a 16 byte path, using
> the canonical name generated 50% less garbage. For a 64 byte path, 25% of
> the garbage. Given the sheer number of times that paths are (re)parsed, the
> cost adds up quickly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5805) TestCheckpoint.testCheckpoint fails intermittently on branch2
[
https://issues.apache.org/jira/browse/HDFS-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-5805:
-
Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
2.9.0
2.8.0
Status: Resolved (was: Patch Available)
> TestCheckpoint.testCheckpoint fails intermittently on branch2
> -
>
> Key: HDFS-5805
> URL: https://issues.apache.org/jira/browse/HDFS-5805
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Mit Desai
>Assignee: Eric Badger
> Fix For: 2.8.0, 2.9.0, 3.0.0-alpha2
>
> Attachments: HDFS-5805.001.patch
>
>
> {noformat}
> java.lang.AssertionError: Bad value for metric GetEditAvgTime
> Expected: gt(0.0)
> got: <0.0>
> at org.junit.Assert.assertThat(Assert.java:780)
> at
> org.apache.hadoop.test.MetricsAsserts.assertGaugeGt(MetricsAsserts.java:341)
> at
> org.apache.hadoop.hdfs.server.namenode.TestCheckpoint.testCheckpoint(TestCheckpoint.java:1070)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10662) Optimize UTF8 string/byte conversions
[ https://issues.apache.org/jira/browse/HDFS-10662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404761#comment-15404761 ] Kihwal Lee commented on HDFS-10662: --- +1 lgtm. > Optimize UTF8 string/byte conversions > - > > Key: HDFS-10662 > URL: https://issues.apache.org/jira/browse/HDFS-10662 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-10662.patch > > > String/byte conversions may take either a Charset instance or its canonical > name. One might think a Charset instance would be faster due to avoiding a > lookup and instantiation of a Charset, but it's not. The canonical string > name variants will cache the string encoder/decoder (obtained from a Charset) > resulting in better performance. > LOG4J2-935 describes a real-world performance boost. I micro-benched a > marginal runtime improvement on jdk 7/8. However for a 16 byte path, using > the canonical name generated 50% less garbage. For a 64 byte path, 25% of > the garbage. Given the sheer number of times that paths are (re)parsed, the > cost adds up quickly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5805) TestCheckpoint.testCheckpoint fails intermittently on branch2
[
https://issues.apache.org/jira/browse/HDFS-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404756#comment-15404756
]
Kihwal Lee commented on HDFS-5805:
--
Committed to trunk, branch-2 and branch-2.8. Thanks for fixing this, [~ebadger].
> TestCheckpoint.testCheckpoint fails intermittently on branch2
> -
>
> Key: HDFS-5805
> URL: https://issues.apache.org/jira/browse/HDFS-5805
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Mit Desai
>Assignee: Eric Badger
> Fix For: 2.8.0, 2.9.0, 3.0.0-alpha2
>
> Attachments: HDFS-5805.001.patch
>
>
> {noformat}
> java.lang.AssertionError: Bad value for metric GetEditAvgTime
> Expected: gt(0.0)
> got: <0.0>
> at org.junit.Assert.assertThat(Assert.java:780)
> at
> org.apache.hadoop.test.MetricsAsserts.assertGaugeGt(MetricsAsserts.java:341)
> at
> org.apache.hadoop.hdfs.server.namenode.TestCheckpoint.testCheckpoint(TestCheckpoint.java:1070)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8901) Use ByteBuffer in striping positional read
[ https://issues.apache.org/jira/browse/HDFS-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404739#comment-15404739 ] Kai Zheng commented on HDFS-8901: - The latest result looks good. Could anyone help do more reviewing? Thanks! Note it's performance related. > Use ByteBuffer in striping positional read > -- > > Key: HDFS-8901 > URL: https://issues.apache.org/jira/browse/HDFS-8901 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Youwei Wang > Attachments: HDFS-8901-v10.patch, HDFS-8901-v2.patch, > HDFS-8901-v3.patch, HDFS-8901-v4.patch, HDFS-8901-v5.patch, > HDFS-8901-v6.patch, HDFS-8901-v7.patch, HDFS-8901-v8.patch, > HDFS-8901-v9.patch, HDFS-8901.v11.patch, HDFS-8901.v12.patch, > HDFS-8901.v13.patch, HDFS-8901.v14.patch, initial-poc.patch > > > Native erasure coder prefers to direct ByteBuffer for performance > consideration. To prepare for it, this change uses ByteBuffer through the > codes in implementing striping position read. It will also fix avoiding > unnecessary data copying between striping read chunk buffers and decode input > buffers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5805) TestCheckpoint.testCheckpoint fails intermittently on branch2
[
https://issues.apache.org/jira/browse/HDFS-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404720#comment-15404720
]
Kihwal Lee commented on HDFS-5805:
--
+1 lgtm
> TestCheckpoint.testCheckpoint fails intermittently on branch2
> -
>
> Key: HDFS-5805
> URL: https://issues.apache.org/jira/browse/HDFS-5805
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Mit Desai
>Assignee: Eric Badger
> Attachments: HDFS-5805.001.patch
>
>
> {noformat}
> java.lang.AssertionError: Bad value for metric GetEditAvgTime
> Expected: gt(0.0)
> got: <0.0>
> at org.junit.Assert.assertThat(Assert.java:780)
> at
> org.apache.hadoop.test.MetricsAsserts.assertGaugeGt(MetricsAsserts.java:341)
> at
> org.apache.hadoop.hdfs.server.namenode.TestCheckpoint.testCheckpoint(TestCheckpoint.java:1070)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8780) Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node.
[
https://issues.apache.org/jira/browse/HDFS-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404728#comment-15404728
]
Konstantin Shvachko commented on HDFS-8780:
---
Yup, that was the porting problem. Fixed.
> Fetching live/dead datanode list with arg true for
> removeDecommissionNode,returns list with decom node.
> ---
>
> Key: HDFS-8780
> URL: https://issues.apache.org/jira/browse/HDFS-8780
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Fix For: 2.7.4
>
> Attachments: HDFS-8780-branch-2.7.patch, HDFS-8780.1.patch,
> HDFS-8780.2.patch, HDFS-8780.3.patch
>
>
> Current implementation:
> ==
> DatanodeManager#removeDecomNodeFromList() , Decommissioned node will be
> removed from dead/live node list only if below conditions are met
> I . If the Include list is not empty.
> II. If include and exclude list does not have decommissioned node and node
> state is decommissioned.
> {code}
> if (!hostFileManager.hasIncludes()) {
> return;
>}
> if ((!hostFileManager.isIncluded(node)) &&
> (!hostFileManager.isExcluded(node))
> && node.isDecommissioned()) {
> // Include list is not empty, an existing datanode does not appear
> // in both include or exclude lists and it has been decommissioned.
> // Remove it from the node list.
> it.remove();
> }
> {code}
> As mentioned in javadoc a datanode cannot be in "already decommissioned
> datanode state".
> Following the steps mentioned in javadoc datanode state is "dead" and not
> decommissioned.
> *Can we avoid the unnecessary checks and have check for the node is in
> decommissioned state then remove from node list. ?*
> Please provide your feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10682) Replace FsDatasetImpl object lock with a separate lock object
[ https://issues.apache.org/jira/browse/HDFS-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404726#comment-15404726 ] Arpit Agarwal commented on HDFS-10682: -- Hi [~fenghua_hu], thanks for the heads up. I've linked HDFS-9968. That one is going to be a more difficult task. Here we just plan to add instrumentation around the lock so we get a better sense of how bad the locking problem is in real clusters. > Replace FsDatasetImpl object lock with a separate lock object > - > > Key: HDFS-10682 > URL: https://issues.apache.org/jira/browse/HDFS-10682 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-10682.001.patch, HDFS-10682.002.patch, > HDFS-10682.003.patch, HDFS-10682.004.patch, HDFS-10682.005.patch, > HDFS-10682.006.patch > > > This Jira proposes to replace the FsDatasetImpl object lock with a separate > lock object. Doing so will make it easier to measure lock statistics like > lock held time and warn about potential lock contention due to slow disk > operations. > In the future we can also consider replacing the lock with a read-write lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5805) TestCheckpoint.testCheckpoint fails intermittently on branch2
[
https://issues.apache.org/jira/browse/HDFS-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-5805:
-
Target Version/s: 2.8.0
> TestCheckpoint.testCheckpoint fails intermittently on branch2
> -
>
> Key: HDFS-5805
> URL: https://issues.apache.org/jira/browse/HDFS-5805
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Mit Desai
>Assignee: Eric Badger
> Attachments: HDFS-5805.001.patch
>
>
> {noformat}
> java.lang.AssertionError: Bad value for metric GetEditAvgTime
> Expected: gt(0.0)
> got: <0.0>
> at org.junit.Assert.assertThat(Assert.java:780)
> at
> org.apache.hadoop.test.MetricsAsserts.assertGaugeGt(MetricsAsserts.java:341)
> at
> org.apache.hadoop.hdfs.server.namenode.TestCheckpoint.testCheckpoint(TestCheckpoint.java:1070)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10707) Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
[ https://issues.apache.org/jira/browse/HDFS-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated HDFS-10707: Attachment: HDFS-10707.branch-2.patch rebase for branch-2 > Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets > - > > Key: HDFS-10707 > URL: https://issues.apache.org/jira/browse/HDFS-10707 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10707.2.patch, HDFS-10707.3.patch, > HDFS-10707.branch-2.patch, HDFS-10707.patch > > > org.apache.commons.io.Charsets is deprecated in favor of > java.nio.charset.StandardCharsets -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-4176) EditLogTailer should call rollEdits with a timeout
[
https://issues.apache.org/jira/browse/HDFS-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404566#comment-15404566
]
Hadoop QA commented on HDFS-4176:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-4176 does not apply to branch-2.3. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821667/HDFS-4176-branch-2.3.patch
|
| JIRA Issue | HDFS-4176 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16293/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> EditLogTailer should call rollEdits with a timeout
> --
>
> Key: HDFS-4176
> URL: https://issues.apache.org/jira/browse/HDFS-4176
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Affects Versions: 2.0.2-alpha, 3.0.0-alpha1
>Reporter: Todd Lipcon
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-4176-branch-2.0.patch, HDFS-4176-branch-2.1.patch,
> HDFS-4176-branch-2.2.patch, HDFS-4176-branch-2.3.patch, HDFS-4176.00.patch,
> HDFS-4176.01.patch, HDFS-4176.02.patch, HDFS-4176.03.patch,
> HDFS-4176.04.patch, namenode.jstack4
>
>
> When the EditLogTailer thread calls rollEdits() on the active NN via RPC, it
> currently does so without a timeout. So, if the active NN has frozen (but not
> actually crashed), this call can hang forever. This can then potentially
> prevent the standby from becoming active.
> This may actually considered a side effect of HADOOP-6762 -- if the RPC were
> interruptible, that would also fix the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-4176) EditLogTailer should call rollEdits with a timeout
[ https://issues.apache.org/jira/browse/HDFS-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-4176: Attachment: HDFS-4176-branch-2.3.patch rebase and upload > EditLogTailer should call rollEdits with a timeout > -- > > Key: HDFS-4176 > URL: https://issues.apache.org/jira/browse/HDFS-4176 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, namenode >Affects Versions: 2.0.2-alpha, 3.0.0-alpha1 >Reporter: Todd Lipcon >Assignee: Lei (Eddy) Xu > Attachments: HDFS-4176-branch-2.0.patch, HDFS-4176-branch-2.1.patch, > HDFS-4176-branch-2.2.patch, HDFS-4176-branch-2.3.patch, HDFS-4176.00.patch, > HDFS-4176.01.patch, HDFS-4176.02.patch, HDFS-4176.03.patch, > HDFS-4176.04.patch, namenode.jstack4 > > > When the EditLogTailer thread calls rollEdits() on the active NN via RPC, it > currently does so without a timeout. So, if the active NN has frozen (but not > actually crashed), this call can hang forever. This can then potentially > prevent the standby from becoming active. > This may actually considered a side effect of HADOOP-6762 -- if the RPC were > interruptible, that would also fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-4176) EditLogTailer should call rollEdits with a timeout
[
https://issues.apache.org/jira/browse/HDFS-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404540#comment-15404540
]
Hadoop QA commented on HDFS-4176:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 11s{color}
| {color:red} HDFS-4176 does not apply to branch-2.2. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821662/HDFS-4176-branch-2.2.patch
|
| JIRA Issue | HDFS-4176 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16292/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> EditLogTailer should call rollEdits with a timeout
> --
>
> Key: HDFS-4176
> URL: https://issues.apache.org/jira/browse/HDFS-4176
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Affects Versions: 2.0.2-alpha, 3.0.0-alpha1
>Reporter: Todd Lipcon
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-4176-branch-2.0.patch, HDFS-4176-branch-2.1.patch,
> HDFS-4176-branch-2.2.patch, HDFS-4176.00.patch, HDFS-4176.01.patch,
> HDFS-4176.02.patch, HDFS-4176.03.patch, HDFS-4176.04.patch, namenode.jstack4
>
>
> When the EditLogTailer thread calls rollEdits() on the active NN via RPC, it
> currently does so without a timeout. So, if the active NN has frozen (but not
> actually crashed), this call can hang forever. This can then potentially
> prevent the standby from becoming active.
> This may actually considered a side effect of HADOOP-6762 -- if the RPC were
> interruptible, that would also fix the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-4176) EditLogTailer should call rollEdits with a timeout
[
https://issues.apache.org/jira/browse/HDFS-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-4176:
Attachment: HDFS-4176-branch-2.2.patch
Update the patch, fix {{TestHdfsConfigFields}}.
[~jingzhao] would be much appreciated to have another review.
> EditLogTailer should call rollEdits with a timeout
> --
>
> Key: HDFS-4176
> URL: https://issues.apache.org/jira/browse/HDFS-4176
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Affects Versions: 2.0.2-alpha, 3.0.0-alpha1
>Reporter: Todd Lipcon
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-4176-branch-2.0.patch, HDFS-4176-branch-2.1.patch,
> HDFS-4176-branch-2.2.patch, HDFS-4176.00.patch, HDFS-4176.01.patch,
> HDFS-4176.02.patch, HDFS-4176.03.patch, HDFS-4176.04.patch, namenode.jstack4
>
>
> When the EditLogTailer thread calls rollEdits() on the active NN via RPC, it
> currently does so without a timeout. So, if the active NN has frozen (but not
> actually crashed), this call can hang forever. This can then potentially
> prevent the standby from becoming active.
> This may actually considered a side effect of HADOOP-6762 -- if the RPC were
> interruptible, that would also fix the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10682) Replace FsDatasetImpl object lock with a separate lock object
[ https://issues.apache.org/jira/browse/HDFS-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404519#comment-15404519 ] Chen Liang commented on HDFS-10682: --- Thanks for the feedbacks, Arpit! Will upload another patch to reflect this. > Replace FsDatasetImpl object lock with a separate lock object > - > > Key: HDFS-10682 > URL: https://issues.apache.org/jira/browse/HDFS-10682 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-10682.001.patch, HDFS-10682.002.patch, > HDFS-10682.003.patch, HDFS-10682.004.patch, HDFS-10682.005.patch, > HDFS-10682.006.patch > > > This Jira proposes to replace the FsDatasetImpl object lock with a separate > lock object. Doing so will make it easier to measure lock statistics like > lock held time and warn about potential lock contention due to slow disk > operations. > In the future we can also consider replacing the lock with a read-write lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10601) Improve log message to include hostname when the NameNode is in safemode
[
https://issues.apache.org/jira/browse/HDFS-10601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404500#comment-15404500
]
Kihwal Lee commented on HDFS-10601:
---
If the namenode http ip address is set to 0.0.0.0 on the namenode, this change
won't be very useful. Setting the http bind address is not mandatory, if the
user can deploy different configs for the NN and the clients. Using
{{NameNode.getRpcServerAddress()}} might be better. After the
{{NameNodeRpcServer}} is initialized, the defaultFS is set and
{{NameNode.getRpcServerAddress()}} will get the address from there afterwards.
> Improve log message to include hostname when the NameNode is in safemode
>
>
> Key: HDFS-10601
> URL: https://issues.apache.org/jira/browse/HDFS-10601
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Minor
> Attachments: HDFS-10601.001.patch, HDFS-10601.002.patch
>
>
> When remote NN operations are involved, it would be nice to have the Namenode
> hostname in safemode notification log.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
[ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404477#comment-15404477 ] Konstantin Shvachko commented on HDFS-10301: Just pushed branch-2.7 > BlockReport retransmissions may lead to storages falsely being declared > zombie if storage report processing happens out of order > > > Key: HDFS-10301 > URL: https://issues.apache.org/jira/browse/HDFS-10301 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.1 >Reporter: Konstantin Shvachko >Assignee: Vinitha Reddy Gankidi >Priority: Critical > Fix For: 2.7.4 > > Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, > HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch, > HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch, > HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch, > HDFS-10301.012.patch, HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch, > HDFS-10301.sample.patch, zombieStorageLogs.rtf > > > When NameNode is busy a DataNode can timeout sending a block report. Then it > sends the block report again. Then NameNode while process these two reports > at the same time can interleave processing storages from different reports. > This screws up the blockReportId field, which makes NameNode think that some > storages are zombie. Replicas from zombie storages are immediately removed, > causing missing blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
[
https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404413#comment-15404413
]
Daryn Sharp commented on HDFS-10301:
bq. According to Rolling upgrade documentation we first upgrade NameNodes, then
DataNodes. So in practice new DNs don't talk to old NNs.
Although the docs claim downgrading the NN requires full downtime, or rolling
downgrade DNs first, we should make an effort to ensure DNs are compatible when
possible. An emergency NN downgrade shouldn't require full downtime when a
failover to the prior release would suffice.
--
I don't like the idea of BRs triggering pruning of storages. That aside, the
patch doesn't appear to close the race. The lock is released after the storage
report is processing and re-acquired to find to find the "zombies". We're back
to out of order processing of heartbeats, which I think is the real problem,
causing false-positives.
How about something like this? {{DatanodeDescriptor}} descriptor tracks the
last {{BlockReportContext#reportId}}. The value is updated when processing a
BR - which has latest value if BR lease let it in. Heartbeat now includes the
last used {{reportId}}. On the NN, if the heartbeat contains this field, NN
will ignore heartbeart if not equal to DND. There's little details like DN
re-registration resetting the field, etc, but wouldn't something simple like
this work?
> BlockReport retransmissions may lead to storages falsely being declared
> zombie if storage report processing happens out of order
>
>
> Key: HDFS-10301
> URL: https://issues.apache.org/jira/browse/HDFS-10301
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.1
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
>Priority: Critical
> Fix For: 2.7.4
>
> Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch,
> HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch,
> HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch,
> HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch,
> HDFS-10301.012.patch, HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch,
> HDFS-10301.sample.patch, zombieStorageLogs.rtf
>
>
> When NameNode is busy a DataNode can timeout sending a block report. Then it
> sends the block report again. Then NameNode while process these two reports
> at the same time can interleave processing storages from different reports.
> This screws up the blockReportId field, which makes NameNode think that some
> storages are zombie. Replicas from zombie storages are immediately removed,
> causing missing blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10672) libhdfs++: reorder directories in src/main/libhdfspp/examples, and add C++ version of cat tool
[
https://issues.apache.org/jira/browse/HDFS-10672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404346#comment-15404346
]
James Clampffer commented on HDFS-10672:
Hey Anatoli, it looks like this no longer applies cleanly to the current head
of HDFS-8707, could you take a look? I let my turnaround time for finishing
reviews get longer than it should be which could have caused this; sorry about
that.
{code}
virtual Status PositionRead(void *buf, size_t buf_size, off_t offset, size_t
*bytes_read) = 0;
virtual Status Read(void *buf, size_t buf_size, size_t *bytes_read) = 0;
{code}
Thanks for changing this and the bits of code the change touched. The old API
that used nbyte as an in/out parameter was a bad design choice for every
real-world situation I tried using it on.
Overall the code looks good. I think there's a few things worth doing make it
a bit better before committing.
1) In cat.c you have some nice uri parsing code. It looks like you are working
on several other file system tools that are going to be using something similar
if they have C versions. I think it'd be worth pulling out that parsing code
into it's own file so it can be shared between your tools and used by others
who need a nice C API for that. You could also write some C wrappers around
the C++ URI parsing code because it's more functionally complete if you want to
go that route. You can defer this work until some of your other utility tools
start landing; I just want to make sure we don't have several tools with the
same copy/pasted block of code.
2) In cat.cpp you use the same C-style URI code. Since part of the value of
this work is to provide good examples of libhdfs API usage I think you should
reuse our URI parsing class from libhdfspp/lib/common/uri.h here.
3) Like Bob mentioned earlier it's worth updating this to use the
ConfigurationLoader to give you an HdfsConfiguration object that can give you
an Options object that reflects what people have in /etc/hadoop/conf or
$HADOOP_CONF_DIR. I can give you a hand here if you need some references on
how to do this (at least the way I'd do it).
4) In some of the unit tests:
{code}
file_info->file_length_ = 1; //To avoid running into EOF
{code}
Best to add at least one test that makes sure we do get the EOF status when
expected. I've tried to get away with similar "small" changes to other tests
without new tests and it always ended up being more pain later on than writing
the test would have.
5) In the changes to status:
{code}
bool invalid_offset() const { return code_ == kInvalidOffset; }
{code}
Not really a blocker, but I try to follow the pattern of is_ to make it really clear that what you're doing is
strictly a test even when the object isn't const qualified.
General comment not related to your changes but could be worth discussing
somewhere:
Factory functions that return a Status and assign to a user supplied pointer
leads to weird looking code (not at all your fault, this is what the API forces)
{code}
FileHandle *file_raw = nullptr;
stat = fs->Open(uri.path, _raw);
if (!stat.ok()) {
cerr << "Could not open file " << uri.path << endl;
return 1;
}
//wrapping file_raw into a unique pointer to guarantee deletion
unique_ptr file(file_raw);
{code}
It'd sure be nice if FileSystem::Open could just return a unique_ptr or raw
pointer IMO. Google's coding standards pretend/assert that exceptions don't
exist. For their code that might be true, but I think this is going to be a
boilerplate pattern that shows up in any code that uses libhdfs++ but is
written to use RAII in idiomatic C++. Do we want to make life easier for those
users with a more idiomatic constructor? One thing you might want to do in the
error catching block is assert that the pointer is still null (or just call
free on it).
> libhdfs++: reorder directories in src/main/libhdfspp/examples, and add C++
> version of cat tool
> --
>
> Key: HDFS-10672
> URL: https://issues.apache.org/jira/browse/HDFS-10672
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
> Attachments: HDFS-10672.HDFS-8707.000.patch,
> HDFS-10672.HDFS-8707.001.patch, HDFS-10672.HDFS-8707.002.patch,
> HDFS-10672.HDFS-8707.003.patch
>
>
> src/main/libhdfspp/examples should be structured like
> examples/language/utility instead of examples/utility/language for easier
> access by different developers.
> Additionally implementing C++ version of cat tool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional
[jira] [Commented] (HDFS-6262) HDFS doesn't raise FileNotFoundException if the source of a rename() is missing
[
https://issues.apache.org/jira/browse/HDFS-6262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404347#comment-15404347
]
Hadoop QA commented on HDFS-6262:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-6262 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12641192/HDFS-6262.2.patch |
| JIRA Issue | HDFS-6262 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16291/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> HDFS doesn't raise FileNotFoundException if the source of a rename() is
> missing
> ---
>
> Key: HDFS-6262
> URL: https://issues.apache.org/jira/browse/HDFS-6262
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Steve Loughran
>Assignee: Akira Ajisaka
> Labels: BB2015-05-TBR
> Attachments: HDFS-6262.2.patch, HDFS-6262.patch
>
>
> HDFS's {{rename(src, dest)}} returns false if src does not exist -all the
> other filesystems raise {{FileNotFoundException}}
> This behaviour is defined in {{FSDirectory.unprotectedRenameTo()}} -the
> attempt is logged, but the operation then just returns false.
> I propose changing the behaviour of {{DistributedFileSystem}} to be the same
> as that of the others -and of {{FileContext}}, which does reject renames with
> nonexistent sources
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6262) HDFS doesn't raise FileNotFoundException if the source of a rename() is missing
[
https://issues.apache.org/jira/browse/HDFS-6262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404338#comment-15404338
]
Andras Bokor commented on HDFS-6262:
[~ajisakaa],
Any update on this ticket? Will it be closed or fixed?
My thoughts:
Based on my previous experience with rename related issues (HADOOP-13082) it
seems the ecosystem strongly based on the current behavior of rename methods
even they are not consistent. Changing in rename behavior can definitely break
compatibility (another discussion is HDFS-10385, I have just close it as
Later). Please check the linked issues.
I am asking because I am going through the points of HDFS-303 and the 3rd point
is covered by this ticket.
Thanks in advance.
> HDFS doesn't raise FileNotFoundException if the source of a rename() is
> missing
> ---
>
> Key: HDFS-6262
> URL: https://issues.apache.org/jira/browse/HDFS-6262
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Steve Loughran
>Assignee: Akira Ajisaka
> Labels: BB2015-05-TBR
> Attachments: HDFS-6262.2.patch, HDFS-6262.patch
>
>
> HDFS's {{rename(src, dest)}} returns false if src does not exist -all the
> other filesystems raise {{FileNotFoundException}}
> This behaviour is defined in {{FSDirectory.unprotectedRenameTo()}} -the
> attempt is logged, but the operation then just returns false.
> I propose changing the behaviour of {{DistributedFileSystem}} to be the same
> as that of the others -and of {{FileContext}}, which does reject renames with
> nonexistent sources
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10683) Make class Token$PrivateToken private
[
https://issues.apache.org/jira/browse/HDFS-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404316#comment-15404316
]
John Zhuge commented on HDFS-10683:
---
{{TestFileCorruption}} passes locally. The link for row {{unit}} is not valid
(error 404).
> Make class Token$PrivateToken private
> -
>
> Key: HDFS-10683
> URL: https://issues.apache.org/jira/browse/HDFS-10683
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: fs, ha, security, security_token
> Attachments: HDFS-10683.001.patch
>
>
> Avoid {{instanceof}} or typecasting of {{Toke.PrivateToken}} by introducing
> an interface method in {{Token}}. Make class {{Toke.PrivateToken}} private.
> Use a factory method instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-303) Make contracts of LocalFileSystem and DistributedFileSystem consistent
[ https://issues.apache.org/jira/browse/HDFS-303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-303 started by Andras Bokor. - > Make contracts of LocalFileSystem and DistributedFileSystem consistent > -- > > Key: HDFS-303 > URL: https://issues.apache.org/jira/browse/HDFS-303 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Tom White >Assignee: Andras Bokor > Attachments: HDFS-303-common-test-case.patch, HDFS-303.patch, > hadoop-4114.patch > > > There are a number of edge cases that the two file system implementations > handle differently. In particular: > * When trying to make a directory under an existing file, HDFS throws an > IOException while LocalFileSystem doesn't. > * The FileSytem#listStatus(Path) method returns null for a non-existent file > on HDFS, while LocalFileSytem returns an empty FileStatus array. > * When trying to rename a non-existent path, LocalFileSystem throws an > IOException, while HDFS returns false. > * When renaming a file or directory to a non-existent directory (e.g. /a/b to > /c/d, where /c doesn't exist) LocalFileSystem succeeds (returns true) while > HDFS fails (false). > * When renaming a file (or directory) as an existing file (or directory) > LocalFileSystem succeeds (returns true) while HDFS fails (false). > We should document the expected behaviour for these cases in FileSystem's > javadoc, and make sure all implementations conform to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-303) Make contracts of LocalFileSystem and DistributedFileSystem consistent
[ https://issues.apache.org/jira/browse/HDFS-303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor reassigned HDFS-303: - Assignee: Andras Bokor > Make contracts of LocalFileSystem and DistributedFileSystem consistent > -- > > Key: HDFS-303 > URL: https://issues.apache.org/jira/browse/HDFS-303 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Tom White >Assignee: Andras Bokor > Attachments: HDFS-303-common-test-case.patch, HDFS-303.patch, > hadoop-4114.patch > > > There are a number of edge cases that the two file system implementations > handle differently. In particular: > * When trying to make a directory under an existing file, HDFS throws an > IOException while LocalFileSystem doesn't. > * The FileSytem#listStatus(Path) method returns null for a non-existent file > on HDFS, while LocalFileSytem returns an empty FileStatus array. > * When trying to rename a non-existent path, LocalFileSystem throws an > IOException, while HDFS returns false. > * When renaming a file or directory to a non-existent directory (e.g. /a/b to > /c/d, where /c doesn't exist) LocalFileSystem succeeds (returns true) while > HDFS fails (false). > * When renaming a file (or directory) as an existing file (or directory) > LocalFileSystem succeeds (returns true) while HDFS fails (false). > We should document the expected behaviour for these cases in FileSystem's > javadoc, and make sure all implementations conform to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10715) NPE when applying AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404201#comment-15404201
]
Hadoop QA commented on HDFS-10715:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFileCreationDelete |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821530/HDFS-10715.001.patch |
| JIRA Issue | HDFS-10715 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 785be055fad4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7fc70c6 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16290/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16290/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16290/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> NPE when applying AvailableSpaceBlockPlacementPolicy
>
>
> Key: HDFS-10715
> URL: https://issues.apache.org/jira/browse/HDFS-10715
> Project: Hadoop HDFS
> Issue Type: Bug
>
[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
[ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404168#comment-15404168 ] Kihwal Lee commented on HDFS-10301: --- [~shv], thanks for the revert, but I think you missed branch-2.7. > BlockReport retransmissions may lead to storages falsely being declared > zombie if storage report processing happens out of order > > > Key: HDFS-10301 > URL: https://issues.apache.org/jira/browse/HDFS-10301 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.1 >Reporter: Konstantin Shvachko >Assignee: Vinitha Reddy Gankidi >Priority: Critical > Fix For: 2.7.4 > > Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, > HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch, > HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch, > HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch, > HDFS-10301.012.patch, HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch, > HDFS-10301.sample.patch, zombieStorageLogs.rtf > > > When NameNode is busy a DataNode can timeout sending a block report. Then it > sends the block report again. Then NameNode while process these two reports > at the same time can interleave processing storages from different reports. > This screws up the blockReportId field, which makes NameNode think that some > storages are zombie. Replicas from zombie storages are immediately removed, > causing missing blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10716) The target node should be removed in balancer when the size of bytes that need to move firstly reduced to less than 0
[
https://issues.apache.org/jira/browse/HDFS-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404111#comment-15404111
]
Hadoop QA commented on HDFS-10716:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
5s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 63m
20s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821604/HDFS-10716.001.patch |
| JIRA Issue | HDFS-10716 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7b3f5bcd5a80 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7fc70c6 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16289/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16289/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The target node should be removed in balancer when the size of bytes that
> need to move firstly reduced to less than 0
> -
>
> Key: HDFS-10716
> URL: https://issues.apache.org/jira/browse/HDFS-10716
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Yiqun Lin
>
[jira] [Commented] (HDFS-10714) Issue in handling checksum errors in write pipeline when fault DN is LAST_IN_PIPELINE
[ https://issues.apache.org/jira/browse/HDFS-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404103#comment-15404103 ] Kihwal Lee commented on HDFS-10714: --- Guessing who is faulty is complicated. Whatever works for a case can break other cases. It is not safe to remove the last guy, since it might be the only one that has valid data up to the ACKed bytes in certain cases. May be we should have the client examine datanodes individually before recreating a pipeline. - Have each datanode perform checksum validation up to the ACKed bytes (needs to be bytes-per-checksum aligned). - For identifying a bad node, directly write to N nodes for some number of packets. Exclude failed nodes and rebuild a pipeline. > Issue in handling checksum errors in write pipeline when fault DN is > LAST_IN_PIPELINE > - > > Key: HDFS-10714 > URL: https://issues.apache.org/jira/browse/HDFS-10714 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > > We had come across one issue, where write is failed even 7 DN’s are available > due to network fault at one datanode which is LAST_IN_PIPELINE. It will be > similar to HDFS-6937 . > Scenario : (DN3 has N/W Fault and Min repl=2). > Write pipeline: > DN1->DN2->DN3 => DN3 Gives ERROR_CHECKSUM ack. And so DN2 marked as bad > DN1->DN4-> DN3 => DN3 Gives ERROR_CHECKSUM ack. And so DN4 is marked as bad > …. > And so on ( all the times DN3 is LAST_IN_PIPELINE) ... Continued till no more > datanodes to construct the pipeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10715) NPE when applying AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10715: -- Status: Patch Available (was: Open) > NPE when applying AvailableSpaceBlockPlacementPolicy > > > Key: HDFS-10715 > URL: https://issues.apache.org/jira/browse/HDFS-10715 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 > Environment: cdh5.8.0 >Reporter: Guangbin Zhu > Attachments: HDFS-10715.001.patch > > > As HDFS-8131 introduced an AvailableSpaceBlockPlacementPolicy, but In some > cases, it caused NPE. > Here are my namenode daemon logs : > 2016-08-02 13:05:03,271 WARN org.apache.hadoop.ipc.Server: IPC Server handler > 13 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.132.89.79:14001 Call#56 Retry#0 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.compareDataNode(AvailableSpaceBlockPlacementPolicy.java:95) > at > org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.chooseDataNode(AvailableSpaceBlockPlacementPolicy.java:80) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:691) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:665) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:572) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:457) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:367) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:242) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:114) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:130) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1606) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3315) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:679) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:214) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:489) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) > I reviewed the source code, and found the bug in method chooseDataNode. > clusterMap.chooseRandom may return null, which cannot compare using equals > a.equals(b) method. > Though this exception can be caught, and then retry another call. I think > this bug should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-8780) Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node.
[
https://issues.apache.org/jira/browse/HDFS-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger reopened HDFS-8780:
---
Please investigate 2.7 unit test failure
> Fetching live/dead datanode list with arg true for
> removeDecommissionNode,returns list with decom node.
> ---
>
> Key: HDFS-8780
> URL: https://issues.apache.org/jira/browse/HDFS-8780
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Fix For: 2.7.4
>
> Attachments: HDFS-8780-branch-2.7.patch, HDFS-8780.1.patch,
> HDFS-8780.2.patch, HDFS-8780.3.patch
>
>
> Current implementation:
> ==
> DatanodeManager#removeDecomNodeFromList() , Decommissioned node will be
> removed from dead/live node list only if below conditions are met
> I . If the Include list is not empty.
> II. If include and exclude list does not have decommissioned node and node
> state is decommissioned.
> {code}
> if (!hostFileManager.hasIncludes()) {
> return;
>}
> if ((!hostFileManager.isIncluded(node)) &&
> (!hostFileManager.isExcluded(node))
> && node.isDecommissioned()) {
> // Include list is not empty, an existing datanode does not appear
> // in both include or exclude lists and it has been decommissioned.
> // Remove it from the node list.
> it.remove();
> }
> {code}
> As mentioned in javadoc a datanode cannot be in "already decommissioned
> datanode state".
> Following the steps mentioned in javadoc datanode state is "dead" and not
> decommissioned.
> *Can we avoid the unnecessary checks and have check for the node is in
> decommissioned state then remove from node list. ?*
> Please provide your feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8224) Any IOException in DataTransfer#run() will run diskError thread even if it is not disk error
[
https://issues.apache.org/jira/browse/HDFS-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403981#comment-15403981
]
Kihwal Lee commented on HDFS-8224:
--
bq. Incidentally, we just saw the same stack trace "java.io.IOException: Could
not create DataChecksum of type 0 with bytesPerChecksum 0" in a cluster. I
wonder if it's caused by a software (HDFS) bug rather than a hardware failure.
We have seen this type of errors caused by flaky disks. The kernel(ext4) would
say delayed block allocations failed and part of the block and/or meta file
ends up being all null. This happens asynchronous to user's I/O activities, so
the writes don't get any errors. Unfortunately including
{{data_err=abort,errors=remount-ro}} in the mount option does not make it go
read-only right away.
> Any IOException in DataTransfer#run() will run diskError thread even if it is
> not disk error
>
>
> Key: HDFS-8224
> URL: https://issues.apache.org/jira/browse/HDFS-8224
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Fix For: 2.8.0
>
>
> This happened in our 2.6 cluster.
> One of the block and its metadata file were corrupted.
> The disk was healthy in this case.
> Only the block was corrupt.
> Namenode tried to copy that block to another datanode but failed with the
> following stack trace:
> 2015-04-20 01:04:04,421
> [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@11319bc4] WARN
> datanode.DataNode: DatanodeRegistration(a.b.c.d,
> datanodeUuid=e8c5135c-9b9f-4d05-a59d-e5525518aca7, infoPort=1006,
> infoSecurePort=0, ipcPort=8020,
> storageInfo=lv=-56;cid=CID-e7f736ac-158e-446e-9091-7e66f3cddf3c;nsid=358250775;c=1428471998571):Failed
> to transfer BP-xxx-1351096255769:blk_2697560713_1107108863999 to
> a1.b1.c1.d1:1004 got
> java.io.IOException: Could not create DataChecksum of type 0 with
> bytesPerChecksum 0
> at
> org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:125)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:175)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:140)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readDataChecksum(BlockMetadataHeader.java:102)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:287)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1989)
> at java.lang.Thread.run(Thread.java:722)
> The following catch block in DataTransfer#run method will treat every
> IOException as disk error fault and run disk errror
> {noformat}
> catch (IOException ie) {
> LOG.warn(bpReg + ":Failed to transfer " + b + " to " +
> targets[0] + " got ", ie);
> // check if there are any disk problem
> checkDiskErrorAsync();
> }
> {noformat}
> This block was never scanned by BlockPoolSliceScanner otherwise it would have
> reported as corrupt block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10716) The target node should be removed in balancer when the size of bytes that need to move firstly reduced to less than 0
[
https://issues.apache.org/jira/browse/HDFS-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10716:
-
Description:
In HDFS-10602, we found a failing case that the balancer moves data always
between 2 DNs. And it made the balancer can't be finished. I debug the code for
this, I found there seems a bug in choosing pending blocks in
{{Dispatcher.Source.chooseNextMove}}.
The codes:
{code}
private PendingMove chooseNextMove() {
for (Iterator i = tasks.iterator(); i.hasNext();) {
final Task task = i.next();
final DDatanode target = task.target.getDDatanode();
final PendingMove pendingBlock = new PendingMove(this, task.target);
if (target.addPendingBlock(pendingBlock)) {
// target is not busy, so do a tentative block allocation
if (pendingBlock.chooseBlockAndProxy()) {
long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
incScheduledSize(-blockSize);
task.size -= blockSize;
// If the size of bytes that need to be moved was first reduced to
less than 0
// it should also be removed.
if (task.size == 0) {
i.remove();
}
return pendingBlock;
//...
{code}
The value of task.size was assigned in {{Balancer#matchSourceWithTargetToMove}}
{code}
long size = Math.min(source.availableSizeToMove(),
target.availableSizeToMove());
final Task task = new Task(target, size);
{code}
This value was depended on the source and target node, and this value will not
always can be reduced to 0 in choosing pending blocks. And then, it will still
move the data to the target node even if the size of bytes that needed to move
has been already reduced less than 0. And finally it will make the data
imbalance again in cluster, then it leads the next balancer.
We can opitimize for this as this title mentioned, I think this can speed the
balancer.
Can see the logs for failling case, or see the HDFS-10602.(Concentrating on the
change record for the scheduled size of target node. That's my added info for
debug, like this).
{code}
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -67, before: 33
{code}
was:
In HDFS-10602, we found a failing case that the balancer moves data always
between 2 DNs. And it made the balancer can't be finished. I debug the code for
this, I found there seems a bug in choosing pending blocks in
{{Dispatcher.Source.chooseNextMove}}.
The codes:
{code}
private PendingMove chooseNextMove() {
for (Iterator i = tasks.iterator(); i.hasNext();) {
final Task task = i.next();
final DDatanode target = task.target.getDDatanode();
final PendingMove pendingBlock = new PendingMove(this, task.target);
if (target.addPendingBlock(pendingBlock)) {
// target is not busy, so do a tentative block allocation
if (pendingBlock.chooseBlockAndProxy()) {
long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
incScheduledSize(-blockSize);
task.size -= blockSize;
// If the size of bytes that need to be moved was first reduced to
less than 0
// it should also be removed.
if (task.size == 0) {
i.remove();
}
return pendingBlock;
//...
{code}
The value of task.size was assigned in {{Balancer#matchSourceWithTargetToMove}}
{code}
long size = Math.min(source.availableSizeToMove(),
target.availableSizeToMove());
final Task task = new Task(target, size);
{code}
This value was depended on the source and target node, and this value will not
always can be reduced to 0 in choosing pending blocks. And then, it will still
move the data to the target node even if the size of bytes that needed to move
has been already reduced less than 0. And finally it will make the data
imbalance again in cluster.
We can opitimize for this as this title mentioned, I think this can speed the
balancer.
Can see the logs for failling case, or see the HDFS-10602.(Concentrating on the
change record for the scheduled size of target node. That's my added info for
debug, like this).
{code}
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -67, before: 33
{code}
> The target node should be removed in balancer when the size of bytes that
> need to move firstly reduced to less than 0
> -
>
> Key: HDFS-10716
> URL: https://issues.apache.org/jira/browse/HDFS-10716
> Project: Hadoop HDFS
>
[jira] [Updated] (HDFS-10716) The target node should be removed in balancer when the size of bytes that need to move firstly reduced to less than 0
[
https://issues.apache.org/jira/browse/HDFS-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10716:
-
Summary: The target node should be removed in balancer when the size of
bytes that need to move firstly reduced to less than 0 (was: The target node
should be removed in balancer when the scheduled size of bytes that firstly
reduced to less than 0)
> The target node should be removed in balancer when the size of bytes that
> need to move firstly reduced to less than 0
> -
>
> Key: HDFS-10716
> URL: https://issues.apache.org/jira/browse/HDFS-10716
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10716.001.patch, failing.log
>
>
> In HDFS-10602, we found a failing case that the balancer moves data always
> between 2 DNs. And it made the balancer can't be finished. I debug the code
> for this, I found there seems a bug in choosing pending blocks in
> {{Dispatcher.Source.chooseNextMove}}.
> The codes:
> {code}
> private PendingMove chooseNextMove() {
> for (Iterator i = tasks.iterator(); i.hasNext();) {
> final Task task = i.next();
> final DDatanode target = task.target.getDDatanode();
> final PendingMove pendingBlock = new PendingMove(this, task.target);
> if (target.addPendingBlock(pendingBlock)) {
> // target is not busy, so do a tentative block allocation
> if (pendingBlock.chooseBlockAndProxy()) {
> long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
> incScheduledSize(-blockSize);
> task.size -= blockSize;
> // If the size of bytes that need to be moved was first reduced
> to less than 0
> // it should also be removed.
> if (task.size == 0) {
> i.remove();
> }
> return pendingBlock;
> //...
> {code}
> The value of task.size was assigned in
> {{Balancer#matchSourceWithTargetToMove}}
> {code}
> long size = Math.min(source.availableSizeToMove(),
> target.availableSizeToMove());
> final Task task = new Task(target, size);
> {code}
> This value was depended on the source and target node, and this value will
> not always can be reduced to 0 in choosing pending blocks. And then, it will
> still move the data to the target node even if the size of bytes that needed
> to move has been already reduced less than 0. And finally it will make the
> data imbalance again in cluster.
> We can opitimize for this as this title mentioned, I think this can speed the
> balancer.
> Can see the logs for failling case, or see the HDFS-10602.(Concentrating on
> the change record for the scheduled size of target node. That's my added info
> for debug, like this).
> {code}
> 2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
> (Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
> move, after: -67, before: 33
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10602) TestBalancer runs timeout intermittently
[
https://issues.apache.org/jira/browse/HDFS-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403941#comment-15403941
]
Yiqun Lin commented on HDFS-10602:
--
I have filed a new jira HDFS-10716 for tracking the case that mentioned in my
latest comment.
> TestBalancer runs timeout intermittently
>
>
> Key: HDFS-10602
> URL: https://issues.apache.org/jira/browse/HDFS-10602
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10602.001.patch, HDFS-10602.002.patch, fail.log,
> pass.log
>
>
> As the jira HDFS-10336 has mentioned, the unit test
> {{TestBalancer#testBalancerWithKeytabs}} will runs too slowly sometimes and
> that leads the timeout. The test {{TestBalancer#testUnknownDatanodeSimple}}
> will also has this problem. These two tests both use the method
> {{testUnknownDatanode}}. We can do some optimization for this method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10716) The target node should be removed in balancer when the scheduled size of bytes that firstly reduced to less than 0
[
https://issues.apache.org/jira/browse/HDFS-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10716:
-
Description:
In HDFS-10602, we found a failing case that the balancer moves data always
between 2 DNs. And it made the balancer can't be finished. I debug the code for
this, I found there seems a bug in choosing pending blocks in
{{Dispatcher.Source.chooseNextMove}}.
The codes:
{code}
private PendingMove chooseNextMove() {
for (Iterator i = tasks.iterator(); i.hasNext();) {
final Task task = i.next();
final DDatanode target = task.target.getDDatanode();
final PendingMove pendingBlock = new PendingMove(this, task.target);
if (target.addPendingBlock(pendingBlock)) {
// target is not busy, so do a tentative block allocation
if (pendingBlock.chooseBlockAndProxy()) {
long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
incScheduledSize(-blockSize);
task.size -= blockSize;
// If the size of bytes that need to be moved was first reduced to
less than 0
// it should also be removed.
if (task.size == 0) {
i.remove();
}
return pendingBlock;
//...
{code}
The value of task.size was assigned in {{Balancer#matchSourceWithTargetToMove}}
{code}
long size = Math.min(source.availableSizeToMove(),
target.availableSizeToMove());
final Task task = new Task(target, size);
{code}
This value was depended on the source and target node, and this value will not
always can be reduced to 0 in choosing pending blocks. And then, it will still
move the data to the target node even if the size of bytes that needed to move
has been already reduced less than 0. And finally it will make the data
imbalance again in cluster.
We can opitimize for this as this title mentioned, I think this can speed the
balancer.
Can see the logs for failling case, or see the HDFS-10602.(Concentrating on the
change record for the scheduled size of target node. That's my added info for
debug, like this).
{code}
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -67, before: 33
{code}
was:
In HDFS-10602, we found a failing case that the balancer moves data always
between 2 DNs. And it made the balancer can't be finished. I debug the code for
this, I found there seems a bug in choosing pending blocks in
{{Dispatcher.Source.chooseNextMove}}.
The codes:
{code}
private PendingMove chooseNextMove() {
for (Iterator i = tasks.iterator(); i.hasNext();) {
final Task task = i.next();
final DDatanode target = task.target.getDDatanode();
final PendingMove pendingBlock = new PendingMove(this, task.target);
if (target.addPendingBlock(pendingBlock)) {
// target is not busy, so do a tentative block allocation
if (pendingBlock.chooseBlockAndProxy()) {
long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
incScheduledSize(-blockSize);
task.size -= blockSize;
// If the size of bytes that need to be moved was first reduced to
less than 0
// it should also be removed.
if (task.size == 0) {
i.remove();
}
return pendingBlock;
//...
{code}
The value of task.size was assigned in {{Balancer#matchSourceWithTargetToMove}}
{code}
long size = Math.min(source.availableSizeToMove(),
target.availableSizeToMove());
final Task task = new Task(target, size);
{code}
This value was depended on the source and target node, and this value will not
always can be reduced to 0 in choosing pending blocks. And then, it will still
move the data to the target node even if the size of bytes that needed to move
has been already reduced less than 0. And finally it will make the data
imbalance again in cluster.
We can opitimize for this as this title mentioned, I think this can speed the
balancer.
Can see the failling logs for this.(Concentrating on the change record for the
scheduled size of target node. That's my added info for debug, like this).
{code}
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -67, before: 33
{code}
> The target node should be removed in balancer when the scheduled size of
> bytes that firstly reduced to less than 0
> --
>
> Key: HDFS-10716
> URL: https://issues.apache.org/jira/browse/HDFS-10716
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>
[jira] [Updated] (HDFS-10716) The target node should be removed in balancer when the scheduled size of bytes that firstly reduced to less than 0
[
https://issues.apache.org/jira/browse/HDFS-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10716:
-
Status: Patch Available (was: Open)
Attach a initial patch, thanks for reviewing.
> The target node should be removed in balancer when the scheduled size of
> bytes that firstly reduced to less than 0
> --
>
> Key: HDFS-10716
> URL: https://issues.apache.org/jira/browse/HDFS-10716
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10716.001.patch, failing.log
>
>
> In HDFS-10602, we found a failing case that the balancer moves data always
> between 2 DNs. And it made the balancer can't be finished. I debug the code
> for this, I found there seems a bug in choosing pending blocks in
> {{Dispatcher.Source.chooseNextMove}}.
> The codes:
> {code}
> private PendingMove chooseNextMove() {
> for (Iterator i = tasks.iterator(); i.hasNext();) {
> final Task task = i.next();
> final DDatanode target = task.target.getDDatanode();
> final PendingMove pendingBlock = new PendingMove(this, task.target);
> if (target.addPendingBlock(pendingBlock)) {
> // target is not busy, so do a tentative block allocation
> if (pendingBlock.chooseBlockAndProxy()) {
> long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
> incScheduledSize(-blockSize);
> task.size -= blockSize;
> // If the size of bytes that need to be moved was first reduced
> to less than 0
> // it should also be removed.
> if (task.size == 0) {
> i.remove();
> }
> return pendingBlock;
> //...
> {code}
> The value of task.size was assigned in
> {{Balancer#matchSourceWithTargetToMove}}
> {code}
> long size = Math.min(source.availableSizeToMove(),
> target.availableSizeToMove());
> final Task task = new Task(target, size);
> {code}
> This value was depended on the source and target node, and this value will
> not always can be reduced to 0 in choosing pending blocks. And then, it will
> still move the data to the target node even if the size of bytes that needed
> to move has been already reduced less than 0. And finally it will make the
> data imbalance again in cluster.
> We can opitimize for this as this title mentioned, I think this can speed the
> balancer.
> Can see the failling logs for this.(Concentrating on the change record for
> the scheduled size of target node. That's my added info for debug, like this).
> {code}
> 2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
> (Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
> move, after: -67, before: 33
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10716) The target node should be removed in balancer when the scheduled size of bytes that firstly reduced to less than 0
[
https://issues.apache.org/jira/browse/HDFS-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10716:
-
Attachment: HDFS-10716.001.patch
> The target node should be removed in balancer when the scheduled size of
> bytes that firstly reduced to less than 0
> --
>
> Key: HDFS-10716
> URL: https://issues.apache.org/jira/browse/HDFS-10716
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10716.001.patch, failing.log
>
>
> In HDFS-10602, we found a failing case that the balancer moves data always
> between 2 DNs. And it made the balancer can't be finished. I debug the code
> for this, I found there seems a bug in choosing pending blocks in
> {{Dispatcher.Source.chooseNextMove}}.
> The codes:
> {code}
> private PendingMove chooseNextMove() {
> for (Iterator i = tasks.iterator(); i.hasNext();) {
> final Task task = i.next();
> final DDatanode target = task.target.getDDatanode();
> final PendingMove pendingBlock = new PendingMove(this, task.target);
> if (target.addPendingBlock(pendingBlock)) {
> // target is not busy, so do a tentative block allocation
> if (pendingBlock.chooseBlockAndProxy()) {
> long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
> incScheduledSize(-blockSize);
> task.size -= blockSize;
> // If the size of bytes that need to be moved was first reduced
> to less than 0
> // it should also be removed.
> if (task.size == 0) {
> i.remove();
> }
> return pendingBlock;
> //...
> {code}
> The value of task.size was assigned in
> {{Balancer#matchSourceWithTargetToMove}}
> {code}
> long size = Math.min(source.availableSizeToMove(),
> target.availableSizeToMove());
> final Task task = new Task(target, size);
> {code}
> This value was depended on the source and target node, and this value will
> not always can be reduced to 0 in choosing pending blocks. And then, it will
> still move the data to the target node even if the size of bytes that needed
> to move has been already reduced less than 0. And finally it will make the
> data imbalance again in cluster.
> We can opitimize for this as this title mentioned, I think this can speed the
> balancer.
> Can see the failling logs for this.(Concentrating on the change record for
> the scheduled size of target node. That's my added info for debug, like this).
> {code}
> 2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
> (Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
> move, after: -67, before: 33
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10716) The target node should be removed in balancer when the scheduled size of bytes that firstly reduced to less than 0
[
https://issues.apache.org/jira/browse/HDFS-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10716:
-
Description:
In HDFS-10602, we found a failing case that the balancer moves data always
between 2 DNs. And it made the balancer can't be finished. I debug the code for
this, I found there seems a bug in choosing pending blocks in
{{Dispatcher.Source.chooseNextMove}}.
The codes:
{code}
private PendingMove chooseNextMove() {
for (Iterator i = tasks.iterator(); i.hasNext();) {
final Task task = i.next();
final DDatanode target = task.target.getDDatanode();
final PendingMove pendingBlock = new PendingMove(this, task.target);
if (target.addPendingBlock(pendingBlock)) {
// target is not busy, so do a tentative block allocation
if (pendingBlock.chooseBlockAndProxy()) {
long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
incScheduledSize(-blockSize);
task.size -= blockSize;
// If the size of bytes that need to be moved was first reduced to
less than 0
// it should also be removed.
if (task.size == 0) {
i.remove();
}
return pendingBlock;
//...
{code}
The value of task.size was assigned in {{Balancer#matchSourceWithTargetToMove}}
{code}
long size = Math.min(source.availableSizeToMove(),
target.availableSizeToMove());
final Task task = new Task(target, size);
{code}
This value was depended on the source and target node, and this value will not
always can be reduced to 0 in choosing pending blocks. And then, it will still
move the data to the target node even if the size of bytes that needed to move
has been already reduced less than 0. And finally it will make the data
imbalance again in cluster.
We can opitimize for this as this title mentioned, I think this can speed the
balancer.
Can see the failling logs for this.(Concentrating on the change record for the
scheduled size of target node. That's my added info for debug).
{code}
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -67, before: 33
{code}
was:
In HDFS-10602, we found a failing case that the balancer moves data always
between 2 DNs. And it made the balancer can't be finished. I debug the code for
this, I found there seems a bug in choosing pending blocks in
{{Dispatcher.Source.chooseNextMove}}.
The codes:
{code}
private PendingMove chooseNextMove() {
for (Iterator i = tasks.iterator(); i.hasNext();) {
final Task task = i.next();
final DDatanode target = task.target.getDDatanode();
final PendingMove pendingBlock = new PendingMove(this, task.target);
if (target.addPendingBlock(pendingBlock)) {
// target is not busy, so do a tentative block allocation
if (pendingBlock.chooseBlockAndProxy()) {
long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
incScheduledSize(-blockSize);
task.size -= blockSize;
// If the size of bytes that need to be moved was first reduced to
less than 0
// it should also be removed.
if (task.size == 0) {
i.remove();
}
return pendingBlock;
//...
{code}
The value of task.size was assigned in {{Balancer#matchSourceWithTargetToMove}}
{code}
long size = Math.min(source.availableSizeToMove(),
target.availableSizeToMove());
final Task task = new Task(target, size);
{code}
This value was depended on the source and target node, and this value will not
always can be reduced to 0 in choosing pending blocks. And then, it will still
move the data to the target node even if the size of bytes that needed to move
has been already reduced less than 0. And finally it will make the data
imbalance again in cluster.
We can opitimize for this as this title mentioned, I think this can speed the
balancer.
Can see the failling logs for this.
> The target node should be removed in balancer when the scheduled size of
> bytes that firstly reduced to less than 0
> --
>
> Key: HDFS-10716
> URL: https://issues.apache.org/jira/browse/HDFS-10716
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: failing.log
>
>
> In HDFS-10602, we found a failing case that the balancer moves data always
> between 2 DNs. And it made the balancer can't be finished. I debug the code
> for this, I found there seems a bug in choosing pending blocks in
>
[jira] [Updated] (HDFS-10716) The target node should be removed in balancer when the scheduled size of bytes that firstly reduced to less than 0
[
https://issues.apache.org/jira/browse/HDFS-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10716:
-
Description:
In HDFS-10602, we found a failing case that the balancer moves data always
between 2 DNs. And it made the balancer can't be finished. I debug the code for
this, I found there seems a bug in choosing pending blocks in
{{Dispatcher.Source.chooseNextMove}}.
The codes:
{code}
private PendingMove chooseNextMove() {
for (Iterator i = tasks.iterator(); i.hasNext();) {
final Task task = i.next();
final DDatanode target = task.target.getDDatanode();
final PendingMove pendingBlock = new PendingMove(this, task.target);
if (target.addPendingBlock(pendingBlock)) {
// target is not busy, so do a tentative block allocation
if (pendingBlock.chooseBlockAndProxy()) {
long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
incScheduledSize(-blockSize);
task.size -= blockSize;
// If the size of bytes that need to be moved was first reduced to
less than 0
// it should also be removed.
if (task.size == 0) {
i.remove();
}
return pendingBlock;
//...
{code}
The value of task.size was assigned in {{Balancer#matchSourceWithTargetToMove}}
{code}
long size = Math.min(source.availableSizeToMove(),
target.availableSizeToMove());
final Task task = new Task(target, size);
{code}
This value was depended on the source and target node, and this value will not
always can be reduced to 0 in choosing pending blocks. And then, it will still
move the data to the target node even if the size of bytes that needed to move
has been already reduced less than 0. And finally it will make the data
imbalance again in cluster.
We can opitimize for this as this title mentioned, I think this can speed the
balancer.
Can see the failling logs for this.(Concentrating on the change record for the
scheduled size of target node. That's my added info for debug, like this).
{code}
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -67, before: 33
{code}
was:
In HDFS-10602, we found a failing case that the balancer moves data always
between 2 DNs. And it made the balancer can't be finished. I debug the code for
this, I found there seems a bug in choosing pending blocks in
{{Dispatcher.Source.chooseNextMove}}.
The codes:
{code}
private PendingMove chooseNextMove() {
for (Iterator i = tasks.iterator(); i.hasNext();) {
final Task task = i.next();
final DDatanode target = task.target.getDDatanode();
final PendingMove pendingBlock = new PendingMove(this, task.target);
if (target.addPendingBlock(pendingBlock)) {
// target is not busy, so do a tentative block allocation
if (pendingBlock.chooseBlockAndProxy()) {
long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
incScheduledSize(-blockSize);
task.size -= blockSize;
// If the size of bytes that need to be moved was first reduced to
less than 0
// it should also be removed.
if (task.size == 0) {
i.remove();
}
return pendingBlock;
//...
{code}
The value of task.size was assigned in {{Balancer#matchSourceWithTargetToMove}}
{code}
long size = Math.min(source.availableSizeToMove(),
target.availableSizeToMove());
final Task task = new Task(target, size);
{code}
This value was depended on the source and target node, and this value will not
always can be reduced to 0 in choosing pending blocks. And then, it will still
move the data to the target node even if the size of bytes that needed to move
has been already reduced less than 0. And finally it will make the data
imbalance again in cluster.
We can opitimize for this as this title mentioned, I think this can speed the
balancer.
Can see the failling logs for this.(Concentrating on the change record for the
scheduled size of target node. That's my added info for debug).
{code}
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -67, before: 33
{code}
> The target node should be removed in balancer when the scheduled size of
> bytes that firstly reduced to less than 0
> --
>
> Key: HDFS-10716
> URL: https://issues.apache.org/jira/browse/HDFS-10716
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Yiqun Lin
>
[jira] [Updated] (HDFS-10716) The target node should be removed in balancer when the scheduled size of bytes that firstly reduced to less than 0
[
https://issues.apache.org/jira/browse/HDFS-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10716:
-
Attachment: failing.log
> The target node should be removed in balancer when the scheduled size of
> bytes that firstly reduced to less than 0
> --
>
> Key: HDFS-10716
> URL: https://issues.apache.org/jira/browse/HDFS-10716
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: failing.log
>
>
> In HDFS-10602, we found a failing case that the balancer moves data always
> between 2 DNs. And it made the balancer can't be finished. I debug the code
> for this, I found there seems a bug in choosing pending blocks in
> {{Dispatcher.Source.chooseNextMove}}.
> The codes:
> {code}
> private PendingMove chooseNextMove() {
> for (Iterator i = tasks.iterator(); i.hasNext();) {
> final Task task = i.next();
> final DDatanode target = task.target.getDDatanode();
> final PendingMove pendingBlock = new PendingMove(this, task.target);
> if (target.addPendingBlock(pendingBlock)) {
> // target is not busy, so do a tentative block allocation
> if (pendingBlock.chooseBlockAndProxy()) {
> long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
> incScheduledSize(-blockSize);
> task.size -= blockSize;
> // If the size of bytes that need to be moved was first reduced
> to less than 0
> // it should also be removed.
> if (task.size == 0) {
> i.remove();
> }
> return pendingBlock;
> //...
> {code}
> The value of task.size was assigned in
> {{Balancer#matchSourceWithTargetToMove}}
> {code}
> long size = Math.min(source.availableSizeToMove(),
> target.availableSizeToMove());
> final Task task = new Task(target, size);
> {code}
> This value was depended on the source and target node, and this value will
> not always can be reduced to 0 in choosing pending blocks. And then, it will
> still move the data to the target node even if the size of bytes that needed
> to move has been already reduced less than 0. And finally it will make the
> data imbalance again in cluster.
> We can opitimize for this as this title mentioned, I think this can speed the
> balancer.
> Can see the failling logs for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10716) The target node should be removed in balancer when the scheduled size of bytes that firstly reduced to less than 0
Yiqun Lin created HDFS-10716:
Summary: The target node should be removed in balancer when the
scheduled size of bytes that firstly reduced to less than 0
Key: HDFS-10716
URL: https://issues.apache.org/jira/browse/HDFS-10716
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer & mover
Reporter: Yiqun Lin
Assignee: Yiqun Lin
In HDFS-10602, we found a failing case that the balancer moves data always
between 2 DNs. And it made the balancer can't be finished. I debug the code for
this, I found there seems a bug in choosing pending blocks in
{{Dispatcher.Source.chooseNextMove}}.
The codes:
{code}
private PendingMove chooseNextMove() {
for (Iterator i = tasks.iterator(); i.hasNext();) {
final Task task = i.next();
final DDatanode target = task.target.getDDatanode();
final PendingMove pendingBlock = new PendingMove(this, task.target);
if (target.addPendingBlock(pendingBlock)) {
// target is not busy, so do a tentative block allocation
if (pendingBlock.chooseBlockAndProxy()) {
long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
incScheduledSize(-blockSize);
task.size -= blockSize;
// If the size of bytes that need to be moved was first reduced to
less than 0
// it should also be removed.
if (task.size == 0) {
i.remove();
}
return pendingBlock;
//...
{code}
The value of task.size was assigned in {{Balancer#matchSourceWithTargetToMove}}
{code}
long size = Math.min(source.availableSizeToMove(),
target.availableSizeToMove());
final Task task = new Task(target, size);
{code}
This value was depended on the source and target node, and this value will not
always can be reduced to 0 in choosing pending blocks. And then, it will still
move the data to the target node even if the size of bytes that needed to move
has been already reduced less than 0. And finally it will make the data
imbalance again in cluster.
We can opitimize for this as this title mentioned, I think this can speed the
balancer.
Can see the failling logs for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10706) Add tool generating FSImage from external store
[
https://issues.apache.org/jira/browse/HDFS-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403784#comment-15403784
]
Hadoop QA commented on HDFS-10706:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 7m
19s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 24s{color} | {color:orange} root: The patch generated 45 new + 6 unchanged -
0 fixed = 51 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 72m
22s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-fs2img in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m
5s{color} | {color:green} hadoop-tools in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821542/HDFS-10706.002.patch |
| JIRA Issue | HDFS-10706 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 2c91f0a241fd 3.13.0-36-lowlatency
[jira] [Commented] (HDFS-8901) Use ByteBuffer in striping positional read
[
https://issues.apache.org/jira/browse/HDFS-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403695#comment-15403695
]
Hadoop QA commented on HDFS-8901:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} root: The patch generated 0 new + 302 unchanged - 9
fixed = 302 total (was 311) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 37s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
56s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestClusterTopology |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821527/HDFS-8901.v14.patch |
| JIRA Issue | HDFS-8901 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6750986d13a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6890d5b |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16286/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit |
[jira] [Commented] (HDFS-10683) Make class Token$PrivateToken private
[
https://issues.apache.org/jira/browse/HDFS-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403650#comment-15403650
]
Hadoop QA commented on HDFS-10683:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 2m
25s{color} | {color:red} hadoop-common in trunk failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} root: The patch generated 0 new + 175 unchanged - 2
fixed = 175 total (was 177) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
53s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}117m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFileCorruption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821520/HDFS-10683.001.patch |
| JIRA Issue | HDFS-10683 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c07c6e69cdec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6890d5b |
| Default Java | 1.8.0_101 |
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/16285/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt
|
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16285/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
[jira] [Commented] (HDFS-10645) Make block report size as a metric and add this metric to datanode web ui
[ https://issues.apache.org/jira/browse/HDFS-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403641#comment-15403641 ] Akira Ajisaka commented on HDFS-10645: -- Mostly looks good to me. Hi [~yuanbo], would you rebase the patch? > Make block report size as a metric and add this metric to datanode web ui > - > > Key: HDFS-10645 > URL: https://issues.apache.org/jira/browse/HDFS-10645 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ui >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Attachments: HDFS-10645.001.patch, HDFS-10645.002.patch, > HDFS-10645.003.patch, HDFS-10645.004.patch, HDFS-10645.005.patch, > HDFS-10645.006.patch, Selection_047.png, Selection_048.png > > > Record block report size as a metric and show it on datanode UI. It's > important for administrators to know the bottleneck of block report, and the > metric is also a good tuning metric. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) A comprehensive and flexible storage policy engine
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403625#comment-15403625 ] Yuanbo Liu commented on HDFS-7343: -- Agree with [~drankye]'s opinion about HDFS-10285. Mover tool has limit in some scenarios, HDFS-10285 provide a way to make rules take effect. This jira is about how to make rules, how to decide a storage policy for a file/directory. To answer those questions, it needs a tremendous amount of work to do, I'm looking forward to your design doc, and hope to get involved in this project. > A comprehensive and flexible storage policy engine > -- > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Kai Zheng >Assignee: Kai Zheng > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-7343) A comprehensive and flexible storage policy engine
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403625#comment-15403625 ] Yuanbo Liu edited comment on HDFS-7343 at 8/2/16 8:37 AM: -- Agree with [~drankye]'s opinion about HDFS-10285. Mover tool has limit in some scenarios, HDFS-10285 provides a way to make rules take effect. This jira is about how to make rules, how to decide a storage policy for a file/directory. To answer those questions, it needs a tremendous amount of work to do, I'm looking forward to your design doc, and hope to get involved in this project. was (Author: yuanbo): Agree with [~drankye]'s opinion about HDFS-10285. Mover tool has limit in some scenarios, HDFS-10285 provide a way to make rules take effect. This jira is about how to make rules, how to decide a storage policy for a file/directory. To answer those questions, it needs a tremendous amount of work to do, I'm looking forward to your design doc, and hope to get involved in this project. > A comprehensive and flexible storage policy engine > -- > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Kai Zheng >Assignee: Kai Zheng > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10706) Add tool generating FSImage from external store
[ https://issues.apache.org/jira/browse/HDFS-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HDFS-10706: - Attachment: HDFS-10706.002.patch > Add tool generating FSImage from external store > --- > > Key: HDFS-10706 > URL: https://issues.apache.org/jira/browse/HDFS-10706 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, tools >Reporter: Chris Douglas > Attachments: HDFS-10706.001.patch, HDFS-10706.002.patch > > > To experiment with provided storage, this provides a tool to map an external > namespace to an FSImage/NN storage. By loading it in a NN, one can access the > remote FS using HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10707) Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
[ https://issues.apache.org/jira/browse/HDFS-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403596#comment-15403596 ] Hudson commented on HDFS-10707: --- SUCCESS: Integrated in Hadoop-trunk-Commit #10191 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10191/]) HDFS-10707. Replace org.apache.commons.io.Charsets with (aajisaka: rev a5fb298e56220a35d61b8d2bda716d8fb8ef8bb7) * hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirMkdirOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/ParameterParser.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/JSONMapProvider.java * hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/JSONProvider.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java * hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSUtils.java > Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets > - > > Key: HDFS-10707 > URL: https://issues.apache.org/jira/browse/HDFS-10707 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10707.2.patch, HDFS-10707.3.patch, HDFS-10707.patch > > > org.apache.commons.io.Charsets is deprecated in favor of > java.nio.charset.StandardCharsets -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10583) Add links to component's configuration UI page under Utilities dropdown
[
https://issues.apache.org/jira/browse/HDFS-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403595#comment-15403595
]
Hadoop QA commented on HDFS-10583:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821529/HDFS-10583.003.patch |
| JIRA Issue | HDFS-10583 |
| Optional Tests | asflicense |
| uname | Linux 0997b5fe0590 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a5fb298 |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16287/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add links to component's configuration UI page under Utilities dropdown
> ---
>
> Key: HDFS-10583
> URL: https://issues.apache.org/jira/browse/HDFS-10583
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, ui
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-10583.001.patch, HDFS-10583.002.patch,
> HDFS-10583.003.patch, conf_link_on_DN.jpg, conf_link_on_NN.jpg
>
>
> When admin wants to explore some configuration properties, such as namenode
> and datanode, it will be helpful to provide an UI page to read them. This is
> extremely useful when nodes are having different configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[
https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403588#comment-15403588
]
Yuanbo Liu commented on HDFS-10285:
---
[~umamaheswararao] Great proposal, thanks for your work.
I have two question about your design:
1.
{quote}
When user calls satisfyStoragePolicy(src) API
{quote}
Is this api only available for java program, or when user uses this command
{code}
hdfs storagepolicies -setStoragePolicy -path -policy
{code}
this api is invoked by default ?
2.
what if inodes exsit in toBeSatisfiedStoragePolicyList meanwhile "mover tool"
takes effect on the directory which contains those inodes?
> Storage Policy Satisfier in Namenode
>
>
> Key: HDFS-10285
> URL: https://issues.apache.org/jira/browse/HDFS-10285
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Affects Versions: 2.7.2
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: Storage-Policy-Satisfier-in-HDFS-May10.pdf
>
>
> Heterogeneous storage in HDFS introduced the concept of storage policy. These
> policies can be set on directory/file to specify the user preference, where
> to store the physical block. When user set the storage policy before writing
> data, then the blocks could take advantage of storage policy preferences and
> stores physical block accordingly.
> If user set the storage policy after writing and completing the file, then
> the blocks would have been written with default storage policy (nothing but
> DISK). User has to run the ‘Mover tool’ explicitly by specifying all such
> file names as a list. In some distributed system scenarios (ex: HBase) it
> would be difficult to collect all the files and run the tool as different
> nodes can write files separately and file can have different paths.
> Another scenarios is, when user rename the files from one effected storage
> policy file (inherited policy from parent directory) to another storage
> policy effected directory, it will not copy inherited storage policy from
> source. So it will take effect from destination file/dir parent storage
> policy. This rename operation is just a metadata change in Namenode. The
> physical blocks still remain with source storage policy.
> So, Tracking all such business logic based file names could be difficult for
> admins from distributed nodes(ex: region servers) and running the Mover tool.
> Here the proposal is to provide an API from Namenode itself for trigger the
> storage policy satisfaction. A Daemon thread inside Namenode should track
> such calls and process to DN as movement commands.
> Will post the detailed design thoughts document soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10285) Storage Policy Satisfier in Namenode
[
https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403588#comment-15403588
]
Yuanbo Liu edited comment on HDFS-10285 at 8/2/16 8:17 AM:
---
[~umamaheswararao] Great proposal, thanks for your work.
I have two questions about your design:
1.
{quote}
When user calls satisfyStoragePolicy(src) API
{quote}
Is this api only available for java program, or when user uses this command
{code}
hdfs storagepolicies -setStoragePolicy -path -policy
{code}
this api is invoked by default ?
2.
what if inodes exsit in toBeSatisfiedStoragePolicyList meanwhile "mover tool"
takes effect on the directory which contains those inodes?
was (Author: yuanbo):
[~umamaheswararao] Great proposal, thanks for your work.
I have two question about your design:
1.
{quote}
When user calls satisfyStoragePolicy(src) API
{quote}
Is this api only available for java program, or when user uses this command
{code}
hdfs storagepolicies -setStoragePolicy -path -policy
{code}
this api is invoked by default ?
2.
what if inodes exsit in toBeSatisfiedStoragePolicyList meanwhile "mover tool"
takes effect on the directory which contains those inodes?
> Storage Policy Satisfier in Namenode
>
>
> Key: HDFS-10285
> URL: https://issues.apache.org/jira/browse/HDFS-10285
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Affects Versions: 2.7.2
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: Storage-Policy-Satisfier-in-HDFS-May10.pdf
>
>
> Heterogeneous storage in HDFS introduced the concept of storage policy. These
> policies can be set on directory/file to specify the user preference, where
> to store the physical block. When user set the storage policy before writing
> data, then the blocks could take advantage of storage policy preferences and
> stores physical block accordingly.
> If user set the storage policy after writing and completing the file, then
> the blocks would have been written with default storage policy (nothing but
> DISK). User has to run the ‘Mover tool’ explicitly by specifying all such
> file names as a list. In some distributed system scenarios (ex: HBase) it
> would be difficult to collect all the files and run the tool as different
> nodes can write files separately and file can have different paths.
> Another scenarios is, when user rename the files from one effected storage
> policy file (inherited policy from parent directory) to another storage
> policy effected directory, it will not copy inherited storage policy from
> source. So it will take effect from destination file/dir parent storage
> policy. This rename operation is just a metadata change in Namenode. The
> physical blocks still remain with source storage policy.
> So, Tracking all such business logic based file names could be difficult for
> admins from distributed nodes(ex: region servers) and running the Mover tool.
> Here the proposal is to provide an API from Namenode itself for trigger the
> storage policy satisfaction. A Daemon thread inside Namenode should track
> such calls and process to DN as movement commands.
> Will post the detailed design thoughts document soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10583) Add links to component's configuration UI page under Utilities dropdown
[ https://issues.apache.org/jira/browse/HDFS-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403587#comment-15403587 ] Weiwei Yang commented on HDFS-10583: Thanks [~vinayrpet], just uploaded v3 to address that. I have verified on my local cluster, both NN, DN and file browser UI shows the new link "Configuration". > Add links to component's configuration UI page under Utilities dropdown > --- > > Key: HDFS-10583 > URL: https://issues.apache.org/jira/browse/HDFS-10583 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, ui >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Minor > Attachments: HDFS-10583.001.patch, HDFS-10583.002.patch, > HDFS-10583.003.patch, conf_link_on_DN.jpg, conf_link_on_NN.jpg > > > When admin wants to explore some configuration properties, such as namenode > and datanode, it will be helpful to provide an UI page to read them. This is > extremely useful when nodes are having different configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10715) NPE when applying AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Guangbin updated HDFS-10715: Attachment: HDFS-10715.001.patch > NPE when applying AvailableSpaceBlockPlacementPolicy > > > Key: HDFS-10715 > URL: https://issues.apache.org/jira/browse/HDFS-10715 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 > Environment: cdh5.8.0 >Reporter: Zhu Guangbin > Attachments: HDFS-10715.001.patch > > > As HDFS-8131 introduced an AvailableSpaceBlockPlacementPolicy, but In some > cases, it caused NPE. > Here are my namenode daemon logs : > 2016-08-02 13:05:03,271 WARN org.apache.hadoop.ipc.Server: IPC Server handler > 13 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.132.89.79:14001 Call#56 Retry#0 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.compareDataNode(AvailableSpaceBlockPlacementPolicy.java:95) > at > org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.chooseDataNode(AvailableSpaceBlockPlacementPolicy.java:80) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:691) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:665) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:572) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:457) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:367) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:242) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:114) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:130) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1606) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3315) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:679) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:214) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:489) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) > I reviewed the source code, and found the bug in method chooseDataNode. > clusterMap.chooseRandom may return null, which cannot compare using equals > a.equals(b) method. > Though this exception can be caught, and then retry another call. I think > this bug should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10583) Add links to component's configuration UI page under Utilities dropdown
[ https://issues.apache.org/jira/browse/HDFS-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-10583: --- Attachment: HDFS-10583.003.patch > Add links to component's configuration UI page under Utilities dropdown > --- > > Key: HDFS-10583 > URL: https://issues.apache.org/jira/browse/HDFS-10583 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, ui >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Minor > Attachments: HDFS-10583.001.patch, HDFS-10583.002.patch, > HDFS-10583.003.patch, conf_link_on_DN.jpg, conf_link_on_NN.jpg > > > When admin wants to explore some configuration properties, such as namenode > and datanode, it will be helpful to provide an UI page to read them. This is > extremely useful when nodes are having different configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10715) NPE when applying AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Guangbin updated HDFS-10715: Summary: NPE when applying AvailableSpaceBlockPlacementPolicy (was: NPE ) > NPE when applying AvailableSpaceBlockPlacementPolicy > > > Key: HDFS-10715 > URL: https://issues.apache.org/jira/browse/HDFS-10715 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 > Environment: cdh5.8.0 >Reporter: Zhu Guangbin > > As HDFS-8131 introduced an AvailableSpaceBlockPlacementPolicy, but In some > cases, it caused NPE. > Here are my namenode daemon logs : > 2016-08-02 13:05:03,271 WARN org.apache.hadoop.ipc.Server: IPC Server handler > 13 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.132.89.79:14001 Call#56 Retry#0 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.compareDataNode(AvailableSpaceBlockPlacementPolicy.java:95) > at > org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.chooseDataNode(AvailableSpaceBlockPlacementPolicy.java:80) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:691) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:665) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:572) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:457) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:367) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:242) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:114) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:130) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1606) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3315) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:679) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:214) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:489) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) > I reviewed the source code, and found the bug in method chooseDataNode. > clusterMap.chooseRandom may return null, which cannot compare using equals > a.equals(b) method. > Though this exception can be caught, and then retry another call. I think > this bug should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10715) NPE
Zhu Guangbin created HDFS-10715: --- Summary: NPE Key: HDFS-10715 URL: https://issues.apache.org/jira/browse/HDFS-10715 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.8.0 Environment: cdh5.8.0 Reporter: Zhu Guangbin As HDFS-8131 introduced an AvailableSpaceBlockPlacementPolicy, but In some cases, it caused NPE. Here are my namenode daemon logs : 2016-08-02 13:05:03,271 WARN org.apache.hadoop.ipc.Server: IPC Server handler 13 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 10.132.89.79:14001 Call#56 Retry#0 java.lang.NullPointerException at org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.compareDataNode(AvailableSpaceBlockPlacementPolicy.java:95) at org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.chooseDataNode(AvailableSpaceBlockPlacementPolicy.java:80) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:691) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:665) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:572) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:457) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:367) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:242) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:114) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:130) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1606) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3315) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:679) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:214) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:489) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) I reviewed the source code, and found the bug in method chooseDataNode. clusterMap.chooseRandom may return null, which cannot compare using equals a.equals(b) method. Though this exception can be caught, and then retry another call. I think this bug should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10707) Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
[ https://issues.apache.org/jira/browse/HDFS-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-10707: - Fix Version/s: 3.0.0-alpha2 > Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets > - > > Key: HDFS-10707 > URL: https://issues.apache.org/jira/browse/HDFS-10707 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10707.2.patch, HDFS-10707.3.patch, HDFS-10707.patch > > > org.apache.commons.io.Charsets is deprecated in favor of > java.nio.charset.StandardCharsets -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10707) Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
[ https://issues.apache.org/jira/browse/HDFS-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403582#comment-15403582 ] Akira Ajisaka commented on HDFS-10707: -- Committed this to trunk. Hi [~vincentpoon], would you rebase the patch for branch-2 and branch-2.8? > Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets > - > > Key: HDFS-10707 > URL: https://issues.apache.org/jira/browse/HDFS-10707 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10707.2.patch, HDFS-10707.3.patch, HDFS-10707.patch > > > org.apache.commons.io.Charsets is deprecated in favor of > java.nio.charset.StandardCharsets -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10707) Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets
[ https://issues.apache.org/jira/browse/HDFS-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403577#comment-15403577 ] Akira Ajisaka commented on HDFS-10707: -- I ran all the failed tests locally and they passed. Committing this. > Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets > - > > Key: HDFS-10707 > URL: https://issues.apache.org/jira/browse/HDFS-10707 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Attachments: HDFS-10707.2.patch, HDFS-10707.3.patch, HDFS-10707.patch > > > org.apache.commons.io.Charsets is deprecated in favor of > java.nio.charset.StandardCharsets -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
