[jira] [Commented] (HDFS-8135) Remove the deprecated FSConstants class
[
https://issues.apache.org/jira/browse/HDFS-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551916#comment-14551916
]
zhangduo commented on HDFS-8135:
HBase uses DistributedFileSystem.setSafeMode to check whether an hdfs cluster
is in safe mode. This method needs {{HdfsConstants.SafeModeAction}}.
I found that there is a {{HdfsUtils.isHealthy}}, we could try to make use of
this method instead of depending on hdfs private classes.
Thanks.
Remove the deprecated FSConstants class
---
Key: HDFS-8135
URL: https://issues.apache.org/jira/browse/HDFS-8135
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Li Lu
Fix For: 2.8.0
Attachments: HDFS-8135-041315.patch
The {{FSConstants}} class has been marked as deprecated since 0.23. There is
no uses of this class in the current code base and it can be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8443) Document dfs.namenode.service.handler.count in hdfs-site.xml
Akira AJISAKA created HDFS-8443: --- Summary: Document dfs.namenode.service.handler.count in hdfs-site.xml Key: HDFS-8443 URL: https://issues.apache.org/jira/browse/HDFS-8443 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Reporter: Akira AJISAKA When dfs.namenode.servicerpc-address is configured, NameNode launches an extra RPC server to handle requests from non-client nodes. dfs.namenode.service.handler.count specifies the number of threads for the server but the parameter is not documented anywhere. I found a mail for asking about the parameter. http://mail-archives.apache.org/mod_mbox/hadoop-user/201505.mbox/%3CE0D5A619-BDEA-44D2-81EB-C32B8464133D%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8413) Directories are not listed recursively when fs.defaultFs is viewFs
[
https://issues.apache.org/jira/browse/HDFS-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajith S updated HDFS-8413:
--
Attachment: HDFS-8413.patch
Directories are not listed recursively when fs.defaultFs is viewFs
--
Key: HDFS-8413
URL: https://issues.apache.org/jira/browse/HDFS-8413
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Ajith S
Assignee: Ajith S
Labels: viewfs
Attachments: HDFS-8413.patch
Mount a cluster on client throught viewFs mount table
Example:
{quote}
property
namefs.defaultFS/name
valueviewfs:value
/property
property
namefs.viewfs.mounttable.default.link./nn1/name
valuehdfs://ns1//value !-- HA nameservice --
/property
property
namefs.viewfs.mounttable.default.link./user/name
valuehdfs://host-72:8020//value
/property
property
{quote}
Try to list the files recursively *(hdfs dfs -ls -R / or hadoop fs -ls -R /)*
only the parent folders are listed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8442) Remove ServerLifecycleListener from kms/server.xml.
nijel created HDFS-8442:
---
Summary: Remove ServerLifecycleListener from kms/server.xml.
Key: HDFS-8442
URL: https://issues.apache.org/jira/browse/HDFS-8442
Project: Hadoop HDFS
Issue Type: Bug
Reporter: nijel
Assignee: nijel
Remove ServerLifecycleListener from kms/server.xml.
From tomcat Tomcat 7.0.9 the support for ServerLifecycleListener is removed
ref : https://tomcat.apache.org/tomcat-7.0-doc/changelog.html
Remove ServerLifecycleListener. This was already removed from server.xml and
with the Lifecycle re-factoring is no longer required. (markt)
So if the build env is with tomcat later than this, kms startup is failing
{code}
SEVERE: Begin event threw exception
java.lang.ClassNotFoundException:
org.apache.catalina.mbeans.ServerLifecycleListener
{code}
can we remove this listener ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8413) Directories are not listed recursively when fs.defaultFs is viewFs
[
https://issues.apache.org/jira/browse/HDFS-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajith S updated HDFS-8413:
--
Status: Patch Available (was: Open)
*Resolution* : decide if the mount point is a directory or not by querying the
target filesystem
Submitting the patch for same. Please review
Directories are not listed recursively when fs.defaultFs is viewFs
--
Key: HDFS-8413
URL: https://issues.apache.org/jira/browse/HDFS-8413
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Ajith S
Assignee: Ajith S
Labels: viewfs
Attachments: HDFS-8413.patch
Mount a cluster on client throught viewFs mount table
Example:
{quote}
property
namefs.defaultFS/name
valueviewfs:value
/property
property
namefs.viewfs.mounttable.default.link./nn1/name
valuehdfs://ns1//value !-- HA nameservice --
/property
property
namefs.viewfs.mounttable.default.link./user/name
valuehdfs://host-72:8020//value
/property
property
{quote}
Try to list the files recursively *(hdfs dfs -ls -R / or hadoop fs -ls -R /)*
only the parent folders are listed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8413) Directories are not listed recursively when fs.defaultFs is viewFs
[
https://issues.apache.org/jira/browse/HDFS-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551940#comment-14551940
]
Hadoop QA commented on HDFS-8413:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 29s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 26s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 7s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 40s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 23m 6s | Tests failed in
hadoop-common. |
| | | 59m 58s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem |
| | hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem |
| | hadoop.security.token.delegation.web.TestWebDelegationToken |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734076/HDFS-8413.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ce53c8e |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11057/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11057/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11057/console |
This message was automatically generated.
Directories are not listed recursively when fs.defaultFs is viewFs
--
Key: HDFS-8413
URL: https://issues.apache.org/jira/browse/HDFS-8413
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Ajith S
Assignee: Ajith S
Labels: viewfs
Attachments: HDFS-8413.patch
Mount a cluster on client throught viewFs mount table
Example:
{quote}
property
namefs.defaultFS/name
valueviewfs:value
/property
property
namefs.viewfs.mounttable.default.link./nn1/name
valuehdfs://ns1//value !-- HA nameservice --
/property
property
namefs.viewfs.mounttable.default.link./user/name
valuehdfs://host-72:8020//value
/property
property
{quote}
Try to list the files recursively *(hdfs dfs -ls -R / or hadoop fs -ls -R /)*
only the parent folders are listed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8427) Remove dataBlockNum and parityBlockNum from BlockInfoStriped
[
https://issues.apache.org/jira/browse/HDFS-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Sasaki updated HDFS-8427:
-
Status: Patch Available (was: Open)
Remove dataBlockNum and parityBlockNum from BlockInfoStriped
Key: HDFS-8427
URL: https://issues.apache.org/jira/browse/HDFS-8427
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8427-HDFS-7285-00.patch,
HDFS-8427-HDFS-7285-01.patch
Remove unnecessary members such as {{dataBlockNum}} and {{parityBlockNum}}
from {{BlockInfoStriped}}. These are included in {{ECShema}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552064#comment-14552064
]
Hadoop QA commented on HDFS-8441:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 39s | Pre-patch HDFS-7285 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 28s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 36s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 14s | The patch appears to introduce 6
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 172m 54s | Tests failed in hadoop-hdfs. |
| | | 214m 15s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.DFSOutputStream.streamer; locked 89% of time
Unsynchronized access at DFSOutputStream.java:89% of time Unsynchronized
access at DFSOutputStream.java:[line 146] |
| | Possible null pointer dereference of arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:[line 194] |
| | Unread field:field be static? At ErasureCodingWorker.java:[line 254] |
| | Should
org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$StripedReader
be a _static_ inner class? At ErasureCodingWorker.java:inner class? At
ErasureCodingWorker.java:[lines 907-914] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:[line 108] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:[line 409] |
| Failed unit tests | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734073/HDFS-8441-HDFS-7285.001.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 4dd4aa5 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/11056/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11056/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11056/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11056/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11056/console |
This message was automatically generated.
Erasure Coding: make condition check earlier for setReplication
---
Key: HDFS-8441
URL: https://issues.apache.org/jira/browse/HDFS-8441
Project: Hadoop HDFS
Issue Type: Sub-task
[jira] [Updated] (HDFS-8427) Remove dataBlockNum and parityBlockNum from BlockInfoStriped
[
https://issues.apache.org/jira/browse/HDFS-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Sasaki updated HDFS-8427:
-
Attachment: HDFS-8427-HDFS-7285-01.patch
Remove dataBlockNum and parityBlockNum from BlockInfoStriped
Key: HDFS-8427
URL: https://issues.apache.org/jira/browse/HDFS-8427
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8427-HDFS-7285-00.patch,
HDFS-8427-HDFS-7285-01.patch
Remove unnecessary members such as {{dataBlockNum}} and {{parityBlockNum}}
from {{BlockInfoStriped}}. These are included in {{ECShema}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8382) Remove chunkSize parameter from initialize method of raw erasure coder
[
https://issues.apache.org/jira/browse/HDFS-8382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552079#comment-14552079
]
Vinayakumar B commented on HDFS-8382:
-
bq. Updated the patch also removing initialize method per the suggestion.
I see {{initialize()}} and {{chunkSize}} in {{AbstractErasureCoder}}
currently this is the only place where chunkSize was passed from ECSchema to
coder. So HDFS-8374 depends on this.
Remove chunkSize parameter from initialize method of raw erasure coder
--
Key: HDFS-8382
URL: https://issues.apache.org/jira/browse/HDFS-8382
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8382-HDFS-7285-v1.patch,
HDFS-8382-HDFS-7285-v2.patch, HDFS-8382-HDFS-7285-v3.patch
Per discussion in HDFS-8347, we need to support encoding/decoding variable
width units data instead of predefined fixed width like {{chunkSize}}. Have
this issue to remove chunkSize in the general raw erasure coder API. Specific
coder will support fixed chunkSize using hard-coded or specific schema
customizing way if necessary, like HitchHiker coder.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8382) Remove chunkSize parameter from initialize method of raw erasure coder
[
https://issues.apache.org/jira/browse/HDFS-8382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552170#comment-14552170
]
Kai Zheng commented on HDFS-8382:
-
Thanks Vinay for the comments. Yes I was going to get all of this done here but
looks like I missed some places. Good catch! Will address them in the following
patch. Thanks.
Remove chunkSize parameter from initialize method of raw erasure coder
--
Key: HDFS-8382
URL: https://issues.apache.org/jira/browse/HDFS-8382
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8382-HDFS-7285-v1.patch,
HDFS-8382-HDFS-7285-v2.patch, HDFS-8382-HDFS-7285-v3.patch
Per discussion in HDFS-8347, we need to support encoding/decoding variable
width units data instead of predefined fixed width like {{chunkSize}}. Have
this issue to remove chunkSize in the general raw erasure coder API. Specific
coder will support fixed chunkSize using hard-coded or specific schema
customizing way if necessary, like HitchHiker coder.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8444) Erasure Coding: fix cannot rename a zone dir
[
https://issues.apache.org/jira/browse/HDFS-8444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8444:
Status: Patch Available (was: Open)
Erasure Coding: fix cannot rename a zone dir
Key: HDFS-8444
URL: https://issues.apache.org/jira/browse/HDFS-8444
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8444-HDFS-7285.001.patch
We create a EC zone {{/my_ec_zone}}.
We want to rename it to {{/myZone}}.
But it failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8444) Erasure Coding: fix cannot rename a zone dir
[
https://issues.apache.org/jira/browse/HDFS-8444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8444:
Attachment: HDFS-8444-HDFS-7285.001.patch
Erasure Coding: fix cannot rename a zone dir
Key: HDFS-8444
URL: https://issues.apache.org/jira/browse/HDFS-8444
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8444-HDFS-7285.001.patch
We create a EC zone {{/my_ec_zone}}.
We want to rename it to {{/myZone}}.
But it failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8131) Implement a space balanced block placement policy
[ https://issues.apache.org/jira/browse/HDFS-8131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552191#comment-14552191 ] Hudson commented on HDFS-8131: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #202 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/202/]) HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java Implement a space balanced block placement policy - Key: HDFS-8131 URL: https://issues.apache.org/jira/browse/HDFS-8131 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Labels: BlockPlacementPolicy Fix For: 2.8.0 Attachments: HDFS-8131-v1.diff, HDFS-8131-v2.diff, HDFS-8131-v3.diff, HDFS-8131.004.patch, HDFS-8131.005.patch, HDFS-8131.006.patch, balanced.png The default block placement policy will choose datanodes for new blocks randomly, which will result in unbalanced space used percent among datanodes after an cluster expansion. The old datanodes always are in high used percent of space and new added ones are in low percent. Through we can used the external balance tool to balance the space used rate, it will cost extra network IO and it's not easy to control the balance speed. An easy solution is to implement an balanced block placement policy which will choose low used percent datanodes for new blocks with a little high possibility. In a not long term, the used percent of datanodes will trend to be balanced. Suggestions and discussions are welcomed. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8404) Pending block replication can get stuck using older genstamp
[ https://issues.apache.org/jira/browse/HDFS-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552189#comment-14552189 ] Hudson commented on HDFS-8404: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #202 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/202/]) HDFS-8404. Pending block replication can get stuck using older genstamp. Contributed by Nathan Roberts. (kihwal: rev 8860e352c394372e4eb3ebdf82ea899567f34e4e) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java Pending block replication can get stuck using older genstamp Key: HDFS-8404 URL: https://issues.apache.org/jira/browse/HDFS-8404 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.6.0, 2.7.0 Reporter: Nathan Roberts Assignee: Nathan Roberts Fix For: 2.7.1 Attachments: HDFS-8404-v0.patch, HDFS-8404-v1.patch If an under-replicated block gets into the pending-replication list, but later the genstamp of that block ends up being newer than the one originally submitted for replication, the block will fail replication until the NN is restarted. It will be safer if processPendingReplications() gets up-to-date blockinfo before resubmitting replication work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8404) Pending block replication can get stuck using older genstamp
[ https://issues.apache.org/jira/browse/HDFS-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552211#comment-14552211 ] Hudson commented on HDFS-8404: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #933 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/933/]) HDFS-8404. Pending block replication can get stuck using older genstamp. Contributed by Nathan Roberts. (kihwal: rev 8860e352c394372e4eb3ebdf82ea899567f34e4e) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java Pending block replication can get stuck using older genstamp Key: HDFS-8404 URL: https://issues.apache.org/jira/browse/HDFS-8404 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.6.0, 2.7.0 Reporter: Nathan Roberts Assignee: Nathan Roberts Fix For: 2.7.1 Attachments: HDFS-8404-v0.patch, HDFS-8404-v1.patch If an under-replicated block gets into the pending-replication list, but later the genstamp of that block ends up being newer than the one originally submitted for replication, the block will fail replication until the NN is restarted. It will be safer if processPendingReplications() gets up-to-date blockinfo before resubmitting replication work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8131) Implement a space balanced block placement policy
[ https://issues.apache.org/jira/browse/HDFS-8131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552213#comment-14552213 ] Hudson commented on HDFS-8131: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #933 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/933/]) HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java Implement a space balanced block placement policy - Key: HDFS-8131 URL: https://issues.apache.org/jira/browse/HDFS-8131 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Labels: BlockPlacementPolicy Fix For: 2.8.0 Attachments: HDFS-8131-v1.diff, HDFS-8131-v2.diff, HDFS-8131-v3.diff, HDFS-8131.004.patch, HDFS-8131.005.patch, HDFS-8131.006.patch, balanced.png The default block placement policy will choose datanodes for new blocks randomly, which will result in unbalanced space used percent among datanodes after an cluster expansion. The old datanodes always are in high used percent of space and new added ones are in low percent. Through we can used the external balance tool to balance the space used rate, it will cost extra network IO and it's not easy to control the balance speed. An easy solution is to implement an balanced block placement policy which will choose low used percent datanodes for new blocks with a little high possibility. In a not long term, the used percent of datanodes will trend to be balanced. Suggestions and discussions are welcomed. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8429) Death of watcherThread making other local read blocked
[
https://issues.apache.org/jira/browse/HDFS-8429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552095#comment-14552095
]
zhouyingchao commented on HDFS-8429:
Colin, thank you for the great comments. In this case, I think the bottom line
is that the death of the watcher thread should not block other threads and the
client side should be indicated to fall through to other ways as quick as
possible.
I created a patch trying to resolve the blocking. Besides that, I also changed
the native getAndClearReadableFds method to throw exception as Colin mentioned.
Please feel free to post your thoughts and comments. Thank you.
Death of watcherThread making other local read blocked
--
Key: HDFS-8429
URL: https://issues.apache.org/jira/browse/HDFS-8429
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: zhouyingchao
Assignee: zhouyingchao
In our cluster, an application is hung when doing a short circuit read of
local hdfs block. By looking into the log, we found the DataNode's
DomainSocketWatcher.watcherThread has exited with following log:
{code}
ERROR org.apache.hadoop.net.unix.DomainSocketWatcher:
Thread[Thread-25,5,main] terminating on unexpected exception
java.lang.NullPointerException
at
org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:463)
at java.lang.Thread.run(Thread.java:662)
{code}
The line 463 is following code snippet:
{code}
try {
for (int fd : fdSet.getAndClearReadableFds()) {
sendCallbackAndRemove(getAndClearReadableFds, entries, fdSet,
fd);
}
{code}
getAndClearReadableFds is a native method which will malloc an int array.
Since our memory is very tight, it looks like the malloc failed and a NULL
pointer is returned.
The bad thing is that other threads then blocked in stack like this:
{code}
DataXceiver for client
unix:/home/work/app/hdfs/c3prc-micloud/datanode/dn_socket [Waiting for
operation #1] daemon prio=10 tid=0x7f0c9c086d90 nid=0x8fc3 waiting on
condition [0x7f09b9856000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0007b0174808 (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:323)
at
org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.createNewMemorySegment(ShortCircuitRegistry.java:322)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitShm(DataXceiver.java:403)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitShm(Receiver.java:214)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:95)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
at java.lang.Thread.run(Thread.java:662)
{code}
IMO, we should exit the DN so that the users can know that something go
wrong and fix it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8444) Erasure Coding: fix cannot rename a zone dir
Walter Su created HDFS-8444:
---
Summary: Erasure Coding: fix cannot rename a zone dir
Key: HDFS-8444
URL: https://issues.apache.org/jira/browse/HDFS-8444
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
We create a EC zone {{/my_ec_zone}}.
We want to rename it to {{/myZone}}.
But it failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8429) Death of watcherThread making other local read blocked
[
https://issues.apache.org/jira/browse/HDFS-8429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhouyingchao updated HDFS-8429:
---
Status: Patch Available (was: Open)
Death of watcherThread making other local read blocked
--
Key: HDFS-8429
URL: https://issues.apache.org/jira/browse/HDFS-8429
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: zhouyingchao
Assignee: zhouyingchao
Attachments: HDFS-8429-001.patch
In our cluster, an application is hung when doing a short circuit read of
local hdfs block. By looking into the log, we found the DataNode's
DomainSocketWatcher.watcherThread has exited with following log:
{code}
ERROR org.apache.hadoop.net.unix.DomainSocketWatcher:
Thread[Thread-25,5,main] terminating on unexpected exception
java.lang.NullPointerException
at
org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:463)
at java.lang.Thread.run(Thread.java:662)
{code}
The line 463 is following code snippet:
{code}
try {
for (int fd : fdSet.getAndClearReadableFds()) {
sendCallbackAndRemove(getAndClearReadableFds, entries, fdSet,
fd);
}
{code}
getAndClearReadableFds is a native method which will malloc an int array.
Since our memory is very tight, it looks like the malloc failed and a NULL
pointer is returned.
The bad thing is that other threads then blocked in stack like this:
{code}
DataXceiver for client
unix:/home/work/app/hdfs/c3prc-micloud/datanode/dn_socket [Waiting for
operation #1] daemon prio=10 tid=0x7f0c9c086d90 nid=0x8fc3 waiting on
condition [0x7f09b9856000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0007b0174808 (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:323)
at
org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.createNewMemorySegment(ShortCircuitRegistry.java:322)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitShm(DataXceiver.java:403)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitShm(Receiver.java:214)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:95)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
at java.lang.Thread.run(Thread.java:662)
{code}
IMO, we should exit the DN so that the users can know that something go
wrong and fix it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8429) Death of watcherThread making other local read blocked
[
https://issues.apache.org/jira/browse/HDFS-8429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhouyingchao updated HDFS-8429:
---
Attachment: HDFS-8429-001.patch
Test following cases :
TestDomainSocket,TestDomainSocketWatcher,TestParallelShortCircuitRead,TestFsDatasetCacheRevocation,TestFatasetCacheRevocation,TestScrLazyPersistFiles,TestParallelShortCircuitReadNoChecksum,TestDFSInputStream,TestBlockReaderFactory,TestParallelUnixDomainRead,TestParallelShortCircuitReadUnCached,TestBlockReaderLocalLegacy,TestPeerCache,TestShortCircuitCache,TestShortCircuitLocalRead,TestBlockReaderLocal,TestParallelShortCircuitLegacyRead,TestTracingShortCircuitLocalRead,TestEnhancedByteBufferAccess
Death of watcherThread making other local read blocked
--
Key: HDFS-8429
URL: https://issues.apache.org/jira/browse/HDFS-8429
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: zhouyingchao
Assignee: zhouyingchao
Attachments: HDFS-8429-001.patch
In our cluster, an application is hung when doing a short circuit read of
local hdfs block. By looking into the log, we found the DataNode's
DomainSocketWatcher.watcherThread has exited with following log:
{code}
ERROR org.apache.hadoop.net.unix.DomainSocketWatcher:
Thread[Thread-25,5,main] terminating on unexpected exception
java.lang.NullPointerException
at
org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:463)
at java.lang.Thread.run(Thread.java:662)
{code}
The line 463 is following code snippet:
{code}
try {
for (int fd : fdSet.getAndClearReadableFds()) {
sendCallbackAndRemove(getAndClearReadableFds, entries, fdSet,
fd);
}
{code}
getAndClearReadableFds is a native method which will malloc an int array.
Since our memory is very tight, it looks like the malloc failed and a NULL
pointer is returned.
The bad thing is that other threads then blocked in stack like this:
{code}
DataXceiver for client
unix:/home/work/app/hdfs/c3prc-micloud/datanode/dn_socket [Waiting for
operation #1] daemon prio=10 tid=0x7f0c9c086d90 nid=0x8fc3 waiting on
condition [0x7f09b9856000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0007b0174808 (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:323)
at
org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.createNewMemorySegment(ShortCircuitRegistry.java:322)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitShm(DataXceiver.java:403)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitShm(Receiver.java:214)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:95)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
at java.lang.Thread.run(Thread.java:662)
{code}
IMO, we should exit the DN so that the users can know that something go
wrong and fix it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8429) Death of watcherThread making other local read blocked
[
https://issues.apache.org/jira/browse/HDFS-8429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552147#comment-14552147
]
Hadoop QA commented on HDFS-8429:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 42s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 5s | The applied patch generated 2
new checkstyle issues (total was 19, now 21). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 40s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 22m 53s | Tests passed in
hadoop-common. |
| | | 60m 2s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734105/HDFS-8429-001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ce53c8e |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11061/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11061/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11061/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11061/console |
This message was automatically generated.
Death of watcherThread making other local read blocked
--
Key: HDFS-8429
URL: https://issues.apache.org/jira/browse/HDFS-8429
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: zhouyingchao
Assignee: zhouyingchao
Attachments: HDFS-8429-001.patch
In our cluster, an application is hung when doing a short circuit read of
local hdfs block. By looking into the log, we found the DataNode's
DomainSocketWatcher.watcherThread has exited with following log:
{code}
ERROR org.apache.hadoop.net.unix.DomainSocketWatcher:
Thread[Thread-25,5,main] terminating on unexpected exception
java.lang.NullPointerException
at
org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:463)
at java.lang.Thread.run(Thread.java:662)
{code}
The line 463 is following code snippet:
{code}
try {
for (int fd : fdSet.getAndClearReadableFds()) {
sendCallbackAndRemove(getAndClearReadableFds, entries, fdSet,
fd);
}
{code}
getAndClearReadableFds is a native method which will malloc an int array.
Since our memory is very tight, it looks like the malloc failed and a NULL
pointer is returned.
The bad thing is that other threads then blocked in stack like this:
{code}
DataXceiver for client
unix:/home/work/app/hdfs/c3prc-micloud/datanode/dn_socket [Waiting for
operation #1] daemon prio=10 tid=0x7f0c9c086d90 nid=0x8fc3 waiting on
condition [0x7f09b9856000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0007b0174808 (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:323)
at
org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.createNewMemorySegment(ShortCircuitRegistry.java:322)
at
[jira] [Updated] (HDFS-8420) Erasure Coding: ECZoneManager#getECZoneInfo is not resolving the path properly if zone dir itself is the snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-8420:
---
Attachment: HDFS-8320-HDFS-7285-01.patch
Erasure Coding: ECZoneManager#getECZoneInfo is not resolving the path
properly if zone dir itself is the snapshottable dir
--
Key: HDFS-8420
URL: https://issues.apache.org/jira/browse/HDFS-8420
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8320-HDFS-7285-00.patch,
HDFS-8320-HDFS-7285-01.patch
Presently the resultant zone dir will come with {{.snapshot}} only when the
zone dir itself is snapshottable dir. It will return the path including the
snapshot name like, {{/zone/.snapshot/snap1}}. Instead could improve this by
returning only path {{/zone}}.
Thanks [~vinayrpet] for the helpful
[discussion|https://issues.apache.org/jira/browse/HDFS-8266?focusedCommentId=14543821page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14543821]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8408) Revisit and refactor ErasureCodingInfo
[
https://issues.apache.org/jira/browse/HDFS-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552220#comment-14552220
]
Tsz Wo Nicholas Sze commented on HDFS-8408:
---
{quote}
There is EncryptionZone.java which provides details about EncryptionZone,
returned with listEncryptionZones().
So IMO, keeping ErasureCodingZoneInfo would be helpful atleast for admin API.
If we can include zoneDir also in HdfsFileStatus, then this may be unnecessary.
{quote}
Thanks for clarifying it. I agree that we should keep something like
ErasureCodingZoneInfo. However, how about renaming it to ErasureCodingZone?
The ec zone related code should correspond to encryption zone code. Otherwise,
it will be quite confusing.
Revisit and refactor ErasureCodingInfo
--
Key: HDFS-8408
URL: https://issues.apache.org/jira/browse/HDFS-8408
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Vinayakumar B
Assignee: Vinayakumar B
As mentioned in HDFS-8375
[here|https://issues.apache.org/jira/browse/HDFS-8375?focusedCommentId=14544618page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14544618]
{{ErasureCodingInfo}} needs a revisit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8408) Revisit and refactor ErasureCodingInfo
[
https://issues.apache.org/jira/browse/HDFS-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552059#comment-14552059
]
Vinayakumar B commented on HDFS-8408:
-
There is a slight difference in directly using the HdfsFileStatus.
HdfsFileStatus as of now carries schema and cellSize, but it doesnot carry the
zonedir. In Serverside {{zone dir}} is the one which stores the schema and
cellsize as xattr. And these zonedirs are not tracked separately in any other
data structure as in EncryptionZoneManager.
As of now, renames are not allowed from one zonedir to another zonedir as
encoding/decoding may have issues, but we can move zone dir's parent dir itself.
So using ErasureCodingZoneInfo admin can get the zone dir and decide to move
zonedir itself
bq. Note that we neither have StorageTypeInfo for StorageType nor
EncryptionZoneInfo for EncryptionZone
There is {{EncryptionZone.java}} which provides details about EncryptionZone,
returned with {{listEncryptionZones()}}.
So IMO, keeping ErasureCodingZoneInfo would be helpful atleast for admin API.
If we can include zoneDir also in HdfsFileStatus, then this may be unnecessary.
Revisit and refactor ErasureCodingInfo
--
Key: HDFS-8408
URL: https://issues.apache.org/jira/browse/HDFS-8408
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Vinayakumar B
Assignee: Vinayakumar B
As mentioned in HDFS-8375
[here|https://issues.apache.org/jira/browse/HDFS-8375?focusedCommentId=14544618page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14544618]
{{ErasureCodingInfo}} needs a revisit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-8443) Document dfs.namenode.service.handler.count in hdfs-site.xml
[ https://issues.apache.org/jira/browse/HDFS-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina reassigned HDFS-8443: Assignee: J.Andreina Document dfs.namenode.service.handler.count in hdfs-site.xml Key: HDFS-8443 URL: https://issues.apache.org/jira/browse/HDFS-8443 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Reporter: Akira AJISAKA Assignee: J.Andreina When dfs.namenode.servicerpc-address is configured, NameNode launches an extra RPC server to handle requests from non-client nodes. dfs.namenode.service.handler.count specifies the number of threads for the server but the parameter is not documented anywhere. I found a mail for asking about the parameter. http://mail-archives.apache.org/mod_mbox/hadoop-user/201505.mbox/%3CE0D5A619-BDEA-44D2-81EB-C32B8464133D%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8382) Remove chunkSize parameter from initialize method of raw erasure coder
[
https://issues.apache.org/jira/browse/HDFS-8382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HDFS-8382:
Attachment: HDFS-8382-HDFS-7285-v3.patch
Updated the patch also removing {{initialize}} method per the suggestion.
Remove chunkSize parameter from initialize method of raw erasure coder
--
Key: HDFS-8382
URL: https://issues.apache.org/jira/browse/HDFS-8382
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8382-HDFS-7285-v1.patch,
HDFS-8382-HDFS-7285-v2.patch, HDFS-8382-HDFS-7285-v3.patch
Per discussion in HDFS-8347, we need to support encoding/decoding variable
width units data instead of predefined fixed width like {{chunkSize}}. Have
this issue to remove chunkSize in the general raw erasure coder API. Specific
coder will support fixed chunkSize using hard-coded or specific schema
customizing way if necessary, like HitchHiker coder.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8382) Remove chunkSize parameter from initialize method of raw erasure coder
[
https://issues.apache.org/jira/browse/HDFS-8382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HDFS-8382:
Status: Open (was: Patch Available)
The patch pending on HADOOP-11847.
Remove chunkSize parameter from initialize method of raw erasure coder
--
Key: HDFS-8382
URL: https://issues.apache.org/jira/browse/HDFS-8382
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8382-HDFS-7285-v1.patch,
HDFS-8382-HDFS-7285-v2.patch, HDFS-8382-HDFS-7285-v3.patch
Per discussion in HDFS-8347, we need to support encoding/decoding variable
width units data instead of predefined fixed width like {{chunkSize}}. Have
this issue to remove chunkSize in the general raw erasure coder API. Specific
coder will support fixed chunkSize using hard-coded or specific schema
customizing way if necessary, like HitchHiker coder.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8441:
Description:
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update quota count
3. support create EC file with replication=0
was:
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update quota count
Erasure Coding: make condition check earlier for setReplication
---
Key: HDFS-8441
URL: https://issues.apache.org/jira/browse/HDFS-8441
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8441-HDFS-7285.001.patch,
HDFS-8441-HDFS-7285.002.patch
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update quota count
3. support create EC file with replication=0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8441:
Description:
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update storage count
3. support create EC file with replication=0
was:
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update quota count
3. support create EC file with replication=0
Erasure Coding: make condition check earlier for setReplication
---
Key: HDFS-8441
URL: https://issues.apache.org/jira/browse/HDFS-8441
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8441-HDFS-7285.001.patch,
HDFS-8441-HDFS-7285.002.patch
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update storage count
3. support create EC file with replication=0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8441:
Attachment: HDFS-8441-HDFS-7285.002.patch
Erasure Coding: make condition check earlier for setReplication
---
Key: HDFS-8441
URL: https://issues.apache.org/jira/browse/HDFS-8441
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8441-HDFS-7285.001.patch,
HDFS-8441-HDFS-7285.002.patch
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update quota count
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552069#comment-14552069
]
Walter Su commented on HDFS-8441:
-
002 patch add support create EC file with replication=0
Erasure Coding: make condition check earlier for setReplication
---
Key: HDFS-8441
URL: https://issues.apache.org/jira/browse/HDFS-8441
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8441-HDFS-7285.001.patch,
HDFS-8441-HDFS-7285.002.patch
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update quota count
3. support create EC file with replication=0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8443) Document dfs.namenode.service.handler.count in hdfs-site.xml
[ https://issues.apache.org/jira/browse/HDFS-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551970#comment-14551970 ] J.Andreina commented on HDFS-8443: -- I would like update the patch. Document dfs.namenode.service.handler.count in hdfs-site.xml Key: HDFS-8443 URL: https://issues.apache.org/jira/browse/HDFS-8443 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Reporter: Akira AJISAKA Assignee: J.Andreina When dfs.namenode.servicerpc-address is configured, NameNode launches an extra RPC server to handle requests from non-client nodes. dfs.namenode.service.handler.count specifies the number of threads for the server but the parameter is not documented anywhere. I found a mail for asking about the parameter. http://mail-archives.apache.org/mod_mbox/hadoop-user/201505.mbox/%3CE0D5A619-BDEA-44D2-81EB-C32B8464133D%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-8382) Remove chunkSize parameter from initialize method of raw erasure coder
[
https://issues.apache.org/jira/browse/HDFS-8382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-8382 started by Kai Zheng.
---
Remove chunkSize parameter from initialize method of raw erasure coder
--
Key: HDFS-8382
URL: https://issues.apache.org/jira/browse/HDFS-8382
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8382-HDFS-7285-v1.patch,
HDFS-8382-HDFS-7285-v2.patch, HDFS-8382-HDFS-7285-v3.patch
Per discussion in HDFS-8347, we need to support encoding/decoding variable
width units data instead of predefined fixed width like {{chunkSize}}. Have
this issue to remove chunkSize in the general raw erasure coder API. Specific
coder will support fixed chunkSize using hard-coded or specific schema
customizing way if necessary, like HitchHiker coder.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8375) Add cellSize as an XAttr to ECZone
[
https://issues.apache.org/jira/browse/HDFS-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552063#comment-14552063
]
Vinayakumar B commented on HDFS-8375:
-
Thanks [~zhz].
Add cellSize as an XAttr to ECZone
--
Key: HDFS-8375
URL: https://issues.apache.org/jira/browse/HDFS-8375
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Fix For: HDFS-7285
Attachments: HDFS-8375-HDFS-7285-01.patch,
HDFS-8375-HDFS-7285-02.patch, HDFS-8375-HDFS-7285-03.patch,
HDFS-8375-HDFS-7285-04.patch
Add {{cellSize}} as an Xattr for ECZone. as discussed
[here|https://issues.apache.org/jira/browse/HDFS-8347?focusedCommentId=14539108page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14539108]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8443) Document dfs.namenode.service.handler.count in hdfs-site.xml
[ https://issues.apache.org/jira/browse/HDFS-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8443: - Attachment: HDFS-8443.1.patch Attached an initial patch. Please review. Document dfs.namenode.service.handler.count in hdfs-site.xml Key: HDFS-8443 URL: https://issues.apache.org/jira/browse/HDFS-8443 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Reporter: Akira AJISAKA Assignee: J.Andreina Attachments: HDFS-8443.1.patch When dfs.namenode.servicerpc-address is configured, NameNode launches an extra RPC server to handle requests from non-client nodes. dfs.namenode.service.handler.count specifies the number of threads for the server but the parameter is not documented anywhere. I found a mail for asking about the parameter. http://mail-archives.apache.org/mod_mbox/hadoop-user/201505.mbox/%3CE0D5A619-BDEA-44D2-81EB-C32B8464133D%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8382) Remove chunkSize parameter from initialize method of raw erasure coder
[
https://issues.apache.org/jira/browse/HDFS-8382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552089#comment-14552089
]
Vinayakumar B commented on HDFS-8382:
-
I see that patch covers 'Raw' coders. But similar treatment to be done for
'coder' classes also. Could it be done in this patch itself. ?
Remove chunkSize parameter from initialize method of raw erasure coder
--
Key: HDFS-8382
URL: https://issues.apache.org/jira/browse/HDFS-8382
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8382-HDFS-7285-v1.patch,
HDFS-8382-HDFS-7285-v2.patch, HDFS-8382-HDFS-7285-v3.patch
Per discussion in HDFS-8347, we need to support encoding/decoding variable
width units data instead of predefined fixed width like {{chunkSize}}. Have
this issue to remove chunkSize in the general raw erasure coder API. Specific
coder will support fixed chunkSize using hard-coded or specific schema
customizing way if necessary, like HitchHiker coder.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8443) Document dfs.namenode.service.handler.count in hdfs-site.xml
[ https://issues.apache.org/jira/browse/HDFS-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8443: - Status: Patch Available (was: Open) Document dfs.namenode.service.handler.count in hdfs-site.xml Key: HDFS-8443 URL: https://issues.apache.org/jira/browse/HDFS-8443 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Reporter: Akira AJISAKA Assignee: J.Andreina Attachments: HDFS-8443.1.patch When dfs.namenode.servicerpc-address is configured, NameNode launches an extra RPC server to handle requests from non-client nodes. dfs.namenode.service.handler.count specifies the number of threads for the server but the parameter is not documented anywhere. I found a mail for asking about the parameter. http://mail-archives.apache.org/mod_mbox/hadoop-user/201505.mbox/%3CE0D5A619-BDEA-44D2-81EB-C32B8464133D%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552227#comment-14552227
]
Kai Zheng commented on HDFS-8441:
-
The patch looks good. One comment regarding the following change. Maybe we
could add a method in {{FSDirectory#isInECZone(src)}} and use it to tell the
condition?
{code}
+final ErasureCodingZoneInfo ecZoneInfo = dir.getECZoneInfo(iip);
+if (ecZoneInfo == null) {
+ blockManager.verifyReplication(src, replication, clientMachine);
+}
{code}
Erasure Coding: make condition check earlier for setReplication
---
Key: HDFS-8441
URL: https://issues.apache.org/jira/browse/HDFS-8441
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8441-HDFS-7285.001.patch,
HDFS-8441-HDFS-7285.002.patch
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update storage count
3. support create EC file with replication=0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552252#comment-14552252
]
Rakesh R commented on HDFS-8441:
Thanks [~walter.k.su] for taking care this. Just one comment:
1) Please add message to the {{fail()}} statement like,
{code}
+ fail(Shouldn't allow to set replication to a file with striped blocks);
{code}
Erasure Coding: make condition check earlier for setReplication
---
Key: HDFS-8441
URL: https://issues.apache.org/jira/browse/HDFS-8441
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8441-HDFS-7285.001.patch,
HDFS-8441-HDFS-7285.002.patch
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update storage count
3. support create EC file with replication=0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-3854) Implement a fence method which should fence the BK shared storage.
[ https://issues.apache.org/jira/browse/HDFS-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552283#comment-14552283 ] Rakesh R commented on HDFS-3854: [~umamaheswararao] like you mentioned, HDFS-3862 has a proposal to skip the fencer logic as BK has already a built-in fencing mechanism. Can I close this issue ? Implement a fence method which should fence the BK shared storage. -- Key: HDFS-3854 URL: https://issues.apache.org/jira/browse/HDFS-3854 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Uma Maheswara Rao G Assignee: Rakesh R Currently when machine down or network down, SSHFence can not ensure that, other node is completely down. So, fence will fail and switch will not happen. [ internally we did work around to return true when machine is not reachable, as BKJM already has fencing] It may be good idea to implement a fence method, which should ensure shared storage fenced propertly and return true. We can plug in this new method in ZKFC fence methods. only pain points what I can see is, we may have to put the BKJM jar in ZKFC lib for running this fence method. thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8427) Remove dataBlockNum and parityBlockNum from BlockInfoStriped
[
https://issues.apache.org/jira/browse/HDFS-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552292#comment-14552292
]
Hadoop QA commented on HDFS-8427:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 46s | Pre-patch HDFS-7285 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 28s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 14s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 12s | The patch appears to introduce 6
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 171m 10s | Tests failed in hadoop-hdfs. |
| | | 214m 11s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.DFSOutputStream.streamer; locked 89% of time
Unsynchronized access at DFSOutputStream.java:89% of time Unsynchronized
access at DFSOutputStream.java:[line 146] |
| | Possible null pointer dereference of arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:[line 194] |
| | Unread field:field be static? At ErasureCodingWorker.java:[line 254] |
| | Should
org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$StripedReader
be a _static_ inner class? At ErasureCodingWorker.java:inner class? At
ErasureCodingWorker.java:[lines 907-914] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:[line 108] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:[line 409] |
| Failed unit tests | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734100/HDFS-8427-HDFS-7285-01.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 4dd4aa5 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/11058/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11058/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11058/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11058/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11058/console |
This message was automatically generated.
Remove dataBlockNum and parityBlockNum from BlockInfoStriped
Key:
[jira] [Commented] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552299#comment-14552299
]
Hadoop QA commented on HDFS-8441:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 20s | Pre-patch HDFS-7285 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 58s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 38s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 40s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 17s | The patch appears to introduce 6
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 19s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 173m 12s | Tests failed in hadoop-hdfs. |
| | | 216m 4s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.DFSOutputStream.streamer; locked 89% of time
Unsynchronized access at DFSOutputStream.java:89% of time Unsynchronized
access at DFSOutputStream.java:[line 146] |
| | Possible null pointer dereference of arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:[line 194] |
| | Unread field:field be static? At ErasureCodingWorker.java:[line 254] |
| | Should
org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$StripedReader
be a _static_ inner class? At ErasureCodingWorker.java:inner class? At
ErasureCodingWorker.java:[lines 907-914] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:[line 108] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:[line 409] |
| Failed unit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734101/HDFS-8441-HDFS-7285.002.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 4dd4aa5 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/11059/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11059/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11059/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11059/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11059/console |
This message was automatically generated.
Erasure Coding: make condition check earlier for setReplication
---
Key: HDFS-8441
URL: https://issues.apache.org/jira/browse/HDFS-8441
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee:
[jira] [Commented] (HDFS-8443) Document dfs.namenode.service.handler.count in hdfs-site.xml
[
https://issues.apache.org/jira/browse/HDFS-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552333#comment-14552333
]
Hadoop QA commented on HDFS-8443:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 44s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 41s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | native | 3m 16s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 164m 13s | Tests failed in hadoop-hdfs. |
| | | 202m 4s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734104/HDFS-8443.1.patch |
| Optional Tests | javadoc javac unit |
| git revision | trunk / ce53c8e |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11060/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11060/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11060/console |
This message was automatically generated.
Document dfs.namenode.service.handler.count in hdfs-site.xml
Key: HDFS-8443
URL: https://issues.apache.org/jira/browse/HDFS-8443
Project: Hadoop HDFS
Issue Type: Improvement
Components: documentation
Reporter: Akira AJISAKA
Assignee: J.Andreina
Attachments: HDFS-8443.1.patch
When dfs.namenode.servicerpc-address is configured, NameNode launches an
extra RPC server to handle requests from non-client nodes.
dfs.namenode.service.handler.count specifies the number of threads for the
server but the parameter is not documented anywhere.
I found a mail for asking about the parameter.
http://mail-archives.apache.org/mod_mbox/hadoop-user/201505.mbox/%3CE0D5A619-BDEA-44D2-81EB-C32B8464133D%40gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8404) Pending block replication can get stuck using older genstamp
[ https://issues.apache.org/jira/browse/HDFS-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552600#comment-14552600 ] Hudson commented on HDFS-8404: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2149 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2149/]) HDFS-8404. Pending block replication can get stuck using older genstamp. Contributed by Nathan Roberts. (kihwal: rev 8860e352c394372e4eb3ebdf82ea899567f34e4e) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java Pending block replication can get stuck using older genstamp Key: HDFS-8404 URL: https://issues.apache.org/jira/browse/HDFS-8404 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.6.0, 2.7.0 Reporter: Nathan Roberts Assignee: Nathan Roberts Fix For: 2.7.1 Attachments: HDFS-8404-v0.patch, HDFS-8404-v1.patch If an under-replicated block gets into the pending-replication list, but later the genstamp of that block ends up being newer than the one originally submitted for replication, the block will fail replication until the NN is restarted. It will be safer if processPendingReplications() gets up-to-date blockinfo before resubmitting replication work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8131) Implement a space balanced block placement policy
[ https://issues.apache.org/jira/browse/HDFS-8131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552602#comment-14552602 ] Hudson commented on HDFS-8131: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2149 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2149/]) HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java Implement a space balanced block placement policy - Key: HDFS-8131 URL: https://issues.apache.org/jira/browse/HDFS-8131 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Labels: BlockPlacementPolicy Fix For: 2.8.0 Attachments: HDFS-8131-v1.diff, HDFS-8131-v2.diff, HDFS-8131-v3.diff, HDFS-8131.004.patch, HDFS-8131.005.patch, HDFS-8131.006.patch, balanced.png The default block placement policy will choose datanodes for new blocks randomly, which will result in unbalanced space used percent among datanodes after an cluster expansion. The old datanodes always are in high used percent of space and new added ones are in low percent. Through we can used the external balance tool to balance the space used rate, it will cost extra network IO and it's not easy to control the balance speed. An easy solution is to implement an balanced block placement policy which will choose low used percent datanodes for new blocks with a little high possibility. In a not long term, the used percent of datanodes will trend to be balanced. Suggestions and discussions are welcomed. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8404) Pending block replication can get stuck using older genstamp
[ https://issues.apache.org/jira/browse/HDFS-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552556#comment-14552556 ] Hudson commented on HDFS-8404: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #201 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/201/]) HDFS-8404. Pending block replication can get stuck using older genstamp. Contributed by Nathan Roberts. (kihwal: rev 8860e352c394372e4eb3ebdf82ea899567f34e4e) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java Pending block replication can get stuck using older genstamp Key: HDFS-8404 URL: https://issues.apache.org/jira/browse/HDFS-8404 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.6.0, 2.7.0 Reporter: Nathan Roberts Assignee: Nathan Roberts Fix For: 2.7.1 Attachments: HDFS-8404-v0.patch, HDFS-8404-v1.patch If an under-replicated block gets into the pending-replication list, but later the genstamp of that block ends up being newer than the one originally submitted for replication, the block will fail replication until the NN is restarted. It will be safer if processPendingReplications() gets up-to-date blockinfo before resubmitting replication work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8131) Implement a space balanced block placement policy
[ https://issues.apache.org/jira/browse/HDFS-8131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552558#comment-14552558 ] Hudson commented on HDFS-8131: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #201 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/201/]) HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java Implement a space balanced block placement policy - Key: HDFS-8131 URL: https://issues.apache.org/jira/browse/HDFS-8131 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Labels: BlockPlacementPolicy Fix For: 2.8.0 Attachments: HDFS-8131-v1.diff, HDFS-8131-v2.diff, HDFS-8131-v3.diff, HDFS-8131.004.patch, HDFS-8131.005.patch, HDFS-8131.006.patch, balanced.png The default block placement policy will choose datanodes for new blocks randomly, which will result in unbalanced space used percent among datanodes after an cluster expansion. The old datanodes always are in high used percent of space and new added ones are in low percent. Through we can used the external balance tool to balance the space used rate, it will cost extra network IO and it's not easy to control the balance speed. An easy solution is to implement an balanced block placement policy which will choose low used percent datanodes for new blocks with a little high possibility. In a not long term, the used percent of datanodes will trend to be balanced. Suggestions and discussions are welcomed. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8444) Erasure Coding: fix cannot rename a zone dir
[
https://issues.apache.org/jira/browse/HDFS-8444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552633#comment-14552633
]
Hadoop QA commented on HDFS-8444:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 11s | Pre-patch HDFS-7285 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 46s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 55s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 38s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 16s | The patch appears to introduce 6
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 21s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 182m 35s | Tests failed in hadoop-hdfs. |
| | | 225m 12s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.DFSOutputStream.streamer; locked 89% of time
Unsynchronized access at DFSOutputStream.java:89% of time Unsynchronized
access at DFSOutputStream.java:[line 146] |
| | Possible null pointer dereference of arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:[line 194] |
| | Unread field:field be static? At ErasureCodingWorker.java:[line 254] |
| | Should
org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$StripedReader
be a _static_ inner class? At ErasureCodingWorker.java:inner class? At
ErasureCodingWorker.java:[lines 907-914] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:[line 108] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:[line 409] |
| Failed unit tests | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734114/HDFS-8444-HDFS-7285.001.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 4dd4aa5 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/11062/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11062/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11062/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11062/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11062/console |
This message was automatically generated.
Erasure Coding: fix cannot rename a zone dir
Key: HDFS-8444
URL: https://issues.apache.org/jira/browse/HDFS-8444
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter:
[jira] [Commented] (HDFS-8443) Document dfs.namenode.service.handler.count in hdfs-site.xml
[ https://issues.apache.org/jira/browse/HDFS-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552620#comment-14552620 ] Arpit Agarwal commented on HDFS-8443: - Thanks for fixing this [~andreina]. I suggest a grammatical fix. +1 otherwise. _thread that listens_ -- _threads that listen_. Document dfs.namenode.service.handler.count in hdfs-site.xml Key: HDFS-8443 URL: https://issues.apache.org/jira/browse/HDFS-8443 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Reporter: Akira AJISAKA Assignee: J.Andreina Attachments: HDFS-8443.1.patch When dfs.namenode.servicerpc-address is configured, NameNode launches an extra RPC server to handle requests from non-client nodes. dfs.namenode.service.handler.count specifies the number of threads for the server but the parameter is not documented anywhere. I found a mail for asking about the parameter. http://mail-archives.apache.org/mod_mbox/hadoop-user/201505.mbox/%3CE0D5A619-BDEA-44D2-81EB-C32B8464133D%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8420) Erasure Coding: ECZoneManager#getECZoneInfo is not resolving the path properly if zone dir itself is the snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552622#comment-14552622
]
Hadoop QA commented on HDFS-8420:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 42s | Pre-patch HDFS-7285 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 37s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 14s | The patch appears to introduce 6
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 174m 24s | Tests failed in hadoop-hdfs. |
| | | 215m 54s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.DFSOutputStream.streamer; locked 89% of time
Unsynchronized access at DFSOutputStream.java:89% of time Unsynchronized
access at DFSOutputStream.java:[line 146] |
| | Possible null pointer dereference of arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:[line 194] |
| | Unread field:field be static? At ErasureCodingWorker.java:[line 254] |
| | Should
org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$StripedReader
be a _static_ inner class? At ErasureCodingWorker.java:inner class? At
ErasureCodingWorker.java:[lines 907-914] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:[line 108] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:[line 409] |
| Failed unit tests | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.TestEncryptedTransfer |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734113/HDFS-8320-HDFS-7285-01.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 4dd4aa5 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/11063/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11063/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11063/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11063/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11063/console |
This message was automatically generated.
Erasure Coding: ECZoneManager#getECZoneInfo is not resolving the path
properly if zone dir itself is the snapshottable dir
--
Key: HDFS-8420
URL: https://issues.apache.org/jira/browse/HDFS-8420
[jira] [Commented] (HDFS-8439) Adding more slow action log in critical read path
[ https://issues.apache.org/jira/browse/HDFS-8439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552970#comment-14552970 ] stack commented on HDFS-8439: - s/cost/duration/g I could also do on commit. I like how you log the culprit or pipeline. Are you running this in production [~liushaohui]? Adding more slow action log in critical read path - Key: HDFS-8439 URL: https://issues.apache.org/jira/browse/HDFS-8439 Project: Hadoop HDFS Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HDFS-8439.001.patch To dig a HBase read spike issue, we'd better to add more slow pread/seek log in read flow to get the abnormal datanodes. Patch will be uploaded soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8427) Remove dataBlockNum and parityBlockNum from BlockInfoStriped
[
https://issues.apache.org/jira/browse/HDFS-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8427:
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
+1. I've committed this to the feature branch. Thanks for the contribution,
[~kaisasak]!
Remove dataBlockNum and parityBlockNum from BlockInfoStriped
Key: HDFS-8427
URL: https://issues.apache.org/jira/browse/HDFS-8427
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8427-HDFS-7285-00.patch,
HDFS-8427-HDFS-7285-01.patch
Remove unnecessary members such as {{dataBlockNum}} and {{parityBlockNum}}
from {{BlockInfoStriped}}. These are included in {{ECShema}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8348) WebHDFS Concat - Support sources list passed a POST body
[ https://issues.apache.org/jira/browse/HDFS-8348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8348: -- Summary: WebHDFS Concat - Support sources list passed a POST body (was: Concat - Support sources list passed a POST body) WebHDFS Concat - Support sources list passed a POST body Key: HDFS-8348 URL: https://issues.apache.org/jira/browse/HDFS-8348 Project: Hadoop HDFS Issue Type: Improvement Components: webhdfs Reporter: vishwajeet dusane Problem - Current webhdfs behavior for concat support sources list to be passed as query parameter. This approach limits on the number of sources list send as query parameter. Proposed Solution - Add support to send sources list as part of the request body instead of query parameter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8446) Separate safemode related operations in GetBlockLocations()
[
https://issues.apache.org/jira/browse/HDFS-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8446:
-
Attachment: HDFS-8446.000.patch
Separate safemode related operations in GetBlockLocations()
---
Key: HDFS-8446
URL: https://issues.apache.org/jira/browse/HDFS-8446
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Attachments: HDFS-8446.000.patch
Currently {{FSNamesystem#GetBlockLocations()}} has some special cases when
the NN is in SafeMode. This jira proposes to refactor the code to improve
readability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a command to get the bandwidth of balancer
[
https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552774#comment-14552774
]
Rakesh R commented on HDFS-7116:
Hi [~ajisakaa], its really an interesting feature and would like to take this
ahead. To begin with, I have gone through the {{-setBalancerBandwidth}}
feature. Here, Namenode is sending the {{DNA_BALANCERBANDWIDTHUPDATE}} command
to the Datanodes as a heartbeat response and Datanode will update the new
bandwidth value. In normal case, after setting the bandwidth all the datanodes
will have the same value. But I could see few corner cases where all the
Datanodes in a cluster not having unique bandwidth value.
Case-1) Newly added Datanodes will not aware about the new value and will have
the default bandwidth value.
Case-2) Assume we have 10 Datanodes, after sending heartbeat responses to 1 to
5 Datanodes, unfortunately Namenode restarted. Since bandwidth value is not
persisted restarted Namenode will not have the new bandwidth value. Now, the
rest of the 6 to 10 Datanodes will be having old value.
Is this expected behavior?. If yes, we have to design the
{{-getBalancerBandwidth}} command in such a way to list each Datanode with the
corresponding bandwidth value like, {{each Datanode = bandwidth value}}. Any
thoughts?
Add a command to get the bandwidth of balancer
--
Key: HDFS-7116
URL: https://issues.apache.org/jira/browse/HDFS-7116
Project: Hadoop HDFS
Issue Type: New Feature
Components: balancer mover
Reporter: Akira AJISAKA
Now reading logs is the only way to check how the balancer bandwidth is set.
It would be useful for administrators if they can get the parameter via CLI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8429) Death of watcherThread making other local read blocked
[
https://issues.apache.org/jira/browse/HDFS-8429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14552875#comment-14552875
]
Colin Patrick McCabe commented on HDFS-8429:
Thanks. We shouldn't need to modify {{DomainSocketWatcher#add}} and
{{DomainSocketWatcher#remove}}, since the finally block clears {{toAdd}} and
{{toRemove}} prior to calling {{signalAll}}. The modifications to the finally
block look reasonable. I think we are going to need a unit test for this as
well. Perhaps simply send an interrupt to a thread in the unit test.
Death of watcherThread making other local read blocked
--
Key: HDFS-8429
URL: https://issues.apache.org/jira/browse/HDFS-8429
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: zhouyingchao
Assignee: zhouyingchao
Attachments: HDFS-8429-001.patch
In our cluster, an application is hung when doing a short circuit read of
local hdfs block. By looking into the log, we found the DataNode's
DomainSocketWatcher.watcherThread has exited with following log:
{code}
ERROR org.apache.hadoop.net.unix.DomainSocketWatcher:
Thread[Thread-25,5,main] terminating on unexpected exception
java.lang.NullPointerException
at
org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:463)
at java.lang.Thread.run(Thread.java:662)
{code}
The line 463 is following code snippet:
{code}
try {
for (int fd : fdSet.getAndClearReadableFds()) {
sendCallbackAndRemove(getAndClearReadableFds, entries, fdSet,
fd);
}
{code}
getAndClearReadableFds is a native method which will malloc an int array.
Since our memory is very tight, it looks like the malloc failed and a NULL
pointer is returned.
The bad thing is that other threads then blocked in stack like this:
{code}
DataXceiver for client
unix:/home/work/app/hdfs/c3prc-micloud/datanode/dn_socket [Waiting for
operation #1] daemon prio=10 tid=0x7f0c9c086d90 nid=0x8fc3 waiting on
condition [0x7f09b9856000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0007b0174808 (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:323)
at
org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.createNewMemorySegment(ShortCircuitRegistry.java:322)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitShm(DataXceiver.java:403)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitShm(Receiver.java:214)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:95)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
at java.lang.Thread.run(Thread.java:662)
{code}
IMO, we should exit the DN so that the users can know that something go
wrong and fix it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8446) Separate safemode related operations in GetBlockLocations()
Haohui Mai created HDFS-8446:
Summary: Separate safemode related operations in
GetBlockLocations()
Key: HDFS-8446
URL: https://issues.apache.org/jira/browse/HDFS-8446
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Currently {{FSNamesystem#GetBlockLocations()}} has some special cases when the
NN is in SafeMode. This jira proposes to refactor the code to improve
readability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8446) Separate safemode related operations in GetBlockLocations()
[
https://issues.apache.org/jira/browse/HDFS-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8446:
-
Status: Patch Available (was: Open)
Separate safemode related operations in GetBlockLocations()
---
Key: HDFS-8446
URL: https://issues.apache.org/jira/browse/HDFS-8446
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Attachments: HDFS-8446.000.patch
Currently {{FSNamesystem#GetBlockLocations()}} has some special cases when
the NN is in SafeMode. This jira proposes to refactor the code to improve
readability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8294) Erasure Coding: Fix Findbug warnings present in erasure coding
[
https://issues.apache.org/jira/browse/HDFS-8294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553247#comment-14553247
]
Zhe Zhang commented on HDFS-8294:
-
Good point Rakesh. I agree. Could you update the patch just for the new
{{StripedBlockUtil}} then?
Erasure Coding: Fix Findbug warnings present in erasure coding
--
Key: HDFS-8294
URL: https://issues.apache.org/jira/browse/HDFS-8294
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Labels: BB2015-05-RFC
Attachments: FindBugs Report in EC feature.html,
HDFS-8294-HDFS-7285.00.patch, HDFS-8294-HDFS-7285.01.patch,
HDFS-8294-HDFS-7285.02.patch, HDFS-8294-HDFS-7285.03.patch,
HDFS-8294-HDFS-7285.04.patch, HDFS-8294-HDFS-7285.05.patch
This jira is to address the findbug issues reported in erasure coding feature.
Attached sheet which contains the details of the findbug issues reported in
the erasure coding feature. I've taken this report from the jenkins build :
https://builds.apache.org/job/PreCommit-HDFS-Build/10848/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8444) Erasure Coding: fix cannot rename a zone dir
[
https://issues.apache.org/jira/browse/HDFS-8444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553474#comment-14553474
]
Walter Su commented on HDFS-8444:
-
jenkins not related. Please review.
Erasure Coding: fix cannot rename a zone dir
Key: HDFS-8444
URL: https://issues.apache.org/jira/browse/HDFS-8444
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8444-HDFS-7285.001.patch
We create a EC zone {{/my_ec_zone}}.
We want to rename it to {{/myZone}}.
But it failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8421) Move startFile() and related operations into FSDirWriteFileOp
[
https://issues.apache.org/jira/browse/HDFS-8421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553416#comment-14553416
]
Jing Zhao commented on HDFS-8421:
-
# We do not need to remove the following check from the beginning of the
startFile call.
{code}
checkOperation(OperationCategory.WRITE);
{code}
# The safemode check should only be in the real file creation section
# The following log seems unnecessary to me: in case of Standby NN this change
will cause a lot of warning msg in the log
{code}
+} catch (IOException e) {
+ NameNode.stateChangeLog.warn(DIR* NameSystem.startFile: + src + +
+ e.getMessage());
{code}
Move startFile() and related operations into FSDirWriteFileOp
-
Key: HDFS-8421
URL: https://issues.apache.org/jira/browse/HDFS-8421
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-8421.000.patch, HDFS-8421.001.patch
This jira proposes to move startFile() and related functions into
FSDirWriteFileOp.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8421) Move startFile() and related operations into FSDirWriteFileOp
[ https://issues.apache.org/jira/browse/HDFS-8421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553421#comment-14553421 ] Jing Zhao commented on HDFS-8421: - +1 after addressing the comments. Move startFile() and related operations into FSDirWriteFileOp - Key: HDFS-8421 URL: https://issues.apache.org/jira/browse/HDFS-8421 Project: Hadoop HDFS Issue Type: Bug Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-8421.000.patch, HDFS-8421.001.patch This jira proposes to move startFile() and related functions into FSDirWriteFileOp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8448) Create REST Interface
Anu Engineer created HDFS-8448: -- Summary: Create REST Interface Key: HDFS-8448 URL: https://issues.apache.org/jira/browse/HDFS-8448 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anu Engineer Assignee: Anu Engineer Create REST interfaces as specified in the architecture document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8344) NameNode doesn't recover lease for files with missing blocks
[
https://issues.apache.org/jira/browse/HDFS-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HDFS-8344:
---
Status: Patch Available (was: Open)
NameNode doesn't recover lease for files with missing blocks
Key: HDFS-8344
URL: https://issues.apache.org/jira/browse/HDFS-8344
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Attachments: HDFS-8344.01.patch, HDFS-8344.02.patch,
HDFS-8344.03.patch
I found another\(?) instance in which the lease is not recovered. This is
reproducible easily on a pseudo-distributed single node cluster
# Before you start it helps if you set. This is not necessary, but simply
reduces how long you have to wait
{code}
public static final long LEASE_SOFTLIMIT_PERIOD = 30 * 1000;
public static final long LEASE_HARDLIMIT_PERIOD = 2 *
LEASE_SOFTLIMIT_PERIOD;
{code}
# Client starts to write a file. (could be less than 1 block, but it hflushed
so some of the data has landed on the datanodes) (I'm copying the client code
I am using. I generate a jar and run it using $ hadoop jar TestHadoop.jar)
# Client crashes. (I simulate this by kill -9 the $(hadoop jar
TestHadoop.jar) process after it has printed Wrote to the bufferedWriter
# Shoot the datanode. (Since I ran on a pseudo-distributed cluster, there was
only 1)
I believe the lease should be recovered and the block should be marked
missing. However this is not happening. The lease is never recovered.
The effect of this bug for us was that nodes could not be decommissioned
cleanly. Although we knew that the client had crashed, the Namenode never
released the leases (even after restarting the Namenode) (even months
afterwards). There are actually several other cases too where we don't
consider what happens if ALL the datanodes die while the file is being
written, but I am going to punt on that for another time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a command to get the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553425#comment-14553425 ] Allen Wittenauer commented on HDFS-7116: Why not just add this to dfsadmin -report? Add a command to get the bandwidth of balancer -- Key: HDFS-7116 URL: https://issues.apache.org/jira/browse/HDFS-7116 Project: Hadoop HDFS Issue Type: New Feature Components: balancer mover Reporter: Akira AJISAKA Now reading logs is the only way to check how the balancer bandwidth is set. It would be useful for administrators if they can get the parameter via CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8448) Create REST Interface for Volumes
[ https://issues.apache.org/jira/browse/HDFS-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-8448: --- Description: Create REST interfaces as specified in the architecture document. This Jira is for creating the Volume Interface was:Create REST interfaces as specified in the architecture document. Summary: Create REST Interface for Volumes (was: Create REST Interface) Create REST Interface for Volumes - Key: HDFS-8448 URL: https://issues.apache.org/jira/browse/HDFS-8448 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anu Engineer Assignee: Anu Engineer Create REST interfaces as specified in the architecture document. This Jira is for creating the Volume Interface -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7609) startup used too much time to load edits
[
https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553292#comment-14553292
]
Jing Zhao commented on HDFS-7609:
-
Thanks for working on this, [~mingma]! Your analysis makes sense to me and I
can also reproduce the same scenario. Your patch also looks good to me.
One extra issue, which exists before your patch, is that the retry cache may
not be hit during the NameNode failover. For example, suppose the following
event sequence:
# a delete request is served by the ANN but the client fails to receive the
response
# NN failover happens
# the first retry request gets StandbyException since the old ANN becomes
standby
# the second retry request is sent to the other NN, which at this time has not
started the transition yet
# no cached entry has been found in the retry cache
# before running {{FSNamesystem#delete}}, the transition starts, which blocks
the {{FSNamesystem#delete}} call
# the {{FSNamesystem#delete}} is called and fails
If the above scenario stands, maybe we can use this chance to fix it? In your
patch, if the NN is in standby state, and there is no retry cache entry, should
we directly throw StandbyException instead of checking it again in
FSNamesystem?
startup used too much time to load edits
Key: HDFS-7609
URL: https://issues.apache.org/jira/browse/HDFS-7609
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.2.0
Reporter: Carrey Zhan
Assignee: Ming Ma
Labels: BB2015-05-RFC
Attachments: HDFS-7609-CreateEditsLogWithRPCIDs.patch,
HDFS-7609.patch, recovery_do_not_use_retrycache.patch
One day my namenode crashed because of two journal node timed out at the same
time under very high load, leaving behind about 100 million transactions in
edits log.(I still have no idea why they were not rolled into fsimage.)
I tryed to restart namenode, but it showed that almost 20 hours would be
needed before finish, and it was loading fsedits most of the time. I also
tryed to restart namenode in recover mode, the loading speed had no different.
I looked into the stack trace, judged that it is caused by the retry cache.
So I set dfs.namenode.enable.retrycache to false, the restart process
finished in half an hour.
I think the retry cached is useless during startup, at least during recover
process.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8210) Ozone: Implement storage container manager
[
https://issues.apache.org/jira/browse/HDFS-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-8210:
Hadoop Flags: Reviewed
Thank you for the new patch, Jitendra. The test failure is unrelated. The
checkstyle nits are mostly not actionable, but we could fix these two:
{code}
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/storagecontainer/protocol/StorageContainer.java:23:
First sentence should end with a period.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/storagecontainer/StorageContainerMap.java:86:
First sentence should end with a period.
{code}
+1 for committing to the feature branch after those are addressed. Thanks!
Ozone: Implement storage container manager
---
Key: HDFS-8210
URL: https://issues.apache.org/jira/browse/HDFS-8210
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: HDFS-8210-HDFS-7240.1.patch,
HDFS-8210-HDFS-7240.2.patch, HDFS-8210-HDFS-7240.3.patch
The storage container manager collects datanode heartbeats, manages
replication and exposes API to lookup containers. This jira implements
storage container manager by re-using the block manager implementation in
namenode. This jira provides initial implementation that works with
datanodes. The additional protocols will be added in subsequent jiras.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a command to get the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553399#comment-14553399 ] Kengo Seki commented on HDFS-7116: -- These are expected behavior, so displaying bandwidth values for each DataNodes seems appropriate. In addition, it may be useful that the command can take a list of DataNodes to be processed, in case the cluster is large. Add a command to get the bandwidth of balancer -- Key: HDFS-7116 URL: https://issues.apache.org/jira/browse/HDFS-7116 Project: Hadoop HDFS Issue Type: New Feature Components: balancer mover Reporter: Akira AJISAKA Now reading logs is the only way to check how the balancer bandwidth is set. It would be useful for administrators if they can get the parameter via CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8344) NameNode doesn't recover lease for files with missing blocks
[
https://issues.apache.org/jira/browse/HDFS-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HDFS-8344:
---
Attachment: HDFS-8344.03.patch
Hi Kihwal! Here's a patch which improves the test to ensure that data that was
once hsynced is read back correctly. Could you please review it?
NameNode doesn't recover lease for files with missing blocks
Key: HDFS-8344
URL: https://issues.apache.org/jira/browse/HDFS-8344
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Attachments: HDFS-8344.01.patch, HDFS-8344.02.patch,
HDFS-8344.03.patch
I found another\(?) instance in which the lease is not recovered. This is
reproducible easily on a pseudo-distributed single node cluster
# Before you start it helps if you set. This is not necessary, but simply
reduces how long you have to wait
{code}
public static final long LEASE_SOFTLIMIT_PERIOD = 30 * 1000;
public static final long LEASE_HARDLIMIT_PERIOD = 2 *
LEASE_SOFTLIMIT_PERIOD;
{code}
# Client starts to write a file. (could be less than 1 block, but it hflushed
so some of the data has landed on the datanodes) (I'm copying the client code
I am using. I generate a jar and run it using $ hadoop jar TestHadoop.jar)
# Client crashes. (I simulate this by kill -9 the $(hadoop jar
TestHadoop.jar) process after it has printed Wrote to the bufferedWriter
# Shoot the datanode. (Since I ran on a pseudo-distributed cluster, there was
only 1)
I believe the lease should be recovered and the block should be marked
missing. However this is not happening. The lease is never recovered.
The effect of this bug for us was that nodes could not be decommissioned
cleanly. Although we knew that the client had crashed, the Namenode never
released the leases (even after restarting the Namenode) (even months
afterwards). There are actually several other cases too where we don't
consider what happens if ALL the datanodes die while the file is being
written, but I am going to punt on that for another time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-8435) createNonRecursive support needed in WebHdfsFileSystem to support HBase
[ https://issues.apache.org/jira/browse/HDFS-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan reassigned HDFS-8435: - Assignee: Jakob Homan createNonRecursive support needed in WebHdfsFileSystem to support HBase --- Key: HDFS-8435 URL: https://issues.apache.org/jira/browse/HDFS-8435 Project: Hadoop HDFS Issue Type: Improvement Components: webhdfs Reporter: Vinoth Sathappan Assignee: Jakob Homan The WebHdfsFileSystem implementation doesn't support createNonRecursive. HBase extensively depends on that for proper functioning. Currently, when the region servers are started over web hdfs, they crash due with - createNonRecursive unsupported for this filesystem class org.apache.hadoop.hdfs.web.SWebHdfsFileSystem at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1137) at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1112) at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1088) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:85) at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWriter(HLogFactory.java:198) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8447) Decouple information of files in GetLocatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-8447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8447:
-
Issue Type: Improvement (was: Bug)
Decouple information of files in GetLocatedBlocks
-
Key: HDFS-8447
URL: https://issues.apache.org/jira/browse/HDFS-8447
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-8447.000.patch
The current implementation of {{BlockManager.getLocatedBlocks()}} requires
the information of files to be passed as parameters. These information does
not affect the results of getting the physical locations of blocks.
This jira proposes to refactor the call so that
{{BlockManager.getLocatedBlocks()}} depends only on the block information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8447) Decouple information of files in GetLocatedBlocks
Haohui Mai created HDFS-8447:
Summary: Decouple information of files in GetLocatedBlocks
Key: HDFS-8447
URL: https://issues.apache.org/jira/browse/HDFS-8447
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
The current implementation of {{BlockManager.getLocatedBlocks()}} requires the
information of files to be passed as parameters. These information does not
affect the results of getting the physical locations of blocks.
This jira proposes to refactor the call so that
{{BlockManager.getLocatedBlocks()}} depends only on the block information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8447) Decouple information of files in GetLocatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-8447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8447:
-
Attachment: HDFS-8447.000.patch
Decouple information of files in GetLocatedBlocks
-
Key: HDFS-8447
URL: https://issues.apache.org/jira/browse/HDFS-8447
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-8447.000.patch
The current implementation of {{BlockManager.getLocatedBlocks()}} requires
the information of files to be passed as parameters. These information does
not affect the results of getting the physical locations of blocks.
This jira proposes to refactor the call so that
{{BlockManager.getLocatedBlocks()}} depends only on the block information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8447) Decouple information of files in GetLocatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-8447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8447:
-
Status: Patch Available (was: Open)
Decouple information of files in GetLocatedBlocks
-
Key: HDFS-8447
URL: https://issues.apache.org/jira/browse/HDFS-8447
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-8447.000.patch
The current implementation of {{BlockManager.getLocatedBlocks()}} requires
the information of files to be passed as parameters. These information does
not affect the results of getting the physical locations of blocks.
This jira proposes to refactor the call so that
{{BlockManager.getLocatedBlocks()}} depends only on the block information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8446) Separate safemode related operations in GetBlockLocations()
[
https://issues.apache.org/jira/browse/HDFS-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553328#comment-14553328
]
Hadoop QA commented on HDFS-8446:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 36s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 51s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 6s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 16s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 164m 25s | Tests failed in hadoop-hdfs. |
| | | 205m 59s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestReservedRawPaths |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734210/HDFS-8446.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4aa730c |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11064/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11064/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11064/console |
This message was automatically generated.
Separate safemode related operations in GetBlockLocations()
---
Key: HDFS-8446
URL: https://issues.apache.org/jira/browse/HDFS-8446
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Attachments: HDFS-8446.000.patch
Currently {{FSNamesystem#GetBlockLocations()}} has some special cases when
the NN is in SafeMode. This jira proposes to refactor the code to improve
readability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8448) Create REST Interface for Volumes
[ https://issues.apache.org/jira/browse/HDFS-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-8448: --- Attachment: hdfs-8448-hdfs-7240.001.patch This patch creates a Volume Interface for the Hadoop Object Store. Create REST Interface for Volumes - Key: HDFS-8448 URL: https://issues.apache.org/jira/browse/HDFS-8448 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anu Engineer Assignee: Anu Engineer Attachments: hdfs-8448-hdfs-7240.001.patch Create REST interfaces as specified in the architecture document. This Jira is for creating the Volume Interface -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8448) Create REST Interface for Volumes
[ https://issues.apache.org/jira/browse/HDFS-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-8448: --- Status: Patch Available (was: Open) Create REST Interface for Volumes - Key: HDFS-8448 URL: https://issues.apache.org/jira/browse/HDFS-8448 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anu Engineer Assignee: Anu Engineer Attachments: hdfs-8448-hdfs-7240.001.patch Create REST interfaces as specified in the architecture document. This Jira is for creating the Volume Interface -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553511#comment-14553511
]
Walter Su commented on HDFS-8441:
-
... Maybe we could add a method in {{FSDirectory#isInECZone(src)}} and use it
to tell the condition?
Good idea. I found {{dir#isInECZone(iip)}} exists. So I add
{{ns#isInEcZone(src)}}. I think usually {{ns}} takes {{src}} arg, and {{dir}}
takes {{iip}}.
... Please add message to the fail() statement like...
Done.
Thank you both. Uploaded 003 patch.
Erasure Coding: make condition check earlier for setReplication
---
Key: HDFS-8441
URL: https://issues.apache.org/jira/browse/HDFS-8441
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8441-HDFS-7285.001.patch,
HDFS-8441-HDFS-7285.002.patch, HDFS-8441-HDFS-7285.003.patch
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update storage count
3. support create EC file with replication=0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8443) Document dfs.namenode.service.handler.count in hdfs-site.xml
[ https://issues.apache.org/jira/browse/HDFS-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553575#comment-14553575 ] J.Andreina commented on HDFS-8443: -- Thanks [~arpitagarwal] for the review comments. Please review the updated patch. Document dfs.namenode.service.handler.count in hdfs-site.xml Key: HDFS-8443 URL: https://issues.apache.org/jira/browse/HDFS-8443 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Reporter: Akira AJISAKA Assignee: J.Andreina Attachments: HDFS-8443.1.patch When dfs.namenode.servicerpc-address is configured, NameNode launches an extra RPC server to handle requests from non-client nodes. dfs.namenode.service.handler.count specifies the number of threads for the server but the parameter is not documented anywhere. I found a mail for asking about the parameter. http://mail-archives.apache.org/mod_mbox/hadoop-user/201505.mbox/%3CE0D5A619-BDEA-44D2-81EB-C32B8464133D%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8344) NameNode doesn't recover lease for files with missing blocks
[
https://issues.apache.org/jira/browse/HDFS-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553625#comment-14553625
]
Hadoop QA commented on HDFS-8344:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 56s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 33s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 22s | The applied patch generated 3
new checkstyle issues (total was 488, now 489). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 10s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 17s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 164m 35s | Tests failed in hadoop-hdfs. |
| | | 208m 17s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestAppendSnapshotTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734291/HDFS-8344.03.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6329bd0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11066/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11066/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11066/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11066/console |
This message was automatically generated.
NameNode doesn't recover lease for files with missing blocks
Key: HDFS-8344
URL: https://issues.apache.org/jira/browse/HDFS-8344
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Attachments: HDFS-8344.01.patch, HDFS-8344.02.patch,
HDFS-8344.03.patch
I found another\(?) instance in which the lease is not recovered. This is
reproducible easily on a pseudo-distributed single node cluster
# Before you start it helps if you set. This is not necessary, but simply
reduces how long you have to wait
{code}
public static final long LEASE_SOFTLIMIT_PERIOD = 30 * 1000;
public static final long LEASE_HARDLIMIT_PERIOD = 2 *
LEASE_SOFTLIMIT_PERIOD;
{code}
# Client starts to write a file. (could be less than 1 block, but it hflushed
so some of the data has landed on the datanodes) (I'm copying the client code
I am using. I generate a jar and run it using $ hadoop jar TestHadoop.jar)
# Client crashes. (I simulate this by kill -9 the $(hadoop jar
TestHadoop.jar) process after it has printed Wrote to the bufferedWriter
# Shoot the datanode. (Since I ran on a pseudo-distributed cluster, there was
only 1)
I believe the lease should be recovered and the block should be marked
missing. However this is not happening. The lease is never recovered.
The effect of this bug for us was that nodes could not be decommissioned
cleanly. Although we knew that the client had crashed, the Namenode never
released the leases (even after restarting the Namenode) (even months
afterwards). There are actually several other cases too where we don't
consider what happens if ALL the datanodes die while the file is being
written, but I am going to punt on that for another time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8210) Ozone: Implement storage container manager
[ https://issues.apache.org/jira/browse/HDFS-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-8210: --- Status: Open (was: Patch Available) Ozone: Implement storage container manager --- Key: HDFS-8210 URL: https://issues.apache.org/jira/browse/HDFS-8210 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-8210-HDFS-7240.1.patch, HDFS-8210-HDFS-7240.2.patch, HDFS-8210-HDFS-7240.3.patch, HDFS-8210-HDFS-7240.4.patch The storage container manager collects datanode heartbeats, manages replication and exposes API to lookup containers. This jira implements storage container manager by re-using the block manager implementation in namenode. This jira provides initial implementation that works with datanodes. The additional protocols will be added in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8210) Ozone: Implement storage container manager
[ https://issues.apache.org/jira/browse/HDFS-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-8210: --- Attachment: HDFS-8210-HDFS-7240.4.patch The attached patch addresses the two style issues. Ozone: Implement storage container manager --- Key: HDFS-8210 URL: https://issues.apache.org/jira/browse/HDFS-8210 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-8210-HDFS-7240.1.patch, HDFS-8210-HDFS-7240.2.patch, HDFS-8210-HDFS-7240.3.patch, HDFS-8210-HDFS-7240.4.patch The storage container manager collects datanode heartbeats, manages replication and exposes API to lookup containers. This jira implements storage container manager by re-using the block manager implementation in namenode. This jira provides initial implementation that works with datanodes. The additional protocols will be added in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8210) Ozone: Implement storage container manager
[ https://issues.apache.org/jira/browse/HDFS-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-8210: --- Status: Patch Available (was: Open) Ozone: Implement storage container manager --- Key: HDFS-8210 URL: https://issues.apache.org/jira/browse/HDFS-8210 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-8210-HDFS-7240.1.patch, HDFS-8210-HDFS-7240.2.patch, HDFS-8210-HDFS-7240.3.patch, HDFS-8210-HDFS-7240.4.patch The storage container manager collects datanode heartbeats, manages replication and exposes API to lookup containers. This jira implements storage container manager by re-using the block manager implementation in namenode. This jira provides initial implementation that works with datanodes. The additional protocols will be added in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8441:
Attachment: HDFS-8441-HDFS-7285.003.patch
Erasure Coding: make condition check earlier for setReplication
---
Key: HDFS-8441
URL: https://issues.apache.org/jira/browse/HDFS-8441
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8441-HDFS-7285.001.patch,
HDFS-8441-HDFS-7285.002.patch, HDFS-8441-HDFS-7285.003.patch
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update storage count
3. support create EC file with replication=0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553526#comment-14553526
]
Kai Zheng commented on HDFS-8441:
-
Thanks for the update. Could we skip the lock and operation check stuffs since
it's private and could be expected to be only used internally?
{code}
+ private boolean isInECZone(String src) throws IOException {
+byte[][] pathComponents =
FSDirectory.getPathComponentsForReservedPath(src);
+readLock();
+try {
+ checkOperation(OperationCategory.READ);
+ src = FSDirectory.resolvePath(src, pathComponents, dir);
+ final INodesInPath iip = dir.getINodesInPath(src, true);
+ return dir.isInECZone(iip);
+} finally {
+ readUnlock();
+}
+ }
+
{code}
Erasure Coding: make condition check earlier for setReplication
---
Key: HDFS-8441
URL: https://issues.apache.org/jira/browse/HDFS-8441
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8441-HDFS-7285.001.patch,
HDFS-8441-HDFS-7285.002.patch, HDFS-8441-HDFS-7285.003.patch
Changes:
1. {{UnsupportedActionException}} is more user-firendly.
2. check condition before update storage count
3. support create EC file with replication=0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8294) Erasure Coding: Fix Findbug warnings present in erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-8294: --- Attachment: HDFS-8294-HDFS-7285.06.patch Erasure Coding: Fix Findbug warnings present in erasure coding -- Key: HDFS-8294 URL: https://issues.apache.org/jira/browse/HDFS-8294 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Rakesh R Assignee: Rakesh R Labels: BB2015-05-RFC Attachments: FindBugs Report in EC feature.html, HDFS-8294-HDFS-7285.00.patch, HDFS-8294-HDFS-7285.01.patch, HDFS-8294-HDFS-7285.02.patch, HDFS-8294-HDFS-7285.03.patch, HDFS-8294-HDFS-7285.04.patch, HDFS-8294-HDFS-7285.05.patch, HDFS-8294-HDFS-7285.06.patch This jira is to address the findbug issues reported in erasure coding feature. Attached sheet which contains the details of the findbug issues reported in the erasure coding feature. I've taken this report from the jenkins build : https://builds.apache.org/job/PreCommit-HDFS-Build/10848/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8020) Erasure Coding: restore BlockGroup and schema info from stripping coding command
[ https://issues.apache.org/jira/browse/HDFS-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553568#comment-14553568 ] Uma Maheswara Rao G commented on HDFS-8020: --- Do we need this JIRA still? we are already sending EC Command to DN and DN already getting required information from it. So, can we we resolve this? Erasure Coding: restore BlockGroup and schema info from stripping coding command Key: HDFS-8020 URL: https://issues.apache.org/jira/browse/HDFS-8020 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Sasaki As a task of HDFS-7344, to process *stripping* coding commands from NameNode or other scheduler services/tools, we need to first be able to restore BlockGroup and schema information in DataNode, which will be used to construct coding work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8020) Erasure Coding: restore BlockGroup and schema info from stripping coding command
[ https://issues.apache.org/jira/browse/HDFS-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-8020: Parent Issue: HDFS-8031 (was: HDFS-7285) Erasure Coding: restore BlockGroup and schema info from stripping coding command Key: HDFS-8020 URL: https://issues.apache.org/jira/browse/HDFS-8020 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Sasaki As a task of HDFS-7344, to process *stripping* coding commands from NameNode or other scheduler services/tools, we need to first be able to restore BlockGroup and schema information in DataNode, which will be used to construct coding work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8442) Remove ServerLifecycleListener from kms/server.xml.
[
https://issues.apache.org/jira/browse/HDFS-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553619#comment-14553619
]
Hadoop QA commented on HDFS-8442:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 40s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | common tests | 1m 38s | Tests passed in
hadoop-kms. |
| | | 36m 7s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734324/HDFS-8442-1.patch |
| Optional Tests | javadoc javac unit |
| git revision | trunk / fb6b38d |
| hadoop-kms test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11073/artifact/patchprocess/testrun_hadoop-kms.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11073/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11073/console |
This message was automatically generated.
Remove ServerLifecycleListener from kms/server.xml.
---
Key: HDFS-8442
URL: https://issues.apache.org/jira/browse/HDFS-8442
Project: Hadoop HDFS
Issue Type: Bug
Reporter: nijel
Assignee: nijel
Attachments: HDFS-8442-1.patch
Remove ServerLifecycleListener from kms/server.xml.
From tomcat Tomcat 7.0.9 the support for ServerLifecycleListener is removed
ref : https://tomcat.apache.org/tomcat-7.0-doc/changelog.html
Remove ServerLifecycleListener. This was already removed from server.xml and
with the Lifecycle re-factoring is no longer required. (markt)
So if the build env is with tomcat later than this, kms startup is failing
{code}
SEVERE: Begin event threw exception
java.lang.ClassNotFoundException:
org.apache.catalina.mbeans.ServerLifecycleListener
{code}
can we remove this listener ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8439) Adding more slow action log in critical read path
[
https://issues.apache.org/jira/browse/HDFS-8439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553577#comment-14553577
]
Liu Shaohui commented on HDFS-8439:
---
[~stack]
{quote}
Are you running this in production?
{quote}
Just in test cluster currently. We are trying to push this to production.
Adding more slow action log in critical read path
-
Key: HDFS-8439
URL: https://issues.apache.org/jira/browse/HDFS-8439
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Attachments: HDFS-8439.001.patch, HDFS-8439.002.patch
To dig a HBase read spike issue, we'd better to add more slow pread/seek log
in read flow to get the abnormal datanodes.
Patch will be uploaded soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8439) Adding more slow action log in critical read path
[ https://issues.apache.org/jira/browse/HDFS-8439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HDFS-8439: -- Attachment: HDFS-8439.002.patch Update for @stack's review - replace cost with duration - Using Time.monotonicNow() instead of System.currentTimeMillis() Adding more slow action log in critical read path - Key: HDFS-8439 URL: https://issues.apache.org/jira/browse/HDFS-8439 Project: Hadoop HDFS Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HDFS-8439.001.patch, HDFS-8439.002.patch To dig a HBase read spike issue, we'd better to add more slow pread/seek log in read flow to get the abnormal datanodes. Patch will be uploaded soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8443) Document dfs.namenode.service.handler.count in hdfs-site.xml
[ https://issues.apache.org/jira/browse/HDFS-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8443: - Attachment: HDFS-8443.2.patch Document dfs.namenode.service.handler.count in hdfs-site.xml Key: HDFS-8443 URL: https://issues.apache.org/jira/browse/HDFS-8443 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Reporter: Akira AJISAKA Assignee: J.Andreina Attachments: HDFS-8443.1.patch, HDFS-8443.2.patch When dfs.namenode.servicerpc-address is configured, NameNode launches an extra RPC server to handle requests from non-client nodes. dfs.namenode.service.handler.count specifies the number of threads for the server but the parameter is not documented anywhere. I found a mail for asking about the parameter. http://mail-archives.apache.org/mod_mbox/hadoop-user/201505.mbox/%3CE0D5A619-BDEA-44D2-81EB-C32B8464133D%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8442) Remove ServerLifecycleListener from kms/server.xml.
[
https://issues.apache.org/jira/browse/HDFS-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nijel updated HDFS-8442:
Status: Patch Available (was: Open)
Remove ServerLifecycleListener from kms/server.xml.
---
Key: HDFS-8442
URL: https://issues.apache.org/jira/browse/HDFS-8442
Project: Hadoop HDFS
Issue Type: Bug
Reporter: nijel
Assignee: nijel
Attachments: HDFS-8442-1.patch
Remove ServerLifecycleListener from kms/server.xml.
From tomcat Tomcat 7.0.9 the support for ServerLifecycleListener is removed
ref : https://tomcat.apache.org/tomcat-7.0-doc/changelog.html
Remove ServerLifecycleListener. This was already removed from server.xml and
with the Lifecycle re-factoring is no longer required. (markt)
So if the build env is with tomcat later than this, kms startup is failing
{code}
SEVERE: Begin event threw exception
java.lang.ClassNotFoundException:
org.apache.catalina.mbeans.ServerLifecycleListener
{code}
can we remove this listener ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8442) Remove ServerLifecycleListener from kms/server.xml.
[
https://issues.apache.org/jira/browse/HDFS-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nijel updated HDFS-8442:
Attachment: HDFS-8442-1.patch
please find the patch
Remove ServerLifecycleListener from kms/server.xml.
---
Key: HDFS-8442
URL: https://issues.apache.org/jira/browse/HDFS-8442
Project: Hadoop HDFS
Issue Type: Bug
Reporter: nijel
Assignee: nijel
Attachments: HDFS-8442-1.patch
Remove ServerLifecycleListener from kms/server.xml.
From tomcat Tomcat 7.0.9 the support for ServerLifecycleListener is removed
ref : https://tomcat.apache.org/tomcat-7.0-doc/changelog.html
Remove ServerLifecycleListener. This was already removed from server.xml and
with the Lifecycle re-factoring is no longer required. (markt)
So if the build env is with tomcat later than this, kms startup is failing
{code}
SEVERE: Begin event threw exception
java.lang.ClassNotFoundException:
org.apache.catalina.mbeans.ServerLifecycleListener
{code}
can we remove this listener ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8441) Erasure Coding: make condition check earlier for setReplication
[
https://issues.apache.org/jira/browse/HDFS-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553586#comment-14553586
]
Hadoop QA commented on HDFS-8441:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 23s | Pre-patch HDFS-7285 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 46s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 54s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 14s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 38s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 17s | The patch appears to introduce 6
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 18s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 48m 33s | Tests failed in hadoop-hdfs. |
| | | 91m 16s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.DFSOutputStream.streamer; locked 89% of time
Unsynchronized access at DFSOutputStream.java:89% of time Unsynchronized
access at DFSOutputStream.java:[line 146] |
| | Possible null pointer dereference of arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:[line 194] |
| | Unread field:field be static? At ErasureCodingWorker.java:[line 254] |
| | Should
org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$StripedReader
be a _static_ inner class? At ErasureCodingWorker.java:inner class? At
ErasureCodingWorker.java:[lines 907-914] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:[line 108] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:[line 409] |
| Failed unit tests | hadoop.hdfs.server.blockmanagement.TestDatanodeManager |
| | hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots |
| | hadoop.hdfs.TestModTime |
| | hadoop.fs.TestUrlStreamHandler |
| | hadoop.hdfs.security.TestDelegationToken |
| | hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolarent |
| | hadoop.hdfs.server.namenode.TestFileLimit |
| | hadoop.hdfs.TestParallelShortCircuitRead |
| | hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfoStriped |
| | hadoop.hdfs.server.namenode.TestEditLogAutoroll |
| | hadoop.TestRefreshCallQueue |
| | hadoop.hdfs.protocolPB.TestPBHelper |
| | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.cli.TestCryptoAdminCLI |
| | hadoop.hdfs.TestSetrepDecreasing |
| | hadoop.hdfs.server.datanode.TestDiskError |
| | hadoop.fs.viewfs.TestViewFsWithAcls |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.server.namenode.TestHostsFiles |
| | hadoop.hdfs.server.datanode.TestTransferRbw |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy |
| | hadoop.fs.contract.hdfs.TestHDFSContractDelete |
| | hadoop.hdfs.server.namenode.TestFileContextAcl |
| | hadoop.fs.TestFcHdfsSetUMask |
| | hadoop.fs.TestUnbuffer |
| | hadoop.hdfs.server.namenode.TestClusterId |
| | hadoop.hdfs.server.namenode.TestDeleteRace |
| | hadoop.hdfs.TestPread |
| | hadoop.hdfs.server.namenode.TestFSDirectory |
|
[jira] [Commented] (HDFS-8407) libhdfs hdfsListDirectory() API has different behavior than documentation
[
https://issues.apache.org/jira/browse/HDFS-8407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553640#comment-14553640
]
Juan Yu commented on HDFS-8407:
---
I think move errno = ret; before the if check will cover both cases, right?
libhdfs hdfsListDirectory() API has different behavior than documentation
-
Key: HDFS-8407
URL: https://issues.apache.org/jira/browse/HDFS-8407
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS
Reporter: Juan Yu
Assignee: Masatake Iwasaki
Attachments: HDFS-8407.001.patch, HDFS-8407.002.patch,
HDFS-8407.003.patch
The documentation says it returns NULL on error, but it could also return
NULL when the directory is empty.
/**
* hdfsListDirectory - Get list of files/directories for a given
* directory-path. hdfsFreeFileInfo should be called to deallocate
memory.
* @param fs The configured filesystem handle.
* @param path The path of the directory.
* @param numEntries Set to the number of files/directories in path.
* @return Returns a dynamically-allocated array of hdfsFileInfo
* objects; NULL on error.
*/
{code}
hdfsFileInfo *pathList = NULL;
...
//Figure out the number of entries in that directory
jPathListSize = (*env)-GetArrayLength(env, jPathList);
if (jPathListSize == 0) {
ret = 0;
goto done;
}
...
if (ret) {
hdfsFreeFileInfo(pathList, jPathListSize);
errno = ret;
return NULL;
}
*numEntries = jPathListSize;
return pathList;
{code}
Either change the implementation to match the doc, or fix the doc to match
the implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8446) Separate safemode related operations in GetBlockLocations()
[
https://issues.apache.org/jira/browse/HDFS-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8446:
-
Attachment: HDFS-8446.001.patch
Separate safemode related operations in GetBlockLocations()
---
Key: HDFS-8446
URL: https://issues.apache.org/jira/browse/HDFS-8446
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Attachments: HDFS-8446.000.patch, HDFS-8446.001.patch
Currently {{FSNamesystem#GetBlockLocations()}} has some special cases when
the NN is in SafeMode. This jira proposes to refactor the code to improve
readability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8020) Erasure Coding: restore BlockGroup and schema info from stripping coding command
[
https://issues.apache.org/jira/browse/HDFS-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553574#comment-14553574
]
Kai Zheng commented on HDFS-8020:
-
Hi Uma, I think we might still need this to make use of the {{ErasureCoder}}
API in next phase. Yes currently it works because we're using the
{{RawErasureCoder}} API directly.
Erasure Coding: restore BlockGroup and schema info from stripping coding
command
Key: HDFS-8020
URL: https://issues.apache.org/jira/browse/HDFS-8020
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Sasaki
As a task of HDFS-7344, to process *stripping* coding commands from NameNode
or other scheduler services/tools, we need to first be able to restore
BlockGroup and schema information in DataNode, which will be used to
construct coding work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
