[jira] [Comment Edited] (HDFS-8898) Create API and command-line argument to get quota and quota usage without detailed content summary
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112515#comment-15112515 ] Kihwal Lee edited comment on HDFS-8898 at 1/22/16 3:16 PM: --- The patch looks to be a correct port of the trunk patch. was (Author: kihwal): The patch looks to be a correct port of the patch. > Create API and command-line argument to get quota and quota usage without > detailed content summary > -- > > Key: HDFS-8898 > URL: https://issues.apache.org/jira/browse/HDFS-8898 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Joep Rottinghuis >Assignee: Ming Ma > Attachments: HDFS-8898-2.patch, HDFS-8898-3.patch, HDFS-8898-4.patch, > HDFS-8898-5.patch, HDFS-8898-6.patch, HDFS-8898-branch-2.patch, > HDFS-8898.patch > > > On large directory structures it takes significant time to iterate through > the file and directory counts recursively to get a complete ContentSummary. > When you want to just check for the quota on a higher level directory it > would be good to have an option to skip the file and directory counts. > Moreover, currently one can only check the quota if you have access to all > the directories underneath. For example, if I have a large home directory > under /user/joep and I host some files for another user in a sub-directory, > the moment they create an unreadable sub-directory under my home I can no > longer check what my quota is. Understood that I cannot check the current > file counts unless I can iterate through all the usage, but for > administrative purposes it is nice to be able to get the current quota > setting on a directory without the need to iterate through and run into > permission issues on sub-directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8898) Create API and command-line argument to get quota and quota usage without detailed content summary
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112532#comment-15112532 ] Kihwal Lee commented on HDFS-8898: -- I've committed this to trunk, branch-2 and branch-2.8. Thanks for working on this Ming, and Vinay for providing the initial review. > Create API and command-line argument to get quota and quota usage without > detailed content summary > -- > > Key: HDFS-8898 > URL: https://issues.apache.org/jira/browse/HDFS-8898 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Joep Rottinghuis >Assignee: Ming Ma > Fix For: 2.8.0 > > Attachments: HDFS-8898-2.patch, HDFS-8898-3.patch, HDFS-8898-4.patch, > HDFS-8898-5.patch, HDFS-8898-6.patch, HDFS-8898-branch-2.patch, > HDFS-8898.patch > > > On large directory structures it takes significant time to iterate through > the file and directory counts recursively to get a complete ContentSummary. > When you want to just check for the quota on a higher level directory it > would be good to have an option to skip the file and directory counts. > Moreover, currently one can only check the quota if you have access to all > the directories underneath. For example, if I have a large home directory > under /user/joep and I host some files for another user in a sub-directory, > the moment they create an unreadable sub-directory under my home I can no > longer check what my quota is. Understood that I cannot check the current > file counts unless I can iterate through all the usage, but for > administrative purposes it is nice to be able to get the current quota > setting on a directory without the need to iterate through and run into > permission issues on sub-directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9172) Erasure Coding: Move DFSStripedIO stream related classes to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112531#comment-15112531
]
Rakesh R commented on HDFS-9172:
bq. Just did 'git merge':
https://github.com/zhe-thoughts/hadoop/tree/HDFS-7285-20150929
Thank you for taking care this [~zhz].
bq. {{ErasureCodingWorker#newBlockReader}} I hard-coded tracer to null.
Its done, please refer
[ErasureCodingWorker.java#L816|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java#L816].
Any other non-ideal code changes required to do in this jira, do you have
anything in your mind?
> Erasure Coding: Move DFSStripedIO stream related classes to hadoop-hdfs-client
> --
>
> Key: HDFS-9172
> URL: https://issues.apache.org/jira/browse/HDFS-9172
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Zhe Zhang
>
> The idea of this jira is to move the striped stream related classes to
> {{hadoop-hdfs-client}} project. This will help to be in sync with the
> HDFS-6200 proposal.
> - DFSStripedInputStream
> - DFSStripedOutputStream
> - StripedDataStreamer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[ https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9643: -- Attachment: HDFS-8643.HDFS-8707.004.patch Thanks for taking another look and catching that cancel call [~bobhansen]. I've changed it to a shutdown() + close(). The docs say calling shutdown first should keep things more portable. > libhdfs++: Support async cancellation of read operations > > > Key: HDFS-9643 > URL: https://issues.apache.org/jira/browse/HDFS-9643 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-8643.HDFS-8707.004.patch, > HDFS-9643.HDFS-8707.000.patch, HDFS-9643.HDFS-8707.001.patch, > HDFS-9643.HDFS-8707.002.patch, HDFS-9643.HDFS-8707.003.patch > > > It should be possible for any thread to cancel operations in progress on a > FileHandle. Any ephemeral objects created by the FileHandle should free > resources as quickly as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8898) Create API and command-line argument to get quota and quota usage without detailed content summary
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8898: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) > Create API and command-line argument to get quota and quota usage without > detailed content summary > -- > > Key: HDFS-8898 > URL: https://issues.apache.org/jira/browse/HDFS-8898 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Joep Rottinghuis >Assignee: Ming Ma > Fix For: 2.8.0 > > Attachments: HDFS-8898-2.patch, HDFS-8898-3.patch, HDFS-8898-4.patch, > HDFS-8898-5.patch, HDFS-8898-6.patch, HDFS-8898-branch-2.patch, > HDFS-8898.patch > > > On large directory structures it takes significant time to iterate through > the file and directory counts recursively to get a complete ContentSummary. > When you want to just check for the quota on a higher level directory it > would be good to have an option to skip the file and directory counts. > Moreover, currently one can only check the quota if you have access to all > the directories underneath. For example, if I have a large home directory > under /user/joep and I host some files for another user in a sub-directory, > the moment they create an unreadable sub-directory under my home I can no > longer check what my quota is. Understood that I cannot check the current > file counts unless I can iterate through all the usage, but for > administrative purposes it is nice to be able to get the current quota > setting on a directory without the need to iterate through and run into > permission issues on sub-directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9525) hadoop utilities need to support provided delegation tokens
[
https://issues.apache.org/jira/browse/HDFS-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112969#comment-15112969
]
Hadoop QA commented on HDFS-9525:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 37s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 24s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 50s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 17s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 15s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 57s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 52s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 1s
{color} | {color:red} root: patch generated 1 new + 341 unchanged - 0 fixed =
342 total (was 341) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 32s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 51s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 2s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
[jira] [Updated] (HDFS-9525) hadoop utilities need to support provided delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-9525: --- Resolution: Fixed Status: Resolved (was: Patch Available) Feedback has been addressed. Committing to trunk. Thanks all! > hadoop utilities need to support provided delegation tokens > --- > > Key: HDFS-9525 > URL: https://issues.apache.org/jira/browse/HDFS-9525 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: HeeSoo Kim >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HDFS-7984.001.patch, HDFS-7984.002.patch, > HDFS-7984.003.patch, HDFS-7984.004.patch, HDFS-7984.005.patch, > HDFS-7984.006.patch, HDFS-7984.007.patch, HDFS-7984.patch, > HDFS-9525.008.patch, HDFS-9525.009.patch, HDFS-9525.009.patch, > HDFS-9525.branch-2.008.patch, HDFS-9525.branch-2.009.patch > > > When using the webhdfs:// filesystem (especially from distcp), we need the > ability to inject a delegation token rather than webhdfs initialize its own. > This would allow for cross-authentication-zone file system accesses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9525) hadoop utilities need to support provided delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112999#comment-15112999 ] Hudson commented on HDFS-9525: -- FAILURE: Integrated in Hadoop-trunk-Commit #9170 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9170/]) HDFS-9525. hadoop utilities need to support provided delegation tokens (aw: rev d22c4239a40a1c7ed49c06038138f0e3f387b4a0) * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/DelegationParam.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java > hadoop utilities need to support provided delegation tokens > --- > > Key: HDFS-9525 > URL: https://issues.apache.org/jira/browse/HDFS-9525 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: HeeSoo Kim >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HDFS-7984.001.patch, HDFS-7984.002.patch, > HDFS-7984.003.patch, HDFS-7984.004.patch, HDFS-7984.005.patch, > HDFS-7984.006.patch, HDFS-7984.007.patch, HDFS-7984.patch, > HDFS-9525.008.patch, HDFS-9525.009.patch, HDFS-9525.009.patch, > HDFS-9525.branch-2.008.patch, HDFS-9525.branch-2.009.patch > > > When using the webhdfs:// filesystem (especially from distcp), we need the > ability to inject a delegation token rather than webhdfs initialize its own. > This would allow for cross-authentication-zone file system accesses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9653) Expose the number of blocks pending deletion through dfsadmin report command
[ https://issues.apache.org/jira/browse/HDFS-9653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113129#comment-15113129 ] Eric Yang commented on HDFS-9653: - WeiWei, In rolling upgrade, the new client may talk to old version of name node. If we can handle the missing data gracefully on the client side, it would be better. > Expose the number of blocks pending deletion through dfsadmin report command > > > Key: HDFS-9653 > URL: https://issues.apache.org/jira/browse/HDFS-9653 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, tools >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-9653-branch-2.001.patch, HDFS-9653.001.patch > > > HDFS-5986 adds *Number of Blocks Pending Deletion* on namenode UI and JMX, > propose to expose this from hdfs dfsadmin -report as well. This is useful > when hadoop admin was not able to access UI (e.g on cloud), he/she can > directly use command to retrieve this information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9260) Improve performance and GC friendliness of startup and FBRs
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Staffan Friberg updated HDFS-9260: -- Attachment: HDFS-9260.012.patch > Improve performance and GC friendliness of startup and FBRs > --- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Attachments: FBR processing.png, HDFS Block and Replica Management > 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch, > HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch, > HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch, > HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch, > HDFSBenchmarks.zip, HDFSBenchmarks2.zip > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9525) hadoop utilities need to support provided delegation tokens
[
https://issues.apache.org/jira/browse/HDFS-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113149#comment-15113149
]
Daryn Sharp commented on HDFS-9525:
---
-1 No, feedback was not addressed, a bug was introduced, and the tests were
changed to verify the new bug occurs. Strikethru on the one point addressed.
bq. -If a code change is necessary, UGI should use
Configuration#getTrimmedStrings- and unconditionally call
Credentials.readTokenStorageFile instead of allowing the user to specify an
invalid setting. Only webhdfs related change is
WebHdfsFileSystem.canRefreshDelegationToken should default to true.
The last and most important point was overlooked and webhdfs is broken. The
tests used to:
# call getfilestatus and verify a token is sent
# clear the token with the comment {{// wipe out internal token to simulate
auth always required}}
# call getfilestatus again to specifically verify no token is sent - because
auth is expected
This patch changed #3 to verify the opposite behavior: the same token as #1 is
passed. Again, just change {{this.canRefreshDelegationToken =
UserGroupInformation.isSecurityEnabled();}} to {{this.canRefreshDelegationToken
= true;}} and it will cause webhdfs to look for a token even if security is
off. Nothing else in webhdfs should require a change.
> hadoop utilities need to support provided delegation tokens
> ---
>
> Key: HDFS-9525
> URL: https://issues.apache.org/jira/browse/HDFS-9525
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: HeeSoo Kim
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HDFS-7984.001.patch, HDFS-7984.002.patch,
> HDFS-7984.003.patch, HDFS-7984.004.patch, HDFS-7984.005.patch,
> HDFS-7984.006.patch, HDFS-7984.007.patch, HDFS-7984.patch,
> HDFS-9525.008.patch, HDFS-9525.009.patch, HDFS-9525.009.patch,
> HDFS-9525.branch-2.008.patch, HDFS-9525.branch-2.009.patch
>
>
> When using the webhdfs:// filesystem (especially from distcp), we need the
> ability to inject a delegation token rather than webhdfs initialize its own.
> This would allow for cross-authentication-zone file system accesses.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9262) Support reconfiguring dfs.datanode.lazywriter.interval.sec without DN restart
[
https://issues.apache.org/jira/browse/HDFS-9262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-9262:
Attachment: HDFS-9262-HDFS-9000.002.patch
V001 addressed your proposal, thanks [~arpitagarwal].
> Support reconfiguring dfs.datanode.lazywriter.interval.sec without DN restart
> -
>
> Key: HDFS-9262
> URL: https://issues.apache.org/jira/browse/HDFS-9262
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-9262-HDFS-9000.002.patch, HDFS-9262.001.patch
>
>
> This is to reconfigure
> {code}
> dfs.datanode.lazywriter.interval.sec
> {code}
> without restarting DN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9330) Support reconfiguring Reconfigure DN deleting duplicate replica on the fly
[
https://issues.apache.org/jira/browse/HDFS-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-9330:
Summary: Support reconfiguring Reconfigure DN deleting duplicate replica on
the fly (was: Reconfigure DN deleting duplicate replica on the fly)
> Support reconfiguring Reconfigure DN deleting duplicate replica on the fly
> --
>
> Key: HDFS-9330
> URL: https://issues.apache.org/jira/browse/HDFS-9330
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-9330.001.patch
>
>
> This is to reconfigure
> {code}
> dfs.datanode.duplicate.replica.deletion
> {code}
> without restarting DN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9330) Support reconfiguring dfs.datanode.duplicate.replica.deletion without DN restart
[
https://issues.apache.org/jira/browse/HDFS-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-9330:
Summary: Support reconfiguring dfs.datanode.duplicate.replica.deletion
without DN restart (was: Support reconfiguring Reconfigure DN deleting
duplicate replica on the fly)
> Support reconfiguring dfs.datanode.duplicate.replica.deletion without DN
> restart
> -
>
> Key: HDFS-9330
> URL: https://issues.apache.org/jira/browse/HDFS-9330
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-9330.001.patch
>
>
> This is to reconfigure
> {code}
> dfs.datanode.duplicate.replica.deletion
> {code}
> without restarting DN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9646) ErasureCodingWorker may fail when recovering data blocks with length less than the first internal block
[
https://issues.apache.org/jira/browse/HDFS-9646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112796#comment-15112796
]
Hudson commented on HDFS-9646:
--
FAILURE: Integrated in Hadoop-trunk-Commit #9169 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9169/])
HDFS-9646. ErasureCodingWorker may fail when recovering data blocks with
(jing9: rev 95363bcc7dae28ba9ae2cd7ee9a258fcb58cd932)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockECRecoveryCommand.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
> ErasureCodingWorker may fail when recovering data blocks with length less
> than the first internal block
> ---
>
> Key: HDFS-9646
> URL: https://issues.apache.org/jira/browse/HDFS-9646
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Takuya Fukudome
>Assignee: Jing Zhao
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HDFS-9646.000.patch, HDFS-9646.001.patch,
> HDFS-9646.002.patch, HDFS-9646.003.patch, HDFS-9646.004.patch,
> test-reconstruct-stripe-file.patch
>
>
> This is reported by [~tfukudom]: ErasureCodingWorker may fail with the
> following exception when recovering a non-full internal block.
> {code}
> 2016-01-06 11:14:44,740 WARN datanode.DataNode
> (ErasureCodingWorker.java:run(467)) - Failed to recover striped block:
> BP-987302662-172.29.4.13-1450757377698:blk_-92233720368
> 54322288_29751
> java.io.IOException: Transfer failed for all targets.
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$ReconstructAndTransferBlock.run(ErasureCodingWorker.java:455)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9674) The HTrace span for OpWriteBlock should record the maxWriteToDisk time
[ https://issues.apache.org/jira/browse/HDFS-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112873#comment-15112873 ] Colin Patrick McCabe commented on HDFS-9674: Thanks, [~zhz]. > The HTrace span for OpWriteBlock should record the maxWriteToDisk time > -- > > Key: HDFS-9674 > URL: https://issues.apache.org/jira/browse/HDFS-9674 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, tracing >Affects Versions: 2.4.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.9.0 > > Attachments: HTRACE-9674.001.patch > > > The HTrace span for OpWriteBlock should record the maxWriteToDisk time. That > way, if the trace span is long, we can figure out whether it is because of a > hard disk or I/O issue, or something else. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9686) Remove useless boxing/unboxing code (Hadoop HDFS)
Kousuke Saruta created HDFS-9686: Summary: Remove useless boxing/unboxing code (Hadoop HDFS) Key: HDFS-9686 URL: https://issues.apache.org/jira/browse/HDFS-9686 Project: Hadoop HDFS Issue Type: Improvement Components: performance Affects Versions: 3.0.0 Reporter: Kousuke Saruta Assignee: Kousuke Saruta Priority: Minor There are lots of places where useless boxing/unboxing occur. To avoid performance issue, let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9686) Remove useless boxing/unboxing code (Hadoop HDFS)
[ https://issues.apache.org/jira/browse/HDFS-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HDFS-9686: - Attachment: HDFS-9686.patch.0 I've attached the initial patch. > Remove useless boxing/unboxing code (Hadoop HDFS) > - > > Key: HDFS-9686 > URL: https://issues.apache.org/jira/browse/HDFS-9686 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > Attachments: HDFS-9686.patch.0 > > > There are lots of places where useless boxing/unboxing occur. > To avoid performance issue, let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9585) Erasure Coding: Wrong limit setting of target ByteBuffer
[ https://issues.apache.org/jira/browse/HDFS-9585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9585: Resolution: Fixed Status: Resolved (was: Patch Available) HDFS-9646 includes the fix. Thanks a lot for reporting the issue and providing the fix, [~lewuathe]! > Erasure Coding: Wrong limit setting of target ByteBuffer > > > Key: HDFS-9585 > URL: https://issues.apache.org/jira/browse/HDFS-9585 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Labels: EC > Attachments: HDFS-9585.01.patch > > > In ErasureCodingWorker, target ByteBuffer should be limit by desired > recovered size. But due to wrong indexing, an output ByteBuffer does not be > limited correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9260) Improve performance and GC friendliness of startup and FBRs
[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112866#comment-15112866
]
Colin Patrick McCabe commented on HDFS-9260:
bq. This comment is simply copied from the method
"processAndHandleReportedBlock" in the same class and not mine (doesn't show up
since I didn't edit that method). I kept it as part of the structure since I
wanted to make sure the algorithm behaves in the same way. So might be best to
address it in a separate bug.
That's fair.
bq. Yes this is a change in behavior compared to earlier. Started down this
path since add on a Set doesn't replace, which unfortunately doesn't match what
the Map API does. I added a "replace" method in the class to be used when a
replace behavior is needed and went through the code to ensure the right method
is called when needed. Not really happy about this choice, perhaps a cleaner
way would be to have a addWithReplace method on the TreeSet and keep the old
add behavior of the ReplicaMap. I believe it would reduce the size of the patch
and only add one "ugly" method on the TreeSet.
That's a good idea. Yes, please, let's have an {{addWithReplace}} method.
Otherwise, there is a risk that we will accidentally change the semantics of
something in an incorrect way (I notice a lot of cases where "add" turns into
"replace" in this patch and it makes me nervous).
I had some minor conflicts applying to trunk, I guess it needs a rebase anyway.
Thanks, [~sfriberg]. Very exciting to see this make progress.
> Improve performance and GC friendliness of startup and FBRs
> ---
>
> Key: HDFS-9260
> URL: https://issues.apache.org/jira/browse/HDFS-9260
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, performance
>Affects Versions: 2.7.1
>Reporter: Staffan Friberg
>Assignee: Staffan Friberg
> Attachments: FBR processing.png, HDFS Block and Replica Management
> 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch,
> HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch,
> HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch,
> HDFS-9260.010.patch, HDFS-9260.011.patch, HDFSBenchmarks.zip,
> HDFSBenchmarks2.zip
>
>
> This patch changes the datastructures used for BlockInfos and Replicas to
> keep them sorted. This allows faster and more GC friendly handling of full
> block reports.
> Would like to hear peoples feedback on this change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9573) SnapshotDiffReport$DiffReportEntry$hashCode should use type field
[
https://issues.apache.org/jira/browse/HDFS-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112792#comment-15112792
]
Hadoop QA commented on HDFS-9573:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 49s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783865/HDFS-9573-002.patch |
| JIRA Issue | HDFS-9573 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9ffcbd6102ee 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
[jira] [Commented] (HDFS-9668) Many long-time BLOCKED threads on FsDatasetImpl in a tiered storage test
[
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112852#comment-15112852
]
Colin Patrick McCabe commented on HDFS-9668:
I am -1 on creating a new FsDatasetSpi subclass for this. However, I am +1 on
the concept of fixing the locking. I think it just needs some reworking so
that it fits with the existing code.
> Many long-time BLOCKED threads on FsDatasetImpl in a tiered storage test
>
>
> Key: HDFS-9668
> URL: https://issues.apache.org/jira/browse/HDFS-9668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-9668-1.patch, execution_time.png
>
>
> During the HBase test on a tiered storage of HDFS (WAL is stored in
> SSD/RAMDISK, and all other files are stored in HDD), we observe many
> long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part
> of the jstack result:
> {noformat}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48521 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread
> t@93336
>java.lang.Thread.State: BLOCKED
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:)
> - waiting to lock <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
>
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread
> t@93335
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
> - locked <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
> {noformat}
> We measured the execution of some operations in FsDatasetImpl during the
> test. Here following is the result.
> !execution_time.png!
> The operations of finalizeBlock, addBlock and createRbw on HDD in a heavy
> load take a really long time.
> It means one slow operation of finalizeBlock, addBlock and createRbw in a
> slow storage can block all the other same operations in the same DataNode,
> especially in HBase when many wal/flusher/compactor are configured.
> We need a finer grained lock mechanism in a new FsDatasetImpl implementation
> and users can choose the implementation by configuring
> "dfs.datanode.fsdataset.factory" in DataNode.
> We can implement the lock by either storage level or block-level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9244) Support nested encryption zones
[
https://issues.apache.org/jira/browse/HDFS-9244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112872#comment-15112872
]
Zhe Zhang commented on HDFS-9244:
-
Thanks for the helpful comments [~szetszwo], [~arpitagarwal], and
[~sanjay.radia]! Really appreciate the discussion.
bq. Rollback must always be allowed for any feature.
Sorry I used "rollback" when I meant "downgrade". With the change, _downgrade_
won't be allowed; I will update the patch to bump the NN layout version.
_Rollback_ will work fine.
bq. will this fix break the EZ trash support introduced by HDFS-8831?
No it won't break {{Trash}} support:
{quote}
The deleted encrypted files will remain encrypted and be moved to .Trash
subdirectory under the root of the encryption zone
{quote}
With nested EZs, the "root of the EZ" will be the nearest ancestor with an EZ
setting. I will extend the HDFS-8831 unit test to demonstrate this.
bq. The main motivation for nested EZ is root + subdirs as per Andrew's
comment.
Yes this is the main motivation.
bq. Is it such a big deal for an admin to set up EZ as he creates the
directories in dirs?
We have received many requests from admins for this support. I think this is a
natural trend as the encryption feature matures and people use it in more
sophisticated ways.
bq. I think nested encryption will complicate things like volumes
I look forward to more discussions under HDFS-. Meanwhile, as Andrew
[commented |
https://issues.apache.org/jira/browse/HDFS-?focusedCommentId=14703974=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14703974],
the complexity of directory-level EZ has already been implemented. The
additional complexity introduced by this change is really minimum: it basically
only relaxes one {{if}} condition check.
>From the ease-of-administration perspective, I think even with nested EZs (and
>other nested policies like erasure coding), we can always enforce volume-level
>uniformity. E.g. we can just disallow creating EZ (or setting EC policy) on a
>file/dir in a volume.
> Support nested encryption zones
> ---
>
> Key: HDFS-9244
> URL: https://issues.apache.org/jira/browse/HDFS-9244
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: encryption
>Reporter: Xiaoyu Yao
>Assignee: Zhe Zhang
> Attachments: HDFS-9244.00.patch, HDFS-9244.01.patch
>
>
> This JIRA is opened to track adding support of nested encryption zone based
> on [~andrew.wang]'s [comment
> |https://issues.apache.org/jira/browse/HDFS-8747?focusedCommentId=14654141=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14654141]
> for certain use cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8949) hdfsOpenFile() in HDFS C API does not support block sizes larger than 2GB
[ https://issues.apache.org/jira/browse/HDFS-8949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HDFS-8949. Resolution: Duplicate Duplicate of HDFS-9541 > hdfsOpenFile() in HDFS C API does not support block sizes larger than 2GB > - > > Key: HDFS-8949 > URL: https://issues.apache.org/jira/browse/HDFS-8949 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Vlad Berindei > > hdfsOpenFile() has an int32 blocksize parameter which restricts the size of > the blocks to 2GB, while FileSystem.create accepts a long blockSize parameter. > https://github.com/apache/hadoop/blob/c1d50a91f7c05e4aaf4655380c8dcd11703ff158/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h#L395 > - int32 blocksize > https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#create(org.apache.hadoop.fs.Path, > boolean, int, short, long) - long blockSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8999) Namenode need not wait for {{blockReceived}} for the last block before completing a file.
[
https://issues.apache.org/jira/browse/HDFS-8999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112820#comment-15112820
]
Jing Zhao commented on HDFS-8999:
-
In the failed test ({{TestDFSClientRetries}} e.g.) I saw the following stack
trace:
{code}
org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
replicated yet: /testIdempotentAllocateBlock
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.validateAddBlock(FSDirWriteFileOp.java:190)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2395)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:797)
at
org.apache.hadoop.hdfs.DFSOutputStream.addBlock(DFSOutputStream.java:917)
at
org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1684)
at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1494)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:594)
{code}
Could you please check if this is caused by the patch, [~szetszwo]?
> Namenode need not wait for {{blockReceived}} for the last block before
> completing a file.
> -
>
> Key: HDFS-8999
> URL: https://issues.apache.org/jira/browse/HDFS-8999
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jitendra Nath Pandey
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h8999_20151228.patch, h8999_20160106.patch,
> h8999_20160106b.patch, h8999_20160106c.patch, h8999_20160111.patch,
> h8999_20160113.patch, h8999_20160114.patch, h8999_20160121.patch,
> h8999_20160121b.patch
>
>
> This comes out of a discussion in HDFS-8763. Pasting [~jingzhao]'s comment
> from the jira:
> {quote}
> ...whether we need to let NameNode wait for all the block_received msgs to
> announce the replica is safe. Looking into the code, now we have
># NameNode knows the DataNodes involved when initially setting up the
> writing pipeline
># If any DataNode fails during the writing, client bumps the GS and
> finally reports all the DataNodes included in the new pipeline to NameNode
> through the updatePipeline RPC.
># When the client received the ack for the last packet of the block (and
> before the client tries to close the file on NameNode), the replica has been
> finalized in all the DataNodes.
> Then in this case, when NameNode receives the close request from the client,
> the NameNode already knows the latest replicas for the block. Currently the
> checkReplication call only counts in all the replicas that NN has already
> received the block_received msg, but based on the above #2 and #3, it may be
> safe to also count in all the replicas in the
> BlockUnderConstructionFeature#replicas?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9676) “hdfs dfs -put”,the return value is 0, but the size of $hdfsPath is 0.
[ https://issues.apache.org/jira/browse/HDFS-9676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112229#comment-15112229 ] DuRuiling commented on HDFS-9676: - Ok, the previous namenode logs has been flushed and I will check it at the next backup, I would like to be able to find the relevant error log information. thanks > “hdfs dfs -put”,the return value is 0, but the size of $hdfsPath is 0. > -- > > Key: HDFS-9676 > URL: https://issues.apache.org/jira/browse/HDFS-9676 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover, datanode, hdfs-client >Affects Versions: 2.5.0 > Environment: Linux version 3.0.101 > 2 namenode, 54 datanode >Reporter: DuRuiling >Assignee: DuRuiling > Original Estimate: 168h > Remaining Estimate: 168h > > I find the bug when I backup about 240T data from local file system to HDFS, > I use command “hdfs dfs -D dfs.replication=3 -put $localPath $hdfsPath” to > backup each file one by one, the return value of this command is 0, then > illustrate the command execute right, I find file path is right, but the size > of $hdfsPath is 0 and the size of $loaclPath is non-zero. I reproduce the bug > when I backup data again. > I write a bash shell to backup all data, according to the $? of the command > to judge the file is backup success and record a detailed log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9503) Replace -namenode option with -fs for NNThroughputBenchmark
[
https://issues.apache.org/jira/browse/HDFS-9503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9503:
Attachment: HDFS-9053.001.patch
> Replace -namenode option with -fs for NNThroughputBenchmark
> ---
>
> Key: HDFS-9503
> URL: https://issues.apache.org/jira/browse/HDFS-9503
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Konstantin Shvachko
>Assignee: Mingliang Liu
> Attachments: HDFS-9053.000.patch, HDFS-9053.001.patch
>
>
> HDFS-7847 introduced a new option {{-namenode}}, which is intended to point
> the benchmark to a remote NameNode. It should use a standard generic option
> {{-fs}} instead, which is routinely used to specify NameNode URI in shell
> commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[ https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113275#comment-15113275 ] Bob Hansen commented on HDFS-9643: -- +1 > libhdfs++: Support async cancellation of read operations > > > Key: HDFS-9643 > URL: https://issues.apache.org/jira/browse/HDFS-9643 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-8643.HDFS-8707.004.patch, > HDFS-9643.HDFS-8707.000.patch, HDFS-9643.HDFS-8707.001.patch, > HDFS-9643.HDFS-8707.002.patch, HDFS-9643.HDFS-8707.003.patch > > > It should be possible for any thread to cancel operations in progress on a > FileHandle. Any ephemeral objects created by the FileHandle should free > resources as quickly as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9687) libhdfs++: potential segfault after teardown
[
https://issues.apache.org/jira/browse/HDFS-9687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Hansen updated HDFS-9687:
-
Attachment: HDFS-9687.HDFS-8707.000.diff
> libhdfs++: potential segfault after teardown
>
>
> Key: HDFS-9687
> URL: https://issues.apache.org/jira/browse/HDFS-9687
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Bob Hansen
>Assignee: Bob Hansen
> Attachments: HDFS-9687.HDFS-8707.000.diff
>
>
> During shutdown, it is possible for an RPC task to outlive the RPC connection
> and segfault when it attempts to signal the engine on timeout.
> In rpc_connection.h, the callback
> {code}
> req->timer().async_wait(std::bind(
> ::HandleRpcTimeout, this, req, _1));
> {code}
> does not retain a reference to shared_this, so the connection can be
> destroyed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9672) o.a.h.hdfs.TestLeaseRecovery2 fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-9672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HDFS-9672: Attachment: HDFS-9672.001.patch > o.a.h.hdfs.TestLeaseRecovery2 fails intermittently > -- > > Key: HDFS-9672 > URL: https://issues.apache.org/jira/browse/HDFS-9672 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0 >Reporter: Mingliang Liu >Assignee: Mingliang Liu > Attachments: HDFS-9672.000.patch, HDFS-9672.001.patch > > > It fails in recent builds, see: > https://builds.apache.org/job/PreCommit-HDFS-Build/14177/testReport/org.apache.hadoop.hdfs/ > https://builds.apache.org/job/PreCommit-HDFS-Build/14147/testReport/org.apache.hadoop.hdfs/ > Failing test methods include: > * > org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart > * org.apache.hadoop.hdfs.TestLeaseRecovery2.testLeaseRecoverByAnotherUser > * org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery > * > org.apache.hadoop.hdfs.TestLeaseRecovery2.org.apache.hadoop.hdfs.TestLeaseRecovery2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9687) libhdfs++: potential segfault after teardown
Bob Hansen created HDFS-9687:
Summary: libhdfs++: potential segfault after teardown
Key: HDFS-9687
URL: https://issues.apache.org/jira/browse/HDFS-9687
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Bob Hansen
Assignee: Bob Hansen
During shutdown, it is possible for an RPC task to outlive the RPC connection
and segfault when it attempts to signal the engine on timeout.
In rpc_connection.h, the callback
{code}
req->timer().async_wait(std::bind(
::HandleRpcTimeout, this, req, _1));
{code}
does not retain a reference to shared_this, so the connection can be destroyed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113282#comment-15113282 ] Yongjun Zhang commented on HDFS-6200: - Hi [~wheat9], Thanks for working on this issue. I wonder whether we should mark this jira as incompatible? --Yongjun > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.8.0 > > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch, > HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch, > HDFS-6200.005.patch, HDFS-6200.006.patch, HDFS-6200.007.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9330) Support reconfiguring dfs.datanode.duplicate.replica.deletion without DN restart
[
https://issues.apache.org/jira/browse/HDFS-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113196#comment-15113196
]
Hadoop QA commented on HDFS-9330:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s {color}
| {color:red} HDFS-9330 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12769672/HDFS-9330.001.patch |
| JIRA Issue | HDFS-9330 |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/14211/console |
This message was automatically generated.
> Support reconfiguring dfs.datanode.duplicate.replica.deletion without DN
> restart
> -
>
> Key: HDFS-9330
> URL: https://issues.apache.org/jira/browse/HDFS-9330
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-9330.001.patch
>
>
> This is to reconfigure
> {code}
> dfs.datanode.duplicate.replica.deletion
> {code}
> without restarting DN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9260) Improve performance and GC friendliness of startup and FBRs
[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113195#comment-15113195
]
Hadoop QA commented on HDFS-9260:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HDFS-9260 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783927/HDFS-9260.012.patch |
| JIRA Issue | HDFS-9260 |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/14210/console |
This message was automatically generated.
> Improve performance and GC friendliness of startup and FBRs
> ---
>
> Key: HDFS-9260
> URL: https://issues.apache.org/jira/browse/HDFS-9260
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, performance
>Affects Versions: 2.7.1
>Reporter: Staffan Friberg
>Assignee: Staffan Friberg
> Attachments: FBR processing.png, HDFS Block and Replica Management
> 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch,
> HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch,
> HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch,
> HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch,
> HDFSBenchmarks.zip, HDFSBenchmarks2.zip
>
>
> This patch changes the datastructures used for BlockInfos and Replicas to
> keep them sorted. This allows faster and more GC friendly handling of full
> block reports.
> Would like to hear peoples feedback on this change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9672) o.a.h.hdfs.TestLeaseRecovery2 fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-9672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113229#comment-15113229
]
Mingliang Liu commented on HDFS-9672:
-
Suppose the {{LeaseManager$Monitor}} is doing blocking I/O via: {{run() ->
FSEditLog#logSync() -> EditLogOutputStream#flush() -> JournalSet#flushAndSync()
-> JournalSet#mapJournalsAndReportErrors()}}, if there is an
InterruptException, the channel will be closed and the journals will be
disabled via {{disableAndReportErrorOnJournals()}}. The test main thread then
closes the edit log because of {{MiniDFSCluster#restartNameNode()}}. During the
closing process, it will call {{FSEditLog#endCurrentLogSegment()}} which calls
{{FSEditLog#logSync()}} internally. At the point, the {{logSync()}} will fail
because all the journals are disabled; the {{System#exit()}} will be called
immediately. This may be a potential bug which happens when the NN is stopping
all active services.
Normally, the in-progress edit log would be finalized by
{{FSEditLog#endCurrentLogSegment}}. For testing purposes, we can disable that
here.
> o.a.h.hdfs.TestLeaseRecovery2 fails intermittently
> --
>
> Key: HDFS-9672
> URL: https://issues.apache.org/jira/browse/HDFS-9672
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-9672.000.patch
>
>
> It fails in recent builds, see:
> https://builds.apache.org/job/PreCommit-HDFS-Build/14177/testReport/org.apache.hadoop.hdfs/
> https://builds.apache.org/job/PreCommit-HDFS-Build/14147/testReport/org.apache.hadoop.hdfs/
> Failing test methods include:
> *
> org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart
> * org.apache.hadoop.hdfs.TestLeaseRecovery2.testLeaseRecoverByAnotherUser
> * org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery
> *
> org.apache.hadoop.hdfs.TestLeaseRecovery2.org.apache.hadoop.hdfs.TestLeaseRecovery2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9330) Support reconfiguring dfs.datanode.duplicate.replica.deletion without DN restart
[
https://issues.apache.org/jira/browse/HDFS-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-9330:
Attachment: HDFS-9330-HDFS-9000.002.patch
Similarly, V002 addressed the issues you have concerns on. Thanks
[~arpitagarwal]
> Support reconfiguring dfs.datanode.duplicate.replica.deletion without DN
> restart
> -
>
> Key: HDFS-9330
> URL: https://issues.apache.org/jira/browse/HDFS-9330
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-9330-HDFS-9000.002.patch, HDFS-9330.001.patch
>
>
> This is to reconfigure
> {code}
> dfs.datanode.duplicate.replica.deletion
> {code}
> without restarting DN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9687) libhdfs++: potential segfault after teardown
[
https://issues.apache.org/jira/browse/HDFS-9687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Hansen updated HDFS-9687:
-
Status: Patch Available (was: Open)
> libhdfs++: potential segfault after teardown
>
>
> Key: HDFS-9687
> URL: https://issues.apache.org/jira/browse/HDFS-9687
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Bob Hansen
>Assignee: Bob Hansen
> Attachments: HDFS-9687.HDFS-8707.000.diff
>
>
> During shutdown, it is possible for an RPC task to outlive the RPC connection
> and segfault when it attempts to signal the engine on timeout.
> In rpc_connection.h, the callback
> {code}
> req->timer().async_wait(std::bind(
> ::HandleRpcTimeout, this, req, _1));
> {code}
> does not retain a reference to shared_this, so the connection can be
> destroyed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9094) Add command line option to ask NameNode reload configuration.
[
https://issues.apache.org/jira/browse/HDFS-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113304#comment-15113304
]
Arpit Agarwal commented on HDFS-9094:
-
Thanks [~xiaobingo], the {{TestAnnotations}} failure is related to the patch.
We will need an {{@Idempotent}} annotation on the {{ReconfigurationProtocol}}
methods. Looks good otherwise.
> Add command line option to ask NameNode reload configuration.
> -
>
> Key: HDFS-9094
> URL: https://issues.apache.org/jira/browse/HDFS-9094
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-9094-HDFS-9000.002.patch,
> HDFS-9094-HDFS-9000.003.patch, HDFS-9094-HDFS-9000.004.patch,
> HDFS-9094-HDFS-9000.005.patch, HDFS-9094-HDFS-9000.006.patch,
> HDFS-9094-HDFS-9000.007.patch, HDFS-9094-HDFS-9000.008.patch,
> HDFS-9094.001.patch
>
>
> This work is going to add DFS admin command that allows reloading NameNode
> configuration. This is sibling work related to HDFS-6808.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9541) Add hdfsStreamBuilder API to libhdfs to support defaultBlockSizes greater than 2 GB
[
https://issues.apache.org/jira/browse/HDFS-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113302#comment-15113302
]
Zhe Zhang commented on HDFS-9541:
-
Thanks Colin. I think having a builder method is a good idea. Patch LGTM
overall. A few minors:
# Should it be {{sizeof(struct hdfsStreamBuilder) + path_len}}? Since
{{bld->path}} already has 1 byte allocated to store the {{\0}}.
{code}
bld = malloc(sizeof(struct hdfsStreamBuilder) + path_len + 1);
{code}
# Should be s/hdfsStreamBuilderFree/hdfsStreamBuilderBuild/
{code}
* It is normally not necessary to call this function since
* hdfsStreamBuilderFree frees the builder.
{code}
# Might be worthwhile adding another {{hdfsOpenFile}} that actually uses
{{int64_t}} as {{defaultBlockSize}}?
> Add hdfsStreamBuilder API to libhdfs to support defaultBlockSizes greater
> than 2 GB
> ---
>
> Key: HDFS-9541
> URL: https://issues.apache.org/jira/browse/HDFS-9541
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: libhdfs
>Affects Versions: 0.20.1
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-9541.001.patch
>
>
> We should have a new API in libhdfs which will support creating files with a
> default block size that is more than 31 bits in size. We should also make
> this a builder API so that it is easy to add more options later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9682) Fix a typo "aplication" in HttpFS document
[ https://issues.apache.org/jira/browse/HDFS-9682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112306#comment-15112306 ] Akira AJISAKA commented on HDFS-9682: - +1, committing this. > Fix a typo "aplication" in HttpFS document > -- > > Key: HDFS-9682 > URL: https://issues.apache.org/jira/browse/HDFS-9682 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Trivial > Attachments: HDFS-9682.001.patch > > > This was found while fixing YARN-4605, > The webhdfs client FileSytem implementation can be used to access HttpFS > using the Hadoop filesystem command (`hadoop fs`) line tool as well as from > Java *aplications* using the Hadoop FileSystem Java API > just a typo fix, trivial change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9682) Fix a typo "aplication" in HttpFS document
[ https://issues.apache.org/jira/browse/HDFS-9682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-9682: Component/s: (was: webhdfs) Summary: Fix a typo "aplication" in HttpFS document (was: Fix a typo "aplication" in webhdfs document) > Fix a typo "aplication" in HttpFS document > -- > > Key: HDFS-9682 > URL: https://issues.apache.org/jira/browse/HDFS-9682 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Trivial > Attachments: HDFS-9682.001.patch > > > This was found while fixing YARN-4605, > The webhdfs client FileSytem implementation can be used to access HttpFS > using the Hadoop filesystem command (`hadoop fs`) line tool as well as from > Java *aplications* using the Hadoop FileSystem Java API > just a typo fix, trivial change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9682) Fix a typo "aplication" in HttpFS document
[ https://issues.apache.org/jira/browse/HDFS-9682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112325#comment-15112325 ] Hudson commented on HDFS-9682: -- FAILURE: Integrated in Hadoop-trunk-Commit #9164 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9164/]) HDFS-9682. Fix a typo "aplication" in HttpFS document. Contributed by (aajisaka: rev 683163468219107afb57a002f2cf2a369442171d) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/index.md > Fix a typo "aplication" in HttpFS document > -- > > Key: HDFS-9682 > URL: https://issues.apache.org/jira/browse/HDFS-9682 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Trivial > Fix For: 2.8.0 > > Attachments: HDFS-9682.001.patch > > > This was found while fixing YARN-4605, > The webhdfs client FileSytem implementation can be used to access HttpFS > using the Hadoop filesystem command (`hadoop fs`) line tool as well as from > Java *aplications* using the Hadoop FileSystem Java API > just a typo fix, trivial change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9682) Fix a typo "aplication" in HttpFS document
[ https://issues.apache.org/jira/browse/HDFS-9682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-9682: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed this to trunk, branch-2, and branch-2.8. Thanks [~cheersyang] for the contribution. > Fix a typo "aplication" in HttpFS document > -- > > Key: HDFS-9682 > URL: https://issues.apache.org/jira/browse/HDFS-9682 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Trivial > Fix For: 2.8.0 > > Attachments: HDFS-9682.001.patch > > > This was found while fixing YARN-4605, > The webhdfs client FileSytem implementation can be used to access HttpFS > using the Hadoop filesystem command (`hadoop fs`) line tool as well as from > Java *aplications* using the Hadoop FileSystem Java API > just a typo fix, trivial change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9629) Update the footer of Web UI to show year 2016
[ https://issues.apache.org/jira/browse/HDFS-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112312#comment-15112312 ] Akira AJISAKA commented on HDFS-9629: - bq. So the consensus here is that we can have the displayed year on the UI to be the software's build year, and if someone build a version released in 2016 today, the UI showing 2017 is acceptable, am I right? Okay. > Update the footer of Web UI to show year 2016 > - > > Key: HDFS-9629 > URL: https://issues.apache.org/jira/browse/HDFS-9629 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen > Labels: supportability > Attachments: HDFS-9629.01.patch, HDFS-9629.02.patch, > HDFS-9629.03.patch, HDFS-9629.04.patch, HDFS-9629.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9684) DataNode stopped sending heartbeat after getting OutOfMemoryError form DataTransfer thread.
[
https://issues.apache.org/jira/browse/HDFS-9684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-9684:
-
Status: Patch Available (was: Open)
> DataNode stopped sending heartbeat after getting OutOfMemoryError form
> DataTransfer thread.
> ---
>
> Key: HDFS-9684
> URL: https://issues.apache.org/jira/browse/HDFS-9684
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Blocker
> Attachments: HDFS-9684.01.patch
>
>
> {noformat}
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlock(DataNode.java:1999)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlocks(DataNode.java:2008)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:657)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:615)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:857)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:671)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:823)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9684) DataNode stopped sending heartbeat after getting OutOfMemoryError form DataTransfer thread.
[
https://issues.apache.org/jira/browse/HDFS-9684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112361#comment-15112361
]
Surendra Singh Lilhore commented on HDFS-9684:
--
Here the problem is, {{DataNode.transferBlock()}} will start one thread to
transfer block.
{code}
new Daemon(new DataTransfer(xferTargets, xferTargetStorageTypes, block,
BlockConstructionStage.PIPELINE_SETUP_CREATE, "")).start();
{code}
but JVM is not able start new thread, so it will throw {{OutOfMemoryError}}
and this will cause to interrupt {{BPServiceActor}} thread.
Attached initial patch...
Please review ...
> DataNode stopped sending heartbeat after getting OutOfMemoryError form
> DataTransfer thread.
> ---
>
> Key: HDFS-9684
> URL: https://issues.apache.org/jira/browse/HDFS-9684
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Blocker
> Attachments: HDFS-9684.01.patch
>
>
> {noformat}
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlock(DataNode.java:1999)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlocks(DataNode.java:2008)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:657)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:615)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:857)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:671)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:823)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9684) DataNode stopped sending heartbeat after getting OutOfMemoryError form DataTransfer thread.
Surendra Singh Lilhore created HDFS-9684:
Summary: DataNode stopped sending heartbeat after getting
OutOfMemoryError form DataTransfer thread.
Key: HDFS-9684
URL: https://issues.apache.org/jira/browse/HDFS-9684
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.7.1
Reporter: Surendra Singh Lilhore
Assignee: Surendra Singh Lilhore
Priority: Blocker
{noformat}
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlock(DataNode.java:1999)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlocks(DataNode.java:2008)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:657)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:615)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:857)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:671)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:823)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9684) DataNode stopped sending heartbeat after getting OutOfMemoryError form DataTransfer thread.
[
https://issues.apache.org/jira/browse/HDFS-9684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-9684:
-
Attachment: HDFS-9684.01.patch
> DataNode stopped sending heartbeat after getting OutOfMemoryError form
> DataTransfer thread.
> ---
>
> Key: HDFS-9684
> URL: https://issues.apache.org/jira/browse/HDFS-9684
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Blocker
> Attachments: HDFS-9684.01.patch
>
>
> {noformat}
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlock(DataNode.java:1999)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlocks(DataNode.java:2008)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:657)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:615)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:857)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:671)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:823)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9672) o.a.h.hdfs.TestLeaseRecovery2 fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-9672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113415#comment-15113415
]
Hadoop QA commented on HDFS-9672:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 37s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 53s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 21s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| JDK v1.7.0_91 Failed junit tests | hadoop.hdfs.TestRenameWhileOpen |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783934/HDFS-9672.001.patch |
| JIRA Issue | HDFS-9672 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 65a23b32063b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64
[jira] [Commented] (HDFS-9687) libhdfs++: potential segfault after teardown
[
https://issues.apache.org/jira/browse/HDFS-9687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113407#comment-15113407
]
Hadoop QA commented on HDFS-9687:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 10m 35s
{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
37s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 13s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 10s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 5s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 54s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 43s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 33s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783944/HDFS-9687.HDFS-8707.000.diff
|
| JIRA Issue | HDFS-9687 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 85be63544b13 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / b1fbec1 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_66
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 |
| JDK v1.7.0_91 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/14215/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Max memory used | 79MB |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/14215/console |
This message was automatically
[jira] [Updated] (HDFS-7101) Potential null dereference in DFSck#doWork()
[
https://issues.apache.org/jira/browse/HDFS-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HDFS-7101:
-
Attachment: HDFS-7101.v1.patch
> Potential null dereference in DFSck#doWork()
>
>
> Key: HDFS-7101
> URL: https://issues.apache.org/jira/browse/HDFS-7101
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Ted Yu
>Assignee: skrho
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HDFS-7101.v1.patch, HDFS-7101_001.patch
>
>
> {code}
> String lastLine = null;
> int errCode = -1;
> try {
> while ((line = input.readLine()) != null) {
> ...
> if (lastLine.endsWith(NamenodeFsck.HEALTHY_STATUS)) {
> errCode = 0;
> {code}
> If readLine() throws exception, lastLine may be null, leading to NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9688) Test the effect of nested encryption zones in HDFS downgrade
Zhe Zhang created HDFS-9688: --- Summary: Test the effect of nested encryption zones in HDFS downgrade Key: HDFS-9688 URL: https://issues.apache.org/jira/browse/HDFS-9688 Project: Hadoop HDFS Issue Type: Test Components: encryption, test Reporter: Zhe Zhang Assignee: Zhe Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7694) FSDataInputStream should support "unbuffer"
[ https://issues.apache.org/jira/browse/HDFS-7694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113588#comment-15113588 ] Ted Yu commented on HDFS-7694: -- Is there compatibility concern for backporting this to 2.6 branch ? Thanks > FSDataInputStream should support "unbuffer" > --- > > Key: HDFS-7694 > URL: https://issues.apache.org/jira/browse/HDFS-7694 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7694.001.patch, HDFS-7694.002.patch, > HDFS-7694.003.patch, HDFS-7694.004.patch, HDFS-7694.005.patch > > > For applications that have many open HDFS (or other Hadoop filesystem) files, > it would be useful to have an API to clear readahead buffers and sockets. > This could be added to the existing APIs as an optional interface, in much > the same way as we added setReadahead / setDropBehind / etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-9686) Remove useless boxing/unboxing code (Hadoop HDFS)
[ https://issues.apache.org/jira/browse/HDFS-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-9686 started by Kousuke Saruta. > Remove useless boxing/unboxing code (Hadoop HDFS) > - > > Key: HDFS-9686 > URL: https://issues.apache.org/jira/browse/HDFS-9686 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > Attachments: HDFS-9686.patch.0 > > > There are lots of places where useless boxing/unboxing occur. > To avoid performance issue, let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9686) Remove useless boxing/unboxing code (Hadoop HDFS)
[ https://issues.apache.org/jira/browse/HDFS-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated HDFS-9686: - Status: Patch Available (was: In Progress) > Remove useless boxing/unboxing code (Hadoop HDFS) > - > > Key: HDFS-9686 > URL: https://issues.apache.org/jira/browse/HDFS-9686 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > Attachments: HDFS-9686.patch.0 > > > There are lots of places where useless boxing/unboxing occur. > To avoid performance issue, let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9262) Support reconfiguring dfs.datanode.lazywriter.interval.sec without DN restart
[
https://issues.apache.org/jira/browse/HDFS-9262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-9262:
Attachment: HDFS-9262-HDFS-9000.003.patch
V003 had minor changes.
> Support reconfiguring dfs.datanode.lazywriter.interval.sec without DN restart
> -
>
> Key: HDFS-9262
> URL: https://issues.apache.org/jira/browse/HDFS-9262
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-9262-HDFS-9000.002.patch,
> HDFS-9262-HDFS-9000.003.patch, HDFS-9262.001.patch
>
>
> This is to reconfigure
> {code}
> dfs.datanode.lazywriter.interval.sec
> {code}
> without restarting DN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9677) Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp
[
https://issues.apache.org/jira/browse/HDFS-9677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113605#comment-15113605
]
Hadoop QA commented on HDFS-9677:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 9 new +
1130 unchanged - 9 fixed = 1139 total (was 1139) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 17s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 13s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 170m 1s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.TestRecoverStripedFile |
| | hadoop.hdfs.TestDFSRemove |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Commented] (HDFS-9094) Add command line option to ask NameNode reload configuration.
[
https://issues.apache.org/jira/browse/HDFS-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113619#comment-15113619
]
Hadoop QA commented on HDFS-9094:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 13s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-hdfs-project: patch generated 2 new + 300
unchanged - 4 fixed = 302 total (was 304) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 47s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 50m 40s
{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 49m 9s
{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 138m 13s {color}
| {color:black}
[jira] [Commented] (HDFS-9686) Remove useless boxing/unboxing code (Hadoop HDFS)
[
https://issues.apache.org/jira/browse/HDFS-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113633#comment-15113633
]
Hadoop QA commented on HDFS-9686:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 19s
{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 42s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 57s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 25s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 34s
{color} | {color:red} hadoop-hdfs-project in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 34s {color}
| {color:red} hadoop-hdfs-project in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 4s
{color} | {color:red} hadoop-hdfs-project in the patch failed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 4s {color}
| {color:red} hadoop-hdfs-project in the patch failed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 39s
{color} | {color:red} hadoop-hdfs-project: patch generated 3 new + 14 unchanged
- 356 fixed = 17 total (was 370) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 26s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 35s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 33s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 25s
{color} | {color:red} hadoop-hdfs
[jira] [Commented] (HDFS-9260) Improve performance and GC friendliness of startup and FBRs
[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113600#comment-15113600
]
Hadoop QA commented on HDFS-9260:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 19 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 26s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 4 new +
704 unchanged - 12 fixed = 708 total (was 716) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 25s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 53m 15s
{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 141m 31s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-8898) Create API and command-line argument to get quota and quota usage without detailed content summary
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112545#comment-15112545 ] Hudson commented on HDFS-8898: -- FAILURE: Integrated in Hadoop-trunk-Commit #9167 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9167/]) HDFS-8898. Create API and command-line argument to get quota and quota (kihwal: rev 2fd19b9674420e025af54a5bed12eb96478f8c48) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-common-project/hadoop-common/src/test/resources/testConf.xml * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestQuotasWithHA.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestQuotaUsage.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsDefaultValue.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Count.java * hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCount.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/CLITestHelper.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ContentSummary.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/QuotaUsage.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java > Create API and command-line argument to get quota and quota usage without > detailed content summary > -- > > Key: HDFS-8898 > URL: https://issues.apache.org/jira/browse/HDFS-8898 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Joep Rottinghuis >Assignee: Ming Ma > Fix For: 2.8.0 > > Attachments: HDFS-8898-2.patch, HDFS-8898-3.patch, HDFS-8898-4.patch, > HDFS-8898-5.patch, HDFS-8898-6.patch, HDFS-8898-branch-2.patch, > HDFS-8898.patch > > > On large directory structures it takes significant time to iterate through > the file and directory counts recursively to get a complete ContentSummary. > When you want to just check for the quota on a higher level directory it > would be good to have an option to skip the file and directory counts. > Moreover, currently one can only check the quota if you have access to all > the directories underneath. For example, if I have a large home directory > under /user/joep and I host some files for another user in a sub-directory, > the moment they create an unreadable sub-directory under my home I can no > longer check what my quota is. Understood that I cannot check the current > file counts unless I can iterate through all the usage, but for > administrative purposes it is nice to be able to get the current quota > setting on a directory without the need to iterate through and run into > permission issues on sub-directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9685) StopDecommission for datanode should remove the underReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112597#comment-15112597
]
Hadoop QA commented on HDFS-9685:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 26s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 54s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 138m 59s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockScanner |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783828/HDFS-9685.001.patch |
| JIRA Issue | HDFS-9685 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7581e76fc7f5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-8898) Create API and command-line argument to get quota and quota usage without detailed content summary
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112622#comment-15112622 ] Allen Wittenauer commented on HDFS-8898: Should this have deprecated the public API created by HDFS-3000? > Create API and command-line argument to get quota and quota usage without > detailed content summary > -- > > Key: HDFS-8898 > URL: https://issues.apache.org/jira/browse/HDFS-8898 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Joep Rottinghuis >Assignee: Ming Ma > Fix For: 2.8.0 > > Attachments: HDFS-8898-2.patch, HDFS-8898-3.patch, HDFS-8898-4.patch, > HDFS-8898-5.patch, HDFS-8898-6.patch, HDFS-8898-branch-2.patch, > HDFS-8898.patch > > > On large directory structures it takes significant time to iterate through > the file and directory counts recursively to get a complete ContentSummary. > When you want to just check for the quota on a higher level directory it > would be good to have an option to skip the file and directory counts. > Moreover, currently one can only check the quota if you have access to all > the directories underneath. For example, if I have a large home directory > under /user/joep and I host some files for another user in a sub-directory, > the moment they create an unreadable sub-directory under my home I can no > longer check what my quota is. Understood that I cannot check the current > file counts unless I can iterate through all the usage, but for > administrative purposes it is nice to be able to get the current quota > setting on a directory without the need to iterate through and run into > permission issues on sub-directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9573) SnapshotDiffReport$DiffReportEntry$hashCode should use type field
[ https://issues.apache.org/jira/browse/HDFS-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-9573: - Summary: SnapshotDiffReport$DiffReportEntry$hashCode should use type field (was: o.a.h.hdfs.protocol.SnapshotDiffReport$DiffReportEntry$hashCode inconsistent with equals) Fix the summary based on Nicholas' comment. > SnapshotDiffReport$DiffReportEntry$hashCode should use type field > - > > Key: HDFS-9573 > URL: https://issues.apache.org/jira/browse/HDFS-9573 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Attachments: HDFS-9573-001.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > DiffReportEntry.equals() uses field "type", but DiffReportEntry.hashCode() > doesn't. This breaks the rules on equals and hashCode: > * if a class overrides equals, it must override hashCode > * when they are both overridden, equals and hashCode must use the same set of > fields -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HDFS-9573) SnapshotDiffReport$DiffReportEntry$hashCode should use type field
[ https://issues.apache.org/jira/browse/HDFS-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reopened HDFS-9573: -- Fix the summary based on Nicholas's comment. > SnapshotDiffReport$DiffReportEntry$hashCode should use type field > - > > Key: HDFS-9573 > URL: https://issues.apache.org/jira/browse/HDFS-9573 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Attachments: HDFS-9573-001.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > DiffReportEntry.equals() uses field "type", but DiffReportEntry.hashCode() > doesn't. This breaks the rules on equals and hashCode: > * if a class overrides equals, it must override hashCode > * when they are both overridden, equals and hashCode must use the same set of > fields -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9573) SnapshotDiffReport$DiffReportEntry$hashCode should use type field
[ https://issues.apache.org/jira/browse/HDFS-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-9573: - Status: Patch Available (was: Reopened) Resubmit the patch. > SnapshotDiffReport$DiffReportEntry$hashCode should use type field > - > > Key: HDFS-9573 > URL: https://issues.apache.org/jira/browse/HDFS-9573 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Attachments: HDFS-9573-001.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > DiffReportEntry.equals() uses field "type", but DiffReportEntry.hashCode() > doesn't. This breaks the rules on equals and hashCode: > * if a class overrides equals, it must override hashCode > * when they are both overridden, equals and hashCode must use the same set of > fields -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9573) SnapshotDiffReport$DiffReportEntry$hashCode should use type field
[
https://issues.apache.org/jira/browse/HDFS-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112720#comment-15112720
]
Hadoop QA commented on HDFS-9573:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HDFS-9573 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12781844/HDFS-9573-001.patch |
| JIRA Issue | HDFS-9573 |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/14207/console |
This message was automatically generated.
> SnapshotDiffReport$DiffReportEntry$hashCode should use type field
> -
>
> Key: HDFS-9573
> URL: https://issues.apache.org/jira/browse/HDFS-9573
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HDFS-9573-001.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> DiffReportEntry.equals() uses field "type", but DiffReportEntry.hashCode()
> doesn't. This breaks the rules on equals and hashCode:
> * if a class overrides equals, it must override hashCode
> * when they are both overridden, equals and hashCode must use the same set of
> fields
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9685) StopDecommission for datanode should remove the underReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Yiqun updated HDFS-9685:
Attachment: HDFS-9685.001.patch
> StopDecommission for datanode should remove the underReplicatedBlocks
> -
>
> Key: HDFS-9685
> URL: https://issues.apache.org/jira/browse/HDFS-9685
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-9685.001.patch
>
>
> When one node removed from exclude file, and its state from
> decommission-in-progress to in service. But the underReplicatedBlocksNum of
> cluster has not been decreased. Most of these underReplicatedBlocks are not
> needed and it will costs namenode much time to deal with. And frequently
> namenode will find there are enough replications. So in
> {{stopDecommissioned}} operation, we should remove neededReplicatedBlocks of
> decomNodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9685) StopDecommission for datanode should remove the underReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Yiqun updated HDFS-9685:
Attachment: HDFS-9685.001.patch
> StopDecommission for datanode should remove the underReplicatedBlocks
> -
>
> Key: HDFS-9685
> URL: https://issues.apache.org/jira/browse/HDFS-9685
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-9685.001.patch
>
>
> When one node removed from exclude file, and its state from
> decommission-in-progress to in service. But the underReplicatedBlocksNum of
> cluster has not been decreased. Most of these underReplicatedBlocks are not
> needed and it will costs namenode much time to deal with. And frequently
> namenode will find there are enough replications. So in
> {{stopDecommissioned}} operation, we should remove neededReplicatedBlocks of
> decomNodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9685) StopDecommission for datanode should remove the underReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112432#comment-15112432
]
Lin Yiqun commented on HDFS-9685:
-
Update the patch. Only when the node' state is decommision-in-progress that
will remove blocks. If the node is already decommisioned, the
underReplicatedBlocks is almost replicated.
{code}
public void stopDecommission(DatanodeDescriptor node) {
if (node.isDecommissionInProgress() || node.isDecommissioned()) {
AdminStates adminState = node.getAdminState();
// Update DN stats maintained by HeartbeatManager
hbManager.stopDecommission(node);
// Over-replicated blocks will be detected and processed when
// the dead node comes back and send in its full block report.
// The original blocks in decomNodes will be removed from
// neededReplications if node is decommission-in-progress.
if (node.isAlive()) {
blockManager.processOverReplicatedBlocksOnReCommission(node);
if (adminState == AdminStates.DECOMMISSION_INPROGRESS) {
removeNeededReplicatedBlocksInDecomNodes(node);
}
}
// Remove from tracking in DecommissionManager
pendingNodes.remove(node);
decomNodeBlocks.remove(node);
} else {
LOG.trace("stopDecommission: Node {} in {}, nothing to do." +
node, node.getAdminState());
}
}
{code}
> StopDecommission for datanode should remove the underReplicatedBlocks
> -
>
> Key: HDFS-9685
> URL: https://issues.apache.org/jira/browse/HDFS-9685
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-9685.001.patch
>
>
> When one node removed from exclude file, and its state from
> decommission-in-progress to in service. But the underReplicatedBlocksNum of
> cluster has not been decreased. Most of these underReplicatedBlocks are not
> needed and it will costs namenode much time to deal with. And frequently
> namenode will find there are enough replications. So in
> {{stopDecommissioned}} operation, we should remove neededReplicatedBlocks of
> decomNodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9685) StopDecommission for datanode should remove the underReplicatedBlocks
Lin Yiqun created HDFS-9685:
---
Summary: StopDecommission for datanode should remove the
underReplicatedBlocks
Key: HDFS-9685
URL: https://issues.apache.org/jira/browse/HDFS-9685
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Lin Yiqun
Assignee: Lin Yiqun
When one node removed from exclude file, and its state from
decommission-in-progress to in service. But the underReplicatedBlocksNum of
cluster has not been decreased. Most of these underReplicatedBlocks are not
needed and it will costs namenode much time to deal with. And frequently
namenode will find there are enough replications. So in {{stopDecommissioned}}
operation, we should remove neededReplicatedBlocks of decomNodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9685) StopDecommission for datanode should remove the underReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Yiqun updated HDFS-9685:
Status: Patch Available (was: Open)
Attach a initial patch. Adding a new method to remove blocks in
stopDecommission.
{code}
void stopDecommission(DatanodeDescriptor node) {
if (node.isDecommissionInProgress() || node.isDecommissioned()) {
LOG.info("Stopping decommissioning of node {}", node);
// Update DN stats maintained by HeartbeatManager
hbManager.stopDecommission(node);
// Over-replicated blocks will be detected and processed when
// the dead node comes back and send in its full block report.
// The original blocks in decomNodes will be removing from
// neededReplications.
if (node.isAlive) {
blockManager.processOverReplicatedBlocksOnReCommission(node);
removeNeededReplicatedBlocksInDecomNodes(node);
}
// Remove from tracking in DecommissionManager
pendingNodes.remove(node);
decomNodeBlocks.remove(node);
} else {
LOG.trace("stopDecommission: Node {} is not decommission in progress " +
"or decommissioned, nothing to do.", node);
}
}
{code}
Kindly reviewing!
> StopDecommission for datanode should remove the underReplicatedBlocks
> -
>
> Key: HDFS-9685
> URL: https://issues.apache.org/jira/browse/HDFS-9685
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>
> When one node removed from exclude file, and its state from
> decommission-in-progress to in service. But the underReplicatedBlocksNum of
> cluster has not been decreased. Most of these underReplicatedBlocks are not
> needed and it will costs namenode much time to deal with. And frequently
> namenode will find there are enough replications. So in
> {{stopDecommissioned}} operation, we should remove neededReplicatedBlocks of
> decomNodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9685) StopDecommission for datanode should remove the underReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Yiqun updated HDFS-9685:
Attachment: (was: HDFS-9685.001.patch)
> StopDecommission for datanode should remove the underReplicatedBlocks
> -
>
> Key: HDFS-9685
> URL: https://issues.apache.org/jira/browse/HDFS-9685
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>
> When one node removed from exclude file, and its state from
> decommission-in-progress to in service. But the underReplicatedBlocksNum of
> cluster has not been decreased. Most of these underReplicatedBlocks are not
> needed and it will costs namenode much time to deal with. And frequently
> namenode will find there are enough replications. So in
> {{stopDecommissioned}} operation, we should remove neededReplicatedBlocks of
> decomNodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8898) Create API and command-line argument to get quota and quota usage without detailed content summary
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112640#comment-15112640 ] Ming Ma commented on HDFS-8898: --- [~aw], HDFS-3000 provides quota write, this jira is to provide quota read. Thanks [~kihwal] and [~vinayrpet]! > Create API and command-line argument to get quota and quota usage without > detailed content summary > -- > > Key: HDFS-8898 > URL: https://issues.apache.org/jira/browse/HDFS-8898 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Joep Rottinghuis >Assignee: Ming Ma > Fix For: 2.8.0 > > Attachments: HDFS-8898-2.patch, HDFS-8898-3.patch, HDFS-8898-4.patch, > HDFS-8898-5.patch, HDFS-8898-6.patch, HDFS-8898-branch-2.patch, > HDFS-8898.patch > > > On large directory structures it takes significant time to iterate through > the file and directory counts recursively to get a complete ContentSummary. > When you want to just check for the quota on a higher level directory it > would be good to have an option to skip the file and directory counts. > Moreover, currently one can only check the quota if you have access to all > the directories underneath. For example, if I have a large home directory > under /user/joep and I host some files for another user in a sub-directory, > the moment they create an unreadable sub-directory under my home I can no > longer check what my quota is. Understood that I cannot check the current > file counts unless I can iterate through all the usage, but for > administrative purposes it is nice to be able to get the current quota > setting on a directory without the need to iterate through and run into > permission issues on sub-directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9525) hadoop utilities need to support provided delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-9525: --- Attachment: HDFS-9525.009.patch -09: * reupload so that precommit will process it > hadoop utilities need to support provided delegation tokens > --- > > Key: HDFS-9525 > URL: https://issues.apache.org/jira/browse/HDFS-9525 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: HeeSoo Kim >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HDFS-7984.001.patch, HDFS-7984.002.patch, > HDFS-7984.003.patch, HDFS-7984.004.patch, HDFS-7984.005.patch, > HDFS-7984.006.patch, HDFS-7984.007.patch, HDFS-7984.patch, > HDFS-9525.008.patch, HDFS-9525.009.patch, HDFS-9525.009.patch, > HDFS-9525.branch-2.008.patch, HDFS-9525.branch-2.009.patch > > > When using the webhdfs:// filesystem (especially from distcp), we need the > ability to inject a delegation token rather than webhdfs initialize its own. > This would allow for cross-authentication-zone file system accesses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8898) Create API and command-line argument to get quota and quota usage without detailed content summary
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112644#comment-15112644 ] Allen Wittenauer commented on HDFS-8898: Argh, yeah, I missed that. Thanks. Just hoping someday we can undo the damage caused by that jira. :( > Create API and command-line argument to get quota and quota usage without > detailed content summary > -- > > Key: HDFS-8898 > URL: https://issues.apache.org/jira/browse/HDFS-8898 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Joep Rottinghuis >Assignee: Ming Ma > Fix For: 2.8.0 > > Attachments: HDFS-8898-2.patch, HDFS-8898-3.patch, HDFS-8898-4.patch, > HDFS-8898-5.patch, HDFS-8898-6.patch, HDFS-8898-branch-2.patch, > HDFS-8898.patch > > > On large directory structures it takes significant time to iterate through > the file and directory counts recursively to get a complete ContentSummary. > When you want to just check for the quota on a higher level directory it > would be good to have an option to skip the file and directory counts. > Moreover, currently one can only check the quota if you have access to all > the directories underneath. For example, if I have a large home directory > under /user/joep and I host some files for another user in a sub-directory, > the moment they create an unreadable sub-directory under my home I can no > longer check what my quota is. Understood that I cannot check the current > file counts unless I can iterate through all the usage, but for > administrative purposes it is nice to be able to get the current quota > setting on a directory without the need to iterate through and run into > permission issues on sub-directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9685) StopDecommission for datanode should remove the underReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112653#comment-15112653
]
Hadoop QA commented on HDFS-9685:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 25s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 39s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 157m 49s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.namenode.TestCacheDirectives |
| JDK v1.7.0_91 Failed junit tests | hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783831/HDFS-9685.001.patch |
| JIRA Issue | HDFS-9685 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d09bf343448b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-9646) ErasureCodingWorker may fail when recovering data blocks with length less than the first internal block
[
https://issues.apache.org/jira/browse/HDFS-9646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112475#comment-15112475
]
Tsz Wo Nicholas Sze commented on HDFS-9646:
---
+1 the latest patch looks good.
> ErasureCodingWorker may fail when recovering data blocks with length less
> than the first internal block
> ---
>
> Key: HDFS-9646
> URL: https://issues.apache.org/jira/browse/HDFS-9646
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Takuya Fukudome
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-9646.000.patch, HDFS-9646.001.patch,
> HDFS-9646.002.patch, HDFS-9646.003.patch, HDFS-9646.004.patch,
> test-reconstruct-stripe-file.patch
>
>
> This is reported by [~tfukudom]: ErasureCodingWorker may fail with the
> following exception when recovering a non-full internal block.
> {code}
> 2016-01-06 11:14:44,740 WARN datanode.DataNode
> (ErasureCodingWorker.java:run(467)) - Failed to recover striped block:
> BP-987302662-172.29.4.13-1450757377698:blk_-92233720368
> 54322288_29751
> java.io.IOException: Transfer failed for all targets.
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$ReconstructAndTransferBlock.run(ErasureCodingWorker.java:455)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9684) DataNode stopped sending heartbeat after getting OutOfMemoryError form DataTransfer thread.
[
https://issues.apache.org/jira/browse/HDFS-9684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112512#comment-15112512
]
Hadoop QA commented on HDFS-9684:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 52s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 6s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 167m 24s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
[jira] [Commented] (HDFS-8898) Create API and command-line argument to get quota and quota usage without detailed content summary
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112515#comment-15112515 ] Kihwal Lee commented on HDFS-8898: -- The patch looks to be a correct port of the patch. > Create API and command-line argument to get quota and quota usage without > detailed content summary > -- > > Key: HDFS-8898 > URL: https://issues.apache.org/jira/browse/HDFS-8898 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Joep Rottinghuis >Assignee: Ming Ma > Attachments: HDFS-8898-2.patch, HDFS-8898-3.patch, HDFS-8898-4.patch, > HDFS-8898-5.patch, HDFS-8898-6.patch, HDFS-8898-branch-2.patch, > HDFS-8898.patch > > > On large directory structures it takes significant time to iterate through > the file and directory counts recursively to get a complete ContentSummary. > When you want to just check for the quota on a higher level directory it > would be good to have an option to skip the file and directory counts. > Moreover, currently one can only check the quota if you have access to all > the directories underneath. For example, if I have a large home directory > under /user/joep and I host some files for another user in a sub-directory, > the moment they create an unreadable sub-directory under my home I can no > longer check what my quota is. Understood that I cannot check the current > file counts unless I can iterate through all the usage, but for > administrative purposes it is nice to be able to get the current quota > setting on a directory without the need to iterate through and run into > permission issues on sub-directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[
https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112629#comment-15112629
]
Hadoop QA commented on HDFS-9643:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
25s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 39s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 42s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 39s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 34s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 0s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783839/HDFS-8643.HDFS-8707.004.patch
|
| JIRA Issue | HDFS-9643 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux ea6ad3c0caef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / b1fbec1 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_66
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 |
| JDK v1.7.0_91 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/14205/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Max memory used | 76MB |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/14205/console |
This message was automatically generated.
> libhdfs++: Support async cancellation of read operations
>
>
> Key: HDFS-9643
> URL: https://issues.apache.org/jira/browse/HDFS-9643
>
[jira] [Updated] (HDFS-9573) SnapshotDiffReport$DiffReportEntry$hashCode should use type field
[ https://issues.apache.org/jira/browse/HDFS-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-9573: - Attachment: HDFS-9573-002.patch Patch 002: * Rebase * Update commit message > SnapshotDiffReport$DiffReportEntry$hashCode should use type field > - > > Key: HDFS-9573 > URL: https://issues.apache.org/jira/browse/HDFS-9573 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Attachments: HDFS-9573-001.patch, HDFS-9573-002.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > DiffReportEntry.equals() uses field "type", but DiffReportEntry.hashCode() > doesn't. This breaks the rules on equals and hashCode: > * if a class overrides equals, it must override hashCode > * when they are both overridden, equals and hashCode must use the same set of > fields -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9646) ErasureCodingWorker may fail when recovering data blocks with length less than the first internal block
[
https://issues.apache.org/jira/browse/HDFS-9646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-9646:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
I've committed this. Thanks all again for the review!
> ErasureCodingWorker may fail when recovering data blocks with length less
> than the first internal block
> ---
>
> Key: HDFS-9646
> URL: https://issues.apache.org/jira/browse/HDFS-9646
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Takuya Fukudome
>Assignee: Jing Zhao
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HDFS-9646.000.patch, HDFS-9646.001.patch,
> HDFS-9646.002.patch, HDFS-9646.003.patch, HDFS-9646.004.patch,
> test-reconstruct-stripe-file.patch
>
>
> This is reported by [~tfukudom]: ErasureCodingWorker may fail with the
> following exception when recovering a non-full internal block.
> {code}
> 2016-01-06 11:14:44,740 WARN datanode.DataNode
> (ErasureCodingWorker.java:run(467)) - Failed to recover striped block:
> BP-987302662-172.29.4.13-1450757377698:blk_-92233720368
> 54322288_29751
> java.io.IOException: Transfer failed for all targets.
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$ReconstructAndTransferBlock.run(ErasureCodingWorker.java:455)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9260) Improve performance and GC friendliness of startup and FBRs
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Staffan Friberg updated HDFS-9260: -- Attachment: HDFS-9260.013.patch > Improve performance and GC friendliness of startup and FBRs > --- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Attachments: FBR processing.png, HDFS Block and Replica Management > 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch, > HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch, > HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch, > HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch, > HDFS-9260.013.patch, HDFSBenchmarks.zip, HDFSBenchmarks2.zip > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7101) Potential null dereference in DFSck#doWork()
[
https://issues.apache.org/jira/browse/HDFS-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113530#comment-15113530
]
Mingliang Liu commented on HDFS-7101:
-
This is a good catch, and the initial value makes sense to me.
If {{readLine()}} throws exception, the following code {{if
(lastLine.endsWith(...)}} will not reach as we don't swallow the exception
here. I think NPE that is caused by {{lastLine}} only happens if the input
stream is empty. Agreed?
> Potential null dereference in DFSck#doWork()
>
>
> Key: HDFS-7101
> URL: https://issues.apache.org/jira/browse/HDFS-7101
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Ted Yu
>Assignee: skrho
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HDFS-7101.v1.patch, HDFS-7101_001.patch
>
>
> {code}
> String lastLine = null;
> int errCode = -1;
> try {
> while ((line = input.readLine()) != null) {
> ...
> if (lastLine.endsWith(NamenodeFsck.HEALTHY_STATUS)) {
> errCode = 0;
> {code}
> If readLine() throws exception, lastLine may be null, leading to NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9677) Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp
[
https://issues.apache.org/jira/browse/HDFS-9677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9677:
Attachment: HDFS-9677.000.patch
The v0 patch is to rename the variables and methods according to the
description. The basic change includes {{BlockIdManager}} and {{FSEditLogOp}}.
The protobuf is intact for backward compatibility, though renaming fields *may
be* safe.
Will refine the patch if it is either too aggressive or incomplete.
> Rename generationStampV1/generationStampV2 to
> legacyGenerationStamp/generationStamp
> ---
>
> Key: HDFS-9677
> URL: https://issues.apache.org/jira/browse/HDFS-9677
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Mingliang Liu
> Attachments: HDFS-9677.000.patch
>
>
> [comment|https://issues.apache.org/jira/browse/HDFS-9542?focusedCommentId=15110531=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15110531]
> from [~drankye] in HDFS-9542:
> {quote}
> Just wonder if it's a good idea to rename: generationStampV1 =>
> legacyGenerationStamp; generationStampV2 => generationStamp, similar for
> other variables, as we have legacy block and block.
> {quote}
> This jira plans to do this rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9677) Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp
[
https://issues.apache.org/jira/browse/HDFS-9677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9677:
Target Version/s: 3.0.0
Status: Patch Available (was: Open)
> Rename generationStampV1/generationStampV2 to
> legacyGenerationStamp/generationStamp
> ---
>
> Key: HDFS-9677
> URL: https://issues.apache.org/jira/browse/HDFS-9677
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Mingliang Liu
> Attachments: HDFS-9677.000.patch
>
>
> [comment|https://issues.apache.org/jira/browse/HDFS-9542?focusedCommentId=15110531=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15110531]
> from [~drankye] in HDFS-9542:
> {quote}
> Just wonder if it's a good idea to rename: generationStampV1 =>
> legacyGenerationStamp; generationStampV2 => generationStamp, similar for
> other variables, as we have legacy block and block.
> {quote}
> This jira plans to do this rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9685) StopDecommission for datanode should remove the underReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113543#comment-15113543
]
Lin Yiqun commented on HDFS-9685:
-
The failed tests seem not related.
> StopDecommission for datanode should remove the underReplicatedBlocks
> -
>
> Key: HDFS-9685
> URL: https://issues.apache.org/jira/browse/HDFS-9685
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-9685.001.patch
>
>
> When one node removed from exclude file, and its state from
> decommission-in-progress to in service. But the underReplicatedBlocksNum of
> cluster has not been decreased. Most of these underReplicatedBlocks are not
> needed and it will costs namenode much time to deal with. And frequently
> namenode will find there are enough replications. So in
> {{stopDecommissioned}} operation, we should remove neededReplicatedBlocks of
> decomNodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9689) Test o.a.h.hdfs.TestRenameWhileOpen fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113566#comment-15113566
]
Hadoop QA commented on HDFS-9689:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 10s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 36s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 58s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.datanode.TestFsDatasetCache |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783975/HDFS-9689.000.patch |
| JIRA Issue | HDFS-9689 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5e9579b1a684
[jira] [Updated] (HDFS-9094) Add command line option to ask NameNode reload configuration.
[ https://issues.apache.org/jira/browse/HDFS-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-9094: Attachment: HDFS-9094-HDFS-9000.009.patch V009 added Idempotent mark, thanks. > Add command line option to ask NameNode reload configuration. > - > > Key: HDFS-9094 > URL: https://issues.apache.org/jira/browse/HDFS-9094 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-9094-HDFS-9000.002.patch, > HDFS-9094-HDFS-9000.003.patch, HDFS-9094-HDFS-9000.004.patch, > HDFS-9094-HDFS-9000.005.patch, HDFS-9094-HDFS-9000.006.patch, > HDFS-9094-HDFS-9000.007.patch, HDFS-9094-HDFS-9000.008.patch, > HDFS-9094-HDFS-9000.009.patch, HDFS-9094.001.patch > > > This work is going to add DFS admin command that allows reloading NameNode > configuration. This is sibling work related to HDFS-6808. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9262) Support reconfiguring dfs.datanode.lazywriter.interval.sec without DN restart
[
https://issues.apache.org/jira/browse/HDFS-9262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113346#comment-15113346
]
Hadoop QA commented on HDFS-9262:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 25s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new +
261 unchanged - 1 fixed = 263 total (was 262) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 29s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 33s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 202m 5s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.TestPread |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.TestRecoverStripedFile |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| |
[jira] [Commented] (HDFS-9687) libhdfs++: potential segfault after teardown
[
https://issues.apache.org/jira/browse/HDFS-9687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113368#comment-15113368
]
James Clampffer commented on HDFS-9687:
---
Looks good to me. If it gets a clean CI run I'll +1.
> libhdfs++: potential segfault after teardown
>
>
> Key: HDFS-9687
> URL: https://issues.apache.org/jira/browse/HDFS-9687
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Bob Hansen
>Assignee: Bob Hansen
> Attachments: HDFS-9687.HDFS-8707.000.diff
>
>
> During shutdown, it is possible for an RPC task to outlive the RPC connection
> and segfault when it attempts to signal the engine on timeout.
> In rpc_connection.h, the callback
> {code}
> req->timer().async_wait(std::bind(
> ::HandleRpcTimeout, this, req, _1));
> {code}
> does not retain a reference to shared_this, so the connection can be
> destroyed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9534) Add CLI command to clear storage policy from a path.
[
https://issues.apache.org/jira/browse/HDFS-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113400#comment-15113400
]
Jing Zhao commented on HDFS-9534:
-
Yeah, for the semantic of the new op, I agree with the comments from
[~cnauroth] and [~arpitagarwal]. +1 to call the op (reset/unset)StoragePolicy.
> Add CLI command to clear storage policy from a path.
>
>
> Key: HDFS-9534
> URL: https://issues.apache.org/jira/browse/HDFS-9534
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Reporter: Chris Nauroth
>Assignee: Xiaobing Zhou
> Attachments: HDFS-9534.001.patch
>
>
> The {{hdfs storagepolicies}} command has sub-commands for
> {{-setStoragePolicy}} and {{-getStoragePolicy}} on a path. However, there is
> no {{-removeStoragePolicy}} to remove a previously set storage policy on a
> path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9682) Fix a typo "aplication" in HttpFS document
[ https://issues.apache.org/jira/browse/HDFS-9682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113457#comment-15113457 ] Weiwei Yang commented on HDFS-9682: --- Thanks [~ajisakaa] > Fix a typo "aplication" in HttpFS document > -- > > Key: HDFS-9682 > URL: https://issues.apache.org/jira/browse/HDFS-9682 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Trivial > Fix For: 2.8.0 > > Attachments: HDFS-9682.001.patch > > > This was found while fixing YARN-4605, > The webhdfs client FileSytem implementation can be used to access HttpFS > using the Hadoop filesystem command (`hadoop fs`) line tool as well as from > Java *aplications* using the Hadoop FileSystem Java API > just a typo fix, trivial change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9689) Test o.a.h.hdfs.TestRenameWhileOpen fails intermittently
Mingliang Liu created HDFS-9689:
---
Summary: Test o.a.h.hdfs.TestRenameWhileOpen fails intermittently
Key: HDFS-9689
URL: https://issues.apache.org/jira/browse/HDFS-9689
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 3.0.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu
The test fails in recent builds, e.g.
https://builds.apache.org/job/PreCommit-HDFS-Build/14063/testReport/org.apache.hadoop.hdfs/TestRenameWhileOpen/
and
https://builds.apache.org/job/PreCommit-HDFS-Build/14212/testReport/org.apache.hadoop.hdfs/TestRenameWhileOpen/testWhileOpenRenameToNonExistentDirectory/
The *Error Message* is like:
{code}
Problem binding to [localhost:60690] java.net.BindException: Address already in
use; For more details see: http://wiki.apache.org/hadoop/BindException
{code}
and *Stacktrace* is:
{code}
java.net.BindException: Problem binding to [localhost:60690]
java.net.BindException: Address already in use; For more details see:
http://wiki.apache.org/hadoop/BindException
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:463)
at sun.nio.ch.Net.bind(Net.java:455)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.hadoop.ipc.Server.bind(Server.java:469)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
at org.apache.hadoop.ipc.Server.(Server.java:2464)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:958)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
at
org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:800)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:392)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:743)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:685)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:884)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:863)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1581)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
at
org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
at
org.apache.hadoop.hdfs.TestRenameWhileOpen.testWhileOpenRenameToNonExistentDirectory(TestRenameWhileOpen.java:332)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9688) Test the effect of nested encryption zones in HDFS downgrade
[
https://issues.apache.org/jira/browse/HDFS-9688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113470#comment-15113470
]
Hadoop QA commented on HDFS-9688:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s {color} | {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 48s
{color} | {color:red} hadoop-hdfs in branch-2.6 failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 48s
{color} | {color:red} hadoop-hdfs in branch-2.6 failed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} branch-2.6 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} branch-2.6 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 12s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2.6 has 273
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} branch-2.6 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s
{color} | {color:green} branch-2.6 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 46s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 46s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 47s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 47s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 2s
{color} | {color:red} The patch has 1212 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 31s
{color} | {color:red} The patch has 95 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 46s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 47s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 37s
{color} | {color:red} Patch generated 74 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 2s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:date2016-01-23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783968/HDFS-9688-branch-2.6.00.patch
|
| JIRA Issue | HDFS-9688 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname |
[jira] [Commented] (HDFS-9330) Support reconfiguring dfs.datanode.duplicate.replica.deletion without DN restart
[
https://issues.apache.org/jira/browse/HDFS-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113503#comment-15113503
]
Hadoop QA commented on HDFS-9330:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new +
285 unchanged - 10 fixed = 287 total (was 295) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 15s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 2s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 174m 6s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| | hadoop.hdfs.server.namenode.TestFileLimit |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.TestInjectionForSimulatedStorage |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.TestFileCreation |
[jira] [Updated] (HDFS-9689) Test o.a.h.hdfs.TestRenameWhileOpen fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9689:
Attachment: HDFS-9689.000.patch
The tests bind the same name node (NN) port when building a new cluster, which
will fail in case of concurrent tests. I see no reason for reusing the NN port
for this test. Meanwhile, we don't need build a new cluster for restarting
purpose, as we can employ {{MiniDFSCluster#restartNameNode()}}.
> Test o.a.h.hdfs.TestRenameWhileOpen fails intermittently
> -
>
> Key: HDFS-9689
> URL: https://issues.apache.org/jira/browse/HDFS-9689
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-9689.000.patch
>
>
> The test fails in recent builds, e.g.
> https://builds.apache.org/job/PreCommit-HDFS-Build/14063/testReport/org.apache.hadoop.hdfs/TestRenameWhileOpen/
> and
> https://builds.apache.org/job/PreCommit-HDFS-Build/14212/testReport/org.apache.hadoop.hdfs/TestRenameWhileOpen/testWhileOpenRenameToNonExistentDirectory/
> The *Error Message* is like:
> {code}
> Problem binding to [localhost:60690] java.net.BindException: Address already
> in use; For more details see: http://wiki.apache.org/hadoop/BindException
> {code}
> and *Stacktrace* is:
> {code}
> java.net.BindException: Problem binding to [localhost:60690]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:463)
> at sun.nio.ch.Net.bind(Net.java:455)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:958)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:800)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:392)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:743)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:685)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:884)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:863)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1581)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at
> org.apache.hadoop.hdfs.TestRenameWhileOpen.testWhileOpenRenameToNonExistentDirectory(TestRenameWhileOpen.java:332)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9689) Test o.a.h.hdfs.TestRenameWhileOpen fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9689:
Affects Version/s: 2.8.0
> Test o.a.h.hdfs.TestRenameWhileOpen fails intermittently
> -
>
> Key: HDFS-9689
> URL: https://issues.apache.org/jira/browse/HDFS-9689
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-9689.000.patch
>
>
> The test fails in recent builds, e.g.
> https://builds.apache.org/job/PreCommit-HDFS-Build/14063/testReport/org.apache.hadoop.hdfs/TestRenameWhileOpen/
> and
> https://builds.apache.org/job/PreCommit-HDFS-Build/14212/testReport/org.apache.hadoop.hdfs/TestRenameWhileOpen/testWhileOpenRenameToNonExistentDirectory/
> The *Error Message* is like:
> {code}
> Problem binding to [localhost:60690] java.net.BindException: Address already
> in use; For more details see: http://wiki.apache.org/hadoop/BindException
> {code}
> and *Stacktrace* is:
> {code}
> java.net.BindException: Problem binding to [localhost:60690]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:463)
> at sun.nio.ch.Net.bind(Net.java:455)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:958)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:800)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:392)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:743)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:685)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:884)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:863)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1581)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at
> org.apache.hadoop.hdfs.TestRenameWhileOpen.testWhileOpenRenameToNonExistentDirectory(TestRenameWhileOpen.java:332)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9689) Test o.a.h.hdfs.TestRenameWhileOpen fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9689:
Status: Patch Available (was: Open)
> Test o.a.h.hdfs.TestRenameWhileOpen fails intermittently
> -
>
> Key: HDFS-9689
> URL: https://issues.apache.org/jira/browse/HDFS-9689
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-9689.000.patch
>
>
> The test fails in recent builds, e.g.
> https://builds.apache.org/job/PreCommit-HDFS-Build/14063/testReport/org.apache.hadoop.hdfs/TestRenameWhileOpen/
> and
> https://builds.apache.org/job/PreCommit-HDFS-Build/14212/testReport/org.apache.hadoop.hdfs/TestRenameWhileOpen/testWhileOpenRenameToNonExistentDirectory/
> The *Error Message* is like:
> {code}
> Problem binding to [localhost:60690] java.net.BindException: Address already
> in use; For more details see: http://wiki.apache.org/hadoop/BindException
> {code}
> and *Stacktrace* is:
> {code}
> java.net.BindException: Problem binding to [localhost:60690]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:463)
> at sun.nio.ch.Net.bind(Net.java:455)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:958)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:800)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:392)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:743)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:685)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:884)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:863)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1581)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at
> org.apache.hadoop.hdfs.TestRenameWhileOpen.testWhileOpenRenameToNonExistentDirectory(TestRenameWhileOpen.java:332)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
