[jira] [Commented] (HDFS-9013) Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
[
https://issues.apache.org/jira/browse/HDFS-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904312#comment-14904312
]
Hudson commented on HDFS-9013:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8502 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8502/])
HDFS-9013. Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from
trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev
a2c76e5f26301d4b01e56b347442f3dec171591d)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
* hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
> Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
> --
>
> Key: HDFS-9013
> URL: https://issues.apache.org/jira/browse/HDFS-9013
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-9013-branch-2.003.patch,
> HDFS-9013-branch-2.004.patch, HDFS-9013.001-branch-2.patch,
> HDFS-9013.001.patch, HDFS-9013.002-branch-2.patch
>
>
> HDFS-8388 added one new metric {{NNStartedTimeInMillis}} to get NN start time
> in milliseconds.
> Now based on [~wheat9] and [~ajisakaa] suggestions now we should deprecate
> {{NameNodeMXBean#getNNStarted}} in branch2 and remove from trunk.
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14709614=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14709614
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14726746=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14726746
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8976) Create HTML5 cluster webconsole for federated cluster
[
https://issues.apache.org/jira/browse/HDFS-8976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904326#comment-14904326
]
Vinayakumar B commented on HDFS-8976:
-
hi [~wheat9],
I wonder whether you could review updated patch?
> Create HTML5 cluster webconsole for federated cluster
> -
>
> Key: HDFS-8976
> URL: https://issues.apache.org/jira/browse/HDFS-8976
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 2.7.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-8976-01.patch, HDFS-8976-02.patch,
> HDFS-8976-03.patch, cluster-health.JPG
>
>
> Since the old jsp variant of cluster web console is no longer present from
> 2.7 onwards, there is a need for HTML 5 web console for overview of overall
> cluster.
> 2.7.1 docs says to check webconsole as below {noformat}Similar to the
> Namenode status web page, when using federation a Cluster Web Console is
> available to monitor the federated cluster at
> http:///dfsclusterhealth.jsp. Any Namenode in the cluster
> can be used to access this web page.{noformat}
> But this is no longer present,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9126) namenode crash in fsimage transfer
[ https://issues.apache.org/jira/browse/HDFS-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengyongping updated HDFS-9126: --- Description: In our product Hadoop cluster,when active namenode begin download/transfer fsimage from standby namenode.some times zkfc monitor health of NameNode socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING ,happen hadoop namenode ha failover. was: In out product Hadoop cluster,when active namenode begin download/transfer fsimage from standby namenode.some times zkfc monitor health of NameNode session timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING ,happen hadoop namenode ha failover. > namenode crash in fsimage transfer > -- > > Key: HDFS-9126 > URL: https://issues.apache.org/jira/browse/HDFS-9126 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover >Affects Versions: 2.6.0 > Environment: OS:Centos 6.5(final) > Hadoop:2.6.0 >Reporter: zengyongping >Priority: Critical > > In our product Hadoop cluster,when active namenode begin download/transfer > fsimage from standby namenode.some times zkfc monitor health of NameNode > socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING > ,happen hadoop namenode ha > failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8739) Move DFSClient to client implementation package
[ https://issues.apache.org/jira/browse/HDFS-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-8739: --- Attachment: HDFS-8739-002.patch > Move DFSClient to client implementation package > --- > > Key: HDFS-8739 > URL: https://issues.apache.org/jira/browse/HDFS-8739 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: build >Reporter: Yi Liu >Assignee: Brahma Reddy Battula > Attachments: HDFS-8739-002.patch, HDFS-8739.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Status: Patch Available (was: Open)
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904337#comment-14904337
]
Hadoop QA commented on HDFS-9044:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 2s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 0s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 6s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 24s | The applied patch generated 2
new checkstyle issues (total was 50, now 51). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 33s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 164m 10s | Tests passed in hadoop-hdfs.
|
| | | 210m 0s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761822/HDFS-9044.4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 7c5c099 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12618/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12618/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12618/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12618/console |
This message was automatically generated.
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8739) Move DFSClient to client implementation package
[ https://issues.apache.org/jira/browse/HDFS-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904409#comment-14904409 ] Brahma Reddy Battula commented on HDFS-8739: Updated patch to fix compile erros in hadoop-hdfs-nfs project... > Move DFSClient to client implementation package > --- > > Key: HDFS-8739 > URL: https://issues.apache.org/jira/browse/HDFS-8739 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: build >Reporter: Yi Liu >Assignee: Brahma Reddy Battula > Attachments: HDFS-8739-002.patch, HDFS-8739.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8882) Erasure Coding: Use datablocks, parityblocks and cell size from ErasureCodingPolicy
[ https://issues.apache.org/jira/browse/HDFS-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-8882: Summary: Erasure Coding: Use datablocks, parityblocks and cell size from ErasureCodingPolicy (was: Use datablocks, parityblocks and cell size from ErasureCodingPolicy) > Erasure Coding: Use datablocks, parityblocks and cell size from > ErasureCodingPolicy > --- > > Key: HDFS-8882 > URL: https://issues.apache.org/jira/browse/HDFS-8882 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-8882-HDFS-7285-01.patch, > HDFS-8882-HDFS-7285-02.patch, HDFS-8882-HDFS-7285-03.patch > > > As part of earlier development, constants were used for datablocks, parity > blocks and cellsize. > Now all these are available in ec zone. Use from there and stop using > constant values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9126) namenode crash in fsimage download/transfer
[ https://issues.apache.org/jira/browse/HDFS-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengyongping updated HDFS-9126: --- Summary: namenode crash in fsimage download/transfer (was: namenode crash in fsimage transfer) > namenode crash in fsimage download/transfer > --- > > Key: HDFS-9126 > URL: https://issues.apache.org/jira/browse/HDFS-9126 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: OS:Centos 6.5(final) > Hadoop:2.6.0 >Reporter: zengyongping >Priority: Critical > > In our product Hadoop cluster,when active namenode begin download/transfer > fsimage from standby namenode.some times zkfc monitor health of NameNode > socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING > ,happen hadoop namenode ha > failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-3599) Better expose when under-construction files are preventing DN decommission
[
https://issues.apache.org/jira/browse/HDFS-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904424#comment-14904424
]
Dave Marion commented on HDFS-3599:
---
I have seen this issue in CDH 5.3.0, which is Hadoop 2.5.0+.
> Better expose when under-construction files are preventing DN decommission
> --
>
> Key: HDFS-3599
> URL: https://issues.apache.org/jira/browse/HDFS-3599
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>
> Filing on behalf of Konstantin Olchanski:
> {quote}
> I have been trying to decommission a data node, but the process
> stalled. I followed the correct instructions, observed my node
> listed in "Decommissioning Nodes", etc, observed "Under Replicated Blocks"
> decrease, etc. But the count went down to "1" and the decommissin process
> stalled.
> There was no visible activity anywhere, nothing was happening (well,
> maybe in some hidden log file somewhere something complained,
> but I did not look).
> It turns out that I had some files stuck in "OPENFORWRITE" mode,
> as reported by "hdfs fsck / -openforwrite -files -blocks -locations -racks":
> {code}
> /users/trinat/data/.fuse_hidden177e0002 0 bytes, 0 block(s),
> OPENFORWRITE: OK
> /users/trinat/data/.fuse_hidden178d0003 0 bytes, 0 block(s),
> OPENFORWRITE: OK
> /users/trinat/data/.fuse_hidden1da30004 0 bytes, 1 block(s),
> OPENFORWRITE: OK
> 0.
> BP-88378204-142.90.119.126-1340494203431:blk_6980480609696383665_20259{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[142.90.111.72:50010|RBW],
> ReplicaUnderConstruction[142.90.119.162:50010|RBW],
> ReplicaUnderConstruction[142.90.119.126:50010|RBW]]} len=0 repl=3
> [/detfac/142.90.111.72:50010, /isac2/142.90.119.162:50010,
> /isac2/142.90.119.126:50010]
> {code}
> After I deleted those files, the decommission process completed successfully.
> Perhaps one can add some visible indication somewhere on the HDFS status web
> page
> that the decommission process is stalled and maybe report why it is stalled?
> Maybe the number of "OPENFORWRITE" files should be listed on the status page
> next to the "Number of Under-Replicated Blocks"? (Since I know that nobody is
> writing
> to my HDFS, the non-zero count would give me a clue that something is wrong).
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8739) Move DFSClient to client implementation package
[
https://issues.apache.org/jira/browse/HDFS-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904296#comment-14904296
]
Hadoop QA commented on HDFS-8739:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 66 new or modified test files. |
| {color:red}-1{color} | javac | 2m 19s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761856/HDFS-8739.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 7c5c099 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12620/console |
This message was automatically generated.
> Move DFSClient to client implementation package
> ---
>
> Key: HDFS-8739
> URL: https://issues.apache.org/jira/browse/HDFS-8739
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Yi Liu
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-8739.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9013) Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
[
https://issues.apache.org/jira/browse/HDFS-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904333#comment-14904333
]
Hudson commented on HDFS-9013:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #434 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/434/])
HDFS-9013. Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from
trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev
a2c76e5f26301d4b01e56b347442f3dec171591d)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
* hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
> Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
> --
>
> Key: HDFS-9013
> URL: https://issues.apache.org/jira/browse/HDFS-9013
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-9013-branch-2.003.patch,
> HDFS-9013-branch-2.004.patch, HDFS-9013.001-branch-2.patch,
> HDFS-9013.001.patch, HDFS-9013.002-branch-2.patch
>
>
> HDFS-8388 added one new metric {{NNStartedTimeInMillis}} to get NN start time
> in milliseconds.
> Now based on [~wheat9] and [~ajisakaa] suggestions now we should deprecate
> {{NameNodeMXBean#getNNStarted}} in branch2 and remove from trunk.
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14709614=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14709614
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14726746=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14726746
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9126) namenode crash in fsimage transfer
[ https://issues.apache.org/jira/browse/HDFS-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengyongping updated HDFS-9126: --- Description: In out product Hadoop cluster,when active namenode begin download/transfer fsimage from standby namenode.some times zkfc monitor health of NameNode session timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING ,happen hadoop namenode ha failover. > namenode crash in fsimage transfer > -- > > Key: HDFS-9126 > URL: https://issues.apache.org/jira/browse/HDFS-9126 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover >Affects Versions: 2.6.0 > Environment: OS:Centos 6.5(final) > Hadoop:2.6.0 >Reporter: zengyongping >Priority: Critical > > In out product Hadoop cluster,when active namenode begin download/transfer > fsimage from standby namenode.some times zkfc monitor health of NameNode > session timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING > ,happen hadoop namenode ha > failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9126) namenode crash in fsimage download/transfer
[ https://issues.apache.org/jira/browse/HDFS-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengyongping updated HDFS-9126: --- Description: In our product Hadoop cluster,when active namenode begin download/transfer fsimage from standby namenode.some times zkfc monitor health of NameNode socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING ,happen hadoop namenode ha failover. was: In our product Hadoop cluster,when active namenode begin download/transfer fsimage from standby namenode.some times zkfc monitor health of NameNode socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING ,happen hadoop namenode ha failover. > namenode crash in fsimage download/transfer > --- > > Key: HDFS-9126 > URL: https://issues.apache.org/jira/browse/HDFS-9126 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: OS:Centos 6.5(final) > Hadoop:2.6.0 >Reporter: zengyongping >Priority: Critical > > In our product Hadoop cluster,when active namenode begin download/transfer > fsimage from standby namenode.some times zkfc monitor health of NameNode > socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING > ,happen hadoop namenode ha failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9126) namenode crash in fsimage download/transfer
[ https://issues.apache.org/jira/browse/HDFS-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengyongping updated HDFS-9126: --- Description: In our product Hadoop cluster,when active namenode begin download/transfer fsimage from standby namenode.some times zkfc monitor health of NameNode socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING ,happen hadoop namenode ha failover,fence old active namenode. was: In our product Hadoop cluster,when active namenode begin download/transfer fsimage from standby namenode.some times zkfc monitor health of NameNode socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING ,happen hadoop namenode ha failover. > namenode crash in fsimage download/transfer > --- > > Key: HDFS-9126 > URL: https://issues.apache.org/jira/browse/HDFS-9126 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: OS:Centos 6.5(final) > Hadoop:2.6.0 >Reporter: zengyongping >Priority: Critical > > In our product Hadoop cluster,when active namenode begin download/transfer > fsimage from standby namenode.some times zkfc monitor health of NameNode > socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING > ,happen hadoop namenode ha failover,fence old active namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9013) Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
[
https://issues.apache.org/jira/browse/HDFS-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904367#comment-14904367
]
Hudson commented on HDFS-9013:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #426 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/426/])
HDFS-9013. Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from
trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev
a2c76e5f26301d4b01e56b347442f3dec171591d)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
* hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java
> Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
> --
>
> Key: HDFS-9013
> URL: https://issues.apache.org/jira/browse/HDFS-9013
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-9013-branch-2.003.patch,
> HDFS-9013-branch-2.004.patch, HDFS-9013.001-branch-2.patch,
> HDFS-9013.001.patch, HDFS-9013.002-branch-2.patch
>
>
> HDFS-8388 added one new metric {{NNStartedTimeInMillis}} to get NN start time
> in milliseconds.
> Now based on [~wheat9] and [~ajisakaa] suggestions now we should deprecate
> {{NameNodeMXBean#getNNStarted}} in branch2 and remove from trunk.
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14709614=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14709614
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14726746=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14726746
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9013) Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
[
https://issues.apache.org/jira/browse/HDFS-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904421#comment-14904421
]
Hudson commented on HDFS-9013:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1166 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1166/])
HDFS-9013. Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from
trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev
a2c76e5f26301d4b01e56b347442f3dec171591d)
* hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
> Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
> --
>
> Key: HDFS-9013
> URL: https://issues.apache.org/jira/browse/HDFS-9013
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-9013-branch-2.003.patch,
> HDFS-9013-branch-2.004.patch, HDFS-9013.001-branch-2.patch,
> HDFS-9013.001.patch, HDFS-9013.002-branch-2.patch
>
>
> HDFS-8388 added one new metric {{NNStartedTimeInMillis}} to get NN start time
> in milliseconds.
> Now based on [~wheat9] and [~ajisakaa] suggestions now we should deprecate
> {{NameNodeMXBean#getNNStarted}} in branch2 and remove from trunk.
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14709614=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14709614
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14726746=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14726746
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9013) Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
[
https://issues.apache.org/jira/browse/HDFS-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-9013:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Release Note: NameNodeMXBean#getNNStarted() metric is deprecated in
branch-2 and removed from trunk.
Status: Resolved (was: Patch Available)
Committed to trunk and branch-2.
Thanks [~surendrasingh] for the contribution.
Thanks [~wheat9] and [~ajisakaa] for initial suggestion.
> Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
> --
>
> Key: HDFS-9013
> URL: https://issues.apache.org/jira/browse/HDFS-9013
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-9013-branch-2.003.patch,
> HDFS-9013-branch-2.004.patch, HDFS-9013.001-branch-2.patch,
> HDFS-9013.001.patch, HDFS-9013.002-branch-2.patch
>
>
> HDFS-8388 added one new metric {{NNStartedTimeInMillis}} to get NN start time
> in milliseconds.
> Now based on [~wheat9] and [~ajisakaa] suggestions now we should deprecate
> {{NameNodeMXBean#getNNStarted}} in branch2 and remove from trunk.
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14709614=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14709614
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14726746=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14726746
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9126) namenode crash in fsimage transfer
[ https://issues.apache.org/jira/browse/HDFS-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengyongping updated HDFS-9126: --- Component/s: (was: auto-failover) namenode > namenode crash in fsimage transfer > -- > > Key: HDFS-9126 > URL: https://issues.apache.org/jira/browse/HDFS-9126 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: OS:Centos 6.5(final) > Hadoop:2.6.0 >Reporter: zengyongping >Priority: Critical > > In our product Hadoop cluster,when active namenode begin download/transfer > fsimage from standby namenode.some times zkfc monitor health of NameNode > socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING > ,happen hadoop namenode ha > failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9126) namenode crash in fsimage download/transfer
[ https://issues.apache.org/jira/browse/HDFS-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengyongping updated HDFS-9126: --- Environment: OS:Centos 6.5(final) Hadoop:2.6.0 namenode ha base 5 journalnode was: OS:Centos 6.5(final) Hadoop:2.6.0 > namenode crash in fsimage download/transfer > --- > > Key: HDFS-9126 > URL: https://issues.apache.org/jira/browse/HDFS-9126 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: OS:Centos 6.5(final) > Hadoop:2.6.0 > namenode ha base 5 journalnode >Reporter: zengyongping >Priority: Critical > > In our product Hadoop cluster,when active namenode begin download/transfer > fsimage from standby namenode.some times zkfc monitor health of NameNode > socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING > ,happen hadoop namenode ha failover,fence old active namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Attachment: HDFS-9100.000.patch
Still testing in my lab. A unit test with multiple networks is difficult.
Will change to patch-available once I finished testing
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9013) Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
[
https://issues.apache.org/jira/browse/HDFS-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904468#comment-14904468
]
Hudson commented on HDFS-9013:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2372 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2372/])
HDFS-9013. Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from
trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev
a2c76e5f26301d4b01e56b347442f3dec171591d)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
* hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
> Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
> --
>
> Key: HDFS-9013
> URL: https://issues.apache.org/jira/browse/HDFS-9013
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-9013-branch-2.003.patch,
> HDFS-9013-branch-2.004.patch, HDFS-9013.001-branch-2.patch,
> HDFS-9013.001.patch, HDFS-9013.002-branch-2.patch
>
>
> HDFS-8388 added one new metric {{NNStartedTimeInMillis}} to get NN start time
> in milliseconds.
> Now based on [~wheat9] and [~ajisakaa] suggestions now we should deprecate
> {{NameNodeMXBean#getNNStarted}} in branch2 and remove from trunk.
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14709614=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14709614
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14726746=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14726746
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904366#comment-14904366
]
J.Andreina commented on HDFS-9044:
--
Checkstyle comments can be ignored , as all the 9 parameters of the method are
been used.
Please review.
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9076) Log full path instead of inodeId in DFSClient#closeAllFilesBeingWritten()
[
https://issues.apache.org/jira/browse/HDFS-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904320#comment-14904320
]
Vinayakumar B commented on HDFS-9076:
-
Can you please update the Log message as below. ?
{code} LOG.error("Failed to " + (abort ? "abort" : "close") + " file: "
+ out.getSrc() + " with inode: " + inodeId, ie);{code}
> Log full path instead of inodeId in DFSClient#closeAllFilesBeingWritten()
> -
>
> Key: HDFS-9076
> URL: https://issues.apache.org/jira/browse/HDFS-9076
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-9076.01.patch, HDFS-9076.patch
>
>
> {code}
>try {
> if (abort) {
> out.abort();
> } else {
> out.close();
> }
> } catch(IOException ie) {
> LOG.error("Failed to " + (abort? "abort": "close") +
> " inode " + inodeId, ie);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9126) namenode crash in fsimage transfer
zengyongping created HDFS-9126: -- Summary: namenode crash in fsimage transfer Key: HDFS-9126 URL: https://issues.apache.org/jira/browse/HDFS-9126 Project: Hadoop HDFS Issue Type: Bug Components: auto-failover Affects Versions: 2.6.0 Environment: OS:Centos 6.5(final) Hadoop:2.6.0 Reporter: zengyongping Priority: Critical -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904476#comment-14904476
]
Brahma Reddy Battula commented on HDFS-8647:
{quote}Ideally HDFS should support auto replication for the case where # of
rack changes from 1 to 2 after NN's restart. But that requires more work and it
is an existing issue. You can open a new jira for that.{quote}
[~mingma] Raised seperate jira to handle this..(HDFS-9127)..will update patch
by keeping current behavior as it is.
> Abstract BlockManager's rack policy into BlockPlacementPolicy
> -
>
> Key: HDFS-8647
> URL: https://issues.apache.org/jira/browse/HDFS-8647
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-8647-001.patch, HDFS-8647-002.patch,
> HDFS-8647-003.patch
>
>
> Sometimes we want to have namenode use alternative block placement policy
> such as upgrade domains in HDFS-7541.
> BlockManager has built-in assumption about rack policy in functions such as
> useDelHint, blockHasEnoughRacks. That means when we have new block placement
> policy, we need to modify BlockManager to account for the new policy. Ideally
> BlockManager should ask BlockPlacementPolicy object instead. That will allow
> us to provide new BlockPlacementPolicy without changing BlockManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9127) Re-replication for files with enough replicas in single rack
Brahma Reddy Battula created HDFS-9127:
--
Summary: Re-replication for files with enough replicas in single
rack
Key: HDFS-9127
URL: https://issues.apache.org/jira/browse/HDFS-9127
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
Found while debugging testcases in HDFS-8647
*Scenario:*
===
Start a cluster with Single rack with three DN's
write a file with RF=3
adde two Nodes with different racks
As per blockplacement policy ([Rack
Awareness|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/RackAwareness.html])
atleast one replica needs to replicate to newly added rack.But it is not
happening..Because of following reason.
{color:blue}
when cluster was single rack,block will be removed from {{neededReplications}}
after 3 replicas.
later, after adding new rack, only replications will happen which are present
in {{neededReplications}}
So for the blocks which already have enough replicas, new rack replications
will not take place..
{color}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904479#comment-14904479
]
Casey Brotherton commented on HDFS-9100:
Created a multi-node cluster with multi-homed networks.
Created one of the nodes with only one ip address, on the subnet
that was not the primary network of the other hosts in the cluster.
Verified that without appropriate configuration, the lone node
could not send data to datanodes.
Confirmed that changing dfs.client.use.datanode.hostname allowed the
lone node to put and read from the datanodes.
Placed data within the HDFS cluster in an unbalanced way. ( dfs.replication=1
on one datanode,
with minimal data spread across two datanodes with dfs.replication=2)
Running the balancer from the lone node causes these errors:
15/09/23 06:07:04 WARN balancer.Dispatcher: Failed to move blk_1073742600_1776
with size=4128368 from 10.17.74.156:50010:DISK to 10.17.74.158:50010:DISK
through 10.17.74.156:50010: Network is unreachable
After the changes, the balancer worked. The logging still showed the IP
address.
15/09/23 06:05:38 INFO balancer.Dispatcher: Successfully moved
blk_1073742139_1315 with size=4128368 from 10.17.74.156:50010:DISK to
10.17.74.158:50010:DISK through 10.17.74.156:50010
This did not test a large environment, where blocks were moved by proxy,
although I don't believe that would cause any problems.
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8341) HDFS mover stuck in loop trying to move corrupt block with no other valid replicas, doesn't move rest of other data blocks
[
https://issues.apache.org/jira/browse/HDFS-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze resolved HDFS-8341.
---
Resolution: Cannot Reproduce
> HDFS mover stuck in loop trying to move corrupt block with no other valid
> replicas, doesn't move rest of other data blocks
> --
>
> Key: HDFS-8341
> URL: https://issues.apache.org/jira/browse/HDFS-8341
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.6.0
> Environment: HDP 2.2
>Reporter: Hari Sekhon
>Priority: Minor
>
> HDFS mover gets stuck looping on a block that fails to move and doesn't
> migrate the rest of the blocks.
> This is preventing recovery of data from a decomissioning external storage
> tier used for archive (we've had problems with that proprietary "hyperscale"
> storage product which is why a couple blocks here and there have checksum
> problems or premature eof as shown below), but this should not prevent moving
> all the other blocks to recover our data:
> {code}hdfs mover -p /apps/hive/warehouse/
> 15/05/07 14:52:50 INFO mover.Mover: namenodes =
> {hdfs://nameservice1=[/apps/hive/warehouse/]}
> 15/05/07 14:52:51 INFO balancer.KeyManager: Block token params received from
> NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
> 15/05/07 14:52:51 INFO block.BlockTokenSecretManager: Setting block keys
> 15/05/07 14:52:51 INFO balancer.KeyManager: Update block keys every 2hrs,
> 30mins, 0sec
> 15/05/07 14:52:52 INFO block.BlockTokenSecretManager: Setting block keys
> 15/05/07 14:52:52 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:52:52 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:52:52 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:52:52 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:52:52 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:52:52 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:52:52 WARN balancer.Dispatcher: Failed to move
> blk_1075156654_1438349 with size=134217728 from :1019:ARCHIVE to
> :1019:DISK through :1019: block move is failed: opReplaceBlock
> BP-120244285--1417023863606:blk_1075156654_1438349 received exception
> java.io.EOFException: Premature EOF: no length prefix available
>
> 15/05/07 14:53:31 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:53:31 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:53:31 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:53:31 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:53:31 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:53:31 INFO net.NetworkTopology: Adding a new node:
> /default-rack/:1019
> 15/05/07 14:53:31 WARN balancer.Dispatcher: Failed to move
> blk_1075156654_1438349 with size=134217728 from :1019:ARCHIVE to
> :1019:DISK through :1019: block move is failed: opReplaceBlock
> BP-120244285--1417023863606:blk_1075156654_1438349 received exception
> java.io.EOFException: Premature EOF: no length prefix available
> ..
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9013) Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
[
https://issues.apache.org/jira/browse/HDFS-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904530#comment-14904530
]
Hudson commented on HDFS-9013:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #407 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/407/])
HDFS-9013. Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from
trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev
a2c76e5f26301d4b01e56b347442f3dec171591d)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java
* hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
> --
>
> Key: HDFS-9013
> URL: https://issues.apache.org/jira/browse/HDFS-9013
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-9013-branch-2.003.patch,
> HDFS-9013-branch-2.004.patch, HDFS-9013.001-branch-2.patch,
> HDFS-9013.001.patch, HDFS-9013.002-branch-2.patch
>
>
> HDFS-8388 added one new metric {{NNStartedTimeInMillis}} to get NN start time
> in milliseconds.
> Now based on [~wheat9] and [~ajisakaa] suggestions now we should deprecate
> {{NameNodeMXBean#getNNStarted}} in branch2 and remove from trunk.
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14709614=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14709614
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14726746=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14726746
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8873) throttle directoryScanner
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904605#comment-14904605 ] Nathan Roberts commented on HDFS-8873: -- Thanks [~templedf] for the update. I am sorry I haven't had a chance to review yet. I plan to do this Thursday AM. > throttle directoryScanner > - > > Key: HDFS-8873 > URL: https://issues.apache.org/jira/browse/HDFS-8873 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: Daniel Templeton > Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch, > HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch > > > The new 2-level directory layout can make directory scans expensive in terms > of disk seeks (see HDFS-8791) for details. > It would be good if the directoryScanner() had a configurable duty cycle that > would reduce its impact on disk performance (much like the approach in > HDFS-8617). > Without such a throttle, disks can go 100% busy for many minutes at a time > (assuming the common case of all inodes in cache but no directory blocks > cached, 64K seeks are required for full directory listing which translates to > 655 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9125) Display help if the command option to "hdfs dfs " is not valid
[
https://issues.apache.org/jira/browse/HDFS-9125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904676#comment-14904676
]
Daniel Templeton commented on HDFS-9125:
I haven't applied the patch and tested it, but it appears to print something
like:
{code}
master:/home/nijel/hadoop-3.0.0-SNAPSHOT/bin # ./hdfs dfs -mkdirs
Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
...
SUBCOMMAND may print help when invoked w/o parameters or with -h.
-mkdirs: Unknown command
{code}
Perhaps it's a minor quibble, but shouldn't we print the error message before
the usage message?
> Display help if the command option to "hdfs dfs " is not valid
> ---
>
> Key: HDFS-9125
> URL: https://issues.apache.org/jira/browse/HDFS-9125
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: nijel
>Assignee: nijel
>Priority: Minor
> Attachments: HDFS-9125_1.patch
>
>
> {noformat}
> master:/home/nijel/hadoop-3.0.0-SNAPSHOT/bin # ./hdfs dfs -mkdirs
> -mkdirs: Unknown command
> {noformat}
> Better to display the help info.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[ https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-9123: -- Description: HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) However, the existing path validation fails if the source path is the root, causing infinite copying, until the disk space is filled up. was: HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) However, if the source path is ended with a '/' path separator, the existing validation for sub-directories fails. For example, copying from / to /abc would cause infinite copying, until the disk space is filled up. > Copying from the root to a subdirectory should be forbidden > --- > > Key: HDFS-9123 > URL: https://issues.apache.org/jira/browse/HDFS-9123 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch > > > HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp > /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, > /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) > However, the existing path validation fails if the source path is the root, > causing infinite copying, until the disk space is filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[ https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-9123: -- Summary: Copying from the root to a subdirectory should be forbidden (was: Validation of a path ended with a '/') > Copying from the root to a subdirectory should be forbidden > --- > > Key: HDFS-9123 > URL: https://issues.apache.org/jira/browse/HDFS-9123 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch > > > HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp > /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, > /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) > However, if the source path is ended with a '/' path separator, the existing > validation for sub-directories fails. For example, copying from / to /abc > would cause infinite copying, until the disk space is filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Validation of a path ended with a '/'
[ https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904581#comment-14904581 ] Wei-Chiu Chuang commented on HDFS-9123: --- [~andreina] Thanks for pointing out. Yes indeed it looks like the case, because when makeQualified() is called, the path is normalized and trailing slashes are removed except for the root. I am going to update the title/description of this jira. > Validation of a path ended with a '/' > - > > Key: HDFS-9123 > URL: https://issues.apache.org/jira/browse/HDFS-9123 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch > > > HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp > /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, > /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) > However, if the source path is ended with a '/' path separator, the existing > validation for sub-directories fails. For example, copying from / to /abc > would cause infinite copying, until the disk space is filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8873) throttle directoryScanner
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904627#comment-14904627 ] Daniel Templeton commented on HDFS-8873: No worries, [~nroberts]. I'm about to upload a new patch anyway. :) > throttle directoryScanner > - > > Key: HDFS-8873 > URL: https://issues.apache.org/jira/browse/HDFS-8873 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: Daniel Templeton > Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch, > HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch > > > The new 2-level directory layout can make directory scans expensive in terms > of disk seeks (see HDFS-8791) for details. > It would be good if the directoryScanner() had a configurable duty cycle that > would reduce its impact on disk performance (much like the approach in > HDFS-8617). > Without such a throttle, disks can go 100% busy for many minutes at a time > (assuming the common case of all inodes in cache but no directory blocks > cached, 64K seeks are required for full directory listing which translates to > 655 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9125) Display help if the command option to "hdfs dfs " is not valid
[
https://issues.apache.org/jira/browse/HDFS-9125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nijel updated HDFS-9125:
Status: Patch Available (was: Open)
> Display help if the command option to "hdfs dfs " is not valid
> ---
>
> Key: HDFS-9125
> URL: https://issues.apache.org/jira/browse/HDFS-9125
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: nijel
>Assignee: nijel
>Priority: Minor
> Attachments: HDFS-9125_1.patch
>
>
> {noformat}
> master:/home/nijel/hadoop-3.0.0-SNAPSHOT/bin # ./hdfs dfs -mkdirs
> -mkdirs: Unknown command
> {noformat}
> Better to display the help info.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9071) chooseTargets in ReplicationWork may pass incomplete srcPath
[ https://issues.apache.org/jira/browse/HDFS-9071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904511#comment-14904511 ] J.Andreina commented on HDFS-9071: -- Thanks [~He Tianyi] for reporting this issue, that was a good catch. bq. I've observed that chooseTargets in ReplicationWork may pass incomplete srcPath (not starting with '/') to block placement policy. I agree . But i doubt , is there any specific reason for going ahead to choose target (in ReplicationWork), after finding that srcpath is invalid/null , in the patch ? > chooseTargets in ReplicationWork may pass incomplete srcPath > > > Key: HDFS-9071 > URL: https://issues.apache.org/jira/browse/HDFS-9071 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 >Reporter: He Tianyi >Assignee: He Tianyi > Attachments: HDFS-9071.0001.patch > > > I've observed that chooseTargets in ReplicationWork may pass incomplete > srcPath (not starting with '/') to block placement policy. > It is possible that srcPath is extensively used in custom placement policy. > In this case, the incomplete srcPath may further cause AssertionError if try > to get INode with it inside placement policy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7858) Improve HA Namenode Failover detection on the client
[ https://issues.apache.org/jira/browse/HDFS-7858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904584#comment-14904584 ] Arun Suresh commented on HDFS-7858: --- [~szetszwo], Unfortunately, I do not remember the specifics, but think it went in to minutes.. > Improve HA Namenode Failover detection on the client > > > Key: HDFS-7858 > URL: https://issues.apache.org/jira/browse/HDFS-7858 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Arun Suresh >Assignee: Arun Suresh > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HDFS-7858.1.patch, HDFS-7858.10.patch, > HDFS-7858.10.patch, HDFS-7858.11.patch, HDFS-7858.12.patch, > HDFS-7858.13.patch, HDFS-7858.2.patch, HDFS-7858.2.patch, HDFS-7858.3.patch, > HDFS-7858.4.patch, HDFS-7858.5.patch, HDFS-7858.6.patch, HDFS-7858.7.patch, > HDFS-7858.8.patch, HDFS-7858.9.patch > > > In an HA deployment, Clients are configured with the hostnames of both the > Active and Standby Namenodes.Clients will first try one of the NNs > (non-deterministically) and if its a standby NN, then it will respond to the > client to retry the request on the other Namenode. > If the client happens to talks to the Standby first, and the standby is > undergoing some GC / is busy, then those clients might not get a response > soon enough to try the other NN. > Proposed Approach to solve this : > 1) Use hedged RPCs to simultaneously call multiple configured NNs to decide > which is the active Namenode. > 2) Subsequent calls, will invoke the previously successful NN. > 3) On failover of the currently active NN, the remaining NNs will be invoked > to decide which is the new active -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[ https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904595#comment-14904595 ] Wei-Chiu Chuang commented on HDFS-9123: --- However, the fact that the source path is the root, does not imply the destination path must be its sub-directory. Correct? I mean they do not necessarily belong to the same file system. > Copying from the root to a subdirectory should be forbidden > --- > > Key: HDFS-9123 > URL: https://issues.apache.org/jira/browse/HDFS-9123 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch > > > HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp > /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, > /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) > However, the existing path validation fails if the source path is the root, > causing infinite copying, until the disk space is filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9013) Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
[
https://issues.apache.org/jira/browse/HDFS-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904559#comment-14904559
]
Hudson commented on HDFS-9013:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2345 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2345/])
HDFS-9013. Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from
trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev
a2c76e5f26301d4b01e56b347442f3dec171591d)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
* hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
> Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk
> --
>
> Key: HDFS-9013
> URL: https://issues.apache.org/jira/browse/HDFS-9013
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-9013-branch-2.003.patch,
> HDFS-9013-branch-2.004.patch, HDFS-9013.001-branch-2.patch,
> HDFS-9013.001.patch, HDFS-9013.002-branch-2.patch
>
>
> HDFS-8388 added one new metric {{NNStartedTimeInMillis}} to get NN start time
> in milliseconds.
> Now based on [~wheat9] and [~ajisakaa] suggestions now we should deprecate
> {{NameNodeMXBean#getNNStarted}} in branch2 and remove from trunk.
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14709614=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14709614
> https://issues.apache.org/jira/browse/HDFS-8388?focusedCommentId=14726746=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14726746
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8882) Erasure Coding: Use datablocks, parityblocks and cell size from ErasureCodingPolicy
[ https://issues.apache.org/jira/browse/HDFS-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904570#comment-14904570 ] Zhe Zhang commented on HDFS-8882: - Thanks Vinay for updating the patch and Walter for the review. I just triggered Jenkins again since branch was updated before last Jenkins run. > Erasure Coding: Use datablocks, parityblocks and cell size from > ErasureCodingPolicy > --- > > Key: HDFS-8882 > URL: https://issues.apache.org/jira/browse/HDFS-8882 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-8882-HDFS-7285-01.patch, > HDFS-8882-HDFS-7285-02.patch, HDFS-8882-HDFS-7285-03.patch > > > As part of earlier development, constants were used for datablocks, parity > blocks and cellsize. > Now all these are available in ec zone. Use from there and stop using > constant values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9105) separate replication config for client to create a file or set replication
[ https://issues.apache.org/jira/browse/HDFS-9105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904640#comment-14904640 ] Chang Li commented on HDFS-9105: Hi [~cmccabe], thanks for comment. What I am trying to achieve is to separate the min replication factor to enforce when create a file and the min replication factor to check when complete block. > separate replication config for client to create a file or set replication > -- > > Key: HDFS-9105 > URL: https://issues.apache.org/jira/browse/HDFS-9105 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: HDFS-9105.patch > > > one scenario when we want this new config: we want to enforce min replication > of value 2, when client try to create a file, it's check against the > replication factor of the new config, which is set to 2, so that client can > not create a file of replication of 1. But when client try to close file and > there are some datanode failures in the pipeline and only 1 datanode remain, > it is checked against old minReplication value, which has value 1, so we will > allow client to close the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9125) Display help if the command option to "hdfs dfs " is not valid
[
https://issues.apache.org/jira/browse/HDFS-9125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nijel updated HDFS-9125:
Attachment: HDFS-9125_1.patch
attached the changes.
Please review
> Display help if the command option to "hdfs dfs " is not valid
> ---
>
> Key: HDFS-9125
> URL: https://issues.apache.org/jira/browse/HDFS-9125
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: nijel
>Assignee: nijel
>Priority: Minor
> Attachments: HDFS-9125_1.patch
>
>
> {noformat}
> master:/home/nijel/hadoop-3.0.0-SNAPSHOT/bin # ./hdfs dfs -mkdirs
> -mkdirs: Unknown command
> {noformat}
> Better to display the help info.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8873) throttle directoryScanner
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904730#comment-14904730 ] Daniel Templeton commented on HDFS-8873: Forgot to mention that this patch tries be fair about oversleeps when throttling, but it doesn't do anything to credit for full seconds lost entirely to oversleeps. For example, if the throttle is set to 100ms, the following could happen: . - thread calls throttle(), no block .0422 - thread calls throttle(), sleep for 588ms .5016 - thread wakes up from oversleep, run limit this second set to 116ms .5219 - thread calls throttle(), sleep for 781ms In the second that the thread wakes up from the oversleep, we try to give it its full run time, but we don't credit it for full seconds lost. Notice also that threads only offer to block once per cycle, so setting a very low throttle limit will just make the threads run one cycle between sleeps. The following could also happen: . - thread calls throttle(), no block .0422 - thread calls throttle(), sleep for 588ms .1970 - thread wakes up from oversleep, run limit this second set to 1070ms .2089 - thread calls throttle(), no block .2101 - thread calls throttle(), sleep for 899ms When the thread wakes up from the oversleep, we give it more than a second's worth of run time because it's so close to the end of the second. (Any value over 999 just means not to throttle the thread that second.) The next second starts the count over, and at .2089, even though it's been running for over 100 consecutive ms, we don't block it, because it hasn't run for 100ms in this second yet. When it checks in the next time, at .2101, we see it's over the 100ms limit and block it for the rest of the second. > throttle directoryScanner > - > > Key: HDFS-8873 > URL: https://issues.apache.org/jira/browse/HDFS-8873 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: Daniel Templeton > Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch, > HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch, > HDFS-8873.006.patch > > > The new 2-level directory layout can make directory scans expensive in terms > of disk seeks (see HDFS-8791) for details. > It would be good if the directoryScanner() had a configurable duty cycle that > would reduce its impact on disk performance (much like the approach in > HDFS-8617). > Without such a throttle, disks can go 100% busy for many minutes at a time > (assuming the common case of all inodes in cache but no directory blocks > cached, 64K seeks are required for full directory listing which translates to > 655 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8053) Move DFSIn/OutputStream and related classes to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8053:
-
Status: Patch Available (was: Open)
> Move DFSIn/OutputStream and related classes to hadoop-hdfs-client
> -
>
> Key: HDFS-8053
> URL: https://issues.apache.org/jira/browse/HDFS-8053
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-8053.000.patch
>
>
> This jira tracks the effort of moving the {{DFSInputStream}} and
> {{DFSOutputSream}} classes from {{hadoop-hdfs}} to {{hadoop-hdfs-client}}
> module.
> Guidelines:
> * As the {{DFSClient}} is heavily coupled to these two classes, we should
> move it together.
> * Related classes should be addressed in separate jiras if they're
> independent and complex enough.
> * The checkstyle warnings can be addressed in [HDFS-8979 |
> https://issues.apache.org/jira/browse/HDFS-8979]
> * Removing the _slf4j_ logger guards when calling {{LOG.debug()}} and
> {{LOG.trace()}} can be addressed in [HDFS-8971 |
> https://issues.apache.org/jira/browse/HDFS-8971].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904865#comment-14904865
]
Hadoop QA commented on HDFS-9100:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 55s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:red}-1{color} | javac | 7m 58s | The applied patch generated 2
additional warning messages. |
| {color:green}+1{color} | javadoc | 10m 12s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 25s | The applied patch generated 1
new checkstyle issues (total was 46, now 47). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 32s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 20s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 193m 30s | Tests failed in hadoop-hdfs. |
| | | 239m 27s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761884/HDFS-9100.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a2c76e5 |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/12622/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12622/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12622/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12622/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12622/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12622/console |
This message was automatically generated.
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8739) Move DFSClient to client implementation package
[
https://issues.apache.org/jira/browse/HDFS-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904737#comment-14904737
]
Hadoop QA commented on HDFS-8739:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 30s | Findbugs (version 3.0.0)
appears to be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 69 new or modified test files. |
| {color:green}+1{color} | javac | 7m 50s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 59s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 49s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 16s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 42s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 18s | The patch appears to introduce 2
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 193m 39s | Tests failed in hadoop-hdfs. |
| {color:red}-1{color} | hdfs tests | 0m 14s | Tests failed in
hadoop-hdfs-nfs. |
| | | 238m 33s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.TestDFSOutputStream |
| | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA |
| | hadoop.fs.TestResolveHdfsSymlink |
| | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.web.TestWebHDFSForHA |
| | hadoop.hdfs.security.TestDelegationToken |
| Failed build | hadoop-hdfs-nfs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761879/HDFS-8739-002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a2c76e5 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12621/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12621/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12621/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-nfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12621/artifact/patchprocess/testrun_hadoop-hdfs-nfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12621/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12621/console |
This message was automatically generated.
> Move DFSClient to client implementation package
> ---
>
> Key: HDFS-8739
> URL: https://issues.apache.org/jira/browse/HDFS-8739
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Yi Liu
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-8739-002.patch, HDFS-8739.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8873) throttle directoryScanner
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8873: --- Attachment: (was: HDFS-8873.006.patch) > throttle directoryScanner > - > > Key: HDFS-8873 > URL: https://issues.apache.org/jira/browse/HDFS-8873 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: Daniel Templeton > Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch, > HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch > > > The new 2-level directory layout can make directory scans expensive in terms > of disk seeks (see HDFS-8791) for details. > It would be good if the directoryScanner() had a configurable duty cycle that > would reduce its impact on disk performance (much like the approach in > HDFS-8617). > Without such a throttle, disks can go 100% busy for many minutes at a time > (assuming the common case of all inodes in cache but no directory blocks > cached, 64K seeks are required for full directory listing which translates to > 655 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8873) throttle directoryScanner
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8873: --- Attachment: HDFS-8873.006.patch > throttle directoryScanner > - > > Key: HDFS-8873 > URL: https://issues.apache.org/jira/browse/HDFS-8873 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: Daniel Templeton > Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch, > HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch, > HDFS-8873.006.patch > > > The new 2-level directory layout can make directory scans expensive in terms > of disk seeks (see HDFS-8791) for details. > It would be good if the directoryScanner() had a configurable duty cycle that > would reduce its impact on disk performance (much like the approach in > HDFS-8617). > Without such a throttle, disks can go 100% busy for many minutes at a time > (assuming the common case of all inodes in cache but no directory blocks > cached, 64K seeks are required for full directory listing which translates to > 655 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9126) namenode crash in fsimage download/transfer
[ https://issues.apache.org/jira/browse/HDFS-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904724#comment-14904724 ] nijel commented on HDFS-9126: - [~pingley] Can you attach the logs or the error messages for more clarity ? > namenode crash in fsimage download/transfer > --- > > Key: HDFS-9126 > URL: https://issues.apache.org/jira/browse/HDFS-9126 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: OS:Centos 6.5(final) > Hadoop:2.6.0 > namenode ha base 5 journalnode >Reporter: zengyongping >Priority: Critical > > In our product Hadoop cluster,when active namenode begin download/transfer > fsimage from standby namenode.some times zkfc monitor health of NameNode > socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING > ,happen hadoop namenode ha failover,fence old active namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9112) Haadmin fails if multiple name service IDs are configured
[
https://issues.apache.org/jira/browse/HDFS-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904789#comment-14904789
]
Daniel Templeton commented on HDFS-9112:
Thanks, Anu, I haven't tested the patch yet, but from reading the changes, here
are my initial comments. (I'm playing with formatting options for the code
macro. Hopefully this comes through as intended.)
{code:title=DFSUtil.java|linenumbers=true|firstline=1148}
// HDFS-6376 Added the ability to support multiple NameService IDs.
// if we have multiple nameService IDs we need to read
// dfs.internal.nameservices and pick that name as the internal Name
// Service Name.
{code}
Love that you're explaining what you're doing, but let's leave the JIRA ID out
of it and don't explain it in terms of what changed. The comment should just
explain the current state of things.
{code:title=DFSUtil.java|linenumbers=true|firstline=1169}
return internalNSIds.toArray(new String[1])[0];
{code}
I think {code}internalNSIds.iterator().next(){code} is the more standard way to
do that.
{code:title=HdfsClientConfigKeys.java|linenumbers=true|firstline=49}
String DFS_INTERNAL_NAMESERVICES_KEY = "dfs.internal.nameservices";
{code}
I realize that you're just following the pattern set with DFS_NAMESERVICES_KEY,
but I really don't like the idea of duplicating key names between DFSConfigKeys
and HdfsClientConfigKeys. If there some reason not to use
DFSConfigKeys.DFS_INTERNAL_NAMESERVICES_KEY?
> Haadmin fails if multiple name service IDs are configured
> -
>
> Key: HDFS-9112
> URL: https://issues.apache.org/jira/browse/HDFS-9112
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HDFS-9112.001.patch, HDFS-9112.002.patch
>
>
> In HDFS-6376 we supported a feature for distcp that allows multiple
> NameService IDs to be specified so that we can copy from two HA enabled
> clusters.
> That confuses haadmin command since we have a check in
> DFSUtil#getNamenodeServiceAddr which fails if it finds more than 1 name in
> that property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9112) Haadmin fails if multiple name service IDs are configured
[ https://issues.apache.org/jira/browse/HDFS-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904828#comment-14904828 ] Anu Engineer commented on HDFS-9112: [~templedf] Thank you very much for your code review and comments. Before we begin, I want to point out that [~jingzhao] commented that we should *not* fix this issue at all, but just add a warning for the user that they should use "-ns" option in the error message.This is based on the discussion in HDFS-6376. So I have a second patch which does exactly that. So I am guessing that second patch will get applied by [~jingzhao]. Meanwhile I will post one more patch that modifies the first code path in case you would like to use it. bq. Love that you're explaining what you're doing, but let's leave the JIRA ID out of it and don't explain it in terms of what changed. The comment should just explain the current state of things. Agreed, let me remove the redundant JIRA number and fix that comment. bq. internalNSIds.iterator().next() It might be, however I was just following the pattern in the nearby code. It is easier to read code if we have same patterns in a file. bq. I really don't like the idea of duplicating key names between DFSConfigKeys and HdfsClientConfigKeys Not really sure about this, but I am guessing this might be due to the fact that we have an on-going refactoring work being done which separates HDFS client into its own jar. > Haadmin fails if multiple name service IDs are configured > - > > Key: HDFS-9112 > URL: https://issues.apache.org/jira/browse/HDFS-9112 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-9112.001.patch, HDFS-9112.002.patch > > > In HDFS-6376 we supported a feature for distcp that allows multiple > NameService IDs to be specified so that we can copy from two HA enabled > clusters. > That confuses haadmin command since we have a check in > DFSUtil#getNamenodeServiceAddr which fails if it finds more than 1 name in > that property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[
https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904747#comment-14904747
]
Yongjun Zhang commented on HDFS-9123:
-
Hi Guys, thanks for the review and comments.
{quote}
However, the fact that the source path is the root, does not imply the
destination path must be its sub-directory. Correct? I mean they do not
necessarily belong to the same file system.
{quote}
Correct.
About
{quote}
should the tests really downgrade exceptions to logs? I'd expect them to signal
failure and be rethrown
{quote}
Good point. Since all the tests in this test file are coded like that, I think
we can create a new jira to fix all of them.
For HDFS-9123, I'd suggest that we should make the newly added test fail when
without the fix (it currently succeeds as I mentioned earlier), and make it
succeed when with the fix.
Do you guys agree?
Thanks.
> Copying from the root to a subdirectory should be forbidden
> ---
>
> Key: HDFS-9123
> URL: https://issues.apache.org/jira/browse/HDFS-9123
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch
>
>
> HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp
> /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz,
> /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc)
> However, the existing path validation fails if the source path is the root,
> causing infinite copying, until the disk space is filled up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8882) Erasure Coding: Use datablocks, parityblocks and cell size from ErasureCodingPolicy
[
https://issues.apache.org/jira/browse/HDFS-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904754#comment-14904754
]
Hadoop QA commented on HDFS-8882:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 2s | Findbugs (version ) appears to
be broken on HDFS-7285. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 26 new or modified test files. |
| {color:green}+1{color} | javac | 7m 56s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 25s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 0s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 6s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 4m 38s | The patch appears to introduce 3
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 77m 37s | Tests failed in hadoop-hdfs. |
| {color:red}-1{color} | hdfs tests | 0m 21s | Tests failed in
hadoop-hdfs-client. |
| | | 123m 49s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-client |
| Failed unit tests | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.TestRead |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.namenode.TestSecondaryWebUi |
| | hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional |
| | hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd |
| | hadoop.hdfs.server.namenode.ha.TestHAStateTransitions |
| | hadoop.hdfs.server.datanode.TestRefreshNamenodes |
| | hadoop.hdfs.TestHdfsAdmin |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN |
| | hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler |
| | hadoop.hdfs.TestClientReportBadBlock |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.server.namenode.TestAuditLogger |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy |
| | hadoop.hdfs.server.blockmanagement.TestDatanodeManager |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement |
| | hadoop.hdfs.server.namenode.TestNameNodeRpcServer |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade |
| | hadoop.cli.TestErasureCodingCLI |
| | hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot |
| | hadoop.hdfs.server.namenode.ha.TestHAFsck |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant |
| | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
| | hadoop.hdfs.TestFileStatusWithECPolicy |
| | hadoop.hdfs.server.namenode.TestHDFSConcat |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.TestRefreshCallQueue |
| | hadoop.hdfs.TestListFilesInDFS |
| | hadoop.hdfs.server.namenode.TestAddBlock |
| | hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold |
| | hadoop.hdfs.server.namenode.TestNameEditsConfigs |
| | hadoop.hdfs.TestMiniDFSCluster |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.security.TestPermissionSymlinks |
| | hadoop.hdfs.TestDFSRollback |
| | hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica |
| | hadoop.hdfs.TestFileConcurrentReader |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.server.datanode.TestDataNodeExit |
| | hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId |
| | hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA |
| | hadoop.hdfs.TestGetFileChecksum |
| |
[jira] [Updated] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[ https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-9123: -- Attachment: HDFS-9123.003.patch Updated the code to deal with this bug as a special case when the source is the root. > Copying from the root to a subdirectory should be forbidden > --- > > Key: HDFS-9123 > URL: https://issues.apache.org/jira/browse/HDFS-9123 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch, > HDFS-9123.003.patch > > > HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp > /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, > /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) > However, the existing path validation fails if the source path is the root, > causing infinite copying, until the disk space is filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8873) throttle directoryScanner
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8873: --- Attachment: HDFS-8873.006.patch New patch to address [~cmccabe]'s comments. Based on the results of previous patch, this one will get flagged with 3 whitespace violations in lines I didn't touch and a fist full of checkstyle issues because of the length of the config key names. :) > throttle directoryScanner > - > > Key: HDFS-8873 > URL: https://issues.apache.org/jira/browse/HDFS-8873 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: Daniel Templeton > Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch, > HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch, > HDFS-8873.006.patch > > > The new 2-level directory layout can make directory scans expensive in terms > of disk seeks (see HDFS-8791) for details. > It would be good if the directoryScanner() had a configurable duty cycle that > would reduce its impact on disk performance (much like the approach in > HDFS-8617). > Without such a throttle, disks can go 100% busy for many minutes at a time > (assuming the common case of all inodes in cache but no directory blocks > cached, 64K seeks are required for full directory listing which translates to > 655 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9125) Display help if the command option to "hdfs dfs " is not valid
[
https://issues.apache.org/jira/browse/HDFS-9125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904786#comment-14904786
]
Hadoop QA commented on HDFS-9125:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 8m 7s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 10s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 14s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 56s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 1s | Tests passed in
hadoop-common. |
| | | 64m 6s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761912/HDFS-9125_1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a2c76e5 |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12624/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12624/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12624/console |
This message was automatically generated.
> Display help if the command option to "hdfs dfs " is not valid
> ---
>
> Key: HDFS-9125
> URL: https://issues.apache.org/jira/browse/HDFS-9125
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: nijel
>Assignee: nijel
>Priority: Minor
> Attachments: HDFS-9125_1.patch
>
>
> {noformat}
> master:/home/nijel/hadoop-3.0.0-SNAPSHOT/bin # ./hdfs dfs -mkdirs
> -mkdirs: Unknown command
> {noformat}
> Better to display the help info.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9126) namenode crash in fsimage download/transfer
[ https://issues.apache.org/jira/browse/HDFS-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengyongping updated HDFS-9126: --- Description: In our product Hadoop cluster,when active namenode begin download/transfer fsimage from standby namenode.some times zkfc monitor health of NameNode socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING ,happen hadoop namenode ha failover,fence old active namenode. zkfc logs: 2015-09-24 11:44:44,739 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hostname1/192.168.10.11:8020: Call From hostname1/192.168.10.11 to hostname1:8020 failed on socket timeout exception: java.net.SocketTimeoutException: 45000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.10.11:22614 remote=hostname1/192.168.10.11:8020]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hostname1/192.168.10.11:8020 entered state: SERVICE_NOT_RESPONDING 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hostname1/192.168.10.11:8020 and marking that fencing is necessary 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2015-09-24 11:44:44,761 INFO org.apache.zookeeper.ZooKeeper: Session: 0x54d81348fe503e3 closed 2015-09-24 11:44:44,761 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x54d81348fe503e3 2015-09-24 11:44:44,764 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down was: In our product Hadoop cluster,when active namenode begin download/transfer fsimage from standby namenode.some times zkfc monitor health of NameNode socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING ,happen hadoop namenode ha failover,fence old active namenode. > namenode crash in fsimage download/transfer > --- > > Key: HDFS-9126 > URL: https://issues.apache.org/jira/browse/HDFS-9126 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: OS:Centos 6.5(final) > Hadoop:2.6.0 > namenode ha base 5 journalnode >Reporter: zengyongping >Priority: Critical > > In our product Hadoop cluster,when active namenode begin download/transfer > fsimage from standby namenode.some times zkfc monitor health of NameNode > socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING > ,happen hadoop namenode ha failover,fence old active namenode. > zkfc logs: > 2015-09-24 11:44:44,739 WARN org.apache.hadoop.ha.HealthMonitor: > Transport-level exception trying to monitor health of NameNode at > hostname1/192.168.10.11:8020: Call From hostname1/192.168.10.11 to > hostname1:8020 failed on socket timeout exception: > java.net.SocketTimeoutException: 45000 millis timeout while waiting for > channel to be ready for read. ch : java.nio.channels.SocketChannel[connected > local=/192.168.10.11:22614 remote=hostname1/192.168.10.11:8020]; For more > details see: http://wiki.apache.org/hadoop/SocketTimeout > 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.HealthMonitor: Entering > state SERVICE_NOT_RESPONDING > 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.ZKFailoverController: Local > service NameNode at hostname1/192.168.10.11:8020 entered state: > SERVICE_NOT_RESPONDING > 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.ZKFailoverController: > Quitting master election for NameNode at hostname1/192.168.10.11:8020 and > marking that fencing is necessary > 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.ActiveStandbyElector: > Yielding from election > 2015-09-24 11:44:44,761 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x54d81348fe503e3 closed > 2015-09-24 11:44:44,761 WARN org.apache.hadoop.ha.ActiveStandbyElector: > Ignoring stale result from old client with sessionId 0x54d81348fe503e3 > 2015-09-24 11:44:44,764 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-9136) TestDFSUpgrade leaks file descriptors.
[
https://issues.apache.org/jira/browse/HDFS-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-9136.
-
Resolution: Not A Problem
I was mistaken. This actually got fixed as part of the test changes done in
HDFS-8846. I'm no longer seeing a problem after that patch. I'll resolve this.
> TestDFSUpgrade leaks file descriptors.
> --
>
> Key: HDFS-9136
> URL: https://issues.apache.org/jira/browse/HDFS-9136
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
>
> HDFS-8480 introduced code in {{TestDFSUpgrade#testPreserveEditLogs}} that
> opens edit log files and reads from them, but these files are never closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8180) AbstractFileSystem Implementation for WebHdfs
[ https://issues.apache.org/jira/browse/HDFS-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905843#comment-14905843 ] Chris Nauroth commented on HDFS-8180: - This patch introduced a test failure on Windows, which has since been fixed in HDFS-9128. > AbstractFileSystem Implementation for WebHdfs > - > > Key: HDFS-8180 > URL: https://issues.apache.org/jira/browse/HDFS-8180 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.6.0 >Reporter: Santhosh G Nayak >Assignee: Santhosh G Nayak > Labels: hadoop > Fix For: 2.8.0 > > Attachments: HDFS-8180-1.patch, HDFS-8180-2.patch, HDFS-8180-3.patch, > HDFS-8180-4.patch > > > Add AbstractFileSystem implementation for WebHdfs to support FileContext APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9133) ExternalBlockReader and ReplicaAccessor need to return -1 on read when at EOF
[
https://issues.apache.org/jira/browse/HDFS-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905849#comment-14905849
]
Hadoop QA commented on HDFS-9133:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 1s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 25s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 58s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 39s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 16s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 194m 38s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 31s | Tests passed in
hadoop-hdfs-client. |
| | | 247m 8s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12762029/HDFS-9133.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 1f707ec |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12647/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12647/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12647/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12647/console |
This message was automatically generated.
> ExternalBlockReader and ReplicaAccessor need to return -1 on read when at EOF
> -
>
> Key: HDFS-9133
> URL: https://issues.apache.org/jira/browse/HDFS-9133
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.8.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-9133.001.patch
>
>
> ExternalBlockReader and ReplicaAccessor need to return -1 on read when at
> EOF, as per the JavaDoc in BlockReader.java.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905859#comment-14905859
]
Hadoop QA commented on HDFS-9100:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 17s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 8m 0s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 3s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 24s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 27s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 35s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 126m 36s | Tests failed in hadoop-hdfs. |
| | | 172m 34s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestSecondaryWebUi |
| | hadoop.hdfs.TestHFlush |
| Timed out tests |
org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12762042/HDFS-9100.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 06d1c90 |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12651/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12651/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12651/console |
This message was automatically generated.
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9137) DeadLock between DataNode#refreshVolumes and BPOfferService#registrationSucceeded
Uma Maheswara Rao G created HDFS-9137: - Summary: DeadLock between DataNode#refreshVolumes and BPOfferService#registrationSucceeded Key: HDFS-9137 URL: https://issues.apache.org/jira/browse/HDFS-9137 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G I can see this code flows between DataNode#refreshVolumes and BPOfferService#registrationSucceeded could cause deadLock. In practice situation may be rare as user calling refreshVolumes at the time DN registration with NN. But seems like issue can happen. Reason for deadLock: 1) refreshVolumes will be called with DN lock and after at the end it will also trigger Block report. In the Block report call, BPServiceActor#triggerBlockReport calls toString on bpos. Here it takes readLock on bpos. DN lock then boos lock 2) BPOfferSetrvice#registrationSucceeded call is taking writeLock on bpos and calling dn.bpRegistrationSucceeded which is again synchronized call on DN. bpos lock and then DN lock. So, this can clearly create dead lock. I think simple fix could be to move triggerBlockReport call outside out DN lock and I feel that call may not be really needed inside DN lock. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8696) Reduce the variances of latency of WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905863#comment-14905863
]
Hadoop QA commented on HDFS-8696:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 9s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 8m 5s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 24s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 23s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 11s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 0m 21s | Tests failed in hadoop-hdfs. |
| | | 46m 34s | |
\\
\\
|| Reason || Tests ||
| Failed build | hadoop-hdfs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12762052/HDFS-8696.007.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 06d1c90 |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12653/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12653/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12653/console |
This message was automatically generated.
> Reduce the variances of latency of WebHDFS
> --
>
> Key: HDFS-8696
> URL: https://issues.apache.org/jira/browse/HDFS-8696
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-8696.004.patch, HDFS-8696.005.patch,
> HDFS-8696.006.patch, HDFS-8696.007.patch, HDFS-8696.1.patch,
> HDFS-8696.2.patch, HDFS-8696.3.patch
>
>
> There is an issue that appears related to the webhdfs server. When making two
> concurrent requests, the DN will sometimes pause for extended periods (I've
> seen 1-300 seconds), killing performance and dropping connections.
> To reproduce:
> 1. set up a HDFS cluster
> 2. Upload a large file (I was using 10GB). Perform 1-byte reads, writing
> the time out to /tmp/times.txt
> {noformat}
> i=1
> while (true); do
> echo $i
> let i++
> /usr/bin/time -f %e -o /tmp/times.txt -a curl -s -L -o /dev/null
> "http://:50070/webhdfs/v1/tmp/bigfile?op=OPEN=root=1";
> done
> {noformat}
> 3. Watch for 1-byte requests that take more than one second:
> tail -F /tmp/times.txt | grep -E "^[^0]"
> 4. After it has had a chance to warm up, start doing large transfers from
> another shell:
> {noformat}
> i=1
> while (true); do
> echo $i
> let i++
> /usr/bin/time -f %e curl -s -L -o /dev/null
> "http://:50070/webhdfs/v1/tmp/bigfile?op=OPEN=root";
> done
> {noformat}
> It's easy to find after a minute or two that small reads will sometimes
> pause for 1-300 seconds. In some extreme cases, it appears that the
> transfers timeout and the DN drops the connection.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9126) namenode crash in fsimage download/transfer
[ https://issues.apache.org/jira/browse/HDFS-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengyongping updated HDFS-9126: --- Environment: OS:Centos 6.5(final) Hadoop:2.6.0 namenode ha base 5 journalnodes was: OS:Centos 6.5(final) Hadoop:2.6.0 namenode ha base 5 journalnode > namenode crash in fsimage download/transfer > --- > > Key: HDFS-9126 > URL: https://issues.apache.org/jira/browse/HDFS-9126 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: OS:Centos 6.5(final) > Hadoop:2.6.0 > namenode ha base 5 journalnodes >Reporter: zengyongping >Priority: Critical > > In our product Hadoop cluster,when active namenode begin download/transfer > fsimage from standby namenode.some times zkfc monitor health of NameNode > socket timeout,zkfs judge active namenode status SERVICE_NOT_RESPONDING > ,happen hadoop namenode ha failover,fence old active namenode. > zkfc logs: > 2015-09-24 11:44:44,739 WARN org.apache.hadoop.ha.HealthMonitor: > Transport-level exception trying to monitor health of NameNode at > hostname1/192.168.10.11:8020: Call From hostname1/192.168.10.11 to > hostname1:8020 failed on socket timeout exception: > java.net.SocketTimeoutException: 45000 millis timeout while waiting for > channel to be ready for read. ch : java.nio.channels.SocketChannel[connected > local=/192.168.10.11:22614 remote=hostname1/192.168.10.11:8020]; For more > details see: http://wiki.apache.org/hadoop/SocketTimeout > 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.HealthMonitor: Entering > state SERVICE_NOT_RESPONDING > 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.ZKFailoverController: Local > service NameNode at hostname1/192.168.10.11:8020 entered state: > SERVICE_NOT_RESPONDING > 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.ZKFailoverController: > Quitting master election for NameNode at hostname1/192.168.10.11:8020 and > marking that fencing is necessary > 2015-09-24 11:44:44,740 INFO org.apache.hadoop.ha.ActiveStandbyElector: > Yielding from election > 2015-09-24 11:44:44,761 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x54d81348fe503e3 closed > 2015-09-24 11:44:44,761 WARN org.apache.hadoop.ha.ActiveStandbyElector: > Ignoring stale result from old client with sessionId 0x54d81348fe503e3 > 2015-09-24 11:44:44,764 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[
https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904908#comment-14904908
]
Mingliang Liu commented on HDFS-9123:
-
This is a good point. I supposed this is the case for {{distcp}} instead of
{{cp}}?
> Copying from the root to a subdirectory should be forbidden
> ---
>
> Key: HDFS-9123
> URL: https://issues.apache.org/jira/browse/HDFS-9123
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch,
> HDFS-9123.003.patch
>
>
> HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp
> /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz,
> /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc)
> However, the existing path validation fails if the source path is the root,
> causing infinite copying, until the disk space is filled up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[
https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904954#comment-14904954
]
Yongjun Zhang commented on HDFS-9123:
-
Hello [~liuml07],
Thanks for the comments.
{quote}
This is a good point. I supposed this is the case for distcp instead of cp?
{quote}
it's for "fs-cp", not distcp here. It also supports copying to different file
systems.
Hi [~jojochuang],
Thanks for the new rev.
Would you please address comment # 1 at
https://issues.apache.org/jira/browse/HDFS-9123?focusedCommentId=14904029=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14904029
?
About my comment #2, I studied the code a bit, and found that FsShell simply
absort all exceptions, and return status code, the client code has no way to
tell what's the exact failure reason if a command failed. So I'm ok that
comment #2 is not addressed. At least, the new test can show that copying from
root to child fail the same way as copying from any parent to child.
Thanks.
> Copying from the root to a subdirectory should be forbidden
> ---
>
> Key: HDFS-9123
> URL: https://issues.apache.org/jira/browse/HDFS-9123
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch,
> HDFS-9123.003.patch
>
>
> HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp
> /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz,
> /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc)
> However, the existing path validation fails if the source path is the root,
> causing infinite copying, until the disk space is filled up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[ https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904925#comment-14904925 ] Mingliang Liu commented on HDFS-9123: - The patch looks good to me. > Copying from the root to a subdirectory should be forbidden > --- > > Key: HDFS-9123 > URL: https://issues.apache.org/jira/browse/HDFS-9123 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch, > HDFS-9123.003.patch > > > HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp > /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, > /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) > However, the existing path validation fails if the source path is the root, > causing infinite copying, until the disk space is filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8976) Create HTML5 cluster webconsole for federated cluster
[
https://issues.apache.org/jira/browse/HDFS-8976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905078#comment-14905078
]
Vinayakumar B commented on HDFS-8976:
-
Thanks [~wheat9].
Regards,
Vinay
> Create HTML5 cluster webconsole for federated cluster
> -
>
> Key: HDFS-8976
> URL: https://issues.apache.org/jira/browse/HDFS-8976
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 2.7.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-8976-01.patch, HDFS-8976-02.patch,
> HDFS-8976-03.patch, cluster-health.JPG
>
>
> Since the old jsp variant of cluster web console is no longer present from
> 2.7 onwards, there is a need for HTML 5 web console for overview of overall
> cluster.
> 2.7.1 docs says to check webconsole as below {noformat}Similar to the
> Namenode status web page, when using federation a Cluster Web Console is
> available to monitor the federated cluster at
> http:///dfsclusterhealth.jsp. Any Namenode in the cluster
> can be used to access this web page.{noformat}
> But this is no longer present,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-8873) throttle directoryScanner
[
https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905122#comment-14905122
]
Colin Patrick McCabe edited comment on HDFS-8873 at 9/23/15 7:47 PM:
-
I guess the short summary is that I'm not happy with the "modulo" solution to
the timekeeping problem since I think its inaccuracy (in the direction of not
running often enough, or at all) will be unacceptable at lower throttle rates.
We have a StopWatch class which should provide a pretty easy solution to this
problem that doesn't suffer from these issues. Just start a StopWatch when you
enable scanning. Check the elapsed duration on the StopWatch before scanning
each block-- if it is above
dfs.datanode.directoryscan.throttle.limit.ms.per.sec, then disable scanning and
go into the other loop, which calls Thread.sleep until 1000 -
dfs.datanode.directoryscan.throttle.limit.ms.per.sec ms have elapsed.
This solution is robust against slightly shorter or longer thread pauses. Like
the modulo solution, it also may run at a slightly lower rate than expected in
the case where we oversleep. Unlike the modulo solution, it does not have a
pathological case where we never get to run at all despite a non-zero throttle
rate.
{code}
399 public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT =
21600;
...
460 int defaultLimit =
461 DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT;
462 String logMsg;
463
464 if (throttleLimitMsPerSec < defaultLimit) {
465 logMsg = String.format(START_MESSAGE_WITH_THROTTLE, firstScanTime,
466 scanPeriodMsecs, throttleLimitMsPerSec);
467 } else {
468 logMsg = String.format(START_MESSAGE, firstScanTime,
scanPeriodMsecs);
469 }
{code}
I'm confused about this code... why are we comparing the directory scan
interval (in seconds) with the throttle limit in milliseconds? It seems like
this was intended to test for the case where the throttle is disabled, but some
wires got crossed?
was (Author: cmccabe):
I guess the short summary is that I'm not happy with the "modulo" solution to
the timekeeping problem since I think its inaccuracy (in the direction of not
running often enough, or at all) will be unacceptable at lower throttle rates.
We have a StopWatch class which should provide a pretty easy solution to this
problem that doesn't suffer from these issues. Just start a StopWatch when you
enable scanning. Check the elapsed duration on the StopWatch before scanning
each block-- if it is above
dfs.datanode.directoryscan.throttle.limit.ms.per.sec, then disable scanning and
go into the other loop, which calls Thread.sleep until 1000 -
dfs.datanode.directoryscan.throttle.limit.ms.per.sec ms have elapsed.
This solution is robust against slightly shorter or longer thread pauses. Like
the module solution, it also may run at a slightly lower rate than expected in
the case where we oversleep. Unlike the modulo solution, it does not have a
pathological case where we never get to run at all despite a non-zero throttle
rate.
{code}
399 public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT =
21600;
...
460 int defaultLimit =
461 DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT;
462 String logMsg;
463
464 if (throttleLimitMsPerSec < defaultLimit) {
465 logMsg = String.format(START_MESSAGE_WITH_THROTTLE, firstScanTime,
466 scanPeriodMsecs, throttleLimitMsPerSec);
467 } else {
468 logMsg = String.format(START_MESSAGE, firstScanTime,
scanPeriodMsecs);
469 }
{code}
I'm confused about this code... why are we comparing the directory scan
interval (in seconds) with the throttle limit in milliseconds? It seems like
this was intended to test for the case where the throttle is disabled, but some
wires got crossed?
> throttle directoryScanner
> -
>
> Key: HDFS-8873
> URL: https://issues.apache.org/jira/browse/HDFS-8873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Nathan Roberts
>Assignee: Daniel Templeton
> Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch,
> HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch,
> HDFS-8873.006.patch
>
>
> The new 2-level directory layout can make directory scans expensive in terms
> of disk seeks (see HDFS-8791) for details.
> It would be good if the directoryScanner() had a configurable duty cycle that
> would reduce its impact on disk performance (much like the approach in
> HDFS-8617).
> Without such a throttle, disks can go 100% busy for many minutes at a time
> (assuming the common case of all inodes in
[jira] [Commented] (HDFS-9112) Haadmin fails if multiple name service IDs are configured
[ https://issues.apache.org/jira/browse/HDFS-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904929#comment-14904929 ] Daniel Templeton commented on HDFS-9112: [~aengineer], don't sweat it. I saw the second patch, but missed that it was a replacement and not intended as additive. In the second patch, "Name Service" should be "Name Services". > Haadmin fails if multiple name service IDs are configured > - > > Key: HDFS-9112 > URL: https://issues.apache.org/jira/browse/HDFS-9112 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-9112.001.patch, HDFS-9112.002.patch > > > In HDFS-6376 we supported a feature for distcp that allows multiple > NameService IDs to be specified so that we can copy from two HA enabled > clusters. > That confuses haadmin command since we have a check in > DFSUtil#getNamenodeServiceAddr which fails if it finds more than 1 name in > that property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[ https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-9123: -- Status: Open (was: Patch Available) > Copying from the root to a subdirectory should be forbidden > --- > > Key: HDFS-9123 > URL: https://issues.apache.org/jira/browse/HDFS-9123 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch, > HDFS-9123.003.patch > > > HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp > /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, > /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) > However, the existing path validation fails if the source path is the root, > causing infinite copying, until the disk space is filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[ https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-9123: -- Status: Patch Available (was: Open) Rev 3. > Copying from the root to a subdirectory should be forbidden > --- > > Key: HDFS-9123 > URL: https://issues.apache.org/jira/browse/HDFS-9123 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch, > HDFS-9123.003.patch > > > HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp > /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, > /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) > However, the existing path validation fails if the source path is the root, > causing infinite copying, until the disk space is filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[ https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905045#comment-14905045 ] Wei-Chiu Chuang commented on HDFS-9123: --- Agree with Yongjun's comment #2. The shell currently is not designed to throw internal exceptions. It would require another lira to make a better test design though. > Copying from the root to a subdirectory should be forbidden > --- > > Key: HDFS-9123 > URL: https://issues.apache.org/jira/browse/HDFS-9123 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch, > HDFS-9123.003.patch, HDFS-9123.004.patch > > > HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp > /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, > /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) > However, the existing path validation fails if the source path is the root, > causing infinite copying, until the disk space is filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8766) Implement a libhdfs(3) compatible API
[ https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-8766: -- Status: Patch Available (was: In Progress) > Implement a libhdfs(3) compatible API > - > > Key: HDFS-8766 > URL: https://issues.apache.org/jira/browse/HDFS-8766 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-8766.HDFS-8707.000.patch > > > Add a synchronous API that is compatible with the hdfs.h header used in > libhdfs and libhdfs3. This will make it possible for projects using > libhdfs/libhdfs3 to relink against libhdfspp with minimal changes. > This also provides a pure C interface that can be linked against projects > that aren't built in C++11 mode for various reasons but use the same > compiler. It also allows many other programming languages to access > libhdfspp through builtin FFI interfaces. > The libhdfs API is very similar to the posix file API which makes it easier > for programs built using posix filesystem calls to be modified to access HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8873) throttle directoryScanner
[
https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905122#comment-14905122
]
Colin Patrick McCabe commented on HDFS-8873:
I guess the short summary is that I'm not happy with the "modulo" solution to
the timekeeping problem since I think its inaccuracy (in the direction of not
running often enough, or at all) will be unacceptable at lower throttle rates.
We have a StopWatch class which should provide a pretty easy solution to this
problem that doesn't suffer from these issues. Just start a StopWatch when you
enable scanning. Check the elapsed duration on the StopWatch before scanning
each block-- if it is above
dfs.datanode.directoryscan.throttle.limit.ms.per.sec, then disable scanning and
go into the other loop, which calls Thread.sleep until 1000 -
dfs.datanode.directoryscan.throttle.limit.ms.per.sec ms have elapsed.
This solution is robust against slightly shorter or longer thread pauses. Like
the module solution, it also may run at a slightly lower rate than expected in
the case where we oversleep. Unlike the modulo solution, it does not have a
pathological case where we never get to run at all despite a non-zero throttle
rate.
{code}
399 public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT =
21600;
...
460 int defaultLimit =
461 DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT;
462 String logMsg;
463
464 if (throttleLimitMsPerSec < defaultLimit) {
465 logMsg = String.format(START_MESSAGE_WITH_THROTTLE, firstScanTime,
466 scanPeriodMsecs, throttleLimitMsPerSec);
467 } else {
468 logMsg = String.format(START_MESSAGE, firstScanTime,
scanPeriodMsecs);
469 }
{code}
I'm confused about this code... why are we comparing the directory scan
interval (in seconds) with the throttle limit in milliseconds? It seems like
this was intended to test for the case where the throttle is disabled, but some
wires got crossed?
> throttle directoryScanner
> -
>
> Key: HDFS-8873
> URL: https://issues.apache.org/jira/browse/HDFS-8873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Nathan Roberts
>Assignee: Daniel Templeton
> Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch,
> HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch,
> HDFS-8873.006.patch
>
>
> The new 2-level directory layout can make directory scans expensive in terms
> of disk seeks (see HDFS-8791) for details.
> It would be good if the directoryScanner() had a configurable duty cycle that
> would reduce its impact on disk performance (much like the approach in
> HDFS-8617).
> Without such a throttle, disks can go 100% busy for many minutes at a time
> (assuming the common case of all inodes in cache but no directory blocks
> cached, 64K seeks are required for full directory listing which translates to
> 655 seconds)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8873) throttle directoryScanner
[
https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905062#comment-14905062
]
Hadoop QA commented on HDFS-8873:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 15s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 15s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 24s | The applied patch generated 8
new checkstyle issues (total was 442, now 443). |
| {color:red}-1{color} | whitespace | 0m 3s | The patch has 3 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 26s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 16s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 138m 33s | Tests failed in hadoop-hdfs. |
| | | 184m 47s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestDFSRename |
| | hadoop.hdfs.TestDatanodeConfig |
| | hadoop.hdfs.TestBlockReaderFactory |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.TestFileCreationDelete |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestFileConcurrentReader |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.TestBlockStoragePolicy |
| | hadoop.hdfs.TestLeaseRecovery |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.tracing.TestTraceAdmin |
| | hadoop.hdfs.TestWriteConfigurationToDFS |
| | hadoop.hdfs.tools.TestStoragePolicyCommands |
| | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.TestPipelines |
| | hadoop.hdfs.tools.TestGetGroups |
| | hadoop.hdfs.TestParallelShortCircuitRead |
| | hadoop.hdfs.TestRenameWhileOpen |
| | hadoop.hdfs.TestBlocksScheduledCounter |
| | hadoop.hdfs.TestRollingUpgradeRollback |
| | hadoop.hdfs.protocolPB.TestPBHelper |
| | hadoop.hdfs.tools.TestDFSAdminWithHA |
| | hadoop.hdfs.TestDataTransferKeepalive |
| | hadoop.hdfs.TestFSInputChecker |
| | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| | hadoop.hdfs.TestRead |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.TestHDFSTrash |
| | hadoop.hdfs.TestExternalBlockReader |
| | hadoop.hdfs.TestDatanodeRegistration |
| | hadoop.hdfs.TestCrcCorruption |
| | hadoop.hdfs.tools.TestDFSHAAdminMiniCluster |
| | hadoop.hdfs.TestSmallBlock |
| | hadoop.hdfs.TestDFSRollback |
| | hadoop.hdfs.TestDFSStorageStateRecovery |
| | hadoop.hdfs.TestDFSUtil |
| | hadoop.hdfs.TestHdfsAdmin |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary |
| | hadoop.hdfs.TestAbandonBlock |
| | hadoop.hdfs.TestConnCache |
| | hadoop.hdfs.TestFileCorruption |
| | hadoop.hdfs.TestHDFSServerPorts |
| | hadoop.hdfs.TestParallelShortCircuitLegacyRead |
| | hadoop.hdfs.TestRollingUpgradeDowngrade |
| | hadoop.hdfs.TestDFSFinalize |
| | hadoop.hdfs.TestLocalDFS |
| | hadoop.hdfs.TestBlockMissingException |
| | hadoop.hdfs.TestFileAppend3 |
| | hadoop.hdfs.TestFileLengthOnClusterRestart |
| | hadoop.hdfs.crypto.TestHdfsCryptoStreams |
| | hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.tools.TestDebugAdmin |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| | hadoop.hdfs.TestDFSPermission |
| | hadoop.hdfs.TestDFSInotifyEventInputStream |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.TestDFSClientExcludedNodes |
| | hadoop.hdfs.TestMiniDFSCluster |
| | hadoop.hdfs.TestParallelRead |
| | hadoop.hdfs.TestPread |
| | hadoop.tracing.TestTracing |
| | hadoop.hdfs.TestReservedRawPaths |
| | hadoop.hdfs.TestEncryptionZonesWithHA |
| | hadoop.hdfs.TestParallelShortCircuitReadNoChecksum |
| | hadoop.hdfs.TestHFlush |
| | hadoop.hdfs.TestParallelUnixDomainRead |
| |
[jira] [Commented] (HDFS-8882) Erasure Coding: Use datablocks, parityblocks and cell size from ErasureCodingPolicy
[ https://issues.apache.org/jira/browse/HDFS-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905068#comment-14905068 ] Vinayakumar B commented on HDFS-8882: - This build again got hit with the Collision of other build on same machine. > Erasure Coding: Use datablocks, parityblocks and cell size from > ErasureCodingPolicy > --- > > Key: HDFS-8882 > URL: https://issues.apache.org/jira/browse/HDFS-8882 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-8882-HDFS-7285-01.patch, > HDFS-8882-HDFS-7285-02.patch, HDFS-8882-HDFS-7285-03.patch > > > As part of earlier development, constants were used for datablocks, parity > blocks and cellsize. > Now all these are available in ec zone. Use from there and stop using > constant values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8766) Implement a libhdfs(3) compatible API
[ https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-8766: -- Attachment: HDFS-8766.HDFS-8707.000.patch Some notes on first patch: -Just enough done to open, read, and close a file. More patches are coming but I'd like some early feedback. -This includes the patch from HDFS-9108 -Two hdfs.h files in the same project will likely lead to issues, I'd like to keep compatibility/hdfs.h named as it is so people don't have to change their includes to link Thoughts on more appropriate errnos would be appreciated. > Implement a libhdfs(3) compatible API > - > > Key: HDFS-8766 > URL: https://issues.apache.org/jira/browse/HDFS-8766 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-8766.HDFS-8707.000.patch > > > Add a synchronous API that is compatible with the hdfs.h header used in > libhdfs and libhdfs3. This will make it possible for projects using > libhdfs/libhdfs3 to relink against libhdfspp with minimal changes. > This also provides a pure C interface that can be linked against projects > that aren't built in C++11 mode for various reasons but use the same > compiler. It also allows many other programming languages to access > libhdfspp through builtin FFI interfaces. > The libhdfs API is very similar to the posix file API which makes it easier > for programs built using posix filesystem calls to be modified to access HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8880) NameNode metrics logging
[ https://issues.apache.org/jira/browse/HDFS-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904915#comment-14904915 ] Arpit Agarwal commented on HDFS-8880: - I hadn't seen it before. Thanks for mentioning it. TheFileContext output format could probably be cleaned up to get the same effect as this change. > NameNode metrics logging > > > Key: HDFS-8880 > URL: https://issues.apache.org/jira/browse/HDFS-8880 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.8.0 > > Attachments: HDFS-8880.01.patch, HDFS-8880.02.patch, > HDFS-8880.03.patch, HDFS-8880.04.patch, namenode-metrics.log > > > The NameNode can periodically log metrics to help debugging when the cluster > is not setup with another metrics monitoring scheme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7270) Add congestion signaling capability to DataNode write protocol
[ https://issues.apache.org/jira/browse/HDFS-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-7270: - Release Note: Introduced a new configuration dfs.pipeline.ecn. When the configuration is turned on, DataNodes will signal in the writing pipelines when they are overloaded. The client can back off based on this congestion signal to avoid overloading the system. > Add congestion signaling capability to DataNode write protocol > -- > > Key: HDFS-7270 > URL: https://issues.apache.org/jira/browse/HDFS-7270 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.7.0 > > Attachments: HDFS-7270.000.patch, HDFS-7270.001.patch, > HDFS-7270.002.patch, HDFS-7270.003.patch, HDFS-7270.004.patch > > > When a client writes to HDFS faster than the disk bandwidth of the DNs, it > saturates the disk bandwidth and put the DNs unresponsive. The client only > backs off by aborting / recovering the pipeline, which leads to failed writes > and unnecessary pipeline recovery. > This jira proposes to add explicit congestion control mechanisms in the > writing pipeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8976) Create HTML5 cluster webconsole for federated cluster
[
https://issues.apache.org/jira/browse/HDFS-8976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904989#comment-14904989
]
Haohui Mai commented on HDFS-8976:
--
Sure. Will do it in a couple of days.
> Create HTML5 cluster webconsole for federated cluster
> -
>
> Key: HDFS-8976
> URL: https://issues.apache.org/jira/browse/HDFS-8976
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 2.7.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-8976-01.patch, HDFS-8976-02.patch,
> HDFS-8976-03.patch, cluster-health.JPG
>
>
> Since the old jsp variant of cluster web console is no longer present from
> 2.7 onwards, there is a need for HTML 5 web console for overview of overall
> cluster.
> 2.7.1 docs says to check webconsole as below {noformat}Similar to the
> Namenode status web page, when using federation a Cluster Web Console is
> available to monitor the federated cluster at
> http:///dfsclusterhealth.jsp. Any Namenode in the cluster
> can be used to access this web page.{noformat}
> But this is no longer present,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9123) Copying from the root to a subdirectory should be forbidden
[ https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-9123: -- Status: Open (was: Patch Available) > Copying from the root to a subdirectory should be forbidden > --- > > Key: HDFS-9123 > URL: https://issues.apache.org/jira/browse/HDFS-9123 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch, > HDFS-9123.003.patch > > > HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp > /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, > /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) > However, the existing path validation fails if the source path is the root, > causing infinite copying, until the disk space is filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9053) Support large directories efficiently using B-Tree
[
https://issues.apache.org/jira/browse/HDFS-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904106#comment-14904106
]
Hadoop QA commented on HDFS-9053:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 45s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 8m 3s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 13s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 2s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 8s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 25s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 56s | Tests passed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 195m 15s | Tests failed in hadoop-hdfs. |
| | | 266m 28s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761615/HDFS-9053.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / cc2b473 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12615/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12615/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12615/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12615/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12615/console |
This message was automatically generated.
> Support large directories efficiently using B-Tree
> --
>
> Key: HDFS-9053
> URL: https://issues.apache.org/jira/browse/HDFS-9053
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
>Priority: Critical
> Attachments: HDFS-9053 (BTree with simple benchmark).patch, HDFS-9053
> (BTree).patch, HDFS-9053.001.patch, HDFS-9053.002.patch
>
>
> This is a long standing issue, we were trying to improve this in the past.
> Currently we use an ArrayList for the children under a directory, and the
> children are ordered in the list, for insert/delete/search, the time
> complexity is O(log n), but insertion/deleting causes re-allocations and
> copies of big arrays, so the operations are costly. For example, if the
> children grow to 1M size, the ArrayList will resize to > 1M capacity, so need
> > 1M * 4bytes = 4M continuous heap memory, it easily causes full GC in HDFS
> cluster where namenode heap memory is already highly used. I recap the 3
> main issues:
> # Insertion/deletion operations in large directories are expensive because
> re-allocations and copies of big arrays.
> # Dynamically allocate several MB continuous heap memory which will be
> long-lived can easily cause full GC problem.
> # Even most children are removed later, but the directory INode still
> occupies same size heap memory, since the ArrayList will never shrink.
> This JIRA is similar to HDFS-7174 created by [~kihwal], but use B-Tree to
> solve the problem suggested by [~shv].
> So the target of this JIRA is to implement a low memory footprint B-Tree and
> use it to replace ArrayList.
> If the elements size is not large (less than the maximum degree of B-Tree
> node), the B-Tree only has one root node which contains an array for the
> elements. And if the size grows large enough, it will split automatically,
> and if elements are removed, then B-Tree nodes can merge automatically (see
> more:
[jira] [Commented] (HDFS-8733) Keep server related definition in hdfs.proto on server side
[
https://issues.apache.org/jira/browse/HDFS-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904109#comment-14904109
]
Hudson commented on HDFS-8733:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2344 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2344/])
HDFS-8733. Keep server related definition in hdfs.proto on server side.
Contributed by Mingliang Liu. (wheat9: rev
7c5c099324d9168114be2f1233d49fdb65a8c1f2)
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
* hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/proto/bkjournal.proto
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/JournalProtocol.proto
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/InterDatanodeProtocol.proto
* hadoop-hdfs-project/hadoop-hdfs/pom.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HdfsServer.proto
* hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/QJournalProtocol.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
> Keep server related definition in hdfs.proto on server side
> ---
>
> Key: HDFS-8733
> URL: https://issues.apache.org/jira/browse/HDFS-8733
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HFDS-8733.000.patch
>
>
> In [HDFS-8726], we moved the protobuf files that define the client-sever
> protocols to {{hadoop-hdfs-client}} module. In {{hdfs.proto}} , there are
> some server related definition. This jira tracks the effort of moving those
> server related definition back to {{hadoop-hdfs}} module. A good place may be
> a new file named {{HdfsServer.proto}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-7529) Consolidate encryption zone related implementation into a single class
[ https://issues.apache.org/jira/browse/HDFS-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R reassigned HDFS-7529: -- Assignee: Rakesh R (was: Haohui Mai) > Consolidate encryption zone related implementation into a single class > -- > > Key: HDFS-7529 > URL: https://issues.apache.org/jira/browse/HDFS-7529 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Rakesh R > Labels: BB2015-05-TBR > Attachments: HDFS-7529-002.patch, HDFS-7529-003.patch, > HDFS-7529-004.patch, HDFS-7529.000.patch, HDFS-7529.001.patch > > > This jira proposes to consolidate encryption zone related implementation to a > single class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Validation of a path ended with a '/'
[ https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904034#comment-14904034 ] Yongjun Zhang commented on HDFS-9123: - BTW, I noticed that your patch rev 2 is not correctly generated. > Validation of a path ended with a '/' > - > > Key: HDFS-9123 > URL: https://issues.apache.org/jira/browse/HDFS-9123 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch > > > HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp > /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz, > /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc) > However, if the source path is ended with a '/' path separator, the existing > validation for sub-directories fails. For example, copying from / to /abc > would cause infinite copying, until the disk space is filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Validation of a path ended with a '/'
[
https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904029#comment-14904029
]
Yongjun Zhang commented on HDFS-9123:
-
Hi [~jojochuang],
Thanks for reporting the issue and the patch. It looks good in general and I
have some minor comments:
1. Suggest to change
{code}
String pathEnd = "";
if(!srcPath.toString().endsWith(Path.SEPARATOR)){
pathEnd = Path.SEPARATOR;
}
if(dstPath.startsWith(srcPath+pathEnd)) {
{code}
to
{code}
if (!srcPath.endsWith(Path.SEPARATOR)) {
srcPath += Path.SEPARATOR;
}
if(dstPath.startsWith(srcPath)) {
{code}
2. The test looks good, i can see that it report the "cp: `/' to `/test/': is a
subdirectory of itself" message with the fix. However, if I remove the fix and
run the same test, it's supposed to fail but it doesn't. Would you please look
into?
Thanks.
> Validation of a path ended with a '/'
> -
>
> Key: HDFS-9123
> URL: https://issues.apache.org/jira/browse/HDFS-9123
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch
>
>
> HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp
> /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz,
> /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc)
> However, if the source path is ended with a '/' path separator, the existing
> validation for sub-directories fails. For example, copying from / to /abc
> would cause infinite copying, until the disk space is filled up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Validation of a path ended with a '/'
[
https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904042#comment-14904042
]
J.Andreina commented on HDFS-9123:
--
Thanks [~jojochuang] . That was a good catch. I have few comments .
bq.if the source path is ended with a '/' path separator, the existing
validation for sub-directories fails
As I have verified, issue of copying indefinitely, exist only if src is just
root ('/"). It will not happen for any other paths ending with '/'.
{noformat}
> ./hdfs dfs -cp /abc/ /abc/abcdef
cp: `/abc' to `/abc/abcdef/abc': is a subdirectory of itself
{noformat}
*So, I feel since this is only for src is root, validation can be updated
addressing this special case.*
*I think below code would fix it correctly*
{code}
- String srcPath = src.fs.makeQualified(src.path).toString();
- String dstPath = dst.fs.makeQualified(target.path).toString();
+ Path srcPath = src.fs.makeQualified(src.path);
+ Path dstPath = dst.fs.makeQualified(target.path);
if (dstPath.equals(srcPath)) {
PathIOException e = new PathIOException(src.toString(),
"are identical");
e.setTargetPath(dstPath.toString());
throw e;
}
- if (dstPath.startsWith(srcPath+Path.SEPARATOR)) {
+ if (srcPath.isRoot() || dstPath.toString().startsWith(srcPath.toString()
+ Path.SEPARATOR)) {
{code}
*Also it would be good if you can update the jira title and summary to reflect
the special case*
> Validation of a path ended with a '/'
> -
>
> Key: HDFS-9123
> URL: https://issues.apache.org/jira/browse/HDFS-9123
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch
>
>
> HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp
> /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz,
> /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc)
> However, if the source path is ended with a '/' path separator, the existing
> validation for sub-directories fails. For example, copying from / to /abc
> would cause infinite copying, until the disk space is filled up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9044:
-
Attachment: HDFS-9044.4.patch
Updated the patch.
Please review.
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9039) Separate client and server side methods of o.a.h.hdfs.NameNodeProxies
[
https://issues.apache.org/jira/browse/HDFS-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904108#comment-14904108
]
Hudson commented on HDFS-9039:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2344 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2344/])
HDFS-9039. Separate client and server side methods of
o.a.h.hdfs.NameNodeProxies. Contributed by Mingliang Liu. (wheat9: rev
63d9f1596c92206cce3b72e3214d2fb5f6242b90)
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/AbstractNNFailoverProxyProvider.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ConfiguredFailoverProxyProvider.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/WrappedFailoverProxyProvider.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/AbstractNNFailoverProxyProvider.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/WrappedFailoverProxyProvider.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
> Separate client and server side methods of o.a.h.hdfs.NameNodeProxies
> -
>
> Key: HDFS-9039
> URL: https://issues.apache.org/jira/browse/HDFS-9039
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-9039.000.patch, HDFS-9039.001.patch,
> HDFS-9039.002.patch
>
>
> Currently the {{org.apache.hadoop.hdfs.NameNodeProxies}} class is used by
> both {{org.apache.hadoop.hdfs.server}} package (for server side protocols)
> and {{DFSClient}} class (for {{ClientProtocol}}). The {{DFSClient}} class
> should be moved to {{hadoop-hdfs-client}} module (see [HDFS-8053 |
> https://issues.apache.org/jira/browse/HDFS-8053]). As the
> {{org.apache.hadoop.hdfs.NameNodeProxies}} class also depends on server side
> protocols (e.g. {{JournalProtocol}} and {{NamenodeProtocol}}), we can't
> simply move this class to the {{hadoo-hdfs-client}} module as well.
> This jira tracks the effort of moving {{ClientProtocol}} related static
> methods in {{org.apache.hadoop.hdfs.NameNodeProxies}} class to
> {{hadoo-hdfs-client}} module. A good place to put these static methods is a
> new class named {{NameNodeProxiesClient}}.
> The checkstyle warnings can be addressed in [HDFS-8979], and removing the
> _slf4j_ logger guards when calling {{LOG.debug()}} and {{LOG.trace()}} can be
> addressed in [HDFS-8971].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Validation of a path ended with a '/'
[
https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904191#comment-14904191
]
Steve Loughran commented on HDFS-9123:
--
should the tests really downgrade exceptions to logs? I'd expect them to signal
failure and be rethrown
{code}
try {
val = shell.run(args1);
} catch (Exception e) {
System.err.println("Exception raised from DFSShell.run " +
e.getLocalizedMessage());
}
{code}
> Validation of a path ended with a '/'
> -
>
> Key: HDFS-9123
> URL: https://issues.apache.org/jira/browse/HDFS-9123
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch
>
>
> HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp
> /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz,
> /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc)
> However, if the source path is ended with a '/' path separator, the existing
> validation for sub-directories fails. For example, copying from / to /abc
> would cause infinite copying, until the disk space is filled up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7529) Consolidate encryption zone related implementation into a single class
[
https://issues.apache.org/jira/browse/HDFS-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904214#comment-14904214
]
Hadoop QA commented on HDFS-7529:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 54s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 8m 1s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 24s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 23s | The applied patch generated 1
new checkstyle issues (total was 354, now 354). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 26s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 33s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 11s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 103m 52s | Tests failed in hadoop-hdfs. |
| | | 149m 46s | |
\\
\\
|| Reason || Tests ||
| Timed out tests | org.apache.hadoop.hdfs.TestExternalBlockReader |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761787/HDFS-7529-004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 7c5c099 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12617/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12617/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12617/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12617/console |
This message was automatically generated.
> Consolidate encryption zone related implementation into a single class
> --
>
> Key: HDFS-7529
> URL: https://issues.apache.org/jira/browse/HDFS-7529
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Rakesh R
> Labels: BB2015-05-TBR
> Attachments: HDFS-7529-002.patch, HDFS-7529-003.patch,
> HDFS-7529-004.patch, HDFS-7529.000.patch, HDFS-7529.001.patch
>
>
> This jira proposes to consolidate encryption zone related implementation to a
> single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8671) Add client support for HTTP/2 stream channels
[
https://issues.apache.org/jira/browse/HDFS-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HDFS-8671:
Attachment: HDFS-8671-v1.patch
Pick code from POC branch.
Plan to run YCSB on HDFS-7966 branch which is more convincing, so we need to
pick the primary code from POC branch first.
Thanks.
> Add client support for HTTP/2 stream channels
> -
>
> Key: HDFS-8671
> URL: https://issues.apache.org/jira/browse/HDFS-8671
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: HDFS-7966
>
> Attachments: HDFS-8671-v0.patch, HDFS-8671-v1.patch
>
>
> {{Http2StreamChannel}} is introduced in HDFS-8515 but can only be used at
> server side.
> Now we implement Http2BlockReader using jetty http2-client in the POC branch,
> but the final version of jetty 9.3.0 only accepts java8.
> So here we plan to extend the functions of {{Http2StreamChannel}} to support
> client side usage and then implement Http2BlockReader based on it. And we
> still use jetty http2-client to write testcases to ensure that our http2
> implementation is valid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8671) Add client support for HTTP/2 stream channels
[
https://issues.apache.org/jira/browse/HDFS-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904138#comment-14904138
]
Hadoop QA commented on HDFS-8671:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761831/HDFS-8671-v1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 7c5c099 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12619/console |
This message was automatically generated.
> Add client support for HTTP/2 stream channels
> -
>
> Key: HDFS-8671
> URL: https://issues.apache.org/jira/browse/HDFS-8671
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: HDFS-7966
>
> Attachments: HDFS-8671-v0.patch, HDFS-8671-v1.patch
>
>
> {{Http2StreamChannel}} is introduced in HDFS-8515 but can only be used at
> server side.
> Now we implement Http2BlockReader using jetty http2-client in the POC branch,
> but the final version of jetty 9.3.0 only accepts java8.
> So here we plan to extend the functions of {{Http2StreamChannel}} to support
> client side usage and then implement Http2BlockReader based on it. And we
> still use jetty http2-client to write testcases to ensure that our http2
> implementation is valid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Validation of a path ended with a '/'
[
https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904162#comment-14904162
]
Hadoop QA commented on HDFS-9123:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 22m 12s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 49s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 58s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 42s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 41s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 4s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 24m 1s | Tests failed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 120m 56s | Tests failed in hadoop-hdfs. |
| | | 196m 50s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.security.ssl.TestReloadingX509TrustManager |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.web.TestWebHDFSOAuth2 |
| Timed out tests | org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761806/HDFS-9123.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 7c5c099 |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12616/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12616/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12616/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12616/console |
This message was automatically generated.
> Validation of a path ended with a '/'
> -
>
> Key: HDFS-9123
> URL: https://issues.apache.org/jira/browse/HDFS-9123
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch
>
>
> HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp
> /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz,
> /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc)
> However, if the source path is ended with a '/' path separator, the existing
> validation for sub-directories fails. For example, copying from / to /abc
> would cause infinite copying, until the disk space is filled up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8733) Keep server related definition in hdfs.proto on server side
[
https://issues.apache.org/jira/browse/HDFS-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904130#comment-14904130
]
Hudson commented on HDFS-8733:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #406 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/406/])
HDFS-8733. Keep server related definition in hdfs.proto on server side.
Contributed by Mingliang Liu. (wheat9: rev
7c5c099324d9168114be2f1233d49fdb65a8c1f2)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/JournalProtocol.proto
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/InterDatanodeProtocol.proto
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/QJournalProtocol.proto
* hadoop-hdfs-project/hadoop-hdfs/pom.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
* hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HdfsServer.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/proto/bkjournal.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
> Keep server related definition in hdfs.proto on server side
> ---
>
> Key: HDFS-8733
> URL: https://issues.apache.org/jira/browse/HDFS-8733
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HFDS-8733.000.patch
>
>
> In [HDFS-8726], we moved the protobuf files that define the client-sever
> protocols to {{hadoop-hdfs-client}} module. In {{hdfs.proto}} , there are
> some server related definition. This jira tracks the effort of moving those
> server related definition back to {{hadoop-hdfs}} module. A good place may be
> a new file named {{HdfsServer.proto}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9039) Separate client and server side methods of o.a.h.hdfs.NameNodeProxies
[
https://issues.apache.org/jira/browse/HDFS-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904129#comment-14904129
]
Hudson commented on HDFS-9039:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #406 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/406/])
HDFS-9039. Separate client and server side methods of
o.a.h.hdfs.NameNodeProxies. Contributed by Mingliang Liu. (wheat9: rev
63d9f1596c92206cce3b72e3214d2fb5f6242b90)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolPB.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/AbstractNNFailoverProxyProvider.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/WrappedFailoverProxyProvider.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/WrappedFailoverProxyProvider.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ConfiguredFailoverProxyProvider.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/AbstractNNFailoverProxyProvider.java
> Separate client and server side methods of o.a.h.hdfs.NameNodeProxies
> -
>
> Key: HDFS-9039
> URL: https://issues.apache.org/jira/browse/HDFS-9039
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-9039.000.patch, HDFS-9039.001.patch,
> HDFS-9039.002.patch
>
>
> Currently the {{org.apache.hadoop.hdfs.NameNodeProxies}} class is used by
> both {{org.apache.hadoop.hdfs.server}} package (for server side protocols)
> and {{DFSClient}} class (for {{ClientProtocol}}). The {{DFSClient}} class
> should be moved to {{hadoop-hdfs-client}} module (see [HDFS-8053 |
> https://issues.apache.org/jira/browse/HDFS-8053]). As the
> {{org.apache.hadoop.hdfs.NameNodeProxies}} class also depends on server side
> protocols (e.g. {{JournalProtocol}} and {{NamenodeProtocol}}), we can't
> simply move this class to the {{hadoo-hdfs-client}} module as well.
> This jira tracks the effort of moving {{ClientProtocol}} related static
> methods in {{org.apache.hadoop.hdfs.NameNodeProxies}} class to
> {{hadoo-hdfs-client}} module. A good place to put these static methods is a
> new class named {{NameNodeProxiesClient}}.
> The checkstyle warnings can be addressed in [HDFS-8979], and removing the
> _slf4j_ logger guards when calling {{LOG.debug()}} and {{LOG.trace()}} can be
> addressed in [HDFS-8971].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8053) Move DFSIn/OutputStream and related classes to hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905337#comment-14905337
]
Hadoop QA commented on HDFS-8053:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 56s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 23 new or modified test files. |
| {color:red}-1{color} | javac | 7m 51s | The applied patch generated 21
additional warning messages. |
| {color:red}-1{color} | javadoc | 10m 21s | The applied patch generated 4
additional warning messages. |
| {color:red}-1{color} | release audit | 0m 17s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 26s | The applied patch generated
269 new checkstyle issues (total was 65, now 334). |
| {color:red}-1{color} | whitespace | 0m 8s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 163m 0s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 41s | Tests passed in
hadoop-hdfs-client. |
| | | 214m 34s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFSNamesystem |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761811/HDFS-8053.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 692d51c |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/12629/artifact/patchprocess/diffJavacWarnings.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/12629/artifact/patchprocess/diffJavadocWarnings.txt
|
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12629/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12629/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12629/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12629/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12629/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12629/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12629/console |
This message was automatically generated.
> Move DFSIn/OutputStream and related classes to hadoop-hdfs-client
> -
>
> Key: HDFS-8053
> URL: https://issues.apache.org/jira/browse/HDFS-8053
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-8053.000.patch
>
>
> This jira tracks the effort of moving the {{DFSInputStream}} and
> {{DFSOutputSream}} classes from {{hadoop-hdfs}} to {{hadoop-hdfs-client}}
> module.
> Guidelines:
> * As the {{DFSClient}} is heavily coupled to these two classes, we should
> move it together.
> * Related classes should be addressed in separate jiras if they're
> independent and complex enough.
> * The checkstyle warnings can be addressed in [HDFS-8979 |
> https://issues.apache.org/jira/browse/HDFS-8979]
> * Removing the _slf4j_ logger guards when calling {{LOG.debug()}} and
> {{LOG.trace()}} can be addressed in [HDFS-8971 |
> https://issues.apache.org/jira/browse/HDFS-8971].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
