[jira] [Commented] (HDFS-10984) Expose nntop output as metrics
[
https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557289#comment-15557289
]
Hadoop QA commented on HDFS-10984:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 185 unchanged - 0 fixed = 187 total (was 185) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.TestRollingUpgrade |
| Timed out junit tests | org.apache.hadoop.tools.TestJMXGet |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10984 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832240/HDFS-10984.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 71153b448b4a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6a38d11 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17066/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17066/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17066/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17066/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT
[jira] [Updated] (HDFS-10968) BlockManager#isInNewRack should consider decommissioning nodes
[
https://issues.apache.org/jira/browse/HDFS-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-10968:
-
Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
Status: Resolved (was: Patch Available)
Thanks for the review, Nicholas! I've committed this into trunk.
> BlockManager#isInNewRack should consider decommissioning nodes
> --
>
> Key: HDFS-10968
> URL: https://issues.apache.org/jira/browse/HDFS-10968
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10968.000.patch
>
>
> For an EC block, it is possible we have enough internal blocks but without
> enough racks. The current reconstruction code calls
> {{BlockManager#isInNewRack}} to check if the target node can increase the
> total rack number for the case, which compares the target node's rack with
> source node racks:
> {code}
> for (DatanodeDescriptor src : srcs) {
> if (src.getNetworkLocation().equals(target.getNetworkLocation())) {
> return false;
> }
> }
> {code}
> However here the {{srcs}} may include a decommissioning node, in which case
> we should allow the target node to be in the same rack with it.
> For e.g., suppose we have 11 nodes: h1 ~ h11, which are located in racks r1,
> r1, r2, r2, r3, r3, r4, r4, r5, r5, r6, respectively. In case that an EC
> block has 9 live internal blocks on (h1~h8 + h11), and one internal block on
> h9 which is to be decommissioned. The current code will not choose h10 for
> reconstruction because isInNewRack thinks h10 is on the same rack with h9.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Status: Patch Available (was: Open) > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Fix For: 2.7.3 > > Attachments: HDFS-10984.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Attachment: (was: HDFS-10984.patch) > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Fix For: 2.7.3 > > Attachments: HDFS-10984.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Attachment: HDFS-10984.patch > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Fix For: 2.7.3 > > Attachments: HDFS-10984.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Status: Open (was: Patch Available) Minor correction. > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Fix For: 2.7.3 > > Attachments: HDFS-10984.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10968) BlockManager#isInNewRack should consider decommissioning nodes
[
https://issues.apache.org/jira/browse/HDFS-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-10968:
---
Hadoop Flags: Reviewed
Description:
For an EC block, it is possible we have enough internal blocks but without
enough racks. The current reconstruction code calls
{{BlockManager#isInNewRack}} to check if the target node can increase the total
rack number for the case, which compares the target node's rack with source
node racks:
{code}
for (DatanodeDescriptor src : srcs) {
if (src.getNetworkLocation().equals(target.getNetworkLocation())) {
return false;
}
}
{code}
However here the {{srcs}} may include a decommissioning node, in which case we
should allow the target node to be in the same rack with it.
For e.g., suppose we have 11 nodes: h1 ~ h11, which are located in racks r1,
r1, r2, r2, r3, r3, r4, r4, r5, r5, r6, respectively. In case that an EC block
has 9 live internal blocks on (h1~h8 + h11), and one internal block on h9 which
is to be decommissioned. The current code will not choose h10 for
reconstruction because isInNewRack thinks h10 is on the same rack with h9.
was:
For an EC block, it is possible we have enough internal blocks but without
enough racks. The current reconstruction code calls {{BlockManager#isNewRack}}
to check if the target node can increase the total rack number for the case,
which compares the target node's rack with source node racks:
{code}
for (DatanodeDescriptor src : srcs) {
if (src.getNetworkLocation().equals(target.getNetworkLocation())) {
return false;
}
}
{code}
However here the {{srcs}} may include a decommissioning node, in which case we
should allow the target node to be in the same rack with it.
For e.g., suppose we have 11 nodes: h1 ~ h11, which are located in racks r1,
r1, r2, r2, r3, r3, r4, r4, r5, r5, r6, respectively. In case that an EC block
has 9 live internal blocks on (h1~h8 + h11), and one internal block on h9 which
is to be decommissioned. The current code will not choose h10 for
reconstruction because isNewRack thinks h10 is on the same rack with h9.
Summary: BlockManager#isInNewRack should consider decommissioning
nodes (was: BlockManager#isNewRack should consider decommissioning nodes)
Good catch!
+1 the patch looks good.
> BlockManager#isInNewRack should consider decommissioning nodes
> --
>
> Key: HDFS-10968
> URL: https://issues.apache.org/jira/browse/HDFS-10968
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-10968.000.patch
>
>
> For an EC block, it is possible we have enough internal blocks but without
> enough racks. The current reconstruction code calls
> {{BlockManager#isInNewRack}} to check if the target node can increase the
> total rack number for the case, which compares the target node's rack with
> source node racks:
> {code}
> for (DatanodeDescriptor src : srcs) {
> if (src.getNetworkLocation().equals(target.getNetworkLocation())) {
> return false;
> }
> }
> {code}
> However here the {{srcs}} may include a decommissioning node, in which case
> we should allow the target node to be in the same rack with it.
> For e.g., suppose we have 11 nodes: h1 ~ h11, which are located in racks r1,
> r1, r2, r2, r3, r3, r4, r4, r5, r5, r6, respectively. In case that an EC
> block has 9 live internal blocks on (h1~h8 + h11), and one internal block on
> h9 which is to be decommissioned. The current code will not choose h10 for
> reconstruction because isInNewRack thinks h10 is on the same rack with h9.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Status: Patch Available (was: Open) Set fix version to 2.7.3 > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Fix For: 2.7.3 > > Attachments: HDFS-10984.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Fix Version/s: 2.7.3 > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Fix For: 2.7.3 > > Attachments: HDFS-10984.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Attachment: HDFS-10984.patch > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Attachments: HDFS-10984.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10984) Expose nntop output as metrics
[
https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557062#comment-15557062
]
Siddharth Wagle edited comment on HDFS-10984 at 10/8/16 3:42 AM:
-
Sample flattened output of the window metrics emitted to Ambari Metrics System:
{quote}
dfs.TopUserOpCounts.windowMs=6.op=listStatus.user=mapred.count
{quote}
was (Author: swagle):
Sample flattened output of the window metrics emitted to Ambari Metrics System:
{quote}
dfs.TopUserOpCounts.windowMs=6.listStatus.user=mapred.count
{quote}
> Expose nntop output as metrics
> -
>
> Key: HDFS-10984
> URL: https://issues.apache.org/jira/browse/HDFS-10984
> Project: Hadoop HDFS
> Issue Type: Task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Siddharth Wagle
>Assignee: Siddharth Wagle
>
> The nntop output is already exposed via JMX with HDFS-6982.
> However external metrics systems do not get this data. It would be valuable
> to track this as a timeseries as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10984) Expose nntop output as metrics
[
https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557062#comment-15557062
]
Siddharth Wagle edited comment on HDFS-10984 at 10/8/16 3:42 AM:
-
Sample flattened output of the window metrics emitted to Ambari Metrics System:
{quote}
dfs.TopUserOpCounts.windowMs=6.listStatus.user=mapred.count
{quote}
was (Author: swagle):
Sample flattened output of the window metrics emitted to Ambari Metrics System:
{quote}
dfs.TopUserOpCounts.6.listStatus.user=mapred.count
{quote}
> Expose nntop output as metrics
> -
>
> Key: HDFS-10984
> URL: https://issues.apache.org/jira/browse/HDFS-10984
> Project: Hadoop HDFS
> Issue Type: Task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Siddharth Wagle
>Assignee: Siddharth Wagle
>
> The nntop output is already exposed via JMX with HDFS-6982.
> However external metrics systems do not get this data. It would be valuable
> to track this as a timeseries as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Attachment: (was: HDFS-10984.patch) > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Attachment: HDFS-10984.patch > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Attachments: HDFS-10984.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10986) DFSAdmin should log detailed error message if any
[
https://issues.apache.org/jira/browse/HDFS-10986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557072#comment-15557072
]
Hadoop QA commented on HDFS-10986:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 200 unchanged - 0 fixed = 201 total (was 200) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
20s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10986 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832234/HDFS-10986.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c344f3c92da9 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6a38d11 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17065/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17065/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17065/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/17065/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17065/console |
| Powered by | Apache Yetus
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Attachment: (was: HDFS-10984.patch) > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Attachments: HDFS-10984.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10983) OIV tool should make an EC file explicit
[ https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-10983: --- Description: The OIV tool's webhdfs interface does not print if a file is striped or not. Also, it prints the file's EC policy ID as replication factor, which is inconsistent to the output of a typical webhdfs call to the cluster, which always shows replication factor of 0 for EC files. Not just webhdfs, but delimiter output does not print if a file is stripped or not either. was: The OIV tool's webhdfs interface does not print if a file is striped or not. Also, it prints the file's EC policy ID as replication factor, which is inconsistent to the output of a typical webhdfs call to the cluster, which always shows replication factor of 0 for EC files. > OIV tool should make an EC file explicit > > > Key: HDFS-10983 > URL: https://issues.apache.org/jira/browse/HDFS-10983 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang > > The OIV tool's webhdfs interface does not print if a file is striped or not. > Also, it prints the file's EC policy ID as replication factor, which is > inconsistent to the output of a typical webhdfs call to the cluster, which > always shows replication factor of 0 for EC files. > Not just webhdfs, but delimiter output does not print if a file is stripped > or not either. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[
https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle updated HDFS-10984:
---
Attachment: HDFS-10984.patch
Sample flattened output of the window metrics emitted to Ambari Metrics System:
{quote}
dfs.TopUserOpCounts.6.listStatus.user=mapred.count
{quote}
> Expose nntop output as metrics
> -
>
> Key: HDFS-10984
> URL: https://issues.apache.org/jira/browse/HDFS-10984
> Project: Hadoop HDFS
> Issue Type: Task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Siddharth Wagle
>Assignee: Siddharth Wagle
> Attachments: HDFS-10984.patch
>
>
> The nntop output is already exposed via JMX with HDFS-6982.
> However external metrics systems do not get this data. It would be valuable
> to track this as a timeseries as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10983) OIV tool should make an EC file explicit
[ https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-10983: --- Summary: OIV tool should make an EC file explicit (was: OIV webhdfs interface should make an EC file explicit) > OIV tool should make an EC file explicit > > > Key: HDFS-10983 > URL: https://issues.apache.org/jira/browse/HDFS-10983 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang > > The OIV tool's webhdfs interface does not print if a file is striped or not. > Also, it prints the file's EC policy ID as replication factor, which is > inconsistent to the output of a typical webhdfs call to the cluster, which > always shows replication factor of 0 for EC files. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10986) DFSAdmin should log detailed error message if any
[
https://issues.apache.org/jira/browse/HDFS-10986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556967#comment-15556967
]
Mingliang Liu commented on HDFS-10986:
--
[~kihwal] and [~brahmareddy] does this make sense to you? Thanks!
> DFSAdmin should log detailed error message if any
> -
>
> Key: HDFS-10986
> URL: https://issues.apache.org/jira/browse/HDFS-10986
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10986.000.patch
>
>
> There are some subcommands in {{DFSAdmin}} that swallow IOException and give
> very limited error message, if any, to the stderr.
> {code}
> $ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9866
> Datanode unreachable.
> $ hdfs dfsadmin -getDatanodeInfo localhost:9866
> Datanode unreachable.
> $ hdfs dfsadmin -evictWriters 127.0.0.1:9866
> $ echo $?
> -1
> {code}
> User is not able to get the exception stack even the LOG level is DEBUG. This
> is not very user friendly. Fortunately, if the port number is not accessible
> (say ), users can infer the detailed error message by IPC logs:
> {code}
> $ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:
> 2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> .
> 2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> 2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
> localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
> retries number: 10
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> ...
> at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
> Datanode unreachable.
> {code}
> We should fix this by providing detailed error message. Actually, the
> {{DFSAdmin#run}} already handles exception carefully, including:
> # set the exit ret value to -1
> # print the error message
> # log the exception stack trace (in DEBUG level)
> All we need to do is to not swallow exceptions without good reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10986) DFSAdmin should log detailed error message if any
[
https://issues.apache.org/jira/browse/HDFS-10986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10986:
-
Attachment: HDFS-10986.000.patch
> DFSAdmin should log detailed error message if any
> -
>
> Key: HDFS-10986
> URL: https://issues.apache.org/jira/browse/HDFS-10986
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10986.000.patch
>
>
> There are some subcommands in {{DFSAdmin}} that swallow IOException and give
> very limited error message, if any, to the stderr.
> {code}
> $ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9866
> Datanode unreachable.
> $ hdfs dfsadmin -getDatanodeInfo localhost:9866
> Datanode unreachable.
> $ hdfs dfsadmin -evictWriters 127.0.0.1:9866
> $ echo $?
> -1
> {code}
> User is not able to get the exception stack even the LOG level is DEBUG. This
> is not very user friendly. Fortunately, if the port number is not accessible
> (say ), users can infer the detailed error message by IPC logs:
> {code}
> $ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:
> 2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> .
> 2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> 2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
> localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
> retries number: 10
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> ...
> at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
> Datanode unreachable.
> {code}
> We should fix this by providing detailed error message. Actually, the
> {{DFSAdmin#run}} already handles exception carefully, including:
> # set the exit ret value to -1
> # print the error message
> # log the exception stack trace (in DEBUG level)
> All we need to do is to not swallow exceptions without good reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10986) DFSAdmin should log detailed error message if any
[
https://issues.apache.org/jira/browse/HDFS-10986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10986:
-
Status: Patch Available (was: Open)
> DFSAdmin should log detailed error message if any
> -
>
> Key: HDFS-10986
> URL: https://issues.apache.org/jira/browse/HDFS-10986
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10986.000.patch
>
>
> There are some subcommands in {{DFSAdmin}} that swallow IOException and give
> very limited error message, if any, to the stderr.
> {code}
> $ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9866
> Datanode unreachable.
> $ hdfs dfsadmin -getDatanodeInfo localhost:9866
> Datanode unreachable.
> $ hdfs dfsadmin -evictWriters 127.0.0.1:9866
> $ echo $?
> -1
> {code}
> User is not able to get the exception stack even the LOG level is DEBUG. This
> is not very user friendly. Fortunately, if the port number is not accessible
> (say ), users can infer the detailed error message by IPC logs:
> {code}
> $ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:
> 2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> .
> 2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> 2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
> localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
> retries number: 10
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> ...
> at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
> Datanode unreachable.
> {code}
> We should fix this by providing detailed error message. Actually, the
> {{DFSAdmin#run}} already handles exception carefully, including:
> # set the exit ret value to -1
> # print the error message
> # log the exception stack trace (in DEBUG level)
> All we need to do is to not swallow exceptions without good reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10986) DFSAdmin should log detailed error message if any
[
https://issues.apache.org/jira/browse/HDFS-10986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10986:
-
Description:
There are some subcommands in {{DFSAdmin}} that swallow IOException and give
very limited error message, if any, to the stderr.
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9866
Datanode unreachable.
$ hdfs dfsadmin -getDatanodeInfo localhost:9866
Datanode unreachable.
$ hdfs dfsadmin -evictWriters 127.0.0.1:9866
$ echo $?
-1
{code}
User is not able to get the exception stack even the LOG level is DEBUG. This
is not very user friendly. Fortunately, if the port number is not accessible
(say ), users can infer the detailed error message by IPC logs:
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:
2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
.
2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
retries number: 10
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
...
at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
Datanode unreachable.
{code}
We should fix this by providing detailed error message. Actually, the
{{DFSAdmin#run}} already handles exception carefully, including:
# set the exit ret value to -1
# print the error message
# log the exception stack trace (in DEBUG level)
All we need to do is to not swallow exceptions without good reason.
was:
There are some subcommands in {{DFSAdmin}} that swallow IOException and give
very limited error message, if any, to the stderr.
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9866
Datanode unreachable.
$ hdfs dfsadmin -getDatanodeInfo localhost:9866
Datanode unreachable.
$ hdfs dfsadmin -evictWriters 127.0.0.1:9866
$ echo $?
-1
{code}
Fortunately, if the port number is not accessible (say ), users can infer
the detailed error message by logs:
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:
2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
.
2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
retries number: 10
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
...
at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
Datanode unreachable.
{code}
We should fix this by providing detailed error message. Actually, the
{{DFSAdmin#run}} already handles exception carefully, including:
# set the exit ret value to -1
# print the error message
# log the exception stack trace (in DEBUG level)
All we need to do is to not swallow exceptions without good reason.
> DFSAdmin should log detailed error message if any
> -
>
> Key: HDFS-10986
> URL: https://issues.apache.org/jira/browse/HDFS-10986
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>
> There are some subcommands in
[jira] [Updated] (HDFS-10986) DFSAdmin should log detailed error message if any
[
https://issues.apache.org/jira/browse/HDFS-10986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10986:
-
Affects Version/s: (was: 2.8.0)
Target Version/s: 3.0.0-alpha2
Description:
There are some subcommands in {{DFSAdmin}} that swallow IOException and give
very limited error message, if any, to the stderr.
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9866
Datanode unreachable.
$ hdfs dfsadmin -getDatanodeInfo localhost:9866
Datanode unreachable.
$ hdfs dfsadmin -evictWriters 127.0.0.1:9866
$ echo $?
-1
{code}
Fortunately, if the port number is not accessible (say ), users can infer
the detailed error message by logs:
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:
2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
.
2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
retries number: 10
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
...
at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
Datanode unreachable.
{code}
We should fix this by providing detailed error message. Actually, the
{{DFSAdmin#run}} already handles exception carefully, including:
# set the exit ret value to -1
# print the error message
# log the exception stack trace (in DEBUG level)
All we need to do is to not swallow exceptions without good reason.
was:
There are some subcommands in {{DFSAdmin}} that swallow IOException and give
very limited error message, if any, to the stderr.
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9866
Datanode unreachable.
$ hdfs dfsadmin -getDatanodeInfo localhost:9866
Datanode unreachable.
$ hdfs dfsadmin -evictWriters 127.0.0.1:9866
$ echo $?
-1
{code}
Fortunately, if the port number is not accessible (say ), users can infer
the detailed error message by logs:
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:
2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
.
2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
retries number: 10
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
...
at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
Datanode unreachable.
{code}
We should fix this by providing detailed error message. Actually, the
{{DFSAdmin#run}} already handles exception carefully. All we need to do is to
not swallow exceptions without good reason.
Summary: DFSAdmin should log detailed error message if any (was:
DFSAdmin should show detailed error message if any)
> DFSAdmin should log detailed error message if any
> -
>
> Key: HDFS-10986
> URL: https://issues.apache.org/jira/browse/HDFS-10986
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>
> There are some subcommands in {{DFSAdmin}} that
[jira] [Commented] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
[
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556929#comment-15556929
]
Hadoop QA commented on HDFS-10972:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 11s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10972 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832219/HDFS-10972.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a62ff306733f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6a38d11 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17064/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17064/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17064/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
[jira] [Commented] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded
[
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556868#comment-15556868
]
Kai Zheng commented on HDFS-10971:
--
Sounds good to me, Andrew and Zhe.
So for this issue, the replication factor should be still copied or handled
accordingly when source file is striped and/or the dest file is to be striped.
> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if
> {{-pr}} is specified. However, if the source file is EC, the replication
> factor (which is EC policy) should not be replicated to the destination file.
> When a HdfsFileStatus is converted to FileStatus, the replication factor is
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the
> replication factor of an EC file results in an IOException: "Requested
> replication factor of 0 is less than the required minimum of 1 for
> /tmp/dst/dest2"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8028) TestNNHandlesBlockReportPerStorage/TestNNHandlesCombinedBlockReport Failed after patched HDFS-7704
[ https://issues.apache.org/jira/browse/HDFS-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556846#comment-15556846 ] Vinitha Reddy Gankidi commented on HDFS-8028: - These tests fail on branch 2-7 after HDFS-7704 but they pass after HDFS-7430.This doesn't need to be fixed in branch-2.7. Initialization values for DN_RESCAN_INTERVAL and DN_RESCAN_EXTRA_WAIT need to be modified as per HDFS-7430 to fix it temporarily. > TestNNHandlesBlockReportPerStorage/TestNNHandlesCombinedBlockReport Failed > after patched HDFS-7704 > -- > > Key: HDFS-8028 > URL: https://issues.apache.org/jira/browse/HDFS-8028 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.7.0 >Reporter: hongyu bi >Assignee: hongyu bi >Priority: Minor > Attachments: HDFS-8028-v0.patch > > > HDFS-7704 makes BadBlockReport asynchronously however > BlockReportTestBase#blockreport_02 doesn't wait for a while after blockreport. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10985) o.a.h.ha.TestZKFailoverController should not use fixed time sleep before assertsions
[
https://issues.apache.org/jira/browse/HDFS-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556842#comment-15556842
]
Hadoop QA commented on HDFS-10985:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
7s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10985 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832223/HDFS-10985.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0f6af6d1bfdb 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e57fa81 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17063/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17063/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> o.a.h.ha.TestZKFailoverController should not use fixed time sleep before
> assertsions
>
>
> Key: HDFS-10985
> URL: https://issues.apache.org/jira/browse/HDFS-10985
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HDFS-10985.000.patch
>
>
> {{TestZKFailoverController#testGracefulFailoverMultipleZKfcs}} uses fixed
> time sleep before
[jira] [Updated] (HDFS-10986) DFSAdmin should show detailed error message if any
[
https://issues.apache.org/jira/browse/HDFS-10986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10986:
-
Description:
There are some subcommands in {{DFSAdmin}} that swallow IOException and give
very limited error message, if any, to the stderr.
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9866
Datanode unreachable.
$ hdfs dfsadmin -getDatanodeInfo localhost:9866
Datanode unreachable.
$ hdfs dfsadmin -evictWriters 127.0.0.1:9866
$ echo $?
-1
{code}
Fortunately, if the port number is not accessible (say ), users can infer
the detailed error message by logs:
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:
2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
.
2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
retries number: 10
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
...
at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
Datanode unreachable.
{code}
We should fix this by providing detailed error message. Actually, the
{{DFSAdmin#run}} already handles exception carefully. All we need to do is to
not swallow exceptions without good reason.
was:
There are some subcommands in {{DFSAdmin}} that swallow IOException and give
very limited error message, if any, to the stderr.
{code}
$ hdfs dfsadmin -getDatanodeInfo localhost:9866
Datanode unreachable.
$ hdfs dfsadmin -evictWriters 127.0.0.1:9866
$ echo $?
-1
{code}
Fortunately, if the port number is not accessible, users can infer the detailed
error message by logs:
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9690
2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
.
2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
retries number: 10
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
...
at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
Datanode unreachable.
{code}
We should fix this by providing detailed error message. Actually, the
{{DFSAdmin#run}} already handles exception carefully. All we need to do is to
not swallow exceptions without good reason.
> DFSAdmin should show detailed error message if any
> --
>
> Key: HDFS-10986
> URL: https://issues.apache.org/jira/browse/HDFS-10986
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>
> There are some subcommands in {{DFSAdmin}} that swallow IOException and give
> very limited error message, if any, to the stderr.
> {code}
> $ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9866
> Datanode unreachable.
> $ hdfs dfsadmin -getDatanodeInfo localhost:9866
> Datanode unreachable.
> $ hdfs dfsadmin -evictWriters 127.0.0.1:9866
> $ echo $?
> -1
> {code}
> Fortunately, if the port number is not accessible (say ),
[jira] [Updated] (HDFS-10986) DFSAdmin should show detailed error message if any
[
https://issues.apache.org/jira/browse/HDFS-10986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10986:
-
Description:
There are some subcommands in {{DFSAdmin}} that swallow IOException and give
very limited error message, if any, to the stderr.
{code}
$ hdfs dfsadmin -getDatanodeInfo localhost:9866
Datanode unreachable.
$ hdfs dfsadmin -evictWriters 127.0.0.1:9866
$ echo $?
-1
{code}
Fortunately, if the port number is not accessible, users can infer the detailed
error message by logs:
{code}
$ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9690
2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
.
2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
retries number: 10
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
...
at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
Datanode unreachable.
{code}
We should fix this by providing detailed error message. Actually, the
{{DFSAdmin#run}} already handles exception carefully. All we need to do is to
not swallow exceptions without good reason.
was:
There are some subcommands in {{DFSAdmin}} that swallow IOException and give
very limited error message, if any, to the stderr.
{code}
$ hdfs dfsadmin -getDatanodeInfo localhost:9866
Datanode unreachable.
$ hdfs dfsadmin -evictWriters 127.0.0.1:9866
$ echo $?
-1
{code}
We should fix this by providing detailed error message. Actually, the
{{DFSAdmin#run}} already handles exception carefully. All we need to do is to
not swallow exceptions without good reason.
> DFSAdmin should show detailed error message if any
> --
>
> Key: HDFS-10986
> URL: https://issues.apache.org/jira/browse/HDFS-10986
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>
> There are some subcommands in {{DFSAdmin}} that swallow IOException and give
> very limited error message, if any, to the stderr.
> {code}
> $ hdfs dfsadmin -getDatanodeInfo localhost:9866
> Datanode unreachable.
> $ hdfs dfsadmin -evictWriters 127.0.0.1:9866
> $ echo $?
> -1
> {code}
> Fortunately, if the port number is not accessible, users can infer the
> detailed error message by logs:
> {code}
> $ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9690
> 2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> .
> 2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> 2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
> localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
> retries number: 10
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> ...
> at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
> Datanode unreachable.
> {code}
> We should fix this by providing detailed error message. Actually, the
> {{DFSAdmin#run}} already handles exception carefully. All we need to do is to
> not swallow exceptions
[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[ https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556828#comment-15556828 ] Hudson commented on HDFS-10797: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10572 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10572/]) HDFS-10797. Disk usage summary of snapshots causes renamed blocks to get (xiao: rev 6a38d118d86b7907009bcec34f1b788d076f1d1c) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java > Disk usage summary of snapshots causes renamed blocks to get counted twice > -- > > Key: HDFS-10797 > URL: https://issues.apache.org/jira/browse/HDFS-10797 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 2.8.0 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, > HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch, > HDFS-10797.006.patch, HDFS-10797.007.patch, HDFS-10797.008.patch, > HDFS-10797.009.patch, HDFS-10797.010.patch, HDFS-10797.010.patch > > > DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how > much disk usage is used by a snapshot by tallying up the files in the > snapshot that have since been deleted (that way it won't overlap with regular > files whose disk usage is computed separately). However that is determined > from a diff that shows moved (to Trash or otherwise) or renamed files as a > deletion and a creation operation that may overlap with the list of blocks. > Only the deletion operation is taken into consideration, and this causes > those blocks to get represented twice in the disk usage tallying. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10986) DFSAdmin should show detailed error message if any
Mingliang Liu created HDFS-10986:
Summary: DFSAdmin should show detailed error message if any
Key: HDFS-10986
URL: https://issues.apache.org/jira/browse/HDFS-10986
Project: Hadoop HDFS
Issue Type: Bug
Components: tools
Affects Versions: 2.8.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu
There are some subcommands in {{DFSAdmin}} that swallow IOException and give
very limited error message, if any, to the stderr.
{code}
$ hdfs dfsadmin -getDatanodeInfo localhost:9866
Datanode unreachable.
$ hdfs dfsadmin -evictWriters 127.0.0.1:9866
$ echo $?
-1
{code}
We should fix this by providing detailed error message. Actually, the
{{DFSAdmin#run}} already handles exception carefully. All we need to do is to
not swallow exceptions without good reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
[ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556790#comment-15556790 ] Konstantin Shvachko commented on HDFS-10301: Hey [~arpitagarwal], how is the review going? Should we wrap this up? I understand [~shahrs87] is waiting for us to proceed with HDFS-10953. > BlockReport retransmissions may lead to storages falsely being declared > zombie if storage report processing happens out of order > > > Key: HDFS-10301 > URL: https://issues.apache.org/jira/browse/HDFS-10301 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.1 >Reporter: Konstantin Shvachko >Assignee: Vinitha Reddy Gankidi >Priority: Critical > Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, > HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch, > HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch, > HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch, > HDFS-10301.012.patch, HDFS-10301.013.patch, HDFS-10301.014.patch, > HDFS-10301.015.patch, HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch, > HDFS-10301.sample.patch, zombieStorageLogs.rtf > > > When NameNode is busy a DataNode can timeout sending a block report. Then it > sends the block report again. Then NameNode while process these two reports > at the same time can interleave processing storages from different reports. > This screws up the blockReportId field, which makes NameNode think that some > storages are zombie. Replicas from zombie storages are immediately removed, > causing missing blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[
https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-10797:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.8.0
Status: Resolved (was: Patch Available)
Committed this to trunk, branch-2 and branch-2.8. Thanks Sean for the great
work here and Jing for the helpful reviews!
I think this is just a bug fix, and the change in {{du}} output is considered
incorrect previously. So not marking this incompatible.
[~mackrorysd], do you mind adding a short release note to the jira? Thanks
> Disk usage summary of snapshots causes renamed blocks to get counted twice
> --
>
> Key: HDFS-10797
> URL: https://issues.apache.org/jira/browse/HDFS-10797
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.8.0
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch,
> HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch,
> HDFS-10797.006.patch, HDFS-10797.007.patch, HDFS-10797.008.patch,
> HDFS-10797.009.patch, HDFS-10797.010.patch, HDFS-10797.010.patch
>
>
> DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how
> much disk usage is used by a snapshot by tallying up the files in the
> snapshot that have since been deleted (that way it won't overlap with regular
> files whose disk usage is computed separately). However that is determined
> from a diff that shows moved (to Trash or otherwise) or renamed files as a
> deletion and a creation operation that may overlap with the list of blocks.
> Only the deletion operation is taken into consideration, and this causes
> those blocks to get represented twice in the disk usage tallying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[ https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-10797: - Component/s: snapshots > Disk usage summary of snapshots causes renamed blocks to get counted twice > -- > > Key: HDFS-10797 > URL: https://issues.apache.org/jira/browse/HDFS-10797 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 2.8.0 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, > HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch, > HDFS-10797.006.patch, HDFS-10797.007.patch, HDFS-10797.008.patch, > HDFS-10797.009.patch, HDFS-10797.010.patch, HDFS-10797.010.patch > > > DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how > much disk usage is used by a snapshot by tallying up the files in the > snapshot that have since been deleted (that way it won't overlap with regular > files whose disk usage is computed separately). However that is determined > from a diff that shows moved (to Trash or otherwise) or renamed files as a > deletion and a creation operation that may overlap with the list of blocks. > Only the deletion operation is taken into consideration, and this causes > those blocks to get represented twice in the disk usage tallying. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[ https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-10797: - Affects Version/s: 2.8.0 > Disk usage summary of snapshots causes renamed blocks to get counted twice > -- > > Key: HDFS-10797 > URL: https://issues.apache.org/jira/browse/HDFS-10797 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 2.8.0 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, > HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch, > HDFS-10797.006.patch, HDFS-10797.007.patch, HDFS-10797.008.patch, > HDFS-10797.009.patch, HDFS-10797.010.patch, HDFS-10797.010.patch > > > DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how > much disk usage is used by a snapshot by tallying up the files in the > snapshot that have since been deleted (that way it won't overlap with regular > files whose disk usage is computed separately). However that is determined > from a diff that shows moved (to Trash or otherwise) or renamed files as a > deletion and a creation operation that may overlap with the list of blocks. > Only the deletion operation is taken into consideration, and this causes > those blocks to get represented twice in the disk usage tallying. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
[
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556745#comment-15556745
]
Mingliang Liu commented on HDFS-10972:
--
# We don't have to call {{restoreStream()}} in {{testGetDatanodeInfo}} if it's
already called in {{tearDown}}. Actually we can remove this method and inline
its code in {{tearDown}}.
# I think we can still collect the out instead of err, and assert that the outs
is empty. We don't have to assert on specific error message as it is free to
change in the future. Asserting the ret value being -1 is enough.
{code}
203 /* collect outputs */
204 scanIntoList(err, outs);
205
206 /* verify results */
207 assertEquals(-1, ret);
208 assertEquals(
209 "expect to see 'Datanode unreachable.'",
210 1, outs.size());
211 assertThat(outs.get(0), containsString("Datanode unreachable"));
{code}
Something like following:
{code}
203 /* collect outputs */
204 scanIntoList(out, outs);
205
206 /* verify results */
207 assertEquals(-1, ret);
208 assertTrue("Unexpected getDatanodeInfo stdout", outs.isEmpty());
{code}
> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10972.000.patch, HDFS-10972.001.patch,
> HDFS-10972.002.patch
>
>
> getDatanodeInfo should be tested in admin CLI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10930) Refactor: Wrap Datanode IO related operations
[
https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556717#comment-15556717
]
Hadoop QA commented on HDFS-10930:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 11 new + 220 unchanged - 4 fixed = 231 total (was 224) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
44s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 2 new + 7
unchanged - 0 fixed = 9 total (was 7) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFileConcurrentReader |
| | hadoop.hdfs.server.datanode.TestTransferRbw |
| | hadoop.hdfs.TestFileAppend3 |
| | hadoop.hdfs.crypto.TestHdfsCryptoStreams |
| | hadoop.hdfs.TestWriteRead |
| | hadoop.hdfs.TestHFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10930 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831591/HDFS-10930.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e4bd9cc6f79a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e57fa81 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17062/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/17062/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17062/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test
[jira] [Updated] (HDFS-10985) o.a.h.ha.TestZKFailoverController should not use fixed time sleep before assertsions
[
https://issues.apache.org/jira/browse/HDFS-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10985:
-
Description:
{{TestZKFailoverController#testGracefulFailoverMultipleZKfcs}} uses fixed time
sleep before assertions. This may fail sometimes though 10 seconds are
generally long enough. I think we can use {{GenericTestUtils.waitFor()}} to
retry the assertions.
If this makes sense, we can address all other places in
{{TestZKFailoverController}}, including {{testGracefulFailover}} and
{{testDontFailoverToUnhealthyNode}}.
was:{{TestZKFailoverController#testGracefulFailoverMultipleZKfcs}} uses fixed
time sleep before assertions. This may fail sometimes though 10 seconds are
generally long enough. I think we can use {{GenericTestUtils.waitFor()}} to
retry the assertions.
> o.a.h.ha.TestZKFailoverController should not use fixed time sleep before
> assertsions
>
>
> Key: HDFS-10985
> URL: https://issues.apache.org/jira/browse/HDFS-10985
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HDFS-10985.000.patch
>
>
> {{TestZKFailoverController#testGracefulFailoverMultipleZKfcs}} uses fixed
> time sleep before assertions. This may fail sometimes though 10 seconds are
> generally long enough. I think we can use {{GenericTestUtils.waitFor()}} to
> retry the assertions.
> If this makes sense, we can address all other places in
> {{TestZKFailoverController}}, including {{testGracefulFailover}} and
> {{testDontFailoverToUnhealthyNode}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10985) o.a.h.ha.TestZKFailoverController should not use fixed time sleep before assertsions
[
https://issues.apache.org/jira/browse/HDFS-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10985:
-
Summary: o.a.h.ha.TestZKFailoverController should not use fixed time sleep
before assertsions (was:
o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs should not
use fixed time sleep before assertsions)
> o.a.h.ha.TestZKFailoverController should not use fixed time sleep before
> assertsions
>
>
> Key: HDFS-10985
> URL: https://issues.apache.org/jira/browse/HDFS-10985
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HDFS-10985.000.patch
>
>
> {{TestZKFailoverController#testGracefulFailoverMultipleZKfcs}} uses fixed
> time sleep before assertions. This may fail sometimes though 10 seconds are
> generally long enough. I think we can use {{GenericTestUtils.waitFor()}} to
> retry the assertions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10985) o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs should not use fixed time sleep before assertsions
[
https://issues.apache.org/jira/browse/HDFS-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10985:
-
Summary:
o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs should not
use fixed time sleep before assertsions (was:
o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails
intermittently)
> o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs should
> not use fixed time sleep before assertsions
> --
>
> Key: HDFS-10985
> URL: https://issues.apache.org/jira/browse/HDFS-10985
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10985.000.patch
>
>
> {{TestZKFailoverController#testGracefulFailoverMultipleZKfcs}} uses fixed
> time sleep before assertions. This may fail sometimes though 10 seconds are
> generally long enough. I think we can use {{GenericTestUtils.waitFor()}} to
> retry the assertions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10985) o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs should not use fixed time sleep before assertsions
[
https://issues.apache.org/jira/browse/HDFS-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10985:
-
Priority: Minor (was: Major)
> o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs should
> not use fixed time sleep before assertsions
> --
>
> Key: HDFS-10985
> URL: https://issues.apache.org/jira/browse/HDFS-10985
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HDFS-10985.000.patch
>
>
> {{TestZKFailoverController#testGracefulFailoverMultipleZKfcs}} uses fixed
> time sleep before assertions. This may fail sometimes though 10 seconds are
> generally long enough. I think we can use {{GenericTestUtils.waitFor()}} to
> retry the assertions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10985) o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10985:
-
Description: {{TestZKFailoverController#testGracefulFailoverMultipleZKfcs}}
uses fixed time sleep before assertions. This may fail sometimes though 10
seconds are generally long enough. I think we can use
{{GenericTestUtils.waitFor()}} to retry the assertions. (was: h6. Error Message
{quote}
Unable to become active. Local node did not get an opportunity to do so from
ZooKeeper, or the local node took too long to transition to active.
{quote}
h6. Stacktrace
{quote}
org.apache.hadoop.ha.ServiceFailedException: Unable to become active. Local
node did not get an opportunity to do so from ZooKeeper, or the local node took
too long to transition to active.
at
org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:690)
at
org.apache.hadoop.ha.ZKFailoverController.access$400(ZKFailoverController.java:62)
at
org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:607)
at
org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:604)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1795)
at
org.apache.hadoop.ha.ZKFailoverController.gracefulFailoverToYou(ZKFailoverController.java:604)
at
org.apache.hadoop.ha.ZKFCRpcServer.gracefulFailover(ZKFCRpcServer.java:94)
at
org.apache.hadoop.ha.TestZKFailoverController.testGracefulFailoverMultipleZKfcs(TestZKFailoverController.java:590)
{quote}
See recent failing build:
#
https://builds.apache.org/job/PreCommit-HADOOP-Build/10705/testReport/org.apache.hadoop.ha/TestZKFailoverController/testGracefulFailoverMultipleZKfcs/
# to add more
I think we can use {{GenericTestUtils.waitFor()}} to retry the assertions.)
> o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails
> intermittently
>
>
> Key: HDFS-10985
> URL: https://issues.apache.org/jira/browse/HDFS-10985
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10985.000.patch
>
>
> {{TestZKFailoverController#testGracefulFailoverMultipleZKfcs}} uses fixed
> time sleep before assertions. This may fail sometimes though 10 seconds are
> generally long enough. I think we can use {{GenericTestUtils.waitFor()}} to
> retry the assertions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
[
https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556694#comment-15556694
]
Hadoop QA commented on HDFS-10967:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 433 unchanged - 0 fixed = 437 total (was 433) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 23s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestSafeMode |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10967 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832206/HDFS-10967.00.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9656bba47972 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e57fa81 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17061/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17061/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17061/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17061/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org
[jira] [Updated] (HDFS-10985) o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10985:
-
Status: Patch Available (was: Open)
> o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails
> intermittently
>
>
> Key: HDFS-10985
> URL: https://issues.apache.org/jira/browse/HDFS-10985
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10985.000.patch
>
>
> h6. Error Message
> {quote}
> Unable to become active. Local node did not get an opportunity to do so from
> ZooKeeper, or the local node took too long to transition to active.
> {quote}
> h6. Stacktrace
> {quote}
> org.apache.hadoop.ha.ServiceFailedException: Unable to become active. Local
> node did not get an opportunity to do so from ZooKeeper, or the local node
> took too long to transition to active.
> at
> org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:690)
> at
> org.apache.hadoop.ha.ZKFailoverController.access$400(ZKFailoverController.java:62)
> at
> org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:607)
> at
> org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:604)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1795)
> at
> org.apache.hadoop.ha.ZKFailoverController.gracefulFailoverToYou(ZKFailoverController.java:604)
> at
> org.apache.hadoop.ha.ZKFCRpcServer.gracefulFailover(ZKFCRpcServer.java:94)
> at
> org.apache.hadoop.ha.TestZKFailoverController.testGracefulFailoverMultipleZKfcs(TestZKFailoverController.java:590)
> {quote}
> See recent failing build:
> #
> https://builds.apache.org/job/PreCommit-HADOOP-Build/10705/testReport/org.apache.hadoop.ha/TestZKFailoverController/testGracefulFailoverMultipleZKfcs/
> # to add more
> I think we can use {{GenericTestUtils.waitFor()}} to retry the assertions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10985) o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10985:
-
Attachment: HDFS-10985.000.patch
[~ste...@apache.org] and [~atm], does this makes sense to you?
> o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails
> intermittently
>
>
> Key: HDFS-10985
> URL: https://issues.apache.org/jira/browse/HDFS-10985
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10985.000.patch
>
>
> h6. Error Message
> {quote}
> Unable to become active. Local node did not get an opportunity to do so from
> ZooKeeper, or the local node took too long to transition to active.
> {quote}
> h6. Stacktrace
> {quote}
> org.apache.hadoop.ha.ServiceFailedException: Unable to become active. Local
> node did not get an opportunity to do so from ZooKeeper, or the local node
> took too long to transition to active.
> at
> org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:690)
> at
> org.apache.hadoop.ha.ZKFailoverController.access$400(ZKFailoverController.java:62)
> at
> org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:607)
> at
> org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:604)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1795)
> at
> org.apache.hadoop.ha.ZKFailoverController.gracefulFailoverToYou(ZKFailoverController.java:604)
> at
> org.apache.hadoop.ha.ZKFCRpcServer.gracefulFailover(ZKFCRpcServer.java:94)
> at
> org.apache.hadoop.ha.TestZKFailoverController.testGracefulFailoverMultipleZKfcs(TestZKFailoverController.java:590)
> {quote}
> See recent failing build:
> #
> https://builds.apache.org/job/PreCommit-HADOOP-Build/10705/testReport/org.apache.hadoop.ha/TestZKFailoverController/testGracefulFailoverMultipleZKfcs/
> # to add more
> I think we can use {{GenericTestUtils.waitFor()}} to retry the assertions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10985) o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10985:
-
Component/s: test
> o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails
> intermittently
>
>
> Key: HDFS-10985
> URL: https://issues.apache.org/jira/browse/HDFS-10985
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>
> h6. Error Message
> {quote}
> Unable to become active. Local node did not get an opportunity to do so from
> ZooKeeper, or the local node took too long to transition to active.
> {quote}
> h6. Stacktrace
> {quote}
> org.apache.hadoop.ha.ServiceFailedException: Unable to become active. Local
> node did not get an opportunity to do so from ZooKeeper, or the local node
> took too long to transition to active.
> at
> org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:690)
> at
> org.apache.hadoop.ha.ZKFailoverController.access$400(ZKFailoverController.java:62)
> at
> org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:607)
> at
> org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:604)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1795)
> at
> org.apache.hadoop.ha.ZKFailoverController.gracefulFailoverToYou(ZKFailoverController.java:604)
> at
> org.apache.hadoop.ha.ZKFCRpcServer.gracefulFailover(ZKFCRpcServer.java:94)
> at
> org.apache.hadoop.ha.TestZKFailoverController.testGracefulFailoverMultipleZKfcs(TestZKFailoverController.java:590)
> {quote}
> See recent failing build:
> #
> https://builds.apache.org/job/PreCommit-HADOOP-Build/10705/testReport/org.apache.hadoop.ha/TestZKFailoverController/testGracefulFailoverMultipleZKfcs/
> # to add more
> I think we can use {{GenericTestUtils.waitFor()}} to retry the assertions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10985) o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails intermittently
Mingliang Liu created HDFS-10985:
Summary:
o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails
intermittently
Key: HDFS-10985
URL: https://issues.apache.org/jira/browse/HDFS-10985
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha
Reporter: Mingliang Liu
Assignee: Mingliang Liu
h6. Error Message
{quote}
Unable to become active. Local node did not get an opportunity to do so from
ZooKeeper, or the local node took too long to transition to active.
{quote}
h6. Stacktrace
{quote}
org.apache.hadoop.ha.ServiceFailedException: Unable to become active. Local
node did not get an opportunity to do so from ZooKeeper, or the local node took
too long to transition to active.
at
org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:690)
at
org.apache.hadoop.ha.ZKFailoverController.access$400(ZKFailoverController.java:62)
at
org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:607)
at
org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:604)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1795)
at
org.apache.hadoop.ha.ZKFailoverController.gracefulFailoverToYou(ZKFailoverController.java:604)
at
org.apache.hadoop.ha.ZKFCRpcServer.gracefulFailover(ZKFCRpcServer.java:94)
at
org.apache.hadoop.ha.TestZKFailoverController.testGracefulFailoverMultipleZKfcs(TestZKFailoverController.java:590)
{quote}
See recent failing build:
#
https://builds.apache.org/job/PreCommit-HADOOP-Build/10705/testReport/org.apache.hadoop.ha/TestZKFailoverController/testGracefulFailoverMultipleZKfcs/
# to add more
I think we can use {{GenericTestUtils.waitFor()}} to retry the assertions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
[ https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556648#comment-15556648 ] Xiaobing Zhou commented on HDFS-10972: -- v002 added test for run on exception. > Add unit test for HDFS command 'dfsadmin -getDatanodeInfo' > -- > > Key: HDFS-10972 > URL: https://issues.apache.org/jira/browse/HDFS-10972 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, shell, test >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10972.000.patch, HDFS-10972.001.patch, > HDFS-10972.002.patch > > > getDatanodeInfo should be tested in admin CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
[ https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10972: - Attachment: HDFS-10972.002.patch > Add unit test for HDFS command 'dfsadmin -getDatanodeInfo' > -- > > Key: HDFS-10972 > URL: https://issues.apache.org/jira/browse/HDFS-10972 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, shell, test >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10972.000.patch, HDFS-10972.001.patch, > HDFS-10972.002.patch > > > getDatanodeInfo should be tested in admin CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10983) OIV webhdfs interface should make an EC file explicit
[ https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10983: --- Description: The OIV tool's webhdfs interface does not print if a file is striped or not. Also, it prints the file's EC policy ID as replication factor, which is inconsistent to the output of a typical webhdfs call to the cluster, which always shows replication factor of 0 for EC files. was: The OIV tool's webhdfs interface does not print if a file is stripped or not. Also, it prints the file's EC policy ID as replication factor, which is inconsistent to the output of a typical webhdfs call to the cluster, which always shows replication factor of 0 for EC files. > OIV webhdfs interface should make an EC file explicit > - > > Key: HDFS-10983 > URL: https://issues.apache.org/jira/browse/HDFS-10983 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang > > The OIV tool's webhdfs interface does not print if a file is striped or not. > Also, it prints the file's EC policy ID as replication factor, which is > inconsistent to the output of a typical webhdfs call to the cluster, which > always shows replication factor of 0 for EC files. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10984) Expose nntop output as metrics
Siddharth Wagle created HDFS-10984: -- Summary: Expose nntop output as metrics Key: HDFS-10984 URL: https://issues.apache.org/jira/browse/HDFS-10984 Project: Hadoop HDFS Issue Type: Task Components: namenode Affects Versions: 2.7.0 Reporter: Siddharth Wagle The nntop output is already exposed via JMX with HDFS-6982. However external metrics systems do not get this data. It would be valuable to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle reassigned HDFS-10984: -- Assignee: Siddharth Wagle > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10924) Add a new instrumented read-write lock
[
https://issues.apache.org/jira/browse/HDFS-10924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556560#comment-15556560
]
Xiao Chen commented on HDFS-10924:
--
Thanks [~jingcheng...@intel.com] for the nice work here, and on the related
HDFS-9668.
Some comments on patch 3, looks good overall:
{{AutoCloseableReadWriteLockWrapper}}:
- Are the constructors of {{AutoCloseableReadLock}} and
{{AutoCloseableWriteLock}} intentionally public? I'm having a hard time coming
up with a scenario of this.
- Maybe we could update the above to constructors to accept the {{lock}}
parameter. This is also more consistent with what the new instrumented lock
class does. (as well as java's {{ReentrantReadWriteLock}} class)
{{InstrumentedAutoCloseableReadWriteLockWrapper}}:
- Please check input variables on
{{InstrumentedAutoCloseableReadWriteLockWrapper}}'s constructor. (e.g. time >
0, logger not null. Do we allow null name?)
- Personally feels the following change would be more readable:
{code}
if (lockWarningThresholdMs - lockHeldTime < 0) {
...
now = clock.monotonicNow();
localLastLogTs = lastLogTimestamp.get();
long deltaSinceLastLog = now - localLastLogTs;
// check should print log or not
if (deltaSinceLastLog - minLoggingGapMs < 0) {
warningsSuppressed.incrementAndGet();
return;
}
{code}
to
{code}
if (lockHeldTime > lockWarningThresholdMs) {
now = clock.monotonicNow();
localLastLogTs = lastLogTimestamp.get();
if (now - localLastLogTs < minLoggingGapMs) {
warningsSuppressed.incrementAndGet();
return;
}
{code}
- Feels like a StringBuilder would be more readable and efficient in
{{logWarning}}, than the current 3 concatenation.
> Add a new instrumented read-write lock
> --
>
> Key: HDFS-10924
> URL: https://issues.apache.org/jira/browse/HDFS-10924
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-10924-2.patch, HDFS-10924-3.patch, HDFS-10924.patch
>
>
> Add a new instrumented read-write lock in hadoop common, so that the
> HDFS-9668 can use this to improve the locking in FsDatasetImpl
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10983) OIV webhdfs interface should make an EC file explicit
[ https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-10983: --- Issue Type: Sub-task (was: Bug) Parent: HDFS-8031 > OIV webhdfs interface should make an EC file explicit > - > > Key: HDFS-10983 > URL: https://issues.apache.org/jira/browse/HDFS-10983 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang > > The OIV tool's webhdfs interface does not print if a file is stripped or not. > Also, it prints the file's EC policy ID as replication factor, which is > inconsistent to the output of a typical webhdfs call to the cluster, which > always shows replication factor of 0 for EC files. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10983) OIV webhdfs interface should make an EC file explicit
Wei-Chiu Chuang created HDFS-10983: -- Summary: OIV webhdfs interface should make an EC file explicit Key: HDFS-10983 URL: https://issues.apache.org/jira/browse/HDFS-10983 Project: Hadoop HDFS Issue Type: Bug Reporter: Wei-Chiu Chuang The OIV tool's webhdfs interface does not print if a file is stripped or not. Also, it prints the file's EC policy ID as replication factor, which is inconsistent to the output of a typical webhdfs call to the cluster, which always shows replication factor of 0 for EC files. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded
[
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556537#comment-15556537
]
Zhe Zhang commented on HDFS-10971:
--
Thanks Andrew. SGTM to expose replication factor 1 in {{FileStatus}}, same as
S3A. Like you mentioned an operator can always check the HdfsFileStatus to find
out the actual durability.
> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if
> {{-pr}} is specified. However, if the source file is EC, the replication
> factor (which is EC policy) should not be replicated to the destination file.
> When a HdfsFileStatus is converted to FileStatus, the replication factor is
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the
> replication factor of an EC file results in an IOException: "Requested
> replication factor of 0 is less than the required minimum of 1 for
> /tmp/dst/dest2"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10759) Change fsimage bool isStriped from boolean to an enum
[
https://issues.apache.org/jira/browse/HDFS-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556534#comment-15556534
]
Jing Zhao commented on HDFS-10759:
--
Yeah I also think the idea is good. But we need to guarantee the compatibility:
the old fsimage should still be supported and new enum types should be easily
added (which means we may need to add UNKNOWN_TYPE in the enum according to the
link).
> Change fsimage bool isStriped from boolean to an enum
> -
>
> Key: HDFS-10759
> URL: https://issues.apache.org/jira/browse/HDFS-10759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2
>Reporter: Ewan Higgs
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10759.0001.patch
>
>
> The new erasure coding project has updated the protocol for fsimage such that
> the {{INodeFile}} has a boolean '{{isStriped}}'. I think this is better as an
> enum or integer since a boolean precludes any future block types.
> For example:
> {code}
> enum BlockType {
> CONTIGUOUS = 0,
> STRIPED = 1,
> }
> {code}
> We can also make this more robust to future changes where there are different
> block types supported in a staged rollout. Here, we would use
> {{UNKNOWN_BLOCK_TYPE}} as the first value since this is the default value.
> See
> [here|http://androiddevblog.com/protocol-buffers-pitfall-adding-enum-values/]
> for more discussion.
> {code}
> enum BlockType {
> UNKNOWN_BLOCK_TYPE = 0,
> CONTIGUOUS = 1,
> STRIPED = 2,
> }
> {code}
> But I'm not convinced this is necessary since there are other enums that
> don't use this approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10980) Optimize check for existence of parent directory
[
https://issues.apache.org/jira/browse/HDFS-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556513#comment-15556513
]
Hudson commented on HDFS-10980:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10571 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10571/])
HDFS-10980. Optimize check for existence of parent directory. (kihwal: rev
e57fa81d9559a93d77fd724f7792326c31a490be)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirMkdirOp.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSymlinkOp.java
> Optimize check for existence of parent directory
>
>
> Key: HDFS-10980
> URL: https://issues.apache.org/jira/browse/HDFS-10980
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10980.patch
>
>
> {{FSDirectory.verifyParentDir()}} uses a {{Path}} object to parse and return
> the parent path. This is very expensive compared to using the path within
> the IIP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10930) Refactor: Wrap Datanode IO related operations
[ https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10930: -- Status: Patch Available (was: Open) > Refactor: Wrap Datanode IO related operations > - > > Key: HDFS-10930 > URL: https://issues.apache.org/jira/browse/HDFS-10930 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-10930.01.patch > > > Datanode IO (Disk/Network) related operations and instrumentations are > currently spilled in many classes such as DataNode.java, BlockReceiver.java, > BlockSender.java, FsDatasetImpl.java, FsVolumeImpl.java, > DirectoryScanner.java, BlockScanner.java, FsDatasetAsyncDiskService.java, > LocalReplica.java, LocalReplicaPipeline.java, Storage.java, etc. > This ticket is opened to consolidate IO related operations for easy > instrumentation, metrics collection, logging and trouble shooting. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10980) Optimize check for existence of parent directory
[
https://issues.apache.org/jira/browse/HDFS-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-10980:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.8.0
Status: Resolved (was: Patch Available)
Committed to trunk, branch-2 and branch-2.8.
> Optimize check for existence of parent directory
>
>
> Key: HDFS-10980
> URL: https://issues.apache.org/jira/browse/HDFS-10980
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10980.patch
>
>
> {{FSDirectory.verifyParentDir()}} uses a {{Path}} object to parse and return
> the parent path. This is very expensive compared to using the path within
> the IIP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
[ https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556406#comment-15556406 ] Zhe Zhang commented on HDFS-10967: -- [~mingma] [~kihwal] [~daryn] I'd like to hear your opinions based on operating large clusters (not sure if you have the heterogeneity issue though). In our case, the Balancer is essentially fighting with application data ingestion. Triaging the 2nd and 3rd replicas at ingestion time should be better than spending the bandwidth to move them later. > Add configuration for BlockPlacementPolicy to avoid near-full DataNodes > --- > > Key: HDFS-10967 > URL: https://issues.apache.org/jira/browse/HDFS-10967 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Labels: balancer > Attachments: HDFS-10967.00.patch, HDFS-10967.poc.patch > > > Large production clusters are likely to have heterogeneous nodes in terms of > storage capacity, memory, and CPU cores. It is not always possible to > proportionally ingest data into DataNodes based on their remaining storage > capacity. Therefore it's possible for a subset of DataNodes to be much closer > to full capacity than the rest. > This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks of > low-storage nodes and _n_ whole racks of high-storage nodes. So It'd be very > useful if we can lower the chance for those near-full DataNodes to become > destinations for the 2nd and 3rd replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10980) Optimize check for existence of parent directory
[
https://issues.apache.org/jira/browse/HDFS-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556402#comment-15556402
]
Kihwal Lee commented on HDFS-10980:
---
+1 looks simple and straightforward.
> Optimize check for existence of parent directory
>
>
> Key: HDFS-10980
> URL: https://issues.apache.org/jira/browse/HDFS-10980
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10980.patch
>
>
> {{FSDirectory.verifyParentDir()}} uses a {{Path}} object to parse and return
> the parent path. This is very expensive compared to using the path within
> the IIP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10801) [SPS]: Protocol buffer changes for sending storage movement commands from NN to DN
[ https://issues.apache.org/jira/browse/HDFS-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556385#comment-15556385 ] Uma Maheswara Rao G commented on HDFS-10801: Latest patch looks good to me. +1 > [SPS]: Protocol buffer changes for sending storage movement commands from NN > to DN > --- > > Key: HDFS-10801 > URL: https://issues.apache.org/jira/browse/HDFS-10801 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R > Attachments: HDFS-10801-HDFS-10285-00.patch, > HDFS-10801-HDFS-10285-01.patch > > > This JIRA is for tracking the work of protocol buffer changes for sending the > storage movement commands from NN to DN -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
[
https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10967:
-
Attachment: HDFS-10967.00.patch
Thought more about the design options. Currently I think the most reasonable
direction is to:
# Apply the logic only when trying to {{chooseRemoteRack}} -- that is when the
2nd and 3rd replicas are being placed. In most cases, triaging the 2nd and 3rd
replicas from near-full DNs is already sufficient to address the balancing
issue.
# Use the percentage of remaining capacity because current NN metrics are
already based on percentage. So admins should be most comfortable operating
based on it (e.g. {{97%}} full).
# Do random selection a few times instead of completely avoiding near-full DNs.
Like mentioned above, imbalance will cause an issue only if a large number of
DNs are near full. So a statistical solution should be sufficient.
> Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
> ---
>
> Key: HDFS-10967
> URL: https://issues.apache.org/jira/browse/HDFS-10967
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Labels: balancer
> Attachments: HDFS-10967.00.patch, HDFS-10967.poc.patch
>
>
> Large production clusters are likely to have heterogeneous nodes in terms of
> storage capacity, memory, and CPU cores. It is not always possible to
> proportionally ingest data into DataNodes based on their remaining storage
> capacity. Therefore it's possible for a subset of DataNodes to be much closer
> to full capacity than the rest.
> This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks of
> low-storage nodes and _n_ whole racks of high-storage nodes. So It'd be very
> useful if we can lower the chance for those near-full DataNodes to become
> destinations for the 2nd and 3rd replicas.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9618) Fix mismatch between log level and guard in BlockManager#computeRecoveryWorkForBlocks
[
https://issues.apache.org/jira/browse/HDFS-9618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556346#comment-15556346
]
Mehran Hassani commented on HDFS-9618:
--
Correct me if I am wrong but I think self4j checks for all the log levels
before making the string so doing IsLevelEnabled() is unnecessary when you use
it.
> Fix mismatch between log level and guard in
> BlockManager#computeRecoveryWorkForBlocks
> -
>
> Key: HDFS-9618
> URL: https://issues.apache.org/jira/browse/HDFS-9618
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: HDFS-9618.001.patch, HDFS-9618.002.patch
>
>
> Debug log message is constructed when {{Logger#isInfoEnabled}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10974) Document replication factor for EC files.
[
https://issues.apache.org/jira/browse/HDFS-10974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556297#comment-15556297
]
Wei-Chiu Chuang commented on HDFS-10974:
I think the easiest approach is to add this note somewhere in the HDFS Erasure
Coding doc so that admins do not panic :) Most tools like -ls, -stat use
FileStatus which is an abstraction for all file systems. Some of the file
systems do not support replication factor (e.g. S3) so this is not unexpected.
> Document replication factor for EC files.
> -
>
> Key: HDFS-10974
> URL: https://issues.apache.org/jira/browse/HDFS-10974
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
>
> The replication factor of an EC file is 0 from a user perspective. This
> affects all user visible commands and APIs, such as {{hdfs dfs -ls}}, {{hdfs
> dfs -stat}}, webhdfs, httpfs...etc.
> In addition, FileSystem#setReplication API for an EC file is ignored. So is
> the command {{hdfs dfs -setrep}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10982) 'errno' set on successful code path in hdfsOpenFileImpl()
[
https://issues.apache.org/jira/browse/HDFS-10982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reassigned HDFS-10982:
-
Assignee: John Zhuge
> 'errno' set on successful code path in hdfsOpenFileImpl()
> -
>
> Key: HDFS-10982
> URL: https://issues.apache.org/jira/browse/HDFS-10982
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 2.7.0
>Reporter: Sailesh Mukil
>Assignee: John Zhuge
> Labels: easyfix
>
> In hdfsOpenFileImpl() in libhdfs/hdfs.c, the following code is used to check
> if the underlying FileSystem class supports direct reads (i.e.
> read(ByteBuffer)):
> {code:java}
> if ((flags & O_WRONLY) == 0) {
> // Try a test read to see if we can do direct reads
> char buf;
> if (readDirect(fs, file, , 0) == 0) {
> // Success - 0-byte read should return 0
> file->flags |= HDFS_FILE_SUPPORTS_DIRECT_READ;
> } else if (errno != ENOTSUP) {
> // Unexpected error. Clear it, don't set the direct flag.
> fprintf(stderr,
> "hdfsOpenFile(%s): WARN: Unexpected error %d when testing "
> "for direct read compatibility\n", path, errno);
> }
> }
> ret = 0;
> {code}
> The S3A connector, specifically S3AInputStream does not support direct reads,
> and therefore it sets 'errno = ENOTSUP' on a call to readDirect().
> This 'errno' should be reset before returning the call because this is not an
> actual error and we're okay with not having direct reads supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10629) Federation Router
[
https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556262#comment-15556262
]
Inigo Goiri commented on HDFS-10629:
[~jakace], the refactor with the separate {{RouterRpcClient}} is cleaner.
However, it looks like the connections are never released now and always marked
as busy.
> Federation Router
> -
>
> Key: HDFS-10629
> URL: https://issues.apache.org/jira/browse/HDFS-10629
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Inigo Goiri
>Assignee: Jason Kace
> Attachments: HDFS-10629-HDFS-10467-002.patch,
> HDFS-10629-HDFS-10467-003.patch, HDFS-10629-HDFS-10467-004.patch,
> HDFS-10629-HDFS-10467-005.patch, HDFS-10629-HDFS-10467-006.patch,
> HDFS-10629.000.patch, HDFS-10629.001.patch
>
>
> Component that routes calls from the clients to the right Namespace. It
> implements {{ClientProtocol}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10429) DataStreamer interrupted warning always appears when using CLI upload file
[
https://issues.apache.org/jira/browse/HDFS-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556255#comment-15556255
]
Zhiyuan Yang commented on HDFS-10429:
-
I'm sorry but I don't think I'm able to handle this for now given my very
limited knowledge about HDFS (and limited time...). It seems you're much more
familiar with this piece of code than me, so feel free to take over this jira
if you want a quick fix.
> DataStreamer interrupted warning always appears when using CLI upload file
> ---
>
> Key: HDFS-10429
> URL: https://issues.apache.org/jira/browse/HDFS-10429
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>Priority: Minor
> Attachments: HDFS-10429.1.patch, HDFS-10429.2.patch
>
>
> Every time I use 'hdfs dfs -put' upload file, this warning is printed:
> {code:java}
> 16/05/18 20:57:56 WARN hdfs.DataStreamer: Caught exception
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1245)
> at java.lang.Thread.join(Thread.java:1319)
> at
> org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:871)
> at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:519)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:696)
> {code}
> The reason is this: originally, DataStreamer::closeResponder always prints a
> warning about InterruptedException; since HDFS-9812,
> DFSOutputStream::closeImpl always forces threads to close, which causes
> InterruptedException.
> A simple fix is to use debug level log instead of warning level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10429) DataStreamer interrupted warning always appears when using CLI upload file
[
https://issues.apache.org/jira/browse/HDFS-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556220#comment-15556220
]
Wei-Chiu Chuang commented on HDFS-10429:
I think that closeThreads(false) should be called before try...completeFile to
allow response processors to close gracefully while the client asks NN to
complete the file. It is possible that the threads are not closed when the
forceful close() is called, in which case reducing the log level from warn to
debug can eliminate user frustration.
> DataStreamer interrupted warning always appears when using CLI upload file
> ---
>
> Key: HDFS-10429
> URL: https://issues.apache.org/jira/browse/HDFS-10429
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>Priority: Minor
> Attachments: HDFS-10429.1.patch, HDFS-10429.2.patch
>
>
> Every time I use 'hdfs dfs -put' upload file, this warning is printed:
> {code:java}
> 16/05/18 20:57:56 WARN hdfs.DataStreamer: Caught exception
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1245)
> at java.lang.Thread.join(Thread.java:1319)
> at
> org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:871)
> at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:519)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:696)
> {code}
> The reason is this: originally, DataStreamer::closeResponder always prints a
> warning about InterruptedException; since HDFS-9812,
> DFSOutputStream::closeImpl always forces threads to close, which causes
> InterruptedException.
> A simple fix is to use debug level log instead of warning level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-3544) Ability to use SimpleRegeratingCode to fix missing blocks
[ https://issues.apache.org/jira/browse/HDFS-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556169#comment-15556169 ] Andrew Wang commented on HDFS-3544: --- The other note is that we have a JIRA to add LRC support (HDFS-7345) but it's patent encumbered by Microsoft. > Ability to use SimpleRegeratingCode to fix missing blocks > - > > Key: HDFS-3544 > URL: https://issues.apache.org/jira/browse/HDFS-3544 > Project: Hadoop HDFS > Issue Type: Improvement > Components: contrib/raid >Reporter: dhruba borthakur > > ReedSolomon encoding (n, k) has n storage nodes and can tolerate n-k > failures. Regenerating a block needs to access k blocks. This is a problem > when n and k are large. Instead, we can use simple regenerating codes (n, k, > f) that does first does ReedSolomon (n,k) and then does XOR with f stripe > size. Then, a single disk failure needs to access only f nodes and f can be > very small. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10982) 'errno' set on successful code path in hdfsOpenFileImpl()
Sailesh Mukil created HDFS-10982:
Summary: 'errno' set on successful code path in hdfsOpenFileImpl()
Key: HDFS-10982
URL: https://issues.apache.org/jira/browse/HDFS-10982
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.7.0
Reporter: Sailesh Mukil
In hdfsOpenFileImpl() in libhdfs/hdfs.c, the following code is used to check if
the underlying FileSystem class supports direct reads (i.e. read(ByteBuffer)):
{code:java}
if ((flags & O_WRONLY) == 0) {
// Try a test read to see if we can do direct reads
char buf;
if (readDirect(fs, file, , 0) == 0) {
// Success - 0-byte read should return 0
file->flags |= HDFS_FILE_SUPPORTS_DIRECT_READ;
} else if (errno != ENOTSUP) {
// Unexpected error. Clear it, don't set the direct flag.
fprintf(stderr,
"hdfsOpenFile(%s): WARN: Unexpected error %d when testing "
"for direct read compatibility\n", path, errno);
}
}
ret = 0;
{code}
The S3A connector, specifically S3AInputStream does not support direct reads,
and therefore it sets 'errno = ENOTSUP' on a call to readDirect().
This 'errno' should be reset before returning the call because this is not an
actual error and we're okay with not having direct reads supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-3544) Ability to use SimpleRegeratingCode to fix missing blocks
[ https://issues.apache.org/jira/browse/HDFS-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-3544. --- Resolution: Won't Fix HDFS-RAID has been removed and native EC support is available at HDFS-7285. Closing as won't fix. > Ability to use SimpleRegeratingCode to fix missing blocks > - > > Key: HDFS-3544 > URL: https://issues.apache.org/jira/browse/HDFS-3544 > Project: Hadoop HDFS > Issue Type: Improvement > Components: contrib/raid >Reporter: dhruba borthakur > > ReedSolomon encoding (n, k) has n storage nodes and can tolerate n-k > failures. Regenerating a block needs to access k blocks. This is a problem > when n and k are large. Instead, we can use simple regenerating codes (n, k, > f) that does first does ReedSolomon (n,k) and then does XOR with f stripe > size. Then, a single disk failure needs to access only f nodes and f can be > very small. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-600) Support for pluggable erasure coding policy for HDFS
[ https://issues.apache.org/jira/browse/HDFS-600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-600. -- Resolution: Won't Fix This one goes back to the HDFS-RAID days, which has been removed. EC has also been implemented natively at HDFS-7285. Closing as won't fix. > Support for pluggable erasure coding policy for HDFS > > > Key: HDFS-600 > URL: https://issues.apache.org/jira/browse/HDFS-600 > Project: Hadoop HDFS > Issue Type: Improvement > Components: contrib/raid >Reporter: dhruba borthakur > > HDFS-503 introduces erasure coding for HDFS files. It currently uses "xor" > algoritm as the Erasure coding algorithm. It would be nice if that Erasure > Coding framework supports a pluggable API to allow plugging in other Erasure > Coding policies. A few of these policies are mentioned by Hong at > https://issues.apache.org/jira/browse/HDFS-503?focusedCommentId=12735011=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12735011 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10759) Change fsimage bool isStriped from boolean to an enum
[
https://issues.apache.org/jira/browse/HDFS-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556155#comment-15556155
]
Andrew Wang commented on HDFS-10759:
Idea seems fine to me, I like evolvable datatypes.
> Change fsimage bool isStriped from boolean to an enum
> -
>
> Key: HDFS-10759
> URL: https://issues.apache.org/jira/browse/HDFS-10759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2
>Reporter: Ewan Higgs
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10759.0001.patch
>
>
> The new erasure coding project has updated the protocol for fsimage such that
> the {{INodeFile}} has a boolean '{{isStriped}}'. I think this is better as an
> enum or integer since a boolean precludes any future block types.
> For example:
> {code}
> enum BlockType {
> CONTIGUOUS = 0,
> STRIPED = 1,
> }
> {code}
> We can also make this more robust to future changes where there are different
> block types supported in a staged rollout. Here, we would use
> {{UNKNOWN_BLOCK_TYPE}} as the first value since this is the default value.
> See
> [here|http://androiddevblog.com/protocol-buffers-pitfall-adding-enum-values/]
> for more discussion.
> {code}
> enum BlockType {
> UNKNOWN_BLOCK_TYPE = 0,
> CONTIGUOUS = 1,
> STRIPED = 2,
> }
> {code}
> But I'm not convinced this is necessary since there are other enums that
> don't use this approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10979) Pass IIP for FSDirDeleteOp methods
[ https://issues.apache.org/jira/browse/HDFS-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556150#comment-15556150 ] Hudson commented on HDFS-10979: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10568 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10568/]) HDFS-10979. Pass IIP for FSDirDeleteOp methods. Contributed by Daryn (kihwal: rev 3565c9af17ab05bf9e7f68b71b6c6850df772bb9) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Pass IIP for FSDirDeleteOp methods > -- > > Key: HDFS-10979 > URL: https://issues.apache.org/jira/browse/HDFS-10979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10979.patch > > > Remove path strings from method signatures and/or replace with IIP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10980) Optimize check for existence of parent directory
[
https://issues.apache.org/jira/browse/HDFS-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556113#comment-15556113
]
Hadoop QA commented on HDFS-10980:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 57m
34s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10980 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832173/HDFS-10980.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bae1215b66a7 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 69620f95 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17060/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17060/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize check for existence of parent directory
>
>
> Key: HDFS-10980
> URL: https://issues.apache.org/jira/browse/HDFS-10980
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10980.patch
>
>
> {{FSDirectory.verifyParentDir()}} uses a {{Path}} object to parse and return
> the parent path. This is very expensive compared to using the path within
> the IIP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10979) Pass IIP for FSDirDeleteOp methods
[ https://issues.apache.org/jira/browse/HDFS-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10979: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.8.0 Status: Resolved (was: Patch Available) Committed to trunk, branch-2 and branch-2.8 > Pass IIP for FSDirDeleteOp methods > -- > > Key: HDFS-10979 > URL: https://issues.apache.org/jira/browse/HDFS-10979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10979.patch > > > Remove path strings from method signatures and/or replace with IIP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10979) Pass IIP for FSDirDeleteOp methods
[ https://issues.apache.org/jira/browse/HDFS-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556010#comment-15556010 ] Kihwal Lee commented on HDFS-10979: --- +1 Looks good. > Pass IIP for FSDirDeleteOp methods > -- > > Key: HDFS-10979 > URL: https://issues.apache.org/jira/browse/HDFS-10979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-10979.patch > > > Remove path strings from method signatures and/or replace with IIP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10827) When there are unrecoverable ec block groups, Namenode Web UI shows "There are X missing blocks." but doesn't show the block names.
[
https://issues.apache.org/jira/browse/HDFS-10827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1949#comment-1949
]
Jing Zhao commented on HDFS-10827:
--
Thanks for working on this, [~tasanuma0829]! The patch looks good to me
overall. Some minors:
# I think we can rename "isCorrupt" to "isMissing". In the outer "while" loop,
we're scanning all the blocks with {{QUEUE_WITH_CORRUPT_BLOCKS}} priority, and
this "if" logic is to select missing blocks.
# We may also want to take into account the decommissioning/decommissioned
internal blocks for the check.
# As follow-on work, maybe we can create a utility function to check if a block
is corrupted/missing, considering this is widely used in
BlockManager/FSNamesystem.
# For the unit test, do you think we can use {{MiniDFSCluster#corruptReplica}}
to simplify the code?
> When there are unrecoverable ec block groups, Namenode Web UI shows "There
> are X missing blocks." but doesn't show the block names.
> ---
>
> Key: HDFS-10827
> URL: https://issues.apache.org/jira/browse/HDFS-10827
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10827.1.patch, HDFS-10827.2.patch, case_2.png,
> case_3.png
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, NN Web UI doesn't show any warns.
> 2) 4~8 out of 9 internal blocks are missing, NN Web UI shows "There are 1
> missing blocks." but doesn't show the block names. (please see case_2.png)
> 3) 9 out of 9 internal blocks are missing, NN Web UI shows "There are 1
> missing blocks." and also shows the block name. (please see case_3.png)
> We should fix the case 2). I think NN Web UI should show the block names
> since the ec block group is unrecoverable.
> The values come from JMX. "There are X missing blocks." is
> {{NumberOfMissingBlocks}} and the block names are {{CorruptFiles}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
[ https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1885#comment-1885 ] Mingliang Liu commented on HDFS-10972: -- Can you add a failing case (e.g. no arguments provided) for this command? Otherwise +1. > Add unit test for HDFS command 'dfsadmin -getDatanodeInfo' > -- > > Key: HDFS-10972 > URL: https://issues.apache.org/jira/browse/HDFS-10972 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, shell, test >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10972.000.patch, HDFS-10972.001.patch > > > getDatanodeInfo should be tested in admin CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10939) Reduce performance penalty of encryption zones
[ https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10939: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed to branch-2 and branch-2.8 as well. > Reduce performance penalty of encryption zones > -- > > Key: HDFS-10939 > URL: https://issues.apache.org/jira/browse/HDFS-10939 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Affects Versions: 2.7 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10939.1.patch, HDFS-10939.2.patch, > HDFS-10939.branch-2.patch, HDFS-10939.patch > > > The encryption zone APIs should be optimized to extensively use IIPs to > eliminate path resolutions. The performance penalties incurred by common > operations like creation of file statuses may be reduced by more extensive > short-circuiting of EZ lookups when no EZs exist. All file creates should > not be subjected to the multi-stage locking performance penalty required only > for EDEK generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10933) Refactor TestFsck
[
https://issues.apache.org/jira/browse/HDFS-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1826#comment-1826
]
Hudson commented on HDFS-10933:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10566 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10566/])
HDFS-10933. Refactor TestFsck. Contributed by Takanobu Asanuma. (weichiu: rev
3059b251d8f37456c5761ecaf73fe6c0c5a59067)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
> Refactor TestFsck
> -
>
> Key: HDFS-10933
> URL: https://issues.apache.org/jira/browse/HDFS-10933
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-10933.1.patch, HDFS-10933.2.patch,
> HDFS-10933.3.patch, HDFS-10933.WIP.1.patch
>
>
> {{TestFsck}} should be refactored.
> - use @Before @After annotations
> - improve loggings
> - fix checkstyle warnings
> etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
[
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1808#comment-1808
]
Hadoop QA commented on HDFS-10972:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 5 new + 1 unchanged - 0 fixed = 6 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m
39s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10972 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832148/HDFS-10972.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d969b39058e9 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c183b9d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17059/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17059/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17059/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
[jira] [Commented] (HDFS-10759) Change fsimage bool isStriped from boolean to an enum
[
https://issues.apache.org/jira/browse/HDFS-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1806#comment-1806
]
Zhe Zhang commented on HDFS-10759:
--
Sorry for the late response. I just labeled this as a 3.0 blocker, and will get
back on this soon.
Pinging [~andrew.wang] [~jingzhao] for additional thoughts.
> Change fsimage bool isStriped from boolean to an enum
> -
>
> Key: HDFS-10759
> URL: https://issues.apache.org/jira/browse/HDFS-10759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2
>Reporter: Ewan Higgs
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10759.0001.patch
>
>
> The new erasure coding project has updated the protocol for fsimage such that
> the {{INodeFile}} has a boolean '{{isStriped}}'. I think this is better as an
> enum or integer since a boolean precludes any future block types.
> For example:
> {code}
> enum BlockType {
> CONTIGUOUS = 0,
> STRIPED = 1,
> }
> {code}
> We can also make this more robust to future changes where there are different
> block types supported in a staged rollout. Here, we would use
> {{UNKNOWN_BLOCK_TYPE}} as the first value since this is the default value.
> See
> [here|http://androiddevblog.com/protocol-buffers-pitfall-adding-enum-values/]
> for more discussion.
> {code}
> enum BlockType {
> UNKNOWN_BLOCK_TYPE = 0,
> CONTIGUOUS = 1,
> STRIPED = 2,
> }
> {code}
> But I'm not convinced this is necessary since there are other enums that
> don't use this approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10759) Change fsimage bool isStriped from boolean to an enum
[
https://issues.apache.org/jira/browse/HDFS-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10759:
-
Labels: hdfs-ec-3.0-must-do (was: )
> Change fsimage bool isStriped from boolean to an enum
> -
>
> Key: HDFS-10759
> URL: https://issues.apache.org/jira/browse/HDFS-10759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2
>Reporter: Ewan Higgs
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10759.0001.patch
>
>
> The new erasure coding project has updated the protocol for fsimage such that
> the {{INodeFile}} has a boolean '{{isStriped}}'. I think this is better as an
> enum or integer since a boolean precludes any future block types.
> For example:
> {code}
> enum BlockType {
> CONTIGUOUS = 0,
> STRIPED = 1,
> }
> {code}
> We can also make this more robust to future changes where there are different
> block types supported in a staged rollout. Here, we would use
> {{UNKNOWN_BLOCK_TYPE}} as the first value since this is the default value.
> See
> [here|http://androiddevblog.com/protocol-buffers-pitfall-adding-enum-values/]
> for more discussion.
> {code}
> enum BlockType {
> UNKNOWN_BLOCK_TYPE = 0,
> CONTIGUOUS = 1,
> STRIPED = 2,
> }
> {code}
> But I'm not convinced this is necessary since there are other enums that
> don't use this approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10980) Optimize check for existence of parent directory
[
https://issues.apache.org/jira/browse/HDFS-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-10980:
---
Status: Patch Available (was: Open)
> Optimize check for existence of parent directory
>
>
> Key: HDFS-10980
> URL: https://issues.apache.org/jira/browse/HDFS-10980
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10980.patch
>
>
> {{FSDirectory.verifyParentDir()}} uses a {{Path}} object to parse and return
> the parent path. This is very expensive compared to using the path within
> the IIP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10980) Optimize check for existence of parent directory
[
https://issues.apache.org/jira/browse/HDFS-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-10980:
---
Attachment: HDFS-10980.patch
Trivial patch to remove String arg and use IIP to check parent inode.
> Optimize check for existence of parent directory
>
>
> Key: HDFS-10980
> URL: https://issues.apache.org/jira/browse/HDFS-10980
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10980.patch
>
>
> {{FSDirectory.verifyParentDir()}} uses a {{Path}} object to parse and return
> the parent path. This is very expensive compared to using the path within
> the IIP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8824) Do not use small blocks for balancing the cluster
[
https://issues.apache.org/jira/browse/HDFS-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8824:
Fix Version/s: 2.7.4
Thanks Nicholas for the work. I think this is a good improvement for branch-2.7
and just backported. Verified {{TestBalancer}}.
> Do not use small blocks for balancing the cluster
> -
>
> Key: HDFS-8824
> URL: https://issues.apache.org/jira/browse/HDFS-8824
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>
> Attachments: h8824_20150727b.patch, h8824_20150811b.patch
>
>
> Balancer gets datanode block lists from NN and then move the blocks in order
> to balance the cluster. It should not use the blocks with small size since
> moving the small blocks generates a lot of overhead and the small blocks do
> not help balancing the cluster much.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10979) Pass IIP for FSDirDeleteOp methods
[
https://issues.apache.org/jira/browse/HDFS-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1739#comment-1739
]
Hadoop QA commented on HDFS-10979:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 267 unchanged - 2 fixed = 268 total (was 269) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 76m
24s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10979 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832142/HDFS-10979.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 08b3976bfaa0 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 459a483 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17058/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17058/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17058/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Pass IIP for FSDirDeleteOp methods
> --
>
> Key: HDFS-10979
> URL: https://issues.apache.org/jira/browse/HDFS-10979
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>
[jira] [Commented] (HDFS-10939) Reduce performance penalty of encryption zones
[
https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1719#comment-1719
]
Hadoop QA commented on HDFS-10939:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 8 new + 403 unchanged - 10 fixed = 411 total (was 413) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 48m
54s{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_111.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HDFS-10939 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832140/HDFS-10939.branch-2.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 75fdfa1b65b3 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / 8bee319 |
| Default Java | 1.7.0_111 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_101
[jira] [Commented] (HDFS-10933) Refactor TestFsck
[
https://issues.apache.org/jira/browse/HDFS-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1684#comment-1684
]
Wei-Chiu Chuang commented on HDFS-10933:
Committed it to trunk. The branch-2 has quite a number of conflicts. Would you
mind to post a branch-2 patch? [~tasanuma0829] thanks very much.
> Refactor TestFsck
> -
>
> Key: HDFS-10933
> URL: https://issues.apache.org/jira/browse/HDFS-10933
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-10933.1.patch, HDFS-10933.2.patch,
> HDFS-10933.3.patch, HDFS-10933.WIP.1.patch
>
>
> {{TestFsck}} should be refactored.
> - use @Before @After annotations
> - improve loggings
> - fix checkstyle warnings
> etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10969) Fix typos in hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1678#comment-1678
]
Hudson commented on HDFS-10969:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10565 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10565/])
HDFS-10969. Fix typos in hdfs-default.xml Contributed by Yiqun Lin (brahma: rev
be3cb10f5301c2d526d0ba37dbe82f426683a801)
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
> Fix typos in hdfs-default.xml
> -
>
> Key: HDFS-10969
> URL: https://issues.apache.org/jira/browse/HDFS-10969
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: HDFS-10969.001.patch
>
>
> There are three typos in file {{hdfs-default.xml}}:
> * In {{dfs.datanode.transferTo.allowed}}: tranfers->transfers
> * In {{dfs.block.local-path-access.user}}: allowd->allowed
> * In {{dfs.datanode.directoryscan.throttle.limit.ms.per.sec}}:
> disbled->disabled
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10933) Refactor TestFsck
[
https://issues.apache.org/jira/browse/HDFS-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1658#comment-1658
]
Wei-Chiu Chuang commented on HDFS-10933:
Committing the 3rd patch.
> Refactor TestFsck
> -
>
> Key: HDFS-10933
> URL: https://issues.apache.org/jira/browse/HDFS-10933
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-10933.1.patch, HDFS-10933.2.patch,
> HDFS-10933.3.patch, HDFS-10933.WIP.1.patch
>
>
> {{TestFsck}} should be refactored.
> - use @Before @After annotations
> - improve loggings
> - fix checkstyle warnings
> etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[ https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1630#comment-1630 ] Xiao Chen commented on HDFS-10797: -- I plan to commit this end of today if no objections. > Disk usage summary of snapshots causes renamed blocks to get counted twice > -- > > Key: HDFS-10797 > URL: https://issues.apache.org/jira/browse/HDFS-10797 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, > HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch, > HDFS-10797.006.patch, HDFS-10797.007.patch, HDFS-10797.008.patch, > HDFS-10797.009.patch, HDFS-10797.010.patch, HDFS-10797.010.patch > > > DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how > much disk usage is used by a snapshot by tallying up the files in the > snapshot that have since been deleted (that way it won't overlap with regular > files whose disk usage is computed separately). However that is determined > from a diff that shows moved (to Trash or otherwise) or renamed files as a > deletion and a creation operation that may overlap with the list of blocks. > Only the deletion operation is taken into consideration, and this causes > those blocks to get represented twice in the disk usage tallying. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8028) TestNNHandlesBlockReportPerStorage/TestNNHandlesCombinedBlockReport Failed after patched HDFS-7704
[ https://issues.apache.org/jira/browse/HDFS-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1619#comment-1619 ] Wei-Chiu Chuang commented on HDFS-8028: --- Hello [~hongyu.bi] would you like to post a new patch? Otherwise I can work on a new one. Thanks! > TestNNHandlesBlockReportPerStorage/TestNNHandlesCombinedBlockReport Failed > after patched HDFS-7704 > -- > > Key: HDFS-8028 > URL: https://issues.apache.org/jira/browse/HDFS-8028 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.7.0 >Reporter: hongyu bi >Assignee: hongyu bi >Priority: Minor > Attachments: HDFS-8028-v0.patch > > > HDFS-7704 makes BadBlockReport asynchronously however > BlockReportTestBase#blockreport_02 doesn't wait for a while after blockreport. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10965) Add unit test for HDFS command 'dfsadmin -printTopology'
[ https://issues.apache.org/jira/browse/HDFS-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1610#comment-1610 ] Xiaobing Zhou commented on HDFS-10965: -- Thanks [~liuml07]. I hold off doing similar changes proposed by HDFS-10972 since they can be reused. > Add unit test for HDFS command 'dfsadmin -printTopology' > > > Key: HDFS-10965 > URL: https://issues.apache.org/jira/browse/HDFS-10965 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, shell, test >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10965.000.patch, HDFS-10965.001.patch, > HDFS-10965.002.patch, HDFS-10965.003.patch > > > DFSAdmin#printTopology should also be tested. This proposes adding it in > TestDFSAdmin. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10969) Fix typos in hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-10969:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.9.0
Status: Resolved (was: Patch Available)
Committed to trunk and branch-2.[~linyiqun] thanks for contribution.
> Fix typos in hdfs-default.xml
> -
>
> Key: HDFS-10969
> URL: https://issues.apache.org/jira/browse/HDFS-10969
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: HDFS-10969.001.patch
>
>
> There are three typos in file {{hdfs-default.xml}}:
> * In {{dfs.datanode.transferTo.allowed}}: tranfers->transfers
> * In {{dfs.block.local-path-access.user}}: allowd->allowed
> * In {{dfs.datanode.directoryscan.throttle.limit.ms.per.sec}}:
> disbled->disabled
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10981) Remove print stream maintained in DFSAdmin
[ https://issues.apache.org/jira/browse/HDFS-10981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10981: - Summary: Remove print stream maintained in DFSAdmin (was: remove print stream maintained in DFSAdmin) > Remove print stream maintained in DFSAdmin > -- > > Key: HDFS-10981 > URL: https://issues.apache.org/jira/browse/HDFS-10981 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs, shell >Affects Versions: 3.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou >Priority: Minor > > The out/err print stream is maintained in DFSAdmin, so that users can pass in > their own streams. Originally, they are used to verify outputs returned by > DFSAdmin, it's preferable to remove them, instead, users can use > System.setOut/setErr. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
