[jira] [Updated] (HDFS-9405) When starting a file, NameNode should generate EDEK in a separate thread
[
https://issues.apache.org/jira/browse/HDFS-9405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-9405:
Attachment: HDFS-9405.06.patch
Whitespace + checkstyle fix.
> When starting a file, NameNode should generate EDEK in a separate thread
>
>
> Key: HDFS-9405
> URL: https://issues.apache.org/jira/browse/HDFS-9405
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: encryption, namenode
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9405.01.patch, HDFS-9405.02.patch,
> HDFS-9405.03.patch, HDFS-9405.04.patch, HDFS-9405.05.patch, HDFS-9405.06.patch
>
>
> {{generateEncryptedDataEncryptionKey}} involves a non-trivial I/O operation
> to the key provider, which could be slow or cause timeout. It should be done
> as a separate thread so as to return a proper error message to the RPC caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9941) Do not log StandbyException on NN, other minor logging fixes
[ https://issues.apache.org/jira/browse/HDFS-9941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-9941: Attachment: HDFS-9941-branch-2.03.patch Thank you for the review [~cnauroth]. branch-2 patch attached.? It also applies cleanly to branch-2.8. > Do not log StandbyException on NN, other minor logging fixes > > > Key: HDFS-9941 > URL: https://issues.apache.org/jira/browse/HDFS-9941 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-9941-branch-2.03.patch, HDFS-9941.01.patch, > HDFS-9941.02.patch, HDFS-9941.03.patch > > > The NameNode can skip logging StandbyException messages. These are seen > regularly in normal operation and convey no useful information. > We no longer log the locations of newly allocated blocks in 2.8.0. The DN IDs > can be useful for debugging so let's add that back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9405) When starting a file, NameNode should generate EDEK in a separate thread
[
https://issues.apache.org/jira/browse/HDFS-9405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191611#comment-15191611
]
Hadoop QA commented on HDFS-9405:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 39s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 47s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 13s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 1s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 7s
{color} | {color:red} root: patch generated 1 new + 199 unchanged - 1 fixed =
200 total (was 200) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 25s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 16s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 32s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 35s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 216m 3s {color}
| {color:black} {color} |
\\
[jira] [Created] (HDFS-9948) Block#toString should not output information from derived classes
Colin Patrick McCabe created HDFS-9948:
--
Summary: Block#toString should not output information from derived
classes
Key: HDFS-9948
URL: https://issues.apache.org/jira/browse/HDFS-9948
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
{{Block#toString}} should not output information from derived classes. Thanks
for [~cnauroth] for spotting this bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9947) Block#toString should not output information from derived classes
Colin Patrick McCabe created HDFS-9947:
--
Summary: Block#toString should not output information from derived
classes
Key: HDFS-9947
URL: https://issues.apache.org/jira/browse/HDFS-9947
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
{{Block#toString}} should not output information from derived classes. Thanks
for [~cnauroth] for spotting this bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9945) Datanode command for evicting writers
[
https://issues.apache.org/jira/browse/HDFS-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191533#comment-15191533
]
Kihwal Lee commented on HDFS-9945:
--
This patch adds a new dfsadmin command that talks to the specified datanode to
stop all writers. Since it is a new method and feature, the risk of breaking
existing code is low.
The only notable change to the existing code is making {{replaceBlock()}}
update the {{blockReceiver}} class variable instead of its own local variable.
We want {{evictWriters()}} to also stop replications and balancer-related write
activities. Because of this change, {{sendOOB()}} was updated to not send if
{{isDatanode}} is true. Prior to this change, {{sendOOB()}} was not called at
all when {{replaceBlock()}} is in progress, because {{blockReceiver}}, the
class variable, was still null.
> Datanode command for evicting writers
> -
>
> Key: HDFS-9945
> URL: https://issues.apache.org/jira/browse/HDFS-9945
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9945.patch
>
>
> It will be useful if there is a command to evict writers from a datanode.
> When a set of datanodes are being decommissioned, they can get blocked by
> slow writers at the end. It was rare in the old days since mapred jobs
> didn't last too long, but with many different types of apps running on
> today's YARN cluster, we are often see very long tail in datanode
> decommissioning.
> I propose a new dfsadmin command, {{evictWriters}}, to be added. I initially
> thought about having namenode automatically telling datanodes on
> decommissioning, but realized that having a command is more flexible. E.g.
> users can choose not to do this at all, choose when to evict writers, or
> whether to try multiple times for whatever reasons.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9926) ozone : Add volume commands to CLI
[ https://issues.apache.org/jira/browse/HDFS-9926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191526#comment-15191526 ] Chris Nauroth commented on HDFS-9926: - The Checkstyle warning is not actionable, and the test failures are unrelated. +1 for patch v003, and I'll commit it later. > ozone : Add volume commands to CLI > -- > > Key: HDFS-9926 > URL: https://issues.apache.org/jira/browse/HDFS-9926 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-9926-HDFS-7240.001.patch, > HDFS-9926-HDFS-7240.002.patch, HDFS-9926-HDFS-7240.003.patch > > > Adds a cli tool which supports volume commands -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9926) ozone : Add volume commands to CLI
[
https://issues.apache.org/jira/browse/HDFS-9926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191519#comment-15191519
]
Chris Nauroth commented on HDFS-9926:
-
The pre-commit run for patch v003 didn't post the results back to the issue
here. I assume that's because of the JIRA instability today. Here is a
copy-paste out of Jenkins:
{code}
-1 overall
| Vote | Subsystem | Runtime | Comment
| 0 |reexec | 0m 30s| Docker mode activated.
| +1 | @author | 0m 0s | The patch does not contain any @author
| ||| tags.
| -1 |test4tests | 0m 0s | The patch doesn't appear to include any
| ||| new or modified tests. Please justify why
| ||| no new tests are needed for this patch.
| ||| Also please list what manual steps were
| ||| performed to verify this patch.
| +1 |mvninstall | 9m 29s| HDFS-7240 passed
| +1 | compile | 0m 46s| HDFS-7240 passed with JDK v1.8.0_74
| +1 | compile | 0m 47s| HDFS-7240 passed with JDK v1.7.0_95
| +1 |checkstyle | 0m 25s| HDFS-7240 passed
| +1 | mvnsite | 1m 0s | HDFS-7240 passed
| +1 |mvneclipse | 0m 18s| HDFS-7240 passed
| -1 | findbugs | 2m 29s| hadoop-hdfs-project/hadoop-hdfs in
| ||| HDFS-7240 has 1 extant Findbugs warnings.
| +1 | javadoc | 1m 34s| HDFS-7240 passed with JDK v1.8.0_74
| +1 | javadoc | 2m 36s| HDFS-7240 passed with JDK v1.7.0_95
| +1 |mvninstall | 0m 51s| the patch passed
| +1 | compile | 0m 41s| the patch passed with JDK v1.8.0_74
| +1 | javac | 0m 41s| the patch passed
| +1 | compile | 0m 44s| the patch passed with JDK v1.7.0_95
| +1 | javac | 0m 44s| the patch passed
| -1 |checkstyle | 0m 18s| hadoop-hdfs-project/hadoop-hdfs: patch
| ||| generated 2 new + 0 unchanged - 0 fixed =
| ||| 2 total (was 0)
| +1 | mvnsite | 0m 53s| the patch passed
| +1 |mvneclipse | 0m 12s| the patch passed
| +1 |whitespace | 0m 0s | Patch has no whitespace issues.
| +1 | findbugs | 2m 23s| the patch passed
| +1 | javadoc | 1m 35s| the patch passed with JDK v1.8.0_74
| +1 | javadoc | 2m 38s| the patch passed with JDK v1.7.0_95
| -1 | unit | 57m 51s | hadoop-hdfs in the patch failed with JDK
| ||| v1.8.0_74.
| -1 | unit | 51m 33s | hadoop-hdfs in the patch failed with JDK
| ||| v1.7.0_95.
| +1 |asflicense | 0m 25s| Patch does not generate ASF License
| ||| warnings.
| || 142m 51s |
Reason | Tests
JDK v1.8.0_74 Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
|
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
hadoop.hdfs.server.datanode.TestBPOfferService
|
hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint
JDK v1.7.0_95 Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
|
hadoop.hdfs.server.datanode.TestBPOfferService
|
hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12792641/HDFS-9926-HDFS-7240.003.patch
|
| JIRA Issue | HDFS-9926 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 832ac1c4cced 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / e4fb9bd |
| Default Java | 1.7.0_95 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_74
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/14791/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
[jira] [Commented] (HDFS-9944) Ozone : Add container dispatcher
[
https://issues.apache.org/jira/browse/HDFS-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191513#comment-15191513
]
Chris Nauroth commented on HDFS-9944:
-
Hi [~anu]. This looks good. Just a few minor comments:
# There is a typo in the method name {{ContainerUtils#unspportedRequest}}.
# There is a typo in package-info.java: "This pacakage is contains Ozone
container implementation."
# The implementation of {{TestContainerManager}} does exactly what a Mockito
mock would do. Do you want to consider removing this class? Then, you could
replace {{new TestContainerManager()}} with {{mock(ContainerManager.class)}}.
Unless you're planning to put more logic into {{TestContainerManager}} in
future patches, using Mockito would reduce the amount of code.
> Ozone : Add container dispatcher
>
>
> Key: HDFS-9944
> URL: https://issues.apache.org/jira/browse/HDFS-9944
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-7240
>
> Attachments: HDFS-9944-HDFS-7240.001.patch
>
>
> This patch takes a request packet from the network layer and delivers it to
> the container manager. ie. OzoneContainerManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9945) Datanode command for evicting writers
[
https://issues.apache.org/jira/browse/HDFS-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-9945:
-
Attachment: HDFS-9945.patch
> Datanode command for evicting writers
> -
>
> Key: HDFS-9945
> URL: https://issues.apache.org/jira/browse/HDFS-9945
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9945.patch
>
>
> It will be useful if there is a command to evict writers from a datanode.
> When a set of datanodes are being decommissioned, they can get blocked by
> slow writers at the end. It was rare in the old days since mapred jobs
> didn't last too long, but with many different types of apps running on
> today's YARN cluster, we are often see very long tail in datanode
> decommissioning.
> I propose a new dfsadmin command, {{evictWriters}}, to be added. I initially
> thought about having namenode automatically telling datanodes on
> decommissioning, but realized that having a command is more flexible. E.g.
> users can choose not to do this at all, choose when to evict writers, or
> whether to try multiple times for whatever reasons.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9945) Datanode command for evicting writers
[
https://issues.apache.org/jira/browse/HDFS-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-9945:
-
Status: Patch Available (was: Open)
> Datanode command for evicting writers
> -
>
> Key: HDFS-9945
> URL: https://issues.apache.org/jira/browse/HDFS-9945
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9945.patch
>
>
> It will be useful if there is a command to evict writers from a datanode.
> When a set of datanodes are being decommissioned, they can get blocked by
> slow writers at the end. It was rare in the old days since mapred jobs
> didn't last too long, but with many different types of apps running on
> today's YARN cluster, we are often see very long tail in datanode
> decommissioning.
> I propose a new dfsadmin command, {{evictWriters}}, to be added. I initially
> thought about having namenode automatically telling datanodes on
> decommissioning, but realized that having a command is more flexible. E.g.
> users can choose not to do this at all, choose when to evict writers, or
> whether to try multiple times for whatever reasons.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9350) Avoid creating temprorary strings in Block.toString() and getBlockName()
[ https://issues.apache.org/jira/browse/HDFS-9350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191501#comment-15191501 ] Chris Nauroth commented on HDFS-9350: - [~cmccabe], that's perfect. +1. > Avoid creating temprorary strings in Block.toString() and getBlockName() > > > Key: HDFS-9350 > URL: https://issues.apache.org/jira/browse/HDFS-9350 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg >Priority: Minor > Fix For: 2.9.0 > > Attachments: HDFS-9350.001.patch > > > Minor change to use StringBuilders directly to avoid creating temporary > strings of Long and Block name when doing toString on a Block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9350) Avoid creating temprorary strings in Block.toString() and getBlockName()
[
https://issues.apache.org/jira/browse/HDFS-9350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191491#comment-15191491
]
Colin Patrick McCabe commented on HDFS-9350:
Good catch, [~cnauroth]. I agree that we should restrict this method to just
printing data from the base class. However, Staffan's goal of reducing the
number of memory allocations is nice. How about this change?
{code}
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/Block.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/Block.java
index 23bfa95..4128ece 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/Block.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/Block.java
@@ -162,7 +162,9 @@ public void setGenerationStamp(long stamp) {
*/
public static String toString(final Block b) {
StringBuilder sb = new StringBuilder();
-b.appendStringTo(sb);
+sb.append(BLOCK_FILE_PREFIX).
+ append(b.blockId).append("_").
+ append(b.generationStamp);
return sb.toString();
}
{code}
> Avoid creating temprorary strings in Block.toString() and getBlockName()
>
>
> Key: HDFS-9350
> URL: https://issues.apache.org/jira/browse/HDFS-9350
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: performance
>Affects Versions: 2.7.1
>Reporter: Staffan Friberg
>Assignee: Staffan Friberg
>Priority: Minor
> Fix For: 2.9.0
>
> Attachments: HDFS-9350.001.patch
>
>
> Minor change to use StringBuilders directly to avoid creating temporary
> strings of Long and Block name when doing toString on a Block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9928) Make HDFS commands guide up to date
[ https://issues.apache.org/jira/browse/HDFS-9928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-9928: -- Attachment: HDFS-9928-branch-2.002.patch Rev02: rebased to 2.9.0. I also had to remove envvars command because HADOOP-12366 just reverted it in branch-2. Other than that it's mostly cosmetic changes. > Make HDFS commands guide up to date > --- > > Key: HDFS-9928 > URL: https://issues.apache.org/jira/browse/HDFS-9928 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 2.9.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Labels: documentation, supportability > Attachments: HDFS-9928-branch-2.002.patch, HDFS-9928.001.patch > > > A few HDFS subcommands and options are missing in the documentation. > # envvars: display computed Hadoop environment variables > I also noticed (in HDFS-9927) that a few OIV options are missing, and I'll be > looking for other missing options as well. > Filling this JIRA to fix them all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9118) Add logging system for libdhfs++
[ https://issues.apache.org/jira/browse/HDFS-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9118: -- Attachment: HDFS-9118.HDFS-8707.002.patch -Now with unit tests -Pluggable loggers for C and stderr. Defaults to stderr with logging level of warning (just like it was before). > Add logging system for libdhfs++ > > > Key: HDFS-9118 > URL: https://issues.apache.org/jira/browse/HDFS-9118 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS-8707 >Reporter: Bob Hansen >Assignee: James Clampffer > Attachments: HDFS-9118.HDFS-8707.000.patch, > HDFS-9118.HDFS-8707.001.patch, HDFS-9118.HDFS-8707.002.patch > > > With HDFS-9505, we've starting logging data from libhdfs++. Consumers of the > library are going to have their own logging infrastructure that we're going > to want to provide data to. > libhdfs++ should have a logging library that: > * Is overridable and can provide sufficient information to work well with > common C++ logging frameworks > * Has a rational default implementation > * Is performant -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9942) Add an HTrace span when refreshing the groups for a username
[ https://issues.apache.org/jira/browse/HDFS-9942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-9942: --- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) > Add an HTrace span when refreshing the groups for a username > > > Key: HDFS-9942 > URL: https://issues.apache.org/jira/browse/HDFS-9942 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.8.0 > > Attachments: HDFS-9942.001.patch > > > We should add an HTrace span when refreshing the groups for a username. This > can be an expensive operation in some cases, and it's good to know if it > delayed a request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9942) Add an HTrace span when refreshing the groups for a username
[ https://issues.apache.org/jira/browse/HDFS-9942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191460#comment-15191460 ] Colin Patrick McCabe commented on HDFS-9942: Thanks, [~iwasakims]. > Add an HTrace span when refreshing the groups for a username > > > Key: HDFS-9942 > URL: https://issues.apache.org/jira/browse/HDFS-9942 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-9942.001.patch > > > We should add an HTrace span when refreshing the groups for a username. This > can be an expensive operation in some cases, and it's good to know if it > delayed a request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9928) Make HDFS commands guide up to date
[ https://issues.apache.org/jira/browse/HDFS-9928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-9928: -- Affects Version/s: (was: 3.0.0) 2.9.0 > Make HDFS commands guide up to date > --- > > Key: HDFS-9928 > URL: https://issues.apache.org/jira/browse/HDFS-9928 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 2.9.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Labels: documentation, supportability > Attachments: HDFS-9928.001.patch > > > A few HDFS subcommands and options are missing in the documentation. > # envvars: display computed Hadoop environment variables > I also noticed (in HDFS-9927) that a few OIV options are missing, and I'll be > looking for other missing options as well. > Filling this JIRA to fix them all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8356) Document missing properties in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated HDFS-8356: - Attachment: HDFS-8356.007.patch - Fixes based on latest feedback from Akira. > Document missing properties in hdfs-default.xml > --- > > Key: HDFS-8356 > URL: https://issues.apache.org/jira/browse/HDFS-8356 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.0 >Reporter: Ray Chiang >Assignee: Ray Chiang > Labels: supportability, test > Attachments: HDFS-8356.001.patch, HDFS-8356.002.patch, > HDFS-8356.003.patch, HDFS-8356.004.patch, HDFS-8356.005.patch, > HDFS-8356.006.patch, HDFS-8356.007.patch > > > The following properties are currently not defined in hdfs-default.xml. These > properties should either be > A) documented in hdfs-default.xml OR > B) listed as an exception (with comments, e.g. for internal use) in the > TestHdfsConfigFields unit test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9350) Avoid creating temprorary strings in Block.toString() and getBlockName()
[ https://issues.apache.org/jira/browse/HDFS-9350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191363#comment-15191363 ] Wei-Chiu Chuang commented on HDFS-9350: --- [~cnauroth], you're absolutely right about it. In HDFS-7284, the original intent was to print just the Block portion, not the subclass. We added the static method because typecasting doesn't change runtime binding. If the goal is to avoid creating temp strings, I think we could create a static appendStringTo() method, and call the static method instead. > Avoid creating temprorary strings in Block.toString() and getBlockName() > > > Key: HDFS-9350 > URL: https://issues.apache.org/jira/browse/HDFS-9350 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg >Priority: Minor > Fix For: 2.9.0 > > Attachments: HDFS-9350.001.patch > > > Minor change to use StringBuilders directly to avoid creating temporary > strings of Long and Block name when doing toString on a Block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9941) Do not log StandbyException on NN, other minor logging fixes
[
https://issues.apache.org/jira/browse/HDFS-9941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191356#comment-15191356
]
Chris Nauroth commented on HDFS-9941:
-
The Checkstyle warning is not practical to address in scope of this patch. The
test failures are unrelated.
While reviewing this, I squinted at the static {{Block#toString}} method for an
hour thinking that I was misunderstanding something about the way Java dynamic
dispatch works. It turns out no, my understanding is fine, and HDFS-9350
introduced a bug in that method. This isn't directly related to the HDFS-9941
patch though, so I left a comment on HDFS-9350 for follow-up.
I am +1 for patch v003 going into trunk, but it doesn't merge cleanly to
branch-2. [~arpitagarwal], would you please post a patch for
branch-2/branch-2.8? Thank you.
> Do not log StandbyException on NN, other minor logging fixes
>
>
> Key: HDFS-9941
> URL: https://issues.apache.org/jira/browse/HDFS-9941
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-9941.01.patch, HDFS-9941.02.patch,
> HDFS-9941.03.patch
>
>
> The NameNode can skip logging StandbyException messages. These are seen
> regularly in normal operation and convey no useful information.
> We no longer log the locations of newly allocated blocks in 2.8.0. The DN IDs
> can be useful for debugging so let's add that back.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9918) Erasure Coding: Sort located striped blocks based on decommissioned states
[ https://issues.apache.org/jira/browse/HDFS-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191355#comment-15191355 ] Rakesh R commented on HDFS-9918: Attached patch for the sorting logic. Kindly review the patch, thanks! > Erasure Coding: Sort located striped blocks based on decommissioned states > -- > > Key: HDFS-9918 > URL: https://issues.apache.org/jira/browse/HDFS-9918 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rakesh R >Assignee: Rakesh R > Attachments: HDFS-9918-001.patch > > > This jira is a follow-on work of HDFS-8786, where we do decommissioning of > datanodes having striped blocks. > Now, after decommissioning it requires to change the ordering of the storage > list so that the decommissioned datanodes should only be last node in list. > For example, assume we have a block group with storage list:- > d0, d1, d2, d3, d4, d5, d6, d7, d8, d9 > mapping to indices > 0, 1, 2, 3, 4, 5, 6, 7, 8, 2 > Here the internal block b2 is duplicated, locating in d2 and d9. If d2 is a > decommissioning node then should switch d2 and d9 in the storage list. > Thanks [~jingzhao] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-8786?focusedCommentId=15180415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15180415] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9918) Erasure Coding: Sort located striped blocks based on decommissioned states
[ https://issues.apache.org/jira/browse/HDFS-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-9918: --- Target Version/s: 3.0.0 Status: Patch Available (was: Open) > Erasure Coding: Sort located striped blocks based on decommissioned states > -- > > Key: HDFS-9918 > URL: https://issues.apache.org/jira/browse/HDFS-9918 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rakesh R >Assignee: Rakesh R > Attachments: HDFS-9918-001.patch > > > This jira is a follow-on work of HDFS-8786, where we do decommissioning of > datanodes having striped blocks. > Now, after decommissioning it requires to change the ordering of the storage > list so that the decommissioned datanodes should only be last node in list. > For example, assume we have a block group with storage list:- > d0, d1, d2, d3, d4, d5, d6, d7, d8, d9 > mapping to indices > 0, 1, 2, 3, 4, 5, 6, 7, 8, 2 > Here the internal block b2 is duplicated, locating in d2 and d9. If d2 is a > decommissioning node then should switch d2 and d9 in the storage list. > Thanks [~jingzhao] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-8786?focusedCommentId=15180415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15180415] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9918) Erasure Coding: Sort located striped blocks based on decommissioned states
[ https://issues.apache.org/jira/browse/HDFS-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-9918: --- Attachment: HDFS-9918-001.patch > Erasure Coding: Sort located striped blocks based on decommissioned states > -- > > Key: HDFS-9918 > URL: https://issues.apache.org/jira/browse/HDFS-9918 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rakesh R >Assignee: Rakesh R > Attachments: HDFS-9918-001.patch > > > This jira is a follow-on work of HDFS-8786, where we do decommissioning of > datanodes having striped blocks. > Now, after decommissioning it requires to change the ordering of the storage > list so that the decommissioned datanodes should only be last node in list. > For example, assume we have a block group with storage list:- > d0, d1, d2, d3, d4, d5, d6, d7, d8, d9 > mapping to indices > 0, 1, 2, 3, 4, 5, 6, 7, 8, 2 > Here the internal block b2 is duplicated, locating in d2 and d9. If d2 is a > decommissioning node then should switch d2 and d9 in the storage list. > Thanks [~jingzhao] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-8786?focusedCommentId=15180415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15180415] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9350) Avoid creating temprorary strings in Block.toString() and getBlockName()
[
https://issues.apache.org/jira/browse/HDFS-9350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191331#comment-15191331
]
Chris Nauroth commented on HDFS-9350:
-
I think this patch should be reverted, because it broke the contract of the
static {{Block#toString}} method. Here is the code before the patch:
{code}
/**
* A helper method to output the string representation of the Block portion of
* a derived class' instance.
*
* @param b the target object
* @return the string representation of the block
*/
public static String toString(final Block b) {
return getBlockName() + "_" + getGenerationStamp();
}
{code}
Here is the code after the patch:
{code}
/**
* A helper method to output the string representation of the Block portion of
* a derived class' instance.
*
* @param b the target object
* @return the string representation of the block
*/
public static String toString(final Block b) {
StringBuilder sb = new StringBuilder();
b.appendStringTo(sb);
return sb.toString();
}
{code}
Notice that the JavaDocs say this method is supposed to provide just the basic
{{Block}} data, even in cases where the parameter is a subclass that may have
overridden {{appendStringTo}}. Having a widening cast to {{Block}} in the
method signature's argument does not influence the outcome of runtime dynamic
dispatch. One specific problem would be {{ReplicaUnderConstruction}}, which
overrides {{appendStringTo}}, so passing an instance of that will not in fact
give the caller the basic {{Block}} data.
Thoughts? Cc'ing [~jojochuang], who I think is the original author of the
static {{Block#toString}} method. He can help confirm the intent of the method.
> Avoid creating temprorary strings in Block.toString() and getBlockName()
>
>
> Key: HDFS-9350
> URL: https://issues.apache.org/jira/browse/HDFS-9350
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: performance
>Affects Versions: 2.7.1
>Reporter: Staffan Friberg
>Assignee: Staffan Friberg
>Priority: Minor
> Fix For: 2.9.0
>
> Attachments: HDFS-9350.001.patch
>
>
> Minor change to use StringBuilders directly to avoid creating temporary
> strings of Long and Block name when doing toString on a Block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9405) When starting a file, NameNode should generate EDEK in a separate thread
[
https://issues.apache.org/jira/browse/HDFS-9405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-9405:
Attachment: HDFS-9405.05.patch
Thanks for explaining [~liuml07], I misread your previous comment. Agree the
extra {{spy()}} is not needed. Patch 5 reflects this.
Surprisingly, I discovered another problem in
{{TestEncryptionZonesWithKMS#testCreateEZPopulatesEDEKCache}} (the case above
my test case):
In HDFS-7209, we added this test, and method {{ValueQueue#getSize}} for testing
purpose. The problem is, when we do {{keyQueues.get(keyName).size()}} to verify
the size, the loading cache itself will always populate the cache because of
the call to {{get}}. In other words, the test always passes I've verified
that by backing up the change and running the test - it's still green.
So, patch 5 here also changed the {{getSize}} method to check size without
loading the cache. Verified both tests behave in a fail-before, pass-after
manner.
> When starting a file, NameNode should generate EDEK in a separate thread
>
>
> Key: HDFS-9405
> URL: https://issues.apache.org/jira/browse/HDFS-9405
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: encryption, namenode
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9405.01.patch, HDFS-9405.02.patch,
> HDFS-9405.03.patch, HDFS-9405.04.patch, HDFS-9405.05.patch
>
>
> {{generateEncryptedDataEncryptionKey}} involves a non-trivial I/O operation
> to the key provider, which could be slow or cause timeout. It should be done
> as a separate thread so as to return a proper error message to the RPC caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9937) Update dfsadmin command line help
[
https://issues.apache.org/jira/browse/HDFS-9937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191317#comment-15191317
]
Wei-Chiu Chuang commented on HDFS-9937:
---
Hi, [~lewuathe]
thank you for working on this. The unit test failures seems unrelated to this
patch.
The patch looks good to me. In addition to checkstyle and whitespace warnings,
just one comment though.
Could you also update the comment of setSafeMode():
{code:title=DFSAdmin#setSafeMode}
/**
* Safe mode maintenance command.
* Usage: hdfs dfsadmin -safemode [enter | leave | get]
* @param argv List of of command line parameters.
* @param idx The index of the command that is being processed.
* @exception IOException if the filesystem does not exist.
*/
public void setSafeMode(String[] argv, int idx) throws IOException {
{code}
Also I feel pretty bad those help messages are not aligned, but maybe it's OK.
> Update dfsadmin command line help
> -
>
> Key: HDFS-9937
> URL: https://issues.apache.org/jira/browse/HDFS-9937
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Kai Sasaki
>Priority: Minor
> Labels: commandline, supportability
> Attachments: HDFS-9937.01.patch
>
>
> dfsadmin command line top-level help menu is not consistent with detailed
> help menu.
> * -safemode missed options -wait and -forceExit
> * -restoreFailedStorage options are not described consistently
> (true/false/check, or Set/Unset/Check?)
> * -setSpaceQuota optionally takes a -storageType parameter, but it's not
> clear what are the available options. (Seems to be (SSD, DISK, ARCHIVE), from
> HdfsQuotaAdminGuide.html)
> * -reconfig seems to also take namenode as parameter
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (HDFS-9946) MiniDFS clusters failing to come up, "timeout can't be negative"
[ https://issues.apache.org/jira/browse/HDFS-9946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran moved HADOOP-12917 to HDFS-9946: --- Affects Version/s: (was: 2.8.0) 2.8.0 Component/s: (was: ipc) test Key: HDFS-9946 (was: HADOOP-12917) Project: Hadoop HDFS (was: Hadoop Common) > MiniDFS clusters failing to come up, "timeout can't be negative" > > > Key: HDFS-9946 > URL: https://issues.apache.org/jira/browse/HDFS-9946 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Critical > > All attempts to bring up minidfs clusters are failing in java.io.IOException: > Failed on local exception: java.io.IOException: Couldn't set up IO streams: > java.lang.IllegalArgumentException: timeout can't be negative; > HADOOP-12672 just altered this code; assuming its the cause and rolling back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9556) libhdfs++: pull Options from default configs by default
[ https://issues.apache.org/jira/browse/HDFS-9556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191193#comment-15191193 ] James Clampffer commented on HDFS-9556: --- Looks good to me. From what I can tell the whitespace and compiler warnings are coming from the URI parsing library rather than your code so that's not an issue. +1 > libhdfs++: pull Options from default configs by default > --- > > Key: HDFS-9556 > URL: https://issues.apache.org/jira/browse/HDFS-9556 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9556.HDFS-8707.000.patch, > HDFS-9556.HDFS-8707.002.patch, HDFS-9556.HDFS-8707.003.patch, > HDFS-9556.HDFS-9932.003.patch > > > Include method to connect to defaultFS from configuration -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-9945) Datanode command for evicting writers
[
https://issues.apache.org/jira/browse/HDFS-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee reassigned HDFS-9945:
Assignee: Kihwal Lee
> Datanode command for evicting writers
> -
>
> Key: HDFS-9945
> URL: https://issues.apache.org/jira/browse/HDFS-9945
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>
> It will be useful if there is a command to evict writers from a datanode.
> When a set of datanodes are being decommissioned, they can get blocked by
> slow writers at the end. It was rare in the old days since mapred jobs
> didn't last too long, but with many different types of apps running on
> today's YARN cluster, we are often see very long tail in datanode
> decommissioning.
> I propose a new dfsadmin command, {{evictWriters}}, to be added. I initially
> thought about having namenode automatically telling datanodes on
> decommissioning, but realized that having a command is more flexible. E.g.
> users can choose not to do this at all, choose when to evict writers, or
> whether to try multiple times for whatever reasons.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9937) Update dfsadmin command line help
[
https://issues.apache.org/jira/browse/HDFS-9937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191017#comment-15191017
]
Hadoop QA commented on HDFS-9937:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 10s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 44s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 35s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 9 new +
224 unchanged - 3 fixed = 233 total (was 227) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 22s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 28s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 3s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests | hadoop.hdfs.TestParallelShortCircuitRead |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.fs.contract.hdfs.TestHDFSContractMkdir |
| | hadoop.hdfs.server.namenode.TestAllowFormat |
| | hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens |
| | hadoop.hdfs.TestBlockStoragePolicy |
| | hadoop.hdfs.server.datanode.TestRefreshNamenodes |
| |
[jira] [Created] (HDFS-9945) Datanode command for evicting writers
Kihwal Lee created HDFS-9945:
Summary: Datanode command for evicting writers
Key: HDFS-9945
URL: https://issues.apache.org/jira/browse/HDFS-9945
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode
Reporter: Kihwal Lee
It will be useful if there is a command to evict writers from a datanode. When
a set of datanodes are being decommissioned, they can get blocked by slow
writers at the end. It was rare in the old days since mapred jobs didn't last
too long, but with many different types of apps running on today's YARN
cluster, we are often see very long tail in datanode decommissioning.
I propose a new dfsadmin command, {{evictWriters}}, to be added. I initially
thought about having namenode automatically telling datanodes on
decommissioning, but realized that having a command is more flexible. E.g.
users can choose not to do this at all, choose when to evict writers, or
whether to try multiple times for whatever reasons.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9937) Update dfsadmin command line help
[ https://issues.apache.org/jira/browse/HDFS-9937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HDFS-9937: - Assignee: Kai Sasaki Status: Patch Available (was: Open) > Update dfsadmin command line help > - > > Key: HDFS-9937 > URL: https://issues.apache.org/jira/browse/HDFS-9937 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Wei-Chiu Chuang >Assignee: Kai Sasaki >Priority: Minor > Labels: commandline, supportability > Attachments: HDFS-9937.01.patch > > > dfsadmin command line top-level help menu is not consistent with detailed > help menu. > * -safemode missed options -wait and -forceExit > * -restoreFailedStorage options are not described consistently > (true/false/check, or Set/Unset/Check?) > * -setSpaceQuota optionally takes a -storageType parameter, but it's not > clear what are the available options. (Seems to be (SSD, DISK, ARCHIVE), from > HdfsQuotaAdminGuide.html) > * -reconfig seems to also take namenode as parameter -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9937) Update dfsadmin command line help
[ https://issues.apache.org/jira/browse/HDFS-9937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HDFS-9937: - Attachment: HDFS-9937.01.patch > Update dfsadmin command line help > - > > Key: HDFS-9937 > URL: https://issues.apache.org/jira/browse/HDFS-9937 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Wei-Chiu Chuang >Priority: Minor > Labels: commandline, supportability > Attachments: HDFS-9937.01.patch > > > dfsadmin command line top-level help menu is not consistent with detailed > help menu. > * -safemode missed options -wait and -forceExit > * -restoreFailedStorage options are not described consistently > (true/false/check, or Set/Unset/Check?) > * -setSpaceQuota optionally takes a -storageType parameter, but it's not > clear what are the available options. (Seems to be (SSD, DISK, ARCHIVE), from > HdfsQuotaAdminGuide.html) > * -reconfig seems to also take namenode as parameter -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9703) DiskBalancer : getBandwidth implementation
[
https://issues.apache.org/jira/browse/HDFS-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190645#comment-15190645
]
Hadoop QA commented on HDFS-9703:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} HDFS-1312 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} HDFS-1312 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} HDFS-1312 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s
{color} | {color:green} HDFS-1312 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 3 new +
179 unchanged - 1 fixed = 182 total (was 180) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 43s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 24s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 171m 34s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| |
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
| JDK v1.7.0_95 Failed junit tests |
hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
| | hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-9941) Do not log StandbyException on NN, other minor logging fixes
[
https://issues.apache.org/jira/browse/HDFS-9941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190620#comment-15190620
]
Hadoop QA commented on HDFS-9941:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new +
83 unchanged - 1 fixed = 84 total (was 84) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 58s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 6s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 154m 14s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| JDK v1.7.0_95 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.TestHFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Issue | HDFS-9941 |
| GITHUB PR | https://github.com/apache/hadoop/pull/83 |
| Optional Tests | asflicense compile javac javadoc
[jira] [Commented] (HDFS-9944) Ozone : Add container dispatcher
[
https://issues.apache.org/jira/browse/HDFS-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190609#comment-15190609
]
Hadoop QA commented on HDFS-9944:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 6m 39s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 8s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 32s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 25s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 51m 47s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 142m 20s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests |
hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint |
| JDK v1.7.0_95 Failed junit tests |
